File size: 10,990 Bytes
2745cf8 27fb574 2745cf8 55615eb 2745cf8 55615eb 2745cf8 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 |
---
license: other
license_name: nvidia-open-model-license
license_link: https://developer.download.nvidia.com/licenses/nvidia-open-model-license-agreement-june-2024.pdf
---
# Model Overview
[[**Github**](https://github.com/NVlabs/RADIO)] [[**CVPR 2025**](https://arxiv.org/abs/2412.07679)] [[**CVPR 2024**](https://arxiv.org/abs/2312.06709)]
## Description
This model performs visual feature extraction.
For instance, RADIO generates image embeddings that can be used by a downstream model to classify images.
C-RADIOv2 models are available in multiple sizes:
* Base (90M parameters).
* Large (320M parameters).
* Huge (653M parameters).
* Gigantic (1.1B parameters).
C-RADIOv2 was trained for 1M steps (400k more steps than v1), using inverse frequency sampling for data balancing, and [PHI Standardization](https://arxiv.org/abs/2410.01680) for teacher distribution balancing.
This model is ready for commercial/non-commercial use.
### License/Terms of Use
GOVERNING TERMS: Use of this model is governed by the [NVIDIA Open Model License Agreement](https://developer.download.nvidia.com/licenses/nvidia-open-model-license-agreement-june-2024.pdf).
## Deployment Geography
Global.
## Use Case
The embeddings generated by this model are expected to be used by a downstream application.
For example:
* Image-level understanding (image classification, curation, etc.).
* Dense processing (semantic segmentation, depth estimation, etc.).
* Integration into a Vision-Language Model.
## Release Date
Huggingface: 03/26/2025 via [RADIO Collection of Models](https://huggingface.co/collections/nvidia/radio-669f77f1dd6b153f007dd1c6).
## References
* \[CVPR 2025\] [**RADIOv2.5: Improved Baselines for Agglomerative Vision Foundation Models**](https://arxiv.org/abs/2412.07679)
* \[CVPR 2024\] [**AM-RADIO: Agglomerative Vision Foundation Model - Reduce All Domains Into One**](https://arxiv.org/abs/2312.06709)
## Model Architecture
**Architecture Type:** Neural Network <br>
**Network Architecture:** Vision Transformer <br>
## Input
**Input Type(s):** Image <br>
**Input Format(s):** Red, Green, Blue (RGB) <br>
**Input Parameters:** Two Dimensional (2D) <br>
**Other Properties Related to Input:** Image resolutions up to 2048x2028 in increments of 16 pixels <br>
## Output
**Output Type(s):** Embeddings <br>
**Output Format:** Tensor <br>
**Output Parameters:** 2D <br>
**Other Properties Related to Output:** Downstream model required to leverage image features <br>
## Usage:
RADIO will return a tuple with two tensors.
The `summary` is similar to the `cls_token` in ViT and is meant to represent the general concept of the entire image.
It has shape `(B,C)` with `B` being the batch dimension, and `C` being some number of channels.
The `spatial_features` represent more localized content which should be suitable for dense tasks such as semantic segmentation, or for integration into an LLM.
```python
import torch
from PIL import Image
from transformers import AutoModel, CLIPImageProcessor
hf_repo = "nvidia/C-RADIOv2-H"
image_processor = CLIPImageProcessor.from_pretrained(hf_repo)
model = AutoModel.from_pretrained(hf_repo, trust_remote_code=True)
model.eval().cuda()
image = Image.open('./assets/radio.png').convert('RGB')
pixel_values = image_processor(images=image, return_tensors='pt', do_resize=True).pixel_values
pixel_values = pixel_values.cuda()
summary, features = model(pixel_values)
```
Spatial features have shape `(B,T,D)` with `T` being the flattened spatial tokens, and `D` being the channels for spatial features. Note that `C!=D` in general.
Converting to a spatial tensor format can be done using the downsampling size of the model, combined with the input tensor shape. For RADIO, the patch size is 16.
```Python
from einops import rearrange
spatial_features = rearrange(spatial_features, 'b (h w) d -> b d h w', h=x.shape[-2] // patch_size, w=x.shape[-1] // patch_size)
```
The resulting tensor will have shape `(B,D,H,W)`, as is typically seen with computer vision models.
## Software Integration
**Runtime Engine(s):**
* TAO- 24.10 <br>
**Supported Hardware Microarchitecture Compatibility:** <br>
* NVIDIA Ampere <br>
* NVIDIA Blackwell <br>
* NVIDIA Jetson <br>
* NVIDIA Hopper <br>
* NVIDIA Lovelace <br>
* NVIDIA Pascal <br>
* NVIDIA Turing <br>
* NVIDIA Volta <br>
**[Preferred/Supported] Operating System(s):** <br>
* Linux
* Linux 4 Tegra
* QNX
* Windows
## Model Version(s)
* C-RADIOv2-B (90M parameters).
* C-RADIOv2-L (320M parameters).
* C-RADIOv2-H (653M parameters).
* C-RADIOv2-G (1.8B parameters).
**Links:**
* https://huggingface.co/nvidia/C-RADIOv2-B
* https://huggingface.co/nvidia/C-RADIOv2-L
* https://huggingface.co/nvidia/C-RADIOv2-H
* https://huggingface.co/nvidia/C-RADIOv2-g
# Training and Evaluation Datasets
## Training Dataset
NV-CC-Img-Text-Dataset <br>
### Data Collection Method by dataset
* Automated <br>
### Labeling Method by dataset
* Not Applicable (no labels are needed)
### Properties
* 700 Million Images <br>
## Evaluation Dataset
**Link:** [ImageNet](https://www.image-net.org/) <br>
### Data Collection Method by dataset
* Automated <br>
### Labeling Method by dataset
* Human <br>
**Properties:** This dataset spans 1000 object classes and contains 1,281,167 training images, 50,000 validation images and 100,000 test images.<br>
## Inference
**Engine:** PyTorch <br>
**Test Hardware:** A100 <br>
## Ethical Considerations
NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
For more detailed information on ethical considerations for this model, please see the Model Card++ Explainability, Bias, Safety & Security, and Privacy Subcards below.
Please report security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/).
### Bias
Field | Response
:---------------------------------------------------------------------------------------------------|:---------------
Participation considerations from adversely impacted groups [protected classes](https://www.senate.ca.gov/content/protected-classes) in model design and testing: | None
Measures taken to mitigate against unwanted bias: | None
### Explainability
Field | Response
:------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------
Intended Application & Domain: | Visual Feature Extraction
Model Type: | Vision Transformer
Intended Users: | Developers of downstream vision applications
Output: | Image embeddings
Describe how the model works: | The model takes an image as input, processes the image through multiple transformer blocks, and outputs summary and patch embeddings.
Name the adversely impacted groups this has been tested to deliver comparable outcomes regardless of: | Not Applicable
Technical Limitations: | This model generates image embeddings that can be used by a downstream model to, for example, classify images. The downstream model must be trained to leverage the visual embeddings.
Verified to have met prescribed NVIDIA quality standards: | Yes
Performance Metrics: | Image classification accuracy, semantic segmentation mean-over-intersection.
Potential Known Risks: | This model is only tested on input resolutions ranging from 256 to 2048, in increments of 16 pixels. Additionally, the generated embeddings might fail to disambiguate differences that appear evident to humans (e.g. two images showing different breeds of dogs might in fact produce very similar embeddings). Domain-specific evaluation is required for the target application.
Licensing: | [NVIDIA Open Model License](https://developer.download.nvidia.com/licenses/nvidia-open-model-license-agreement-june-2024.pdf)
### Privacy
Field | Response
:----------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------
Generatable or reverse engineerable personal data? | None
Personal data used to create this model? | None
How often is dataset reviewed? | Before Every Release
Is there provenance for all datasets used in training? | Yes
Does data labeling (annotation, metadata) comply with privacy laws? | Yes
Is data compliant with data subject requests for data correction or removal, if such a request was made? | Yes
### Safety
Field | Response
:---------------------------------------------------|:----------------------------------
Model Application(s): | Generation of visual embeddings
Describe the life critical impact (if present). | Not Applicable
Use Case Restrictions: | Abide by NVIDIA Open Model License Agreement
Model and dataset restrictions: | The Principle of least privilege (PoLP) is applied limiting access for dataset generation and model development. Restrictions enforce dataset access during training, and dataset license constraints adhered to. |