ChihHsuan-Yang
commited on
Commit
•
595ba8c
1
Parent(s):
824c735
updated readme
Browse files
README.md
CHANGED
@@ -6,6 +6,8 @@ tags:
|
|
6 |
- zero-shot-image-classification
|
7 |
- clip
|
8 |
- biology
|
|
|
|
|
9 |
- CV
|
10 |
- images
|
11 |
- animals
|
@@ -17,6 +19,7 @@ tags:
|
|
17 |
- multimodal
|
18 |
- knowledge-guided
|
19 |
datasets:
|
|
|
20 |
- imageomics/TreeOfLife-10M
|
21 |
- iNat21
|
22 |
- BIOSCAN-1M
|
@@ -46,7 +49,7 @@ ARBORCLIP is a new suite of vision-language foundation models for biodiversity.
|
|
46 |
- **License:** MIT
|
47 |
- **Fine-tuned from model:** [OpenAI CLIP](https://github.com/mlfoundations/open_clip), [MetaCLIP](https://github.com/facebookresearch/MetaCLIP), [BioCLIP](https://github.com/Imageomics/BioCLIP)
|
48 |
|
49 |
-
These models were developed for the benefit of the AI community as an open-source product,
|
50 |
|
51 |
|
52 |
### Model Description
|
@@ -54,12 +57,15 @@ These models were developed for the benefit of the AI community as an open-sourc
|
|
54 |
ArborCLIP is based on OpenAI's [CLIP](https://openai.com/research/clip) model.
|
55 |
The models were trained on [ARBORETUM-40M](https://baskargroup.github.io/Arboretum/) for the following configurations:
|
56 |
|
57 |
-
- **ARBORCLIP-O:** Trained a ViT-B/16 backbone initialized from the [OpenCLIP's](https://github.com/mlfoundations/open_clip) checkpoint.
|
58 |
-
- **ARBORCLIP-B:** Trained a ViT-B/16 backbone initialized from the [BioCLIP's](https://github.com/Imageomics/BioCLIP) checkpoint.
|
59 |
-
- **ARBORCLIP-M:** Trained a ViT-L/14 backbone initialized from the [MetaCLIP's](https://github.com/facebookresearch/MetaCLIP) checkpoint.
|
60 |
|
61 |
|
62 |
-
To access the checkpoints of the above models, go to the `Files and versions` tab and download the weights. These weights can be directly used for zero-shot classification and finetuning. The filenames correspond to the specific model weights -
|
|
|
|
|
|
|
63 |
|
64 |
### Model Training
|
65 |
**See the [Model Training](https://github.com/baskargroup/Arboretum?tab=readme-ov-file#model-training) section on the [Github](https://github.com/baskargroup/Arboretum) for examples of how to use ArborCLIP models in zero-shot image classification tasks.**
|
@@ -128,8 +134,7 @@ Additionally, models trained on web-scraped data excel at classifying at the hig
|
|
128 |
level (kingdom), while models begin to benefit from specialist datasets like [ARBORETUM-40M](https://baskargroup.github.io/Arboretum/) and
|
129 |
[Tree-of-Life-10M](https://huggingface.co/datasets/imageomics/TreeOfLife-10M) at the lower taxonomic levels (order and species). From a practical standpoint, `ArborCLIP` is highly accurate at the species level, and higher-level taxa can be deterministically derived from lower ones.
|
130 |
|
131 |
-
Addressing these limitations will further enhance the applicability of models like `ArborCLIP` in
|
132 |
-
real-world biodiversity monitoring tasks.
|
133 |
|
134 |
### Acknowledgements
|
135 |
This work was supported by the AI Research Institutes program supported by the NSF and USDA-NIFA under [AI Institute: for Resilient Agriculture](https://aiira.iastate.edu/), Award No. 2021-67021-35329. This was also
|
|
|
6 |
- zero-shot-image-classification
|
7 |
- clip
|
8 |
- biology
|
9 |
+
- biodiversity
|
10 |
+
- agronomy
|
11 |
- CV
|
12 |
- images
|
13 |
- animals
|
|
|
19 |
- multimodal
|
20 |
- knowledge-guided
|
21 |
datasets:
|
22 |
+
- Arboretum
|
23 |
- imageomics/TreeOfLife-10M
|
24 |
- iNat21
|
25 |
- BIOSCAN-1M
|
|
|
49 |
- **License:** MIT
|
50 |
- **Fine-tuned from model:** [OpenAI CLIP](https://github.com/mlfoundations/open_clip), [MetaCLIP](https://github.com/facebookresearch/MetaCLIP), [BioCLIP](https://github.com/Imageomics/BioCLIP)
|
51 |
|
52 |
+
These models were developed for the benefit of the AI community as an open-source product. Thus, we request that any derivative products are also open-source.
|
53 |
|
54 |
|
55 |
### Model Description
|
|
|
57 |
ArborCLIP is based on OpenAI's [CLIP](https://openai.com/research/clip) model.
|
58 |
The models were trained on [ARBORETUM-40M](https://baskargroup.github.io/Arboretum/) for the following configurations:
|
59 |
|
60 |
+
- **ARBORCLIP-O:** Trained a ViT-B/16 backbone initialized from the [OpenCLIP's](https://github.com/mlfoundations/open_clip) checkpoint. The training was conducted for 40 epochs.
|
61 |
+
- **ARBORCLIP-B:** Trained a ViT-B/16 backbone initialized from the [BioCLIP's](https://github.com/Imageomics/BioCLIP) checkpoint. The training was conducted for 8 epochs.
|
62 |
+
- **ARBORCLIP-M:** Trained a ViT-L/14 backbone initialized from the [MetaCLIP's](https://github.com/facebookresearch/MetaCLIP) checkpoint. The training was conducted for 12 epochs.
|
63 |
|
64 |
|
65 |
+
To access the checkpoints of the above models, go to the `Files and versions` tab and download the weights. These weights can be directly used for zero-shot classification and finetuning. The filenames correspond to the specific model weights -
|
66 |
+
- **ARBORCLIP-O:** - `arborclip-vit-b-16-from-openai-epoch-40.pt`,
|
67 |
+
- **ARBORCLIP-B:** - `arborclip-vit-b-16-from-bioclip-epoch-8.pt`
|
68 |
+
- **ARBORCLIP-M** - `arborclip-vit-l-14-from-metaclip-epoch-12.pt`
|
69 |
|
70 |
### Model Training
|
71 |
**See the [Model Training](https://github.com/baskargroup/Arboretum?tab=readme-ov-file#model-training) section on the [Github](https://github.com/baskargroup/Arboretum) for examples of how to use ArborCLIP models in zero-shot image classification tasks.**
|
|
|
134 |
level (kingdom), while models begin to benefit from specialist datasets like [ARBORETUM-40M](https://baskargroup.github.io/Arboretum/) and
|
135 |
[Tree-of-Life-10M](https://huggingface.co/datasets/imageomics/TreeOfLife-10M) at the lower taxonomic levels (order and species). From a practical standpoint, `ArborCLIP` is highly accurate at the species level, and higher-level taxa can be deterministically derived from lower ones.
|
136 |
|
137 |
+
Addressing these limitations will further enhance the applicability of models like `ArborCLIP` in real-world biodiversity monitoring tasks.
|
|
|
138 |
|
139 |
### Acknowledgements
|
140 |
This work was supported by the AI Research Institutes program supported by the NSF and USDA-NIFA under [AI Institute: for Resilient Agriculture](https://aiira.iastate.edu/), Award No. 2021-67021-35329. This was also
|