Create README.md

README.md

@@ -0,0 +1,77 @@
+---
+license: mit
+---
+# TAC depth encoder
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+This model is used for encoding a depth image into a dense feature.
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+The model is pre-trained with RGB-D contrastive objectives, named TAC. 
+Different from InfoNCE-based loss fuctions, TAC leverages the similarity between videos frames and estimate a similarity matrix as soft labels.
+The backbone of this version is ViT-B/32.
+The pre-training is conducted on a new unified RGB-D database, UniRGBD.
+
+### Model Sources
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [TAC](https://github.com/RavenKiller/TAC)
+- **Paper:** [Learning Depth Representation from RGB-D Videos by Time-Aware Contrastive Pre-training](https://ieeexplore.ieee.org/document/10288539)
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Uses
+
+```
+from transformers import CLIPImageProcessor, CLIPVisionModel, CLIPVisionConfig
+import numpy as np
+tac_depth_model = CLIPVisionModel.from_pretrained("RavenK/TAC-ViT-base")
+tac_depth_processor = CLIPImageProcessor.from_pretrained("RavenK/TAC-ViT-base")
+
+# Assume test.png is a depth image with a scale factor 1000
+MIN_DEPTH = 0.0
+MAX_DEPTH = 10.0
+DEPTH_SCALE = 1000
+
+depth_path = "test.png"
+depth = Image.open(depth_path)
+depth = np.array(depth).astype("float32") / DEPTH_SCALE # to meters
+depth = np.clip(depth, MIN_DEPTH, MAX_DEPTH) # clip to [MIN_DEPTH, MAX_DEPTH]
+depth = (depth - MIN_DEPTH) / (MAX_DEPTH - MIN_DEPTH) # normalize to [0,1]
+depth = np.expand_dims(depth, axis=2).repeat(3, axis=2) # extend to 3 channels
+depth = tac_depth_processor(depth, do_rescale=False, return_tensors="pt").pixel_values # preprocess (resize, normalize and to tensor)
+
+outputs = tac_depth_model(pixel_values=depth)
+outputs = outputs["last_hidden_state"][:, 0, :] # get embedding without FC. may be used for other downstream fine-tuning
+```
+
+### Other Uses
+
+Please refer to our code repository to get more details.
+
+## Citation
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+```
+@ARTICLE{10288539,
+ author={He, Zongtao and Wang, Liuyi and Dang, Ronghao and Li, Shu and Yan, Qingqing and Liu, Chengju and Chen, Qijun},
+ journal={IEEE Transactions on Circuits and Systems for Video Technology}, 
+ title={Learning Depth Representation from RGB-D Videos by Time-Aware Contrastive Pre-training}, 
+ year={2023},
+ volume={},
+ number={},
+ pages={1-1},
+ doi={10.1109/TCSVT.2023.3326373}}
+```
+
+