File size: 2,906 Bytes
2adb382 8a2ea80 2adb382 3bc73cb 2adb382 b0c2da9 2adb382 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 |
---
license: mit
---
# TAC depth encoder
<!-- Provide a quick summary of what the model is/does. -->
This model is used for encoding a depth image into a dense feature.
**Caution,** the model does not contain the last FC layer.
So, the output features are not aligned with RGB.
## Model Details
### Model Description
<!-- Provide a longer summary of what this model is. -->
The model is pre-trained with RGB-D contrastive objectives, named TAC.
Different from InfoNCE-based loss fuctions, TAC leverages the similarity between videos frames and estimate a similarity matrix as soft labels.
The backbone of this version is ViT-B/32.
The pre-training is conducted on a new unified RGB-D database, UniRGBD.
### Model Sources
<!-- Provide the basic links for the model. -->
- **Repository:** [TAC](https://github.com/RavenKiller/TAC)
- **Paper:** [Learning Depth Representation from RGB-D Videos by Time-Aware Contrastive Pre-training](https://ieeexplore.ieee.org/document/10288539)
## Uses
<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
### Direct Uses
```
from transformers import CLIPImageProcessor, CLIPVisionModel, CLIPVisionConfig
import numpy as np
tac_depth_model = CLIPVisionModel.from_pretrained("RavenK/TAC-ViT-base")
tac_depth_processor = CLIPImageProcessor.from_pretrained("RavenK/TAC-ViT-base")
# Assume test.png is a depth image with a scale factor 1000
MIN_DEPTH = 0.0
MAX_DEPTH = 10.0
DEPTH_SCALE = 1000
depth_path = "test.png"
depth = Image.open(depth_path)
depth = np.array(depth).astype("float32") / DEPTH_SCALE # to meters
depth = np.clip(depth, MIN_DEPTH, MAX_DEPTH) # clip to [MIN_DEPTH, MAX_DEPTH]
depth = (depth - MIN_DEPTH) / (MAX_DEPTH - MIN_DEPTH) # normalize to [0,1]
depth = np.expand_dims(depth, axis=2).repeat(3, axis=2) # extend to 3 channels
depth = tac_depth_processor(depth, do_rescale=False, return_tensors="pt").pixel_values # preprocess (resize, normalize and to tensor)
outputs = tac_depth_model(pixel_values=depth)
outputs = outputs["last_hidden_state"][:, 0, :] # get embedding without FC. may be used for other downstream fine-tuning
```
### Other Uses
Please refer to the [demo](https://github.com/RavenKiller/TAC/blob/main/scripts/demo.ipynb) in our code repository.
## Citation
<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
```
@ARTICLE{10288539,
author={He, Zongtao and Wang, Liuyi and Dang, Ronghao and Li, Shu and Yan, Qingqing and Liu, Chengju and Chen, Qijun},
journal={IEEE Transactions on Circuits and Systems for Video Technology},
title={Learning Depth Representation From RGB-D Videos by Time-Aware Contrastive Pre-Training},
year={2024},
volume={34},
number={6},
pages={4143-4158},
doi={10.1109/TCSVT.2023.3326373}}
```
|