Edit model card

TAC depth encoder

This model is used for encoding a depth image into a dense feature.

Caution, the model does not contain the last FC layer. So, the output features are not aligned with RGB.

Model Details

Model Description

The model is pre-trained with RGB-D contrastive objectives, named TAC. Different from InfoNCE-based loss fuctions, TAC leverages the similarity between videos frames and estimate a similarity matrix as soft labels. The backbone of this version is ViT-B/32. The pre-training is conducted on a new unified RGB-D database, UniRGBD.

Model Sources

Uses

Direct Uses

from transformers import CLIPImageProcessor, CLIPVisionModel, CLIPVisionConfig
import numpy as np
tac_depth_model = CLIPVisionModel.from_pretrained("RavenK/TAC-ViT-base")
tac_depth_processor = CLIPImageProcessor.from_pretrained("RavenK/TAC-ViT-base")

# Assume test.png is a depth image with a scale factor 1000
MIN_DEPTH = 0.0
MAX_DEPTH = 10.0
DEPTH_SCALE = 1000

depth_path = "test.png"
depth = Image.open(depth_path)
depth = np.array(depth).astype("float32") / DEPTH_SCALE  # to meters
depth = np.clip(depth, MIN_DEPTH, MAX_DEPTH) # clip to [MIN_DEPTH, MAX_DEPTH]
depth = (depth - MIN_DEPTH) / (MAX_DEPTH - MIN_DEPTH) # normalize to [0,1]
depth = np.expand_dims(depth, axis=2).repeat(3, axis=2) # extend to 3 channels
depth = tac_depth_processor(depth, do_rescale=False, return_tensors="pt").pixel_values # preprocess (resize, normalize and to tensor)

outputs = tac_depth_model(pixel_values=depth)
outputs = outputs["last_hidden_state"][:, 0, :] # get embedding without FC. may be used for other downstream fine-tuning

Other Uses

Please refer to the demo in our code repository.

Citation

@ARTICLE{10288539,
  author={He, Zongtao and Wang, Liuyi and Dang, Ronghao and Li, Shu and Yan, Qingqing and Liu, Chengju and Chen, Qijun},
  journal={IEEE Transactions on Circuits and Systems for Video Technology}, 
  title={Learning Depth Representation From RGB-D Videos by Time-Aware Contrastive Pre-Training}, 
  year={2024},
  volume={34},
  number={6},
  pages={4143-4158},
  doi={10.1109/TCSVT.2023.3326373}}
Downloads last month
6
Safetensors
Model size
87.5M params
Tensor type
F32
·
Inference Examples
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.