added force field decoder for gelsight

Browse files

Files changed (7) hide show

README.md +44 -0
gelsight_t1_forcefield_dino_vitbase_bg/checkpoints/epoch-0010.pth +3 -0
gelsight_t1_forcefield_dino_vitbase_bg/checkpoints/epoch-0014.pth +3 -0
gelsight_t1_forcefield_dino_vitbase_bg/checkpoints/epoch-0021.pth +3 -0
gelsight_t1_forcefield_dino_vitbase_bg/checkpoints/last.ckpt +3 -0
gelsight_t1_forcefield_dino_vitbase_bg/config.yaml +159 -0
gelsight_t1_forcefield_dino_vitbase_bg/config_tree.log +178 -0

README.md ADDED Viewed

	@@ -0,0 +1,44 @@

+---
+license: cc-by-nc-4.0
+tags:
+- sparsh
+- force field
+- gelsight mini
+---
+# Sparsh (DINO) + force field decoder for GelSight mini sensor
+We decode the touch representations from Sparsh into normal and shear force fields. This allows us to gather an intuition about what the representations capture in terms of forces in a way that is interpretable for humans.
+## How to Use
+For testing Sparsh(DINO) + force field decoder live, you only need a GelSight mini sensor. Follow these steps to run the demo:
+1. Clone the [sparsh repo](https://github.com/facebookresearch/sparsh.git)
+2. Create a folder for downloading the task checkpoints. For example, `${YOUR_PATH}/outputs_sparsh/checkpoints`.
+3. Download the Sparsh (DINO) base [checkpoint](https://huggingface.co/facebook/sparsh-dino-base)
+4. Download the decoder checkpoints from this repo.
+5. Connect the sensor to your PC.
+6. Make sure the device is recognized by the OS (you can use Cheese in Linux to see the video that the sensor is streaming).
+7. Running the demo for GelSight mini (please refer to the Sparsh repo README for more information about how to setup the path configs):
+```bash
+python demo_forcefield.py +experiment=digit/downstream_task/forcefield/gelsight_dino paths=${YOUR_PATH_CONFIG} paths.output_dir=${YOUR_PATH}/outputs_sparsh/checkpoints/ test.demo.gelsight_device_id=${YOUR_GELSIGHT_VIDEO_ID}`
+```
+The GelSight mini is recognized as a webcam. You can get the video ID by checking in a terminal `ls -l /dev/video*`.
+8. Take the sensor and slide it across the edge of a table, or across objects with interesting textures! Look at the normal field to localize where you're making contact on the sensor's surface. Look at the shear field to gather an intuition about the direction of the shear force that you applied while sliding the sensor. For example, slide the sensor over an edge up and down to get translational shear or rotate the sensor in place to see torsional slip!
+### BibTeX entry and citation info
+```bibtex
+    @inproceedings{
+    higuera2024sparsh,
+    title={Sparsh: Self-supervised touch representations for vision-based tactile sensing},
+    author={Carolina Higuera and Akash Sharma and Chaithanya Krishna Bodduluri and Taosha Fan and Patrick Lancaster and Mrinal Kalakrishnan and Michael Kaess and Byron Boots and Mike Lambeta and Tingfan Wu and Mustafa Mukadam},
+    booktitle={8th Annual Conference on Robot Learning},
+    year={2024},
+    url={https://openreview.net/forum?id=xYJn2e1uu8}
+}
+```

gelsight_t1_forcefield_dino_vitbase_bg/checkpoints/epoch-0010.pth ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:2342576673a2e731790e1a02b059f12572e5946aa25b48a172c30494c958d2ea
+size 14695019

gelsight_t1_forcefield_dino_vitbase_bg/checkpoints/epoch-0014.pth ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:fdc987d0a960ec6c9207ad7712693b807bc68743dc95500b78b94db427092081
+size 14695019

gelsight_t1_forcefield_dino_vitbase_bg/checkpoints/epoch-0021.pth ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:dc8a697da3131adaffd54bc4b03e8fb137a0affca689c296d145dd54ad0151c3
+size 14695019

gelsight_t1_forcefield_dino_vitbase_bg/checkpoints/last.ckpt ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:d62ad4094fec1d32a22ce180e7e099b901cac210c04cff1d3d974019f751d997
+size 541797495

gelsight_t1_forcefield_dino_vitbase_bg/config.yaml ADDED Viewed

	@@ -0,0 +1,159 @@

+paths:
+  data_root: /media/chiguera/GUM/
+  encoder_checkpoint_root: /media/chiguera/2TB/sparsh/encoders_460k/
+  log_dir: /media/chiguera/GUM/tactile_ssl/outputs_sparsh/${experiment_name}/logs/
+  output_dir: /media/chiguera/GUM/tactile_ssl/outputs_sparsh/${experiment_name}/
+  work_dir: ${hydra:runtime.cwd}
+wandb:
+  project: ${task_name}_${sensor}
+  entity: chiguera
+  save_dir: ${paths.output_dir}
+  id: ${hydra:job.id}_${experiment_name}
+  tags:
+  - ${ssl_name}
+  group: null
+  notes: null
+data:
+  train_val_split: null
+  train_data_budget: ${train_data_budget}
+  val_data_budget: ${val_data_budget}
+  max_train_data: null
+  sensor: gelsight_mini
+  dataset:
+    _target_: tactile_ssl.data.vision_tactile_forcefield.VisionTactileBackboneDataset
+    config:
+      sensor: ${data.sensor}
+      remove_bg: true
+      out_format: concat_ch_img
+      num_frames: 2
+      frame_stride: 5
+      path_dataset: ${paths.data_root}/datasets/gelsight/Object-Slide/
+      path_bgs: ${paths.data_root}/datasets/gelsight/Object-Slide/bgs/
+      list_datasets:
+      - strawberry/dataset_0
+      - strawberry/dataset_1
+      - strawberry/dataset_2
+      - plum/dataset_0
+      - plum/dataset_1
+      - plum/dataset_2
+      - icecream_cup/dataset_0
+      - icecream_cup/dataset_1
+      - icecream_cup/dataset_2
+      - donut/dataset_0
+      - donut/dataset_1
+      - donut/dataset_2
+      - cookie2/dataset_0
+      - cookie2/dataset_1
+      - cookie2/dataset_2
+      - bread/dataset_0
+      - bread/dataset_1
+      - bread/dataset_2
+      - banana/dataset_0
+      - banana/dataset_1
+      - banana/dataset_2
+      - bagel/dataset_0
+      - bagel/dataset_1
+      - bagel/dataset_2
+      list_datasets_test:
+      - bagel/dataset_3
+      - banana/dataset_3
+      - bread/dataset_3
+      - cookie2/dataset_3
+      - donut/dataset_3
+      - icecream_cup/dataset_3
+      - plum/dataset_3
+      - strawberry/dataset_3
+      look_in_folder: false
+      transforms:
+        with_augmentation: false
+        resize:
+        - 224
+        - 224
+        p_flip: 0.0
+        p_crop: 0.0
+        p_rot: 0.0
+  train_dataloader:
+    batch_size: 20
+    num_workers: 2
+    drop_last: true
+    pin_memory: true
+    persistent_workers: true
+    shuffle: true
+  val_dataloader:
+    batch_size: 20
+    num_workers: 2
+    drop_last: true
+    pin_memory: true
+    persistent_workers: true
+task:
+  _target_: tactile_ssl.downstream_task.ForceFieldModuleSL
+  model_encoder:
+    _target_: tactile_ssl.model.vit_${ssl_model_size}
+    img_size:
+    - 224
+    - 224
+    in_chans: 6
+    pos_embed_fn: sinusoidal
+    num_register_tokens: 1
+  model_task:
+    _target_: tactile_ssl.downstream_task.ForceFieldDecoderSL
+    embed_dim: ${ssl_model_size}
+  checkpoint_encoder: ${paths.encoder_checkpoint_root}/${ssl_name}_vit${ssl_model_size}.ckpt
+  checkpoint_task: /media/chiguera/GUM/tactile_ssl/outputs_sparsh/digit_t1_forcefield_dino_vitbase_bg/checkpoints/epoch-0031.pth
+  train_encoder: false
+  optim_cfg:
+    _partial_: true
+    _target_: torch.optim.Adam
+    lr: 0.0001
+  scheduler_cfg: null
+  ssl_config:
+    img_sz:
+    - 224
+    - 224
+    pose_estimator:
+      num_encoder_layers: 18
+    loss:
+      with_mask_supervision: false
+      with_sl_supervision: false
+      with_ssim: true
+      disparity_smoothness: 0.001
+      min_depth: 0.1
+      max_depth: 100.0
+  encoder_type: ${ssl_name}
+ssl_name: dino
+sensor: gelsight
+ckpt_path: null
+task_name: t1_forcefield
+ssl_model_size: base
+train_data_budget: 1.0
+val_data_budget: 1.0
+experiment_name: ${sensor}_${task_name}_${ssl_name}_vit${ssl_model_size}_bg
+seed: 42
+data_out_format: concat_ch_img
+num_frames: 2
+frame_stride: 5
+trainer:
+  max_epochs: 21
+  validation_frequency: 2
+  sanity_validate: false
+  save_checkpoint_dir: ${paths.output_dir}/checkpoints
+  checkpoint_interval_type: log
+  max_task_checkpoints: 10
+  save_probe_weights_only: true
+  limit_train_batches: 500
+  limit_val_batches: 150
+  use_distributed_sampler: false
+  devices:
+  - 0
+test:
+  data:
+    dataset_name:
+    - cookie2/dataset_0
+    batch_size: 1
+  tester:
+    _partial_: true
+    _target_: tactile_ssl.test.TestForceField
+  demo:
+    _partial_: true
+    _target_: tactile_ssl.test.DemoForceField
+  path_outputs: null

gelsight_t1_forcefield_dino_vitbase_bg/config_tree.log ADDED Viewed

	@@ -0,0 +1,178 @@

+CONFIG
+├── data
+│   └── train_val_split: null
+│       train_data_budget: 1.0
+│       val_data_budget: 1.0
+│       max_train_data: null
+│       sensor: gelsight_mini
+│       dataset:
+│         _target_: tactile_ssl.data.vision_tactile_forcefield.VisionTactileBackboneDataset
+│         config:
+│           sensor: gelsight_mini
+│           remove_bg: true
+│           out_format: concat_ch_img
+│           num_frames: 2
+│           frame_stride: 5
+│           path_dataset: /media/chiguera/GUM//datasets/gelsight/Object-Slide/
+│           path_bgs: /media/chiguera/GUM//datasets/gelsight/Object-Slide/bgs/
+│           list_datasets:
+│           - strawberry/dataset_0
+│           - strawberry/dataset_1
+│           - strawberry/dataset_2
+│           - plum/dataset_0
+│           - plum/dataset_1
+│           - plum/dataset_2
+│           - icecream_cup/dataset_0
+│           - icecream_cup/dataset_1
+│           - icecream_cup/dataset_2
+│           - donut/dataset_0
+│           - donut/dataset_1
+│           - donut/dataset_2
+│           - cookie2/dataset_0
+│           - cookie2/dataset_1
+│           - cookie2/dataset_2
+│           - bread/dataset_0
+│           - bread/dataset_1
+│           - bread/dataset_2
+│           - banana/dataset_0
+│           - banana/dataset_1
+│           - banana/dataset_2
+│           - bagel/dataset_0
+│           - bagel/dataset_1
+│           - bagel/dataset_2
+│           list_datasets_test:
+│           - bagel/dataset_3
+│           - banana/dataset_3
+│           - bread/dataset_3
+│           - cookie2/dataset_3
+│           - donut/dataset_3
+│           - icecream_cup/dataset_3
+│           - plum/dataset_3
+│           - strawberry/dataset_3
+│           look_in_folder: false
+│           transforms:
+│             with_augmentation: false
+│             resize:
+│             - 224
+│             - 224
+│             p_flip: 0.0
+│             p_crop: 0.0
+│             p_rot: 0.0
+│       train_dataloader:
+│         batch_size: 20
+│         num_workers: 2
+│         drop_last: true
+│         pin_memory: true
+│         persistent_workers: true
+│         shuffle: true
+│       val_dataloader:
+│         batch_size: 20
+│         num_workers: 2
+│         drop_last: true
+│         pin_memory: true
+│         persistent_workers: true
+│
+├── trainer
+│   └── max_epochs: 21
+│       validation_frequency: 2
+│       sanity_validate: false
+│       save_checkpoint_dir: /media/chiguera/GUM/tactile_ssl/outputs_sparsh/gelsight_t1_forcefield_dino_vitb
+│       checkpoint_interval_type: log
+│       max_task_checkpoints: 10
+│       save_probe_weights_only: true
+│       limit_train_batches: 500
+│       limit_val_batches: 150
+│       use_distributed_sampler: false
+│       devices:
+│       - 0
+│
+├── paths
+│   └── data_root: /media/chiguera/GUM/
+│       encoder_checkpoint_root: /media/chiguera/2TB/sparsh/encoders_460k/
+│       log_dir: /media/chiguera/GUM/tactile_ssl/outputs_sparsh/gelsight_t1_forcefield_dino_vitbase_bg/logs/
+│       output_dir: /media/chiguera/GUM/tactile_ssl/outputs_sparsh/gelsight_t1_forcefield_dino_vitbase_bg/
+│       work_dir: /media/chiguera/2TB/sparsh/tactile-ssl
+│
+├── wandb
+│   └── project: t1_forcefield_gelsight
+│       entity: chiguera
+│       save_dir: /media/chiguera/GUM/tactile_ssl/outputs_sparsh/gelsight_t1_forcefield_dino_vitbase_bg/
+│       id: 2024.09.30_11-38_gelsight_t1_forcefield_dino_vitbase_bg
+│       tags:
+│       - dino
+│       group: null
+│       notes: null
+│
+├── task
+│   └── _target_: tactile_ssl.downstream_task.ForceFieldModuleSL
+│       model_encoder:
+│         _target_: tactile_ssl.model.vit_base
+│         img_size:
+│         - 224
+│         - 224
+│         in_chans: 6
+│         pos_embed_fn: sinusoidal
+│         num_register_tokens: 1
+│       model_task:
+│         _target_: tactile_ssl.downstream_task.ForceFieldDecoderSL
+│         embed_dim: base
+│       checkpoint_encoder: /media/chiguera/2TB/sparsh/encoders_460k//dino_vitbase.ckpt
+│       checkpoint_task: /media/chiguera/GUM/tactile_ssl/outputs_sparsh/digit_t1_forcefield_dino_vitbase_bg/
+│       train_encoder: false
+│       optim_cfg:
+│         _partial_: true
+│         _target_: torch.optim.Adam
+│         lr: 0.0001
+│       scheduler_cfg: null
+│       ssl_config:
+│         img_sz:
+│         - 224
+│         - 224
+│         pose_estimator:
+│           num_encoder_layers: 18
+│         loss:
+│           with_mask_supervision: false
+│           with_sl_supervision: false
+│           with_ssim: true
+│           disparity_smoothness: 0.001
+│           min_depth: 0.1
+│           max_depth: 100.0
+│       encoder_type: dino
+│
+├── ssl_name
+│   └── dino
+├── sensor
+│   └── gelsight
+├── ckpt_path
+│   └── None
+├── task_name
+│   └── t1_forcefield
+├── ssl_model_size
+│   └── base
+├── train_data_budget
+│   └── 1.0
+├── val_data_budget
+│   └── 1.0
+├── experiment_name
+│   └── gelsight_t1_forcefield_dino_vitbase_bg
+├── seed
+│   └── 42
+├── data_out_format
+│   └── concat_ch_img
+├── num_frames
+│   └── 2
+├── frame_stride
+│   └── 5
+└── test
+    └── data:
+          dataset_name:
+          - cookie2/dataset_0
+          batch_size: 1
+        tester:
+          _partial_: true
+          _target_: tactile_ssl.test.TestForceField
+        demo:
+          _partial_: true
+          _target_: tactile_ssl.test.DemoForceField
+        path_outputs: null