Documentation

Python scripts

These files are for our monocular 3D Tracking pipeline:

main.py Execute 3D photo inpainting

mesh.py Functions about context-aware depth inpainting

mesh_tools.py Some common functions used in mesh.py

utils.py Some common functions used in image preprocessing, data loading

networks.py Network architectures of inpainting model

MiDaS/

run.py Execute depth estimation

monodepth_net.py Network architecture of depth estimation model

MiDaS_utils.py Some common functions in depth estimation

Configuration

argument.yml

depth_edge_model_ckpt: checkpoints/EdgeModel.pth
- Pretrained model of depth-edge inpainting
depth_feat_model_ckpt: checkpoints/DepthModel.pth
- Pretrained model of depth inpainting
rgb_feat_model_ckpt: checkpoints/ColorModel.pth
- Pretrained model of color inpainting
MiDaS_model_ckpt: MiDaS/model.pt
- Pretrained model of depth estimation
use_boostmonodepth: True
- Use BoostMonocularDepth to get sharper monocular depth estimation
fps: 40
- Frame per second of output rendered video
num_frames: 240
- Total number of frames in output rendered video
x_shift_range: [-0.03, -0.03, -0.03]
- The translations on x-axis of output rendered videos.
- This parameter is a list. Each element corresponds to a specific camera motion.
y_shift_range: [-0.00, -0.00, -0.03]
- The translations on y-axis of output rendered videos.
- This parameter is a list. Each element corresponds to a specific camera motion.
z_shift_range: [-0.07, -0.07, -0.07]
- The translations on z-axis of output rendered videos.
- This parameter is a list. Each element corresponds to a specific camera motion.
traj_types: ['straight-line', 'circle', 'circle']
- The type of camera trajectory.
- This parameter is a list.
- Currently, we only privode straight-line and circle.
video_postfix: ['zoom-in', 'swing', 'circle']
- The postfix of video.
- This parameter is a list.
Note that the number of elements in x_shift_range, y_shift_range, z_shift_range, traj_types and video_postfix should be equal.
specific: ''
- The specific image name, use this to specify the image to be executed. By default, all the image in the folder will be executed.
longer_side_len: 960
- The length of larger dimension in output resolution.
src_folder: image
- Input image directory.
depth_folder: depth
- Estimated depth directory.
mesh_folder: mesh
- Output 3-D mesh directory.
video_folder: video
- Output rendered video directory
load_ply: False
- Action to load existed mesh (.ply) file
save_ply: True
- Action to store the output mesh (.ply) file
- Disable this option save_ply: False to reduce the computational time.
inference_video: True
- Action to rendered the output video
gpu_ids: 0
- The ID of working GPU. Leave it blank or negative to use CPU.
offscreen_rendering: True
- If you're executing the process in a remote server (via ssh), please switch on this flag.
- Sometimes, using off-screen rendering result in longer execution time.
img_format: '.jpg'
- Input image format.
depth_format: '.npy'
- Input depth (disparity) format. Use NumPy array file as default.
- If the user wants to edit the depth (disparity) map manually, we provide .png format depth (disparity) map.
  - Remember to switch this parameter from .npy to .png when using depth (disparity) map with .png format.
require_midas: True
- Set it to True if the user wants to use depth map estimated by MiDaS.
- Set it to False if the user wants to use manually edited depth map.
- If the user wants to edit the depth (disparity) map manually, we provide .png format depth (disparity) map.
  - Remember to switch this parameter from True to False when using manually edited depth map.
depth_threshold: 0.04
- A threshold in disparity, adjacent two pixels are discontinuity pixels if the difference between them excceed this number.
ext_edge_threshold: 0.002
- The threshold to define inpainted depth edge. A pixel in inpainted edge map belongs to extended depth edge if the value of that pixel exceeds this number,
sparse_iter: 5
- Total iteration numbers of bilateral median filter
filter_size: [7, 7, 5, 5, 5]
- Window size of bilateral median filter in each iteration.
sigma_s: 4.0
- Intensity term of bilateral median filter
sigma_r: 0.5
- Spatial term of bilateral median filter
redundant_number: 12
- The number defines short segments. If a depth edge is shorter than this number, it is a short segment and removed.
background_thickness: 70
- The thickness of synthesis area.
context_thickness: 140
- The thickness of context area.
background_thickness_2: 70
- The thickness of synthesis area when inpaint second time.
context_thickness_2: 70
- The thickness of context area when inpaint second time.
discount_factor: 1.00
log_depth: True
- The scale of depth inpainting. If true, performing inpainting in log scale. Otherwise, performing in linear scale.
largest_size: 512
- The largest size of inpainted image patch.
depth_edge_dilate: 10
- The thickness of dilated synthesis area.
depth_edge_dilate_2: 5
- The thickness of dilated synthesis area when inpaint second time.
extrapolate_border: True
- Action to extrapolate out-side the border.
extrapolation_thickness: 60
- The thickness of extrapolated area.
repeat_inpaint_edge: True
- Action to apply depth edge inpainting model repeatedly. Sometimes inpainting depth edge once results in short inpinated edge, apply depth edge inpainting repeatedly could help you prolong the inpainted depth edge.
crop_border: [0.03, 0.03, 0.05, 0.03]
- The fraction of pixels to crop out around the borders [top, left, bottom, right].
anti_flickering: True
- Action to avoid flickering effect in the output video.
- This may result in longer computational time in rendering phase.