File size: 5,042 Bytes
09cf70b 0c3c527 09cf70b 0c3c527 494c25e 0c3c527 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 |
---
license: mit
language:
- en
tags:
- robotics
- motion planning
---
# Neural MP
Neural MP is a machine learning-based motion planning system for robotic manipulation tasks. It combines neural networks trained on large-scale simulated data with lightweight optimization techniques to generate efficient, collision-free trajectories. Neural MP is designed to generalize across diverse environments and obstacle configurations, making it suitable for both simulated and real-world robotic applications. This repository contains the model weights for Neural MP.
All Neural MP checkpoints, as well as our [codebase](https://github.com/mihdalal/neuralmotionplanner) are released under an MIT License.
For full details, please read our [paper](https://mihdalal.github.io/neuralmotionplanner/resources/paper.pdf) and see [our project page](https://mihdalal.github.io/neuralmotionplanner/).
## Model Summary
- **Developed by:** The Neural MP team consisting of researchers from Carnegie Mellon University.
- **Language(s) (NLP):** en
- **License:** MIT
- **Pretraining Dataset:** Coming soon
- **Repository:** [https://github.com/mihdalal/neuralmotionplanner](https://github.com/mihdalal/neuralmotionplanner)
- **Paper:** Coming soon
- **Project Page & Videos:** [https://mihdalal.github.io/neuralmotionplanner/](https://mihdalal.github.io/neuralmotionplanner/)
## Installation
Please read [here](https://github.com/mihdalal/neural_mp?tab=readme-ov-file#installation-instructions) for detailed instructions
## Usage
Neural MP model takes in 3D point cloud and start & goal angles of the Franka robot as input, and predict 7-DoF delta joint actions. We provide a wrapper class [NeuralMP](https://github.com/mihdalal/neural_mp/blob/master/neural_mp/real_utils/neural_motion_planner.py) for inference and deploy our model in the real world.
Here's an deployment example with the Manimo Franka control library:
Note: using Manimo is not required, you may use other Franka control libraries by creating a wrapper class which inherits from FrankaRealEnv (see [franka_real_env.py](https://github.com/mihdalal/neural_mp/blob/master/neural_mp/envs/franka_real_env.py))
```python
import argparse
import numpy as np
from neural_mp.envs.franka_real_env import FrankaRealEnvManimo
from neural_mp.real_utils.neural_motion_planner import NeuralMP
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument(
"--mdl_url",
type=str,
default="mihdalal/NeuralMP",
help="hugging face url to load the neural_mp model",
)
parser.add_argument(
"--cache-name",
type=str,
default="scene1_single_blcok",
help="Specify the scene cache file with pcd and rgb data",
)
parser.add_argument(
"--use-cache",
action="store_true",
help=("If set, will use pre-stored point clouds"),
)
parser.add_argument(
"--debug-combined-pcd",
action="store_true",
help=("If set, will show visualization of the combined pcd"),
)
parser.add_argument(
"--denoise-pcd",
action="store_true",
help=("If set, will apply denoising to the pcds"),
)
parser.add_argument(
"--train-mode", action="store_true", help=("If set, will eval with policy in training mode")
)
parser.add_argument(
"--tto", action="store_true", help=("If set, will apply test time optimization")
)
parser.add_argument(
"--in-hand", action="store_true", help=("If set, will enable in hand mode for eval")
)
parser.add_argument(
"--in-hand-params",
nargs="+",
type=float,
default=[0.1, 0.1, 0.1, 0.0, 0.0, 0.1, 0.0, 0.0, 0.0, 1.0],
help="Specify the bounding box of the in hand object. 10 params in total [size(xyz), pos(xyz), ori(xyzw)] 3+3+4.",
)
args = parser.parse_args()
env = FrankaRealEnvManimo()
neural_mp = NeuralMP(
env=env,
model_url=args.mdl_url,
train_mode=args.train_mode,
in_hand=args.in_hand,
in_hand_params=args.in_hand_params,
visualize=True,
)
points, colors = neural_mp.get_scene_pcd(
use_cache=args.use_cache,
cache_name=args.cache_name,
debug_combined_pcd=args.debug_combined_pcd,
denoise=args.denoise_pcd,
)
# specify start and goal configurations
start_config = np.array([-0.538, 0.628, -0.061, -1.750, 0.126, 2.418, 1.610])
goal_config = np.array([1.067, 0.847, -0.591, -1.627, 0.623, 2.295, 2.580])
if args.tto:
trajectory = neural_mp.motion_plan_with_tto(
start_config=start_config,
goal_config=goal_config,
points=points,
colors=colors,
)
else:
trajectory = neural_mp.motion_plan(
start_config=start_config,
goal_config=goal_config,
points=points,
colors=colors,
)
success, joint_error = neural_mp.execute_motion_plan(trajectory, speed=0.2)
``` |