README.md

metadata

title: ChatGPT Robotics
emoji: 🤖
colorFrom: blue
colorTo: red
sdk: gradio
sdk_version: 3.19.1
app_file: app.py
pinned: false
license: mit

ChatGPT + Robotics Gradio demo.

This is an accompaniment to our work on using ChatGPT for robotics applications. This demonstration space allows you to interact with ChatGPT using prompts and high-level APIs specified to robotics scenarios, and get the model to generate code that solves robotics problems.

For details, please check out our blog post: https://aka.ms/ChatGPT-Robotics
and our paper: https://www.microsoft.com/en-us/research/uploads/prod/2023/02/ChatGPT___Robotics.pdf

We encourage you to check out our main repo to access a full robot simulator with ChatGPT integration:
https://github.com/microsoft/PromptCraft-Robotics

Setup

1. Make an OpenAI ChatGPT account at chat.openai.com
2. Navigate to https://chat.openai.com/api/auth/session and copy the accessToken.

Usage

1. Paste the access token in the ChatGPT Login textbox and click Login.
2. Choose a sample prompt to start with. We provide the following prompts

airsim: Drone navigation scenario in the AirSim simulator.
embodied_agent: Embodied agent scenario with discrete actions (left, right, forward), and perception API for object detection.
embodied_agent_closed_loop: Embodied agent scenario but instead of interacting with ChatGPT normally, the user is expected to provide observations of the scene (converted into text) at every step, such as the locations of objects around.
manipulation: Robot arm scenario that is equipped with a suction pump that can pick up/release objects.
real_drone_sample: Drone navigation scenario for the DJI Tello drone as seen in the paper.

3. "Display Prompt" can be optionally used if you wish to see the prompt.
4. Click on Initialize, and then continue the conversation in the following chat box.