ChatGPT for Robotics: Design Principles and Model Abilities

December 2023

tl;dr: A pipeline to use ChatGPT for robotics tasks via prompt engineering, and writing high level code for execution. Similar to CaP (code as policies).

Overall impression

Robotics systems, unlike text-only apps, require deep understanding of real-world physics, environmental context, and the ability to perform physical actions.

LLM’s out-of-the-box understanding of basic concepts (control, camera geometry, physical form factors) makes it an excellenet choice to build generalizable and user-friendly robotics pipeline.

PromptCraft replaces a specialized engineer-in-the-loop with a user-on-the-loop. –> How to polish the interaction between user and the robot or automate as much as possible is the key to real world application (productization).

PromptCraft is NOT a fully automated process, and needs human on the loop to monitor and intervene in case of unexpected behavior generated by LLM, especially so for safety-critical application.

PromptCraft is not using VLM, but rather only LLM.

Key ideas

Technical details