---
title: "From Request to Robot Action: Inside the Franka Agentic Cookbook"
description: "Agentic AI for robotics: the Franka Cookbook turns language into robot actions on the FR3 Duo, tested in an NVIDIA Isaac Sim digital twin."
image: https://franka.de/hubfs/Franka%20Cookbook%20Virtual%20Twin.png
---
[](https://franka.de?hsLang=en)
[](https://franka.de/?hsLang=en)
Menu
- [Products](https://franka.de/products-overview?hsLang=en)
- [Franka Research 3](https://franka.de/franka-research-3?hsLang=en)
- [Diana 7 by Agile Robots](https://franka.de/diana-7?hsLang=en)
- [Accessories](https://franka.de/accessories?hsLang=en)
- [Prototypes](https://franka.de/product-prototypes?hsLang=en)
- [Franka Research 3 Duo](https://franka.de/fr3-duo?hsLang=en)
- [Mobile FR3 Duo](https://franka.de/mobile-fr3-duo?hsLang=en)
- [Tactile Mobile Robot](https://franka.de/tmr?hsLang=en)
- [Franka GELLO and GELLO Duo](https://franka.de/gello?hsLang=en)
- [Applications](https://franka.de/applications?hsLang=en)
- [Physical AI](https://franka.de/applications?hsLang=en#physical-ai-applications)
- [Research](https://franka.de/applications?hsLang=en#application-research)
- [Service / Innovative](https://franka.de/applications?hsLang=en#application-innovation)
- [Art](https://franka.de/art?hsLang=en)
- [Industry](https://franka.de/applications?hsLang=en#application-industry)
- [Resources](https://franka.de/resources?hsLang=en)
- [Product Documentation](https://franka.de/documents?hsLang=en)
- [Development](https://franka.de/develop?hsLang=en)
- [Community Contributions](https://franka.de/community?hsLang=en)
- [Community Forum](https://www.franka-community.de/)
- [Webinars](https://franka.de/webinars?hsLang=en)
- [Careers](https://franka.de/careers?hsLang=en)
- [Company](https://franka.de/company?hsLang=en)
- [About](https://franka.de/company?hsLang=en)
- [Contact](https://franka.de/contact?hsLang=en)
- [News](https://franka.de/news)
- [Events](https://franka.de/events?hsLang=en)
- [Sales Partners](https://franka.de/partners?hsLang=en)
English
- [Deutsch – Deutschland](https://lp.franka.de/de-de/news/von-der-anfrage-zur-roboteraktion-ein-blick-in-das-franka-agentic-cookbook)
- [English](https://franka.de/news/from-request-to-robot-action-inside-the-franka-agentic-cookbook)
- [中文 (Chinese)](https://www.franka-robotics.cn)
# From Request to Robot Action: Inside the Franka Agentic Cookbook
Published by [Özlem Odeh](https://franka.de/news/author/özlem-odeh) on 21.10.2026
“Read the technical drawing and build the depicted construction using the available parts.”
For a person, this instruction sounds straightforward. For a robotic system, it involves several challenges. The system must understand the request, interpret the technical drawing, locate the parts, determine their positions, calculate suitable grasps, plan collision-free trajectories and execute the assembly sequence.
The Franka Cookbook brings these capabilities together in an agentic control architecture for a [FR3 Duo](https://franka.de/de/fr3-duo?hsLang=en) cells, a robot workstation with two Franka Research 3 arms. Its name also offers a useful way to understand the system. The name “Cookbook” reflects how reusable skills can be combined into a complete task, much like cooking techniques are combined in a recipe. The orchestrator acts as the head chef, coordinating specialist agents with defined roles and tools.
Franka Robotics, an [Agile Robots](https://www.agile-robots.com/en/) company, and NVIDIA are demonstrating how this principle can be applied to a physical robotic system.
## The orchestrator coordinates but does not act
At the center of the network is an orchestrator agent. It receives the operator’s request, breaks it down into steps and decides which specialist should handle each one.
Like the head chef in a kitchen, the orchestrator coordinates the work without taking over the individual stations. It has no direct access to the robot tools. Every action must be delegated to one of several specialist agents, including:
- The **perception agent** that uses camera data to detect objects and estimate their position and orientation.
- The **embodiment agent** that handles geometry, coordinate transformations and trajectory planning.
- The **desk agent** that prepares the physical robot by managing system functions such as operating mode and brakes.
- The **libfranka agent** that reads the live robot state and executes planned movements on the physical arms.
- The **simulation agent** that controls the FR3 Duo digital twin, including its arms and grippers.
Each specialist accesses only the tools required for its role through a dedicated Model Context Protocol (MCP) server. MCP is an open standard for connecting AI agents to external tools. This is a central principle of the architecture. The orchestrator can plan a complex task, while the individual agents remain focused on clearly defined parts of the system.
*Graphic source NVIDIA: How Cookbook-agents work together to fulfil the task*
In the assembly example, an operator could ask, “Which parts shown in this drawing are available on the table?” The orchestrator would involve the perception agent to locate the parts.[NVIDIA FoundationPose](https://catalog.ngc.nvidia.com/orgs/nvidia/isaac/models/foundationpose/) would estimate their 3D positions and orientations.
Questions such as “Can either arm reach this part?” or “How could the robot grasp it and place it here?” would involve the embodiment agent, which evaluates reachability and plans movements using the observed positions and the robot’s kinematic model. A request to “Try the planned movement in the digital twin” would involve the simulation agent, which controls the digital twin in the[NVIDIA Isaac Sim](https://developer.nvidia.com/isaac/sim) open simulation framework.
Through a single conversational interface, the operator can inspect available parts, assess reachability and test planned movements in the digital twin, while the orchestrator coordinates the underlying capabilities.
*FR3 Duo Setup performing a task in the virtual environment*
If the agents are the kitchen team, NVIDIA technologies provide their equipment: [NVIDIA NIM](https://developer.nvidia.com/nim) supplies the AI models that allow the orchestrator and the specialists to understand and plan, NVIDIA FoundationPose determines where an object is and how it lies, and[NVIDIA Isaac Sim](https://developer.nvidia.com/isaac/sim) provides the digital twin for testing robot workflows. Franka’s FR3 Duo supplies the physical platform, with Franka software connecting the system to the robot arms. Together, these components let developers explore how AI-driven task coordination connects to perception, simulation and robot control.
## From the test kitchen to the real cell
Before a new recipe is served, it can be developed and refined in a test kitchen. The Cookbook applies a similar principle through a digital twin of the FR3 Duo cell.
The agent network supports two operating modes. In simulation, the agents interact with the robot and its surrounding environment, both represented in the digital twin. This allows developers to develop and evaluate workflows without access to the physical cell. In hardware mode, the network connects to the real FR3 Duo and its live robot and sensor data.
The two control paths use separate services and communication networks. The simulation agent only receives tools for controlling the digital twin and cannot move a physical arm. Real motion is handled by agents and interfaces dedicated to the hardware.
The same overall architecture can therefore support virtual development and physical execution while maintaining a clear boundary between the two environments.
## Why this matters in practice
For manufacturers and system integrators, the approach addresses several everyday challenges. Instead of programming every movement step by step, operators can describe a task in natural language, and the system works out the necessary steps. Because each agent has a clearly defined role and only the tools it needs, errors are easier to trace and unintended actions are less likely.
Testing workflows in the digital twin first means that problems can be identified before they reach the real cell, reducing commissioning effort and downtime. As a reusable reference, the Cookbook can also shorten the path from an idea to a working application and lower the integration effort for new robotic tasks.
## A reusable reference
The partners are preparing the Cookbook for public release as an open reference stack, with code, architectural documentation and examples. Future extensions could include broader use of [NVIDIA OpenShell](https://build.nvidia.com/openshell) for agent runtime security and additional specialist capabilities. The planned release will give developers and partners a starting point for exploring these directions and adapting the architecture to their own robot workflows.
[Back to Blog](https://franka.de/news)
- [Tweet](https://twitter.com/share)
## Related Articles
[](https://franka.de/news/franka-robotics-launches-franka-ai-companion?hsLang=en)
## [Franka Robotics Launches Franka AI Companion: Empowering Innovation in Robotics Research](https://franka.de/news/franka-robotics-launches-franka-ai-companion?hsLang=en)
[Read More](https://franka.de/news/franka-robotics-launches-franka-ai-companion?hsLang=en)
[](https://franka.de/news/real-world-robotics-insights-from-fr3-duo-mobile-showcase?hsLang=en)
## [Towards Real-World Robotics: Takeaways from Launching the FR3 Duo Prototype and its Mobile Counterpart](https://franka.de/news/real-world-robotics-insights-from-fr3-duo-mobile-showcase?hsLang=en)
At Automatica 2025, we presented two new prototypes that drew strong interest from roboticists: the
[Read More](https://franka.de/news/real-world-robotics-insights-from-fr3-duo-mobile-showcase?hsLang=en)
[](https://franka.de/news/franka-robotics-advance-autonomous-medical-robotics-built-on-nvidia?hsLang=en)
## [Franka Robotics Advance Autonomous Medical Robotics Built on NVIDIA](https://franka.de/news/franka-robotics-advance-autonomous-medical-robotics-built-on-nvidia?hsLang=en)
Franka Robotics is advancing automation in healthcare, by integrating Franka Robotics’ systems with...
[Read More](https://franka.de/news/franka-robotics-advance-autonomous-medical-robotics-built-on-nvidia?hsLang=en)
## FOLLOW US
1. [Follow us on Facebook](https://www.facebook.com/frankarobotics/)
2. [Follow us on LinkedIn](https://www.linkedin.com/company/frankarobotics)
3. [Follow us on X](https://twitter.com/FRANKAROBOTICS)
4. [Follow us on Instagram](https://www.instagram.com/frankarobotics/)
5. [Follow us on YouTube](https://www.youtube.com/channel/UCjvmirH4BRbWoQwYO4EmXFQ)
6. [Contact us](https://franka.de/contact?hsLang=en)
1. [Imprint](https://franka.de/imprint?hsLang=en)
2. [Privacy Policy](https://franka.de/privacy-policy?hsLang=en)
3. [GTC](https://franka.de/hubfs/New_GTC_EN_FrankaRobotics_040226.pdf?hsLang=en)
4. [AGB](https://franka.de/hubfs/New_GTC_GER_FrankaRobotics_040226.pdf?hsLang=en)
Franka Robotics GmbH
Koppstraße 12
81379 Munich – Germany
Phone: +49 89 2006069 20
Email: [info@franka.de](mailto:sales@franka.de)
Mo. - Fr. 8:30am - 5:00pm CEST
© 2026 Franka Robotics GmbH.
All rights reserved
```json
{
"@context" : "https://schema.org",
"@type" : "VideoObject",
"contentUrl" : "https://24883234.fs1.hubspotusercontent-eu1.net/hubfs/24883234/video_assets/494487000260/inherited/web_optimized.mp4",
"dateModified" : "2026-10-19T12:07:41.382Z",
"duration" : "PT1M33S",
"height" : 720,
"name" : "Franka_Cookbook_digital",
"thumbnailUrl" : "https://24883234.fs1.hubspotusercontent-eu1.net/hubfs/24883234/Franka_Cookbook_digital.mp4/medium.jpg",
"uploadDate" : "2026-10-19T12:07:41.380Z",
"width" : 1280
}
```