Skip to content

Robot Learning: Complete Workflow

Welcome to the complete guide for training intelligent robotic systems. This documentation provides a holistic, end-to-end workflow from data collection to deployment.

Complete Workflow Overview

Training robot learning systems follows a systematic workflow:

graph TD
    A[1. Data Collection] --> B[2. Dataset Preparation]
    B --> C[3. Simulation Setup]
    C --> D[4. Training]
    D --> E[5. Evaluation]
    E --> F{Good Performance?}
    F -->|No| G[Adjust & Retrain]
    G --> D
    F -->|Yes| H[6. Real Robot Deployment]
    H --> I[Monitor & Iterate]
    I --> A

Step-by-Step Guide

1. Data Collection

Collect high-quality demonstrations or prepare for environment interaction.

Methods: - Teleoperation for demonstrations - Kinesthetic teaching - VR interfaces - Existing datasets

→ Data Collection Guide

2. Dataset Preparation

Format your data using the LeRobot standard for compatibility and reproducibility.

What you'll do: - Structure episodes and observations - Store multi-modal data (images, states, actions) - Add metadata and annotations - Validate dataset format

→ LeRobot Format | → Format Specification

3. Simulation Setup

Set up your simulation environment for safe, fast training.

Choose your simulator: - IsaacSim: For photorealistic rendering and complex sensors - IsaacLab: For fast, GPU-accelerated RL training - Newton: For maximum speed with new physics engine

→ Simulators Overview | → Comparison

4. Training

Train your robot policy using the appropriate learning method.

Training Methods:

Multi-modal models for language-conditioned robot control.

When to use: - Natural language task specification needed - Multi-modal perception (vision + language) - Zero-shot generalization desired

→ VLA Training Guide

Learn through trial-and-error interaction with environment.

When to use: - Reward function can be specified - Exploration is safe (in simulation) - Optimization beyond demonstrations needed

→ RL Training Guide

Learn from expert demonstrations.

When to use: - High-quality demonstrations available - Reward function hard to specify - Sample efficiency critical

→ IL Training Guide

5. Evaluation & Testing

Thoroughly test your trained policy before real-world deployment.

Evaluation steps: - Success rate measurement - Robustness testing with domain randomization - Diverse scenario testing - Failure mode analysis

→ Evaluation Guide | → Best Practices

6. Deployment

Deploy your trained policy to real robots.

Deployment checklist: - Safety checks implemented - Workspace boundaries configured - Emergency stop tested - Monitoring enabled - Gradual rollout plan

→ Deployment Guide

Learning Paradigms

Vision-Language-Action (VLA)

Multi-modal models that combine vision, language, and action generation.

  • End-to-end learning from pixels to actions
  • Natural language task specification
  • Pre-training on internet-scale data
  • Zero-shot generalization to new tasks

Learn more →

Reinforcement Learning (RL)

Optimize policies through environment interaction.

  • Model-free methods (PPO, SAC, TD3)
  • Model-based methods (Dreamer, MBPO)
  • Exploration strategies
  • Reward engineering

Learn more →

Imitation Learning (IL)

Learn from demonstrations without explicit rewards.

  • Behavioral cloning
  • DAgger (interactive learning)
  • Inverse reinforcement learning
  • Generative policies

Learn more →

Essential Tools

LeRobot Dataset Format

Standardized format for robotics datasets.

  • Unified data structure
  • Multi-modal support (images, states, actions)
  • Efficient storage with Parquet + video
  • Hugging Face integration

Learn more →

Simulators

Physics simulation for safe, fast training.

Simulator Best For Speed
IsaacSim Perception, visualization Medium
IsaacLab Fast RL training Very Fast
Newton Maximum speed, differentiable Fastest

Learn more →

Quick Start Paths

Path 1: Imitation Learning

1. Collect demonstrations → Data Collection
2. Format as LeRobot → Dataset Preparation
3. Train BC policy → IL Training
4. Test in simulation → Evaluation
5. Deploy to robot → Deployment

Time estimate: 1-2 weeks

Path 2: Reinforcement Learning

1. Set up simulator → Simulation Setup
2. Define reward → RL Introduction
3. Train with PPO/SAC → RL Training
4. Domain randomization → Best Practices
5. Sim-to-real transfer → Deployment

Time estimate: 2-4 weeks

Path 3: Vision-Language-Action

1. Collect multi-modal data → Data Collection
2. Prepare LeRobot dataset → Dataset Preparation
3. Train VLA model → VLA Training
4. Language generalization testing → Evaluation
5. Natural language deployment → Deployment

Time estimate: 3-6 weeks

Key Concepts

  • Multi-modal Learning: Combining vision, language, and proprioceptive inputs
  • Sim-to-Real Transfer: Training in simulation, deploying on real robots
  • Domain Randomization: Varying simulation parameters for robust policies
  • Data Efficiency: Learning from limited data through smart algorithms
  • Generalization: Performing well on unseen tasks and environments

Next Steps

New to robot learning? 1. Start with Getting Started Guide 2. Follow one of the Quick Start Paths above 3. Read Best Practices

Ready to dive in? - VLA Models - Multi-modal learning - RL Algorithms - Policy optimization - IL Methods - Learning from demonstrations - LeRobot Format - Dataset standard - Simulators - Training environments

Support