Skip to content

Introduction to Imitation Learning

What is Imitation Learning?

Imitation Learning (IL) is a paradigm where agents learn to perform tasks by observing and mimicking expert behavior.

Motivation

The Reward Specification Problem

Defining good reward functions for RL is hard:

# Too sparse
reward = 1 if task_complete else 0

# Too dense (might encourage shortcuts)
reward = -distance_to_goal + action_smoothness - collision_penalty + ...

IL sidesteps this by learning directly from demonstrations.

Formal Framework

Problem Setup

Given: - Expert policy \(\pi^*\) (unknown) - Demonstrations \(\mathcal{D} = \{(s_1, a_1), (s_2, a_2), ..., (s_N, a_N)\}\) - Sampled from \(\pi^*\)

Goal: - Learn policy \(\hat{\pi}\) that mimics \(\pi^*\)

Objective

Minimize expected difference between learned and expert policy:

\[ \min_{\pi} \mathbb{E}_{s \sim d^{\pi^*}} [c(s, \pi(s), \pi^*(s))] \]

where \(d^{\pi^*}\) is state distribution under expert policy, and \(c\) is a cost function.

The Distribution Shift Problem

Compounding Errors

Behavioral cloning suffers from distribution shift:

  1. Policy trained on expert state distribution
  2. Deployment: policy makes small mistakes
  3. Reaches states never seen in training
  4. Makes larger mistakes (no training data)
  5. Errors compound over time

Distribution shift compounding into catastrophic failure

Mathematical Analysis

Error bound for behavioral cloning:

\[ J(\pi^*) - J(\hat{\pi}) \leq \epsilon T^2 \]
  • \(\epsilon\): Per-step error
  • \(T\): Episode length
  • Quadratic in horizon!

IL Algorithm Families

1. Behavioral Cloning

Approach: Supervised learning

Algorithm: 

Input: Expert demonstrations D = {(s, a)}
Output: Policy π

1. Collect dataset D from expert
2. Train policy π(a|s) to maximize likelihood:
   max_π Σ log π(a|s) for (s,a) in D

When to use: - Demonstrations are on-policy (cover deployment distribution) - Short horizon tasks - Expert is consistent

2. Interactive IL (DAgger)

Approach: Query expert during training

Algorithm: 

Initialize: Dataset D = D_expert
for iteration in 1..N:
    1. Train policy π on D
    2. Execute π, collect states S
    3. Query expert for actions A on states S
    4. Add (S, A) to D
    5. Retrain

When to use: - Expert available during training - Need to handle distribution shift - Can afford interactive learning

3. Inverse Reinforcement Learning

Approach: Learn reward, then optimize it

Algorithm: 

1. Infer reward function R from demonstrations
2. Use RL to find policy that maximizes R

When to use: - Want interpretable reward - Need generalization to new scenarios - Have compute for RL step

Theoretical Foundations

Performance Bounds

DAgger Theorem:

With DAgger, error is linear in horizon:

\[ J(\pi^*) - J(\hat{\pi}) \leq \epsilon T \]

Much better than BC's quadratic bound!

Sample Complexity

Number of demonstrations needed scales with:

  • State space dimensionality
  • Action space complexity
  • Task horizon
  • Desired performance level

Practical Considerations

Multi-Modal Actions

Demonstrations may show multiple valid strategies:

# Example: Two valid grasps for same object
demonstration_1: approach_from_top()
demonstration_2: approach_from_side()

# BC will average these actions (bad!)
learned_action: approach_from_middle() # Fails!

Solutions: - Mixture of Experts - Action discretization - Diffusion policies - Energy-based models

Suboptimal Demonstrations

Real demonstrations are rarely perfect:

# Handle suboptimal data
def train_with_confidence_weights(demonstrations):
    for demo in demonstrations:
        # Weight by demonstration quality
        weight = estimate_optimality(demo)
        loss = weight * bc_loss(demo)

Modern Extensions

Generative Behavioral Cloning

Use generative models for multi-modal distributions:

  • VAE: Learn latent space of actions
  • Diffusion: Iteratively denoise to action
  • Flow Matching: Transform noise to action

One-Shot Imitation

Learn from single demonstration:

  • Meta-learning approaches
  • Context-conditioned policies
  • Vision-language models

Learning from Videos

Learn without action labels:

  • Video prediction models
  • Inverse models
  • Time-contrastive learning

Next Steps