- Advanced
- 8 to 12 hours
- 14 min read
- You marked this lesson done
- Updated
After this lesson you can
- Deliver a trained policy for one simulated task in a repository that a stranger can rebuild
- Evaluate it with fixed seeds, baselines and a stress test, and report each result with an interval
- Write a failure report from watched videos, with a taxonomy and a list of what the results do not show
- Present the work in a README, a short animation and a written report
- Review a peer's capstone against a rubric and write useful comments
Read these first
Supports: the Robot learning track and the Master Developer Product Package, as the evidence that you can deliver a learned policy and defend what it does. Time: about eight to twelve hours, mostly waiting for training.
1. Why this matters
This is the piece you can show. A reviewer will not read your notes from five lessons. They will open one repository, read one README, run one command and look at one report. If it is honest, repeatable and clear about its limits, it proves more than a flashy demo.
The capstone tests the habit this track teaches: you decide what success means before you train, you measure against baselines, and you write down what went wrong. A policy that scores zero, honestly reported, can pass. A high score with no baseline, no seeds and no failure report cannot.
Nothing here needs a robot. Everything runs in simulation on public data.
What the PDM decides here
| You decide | You may delegate to an agent | You verify | Other disciplines involved |
|---|---|---|---|
| The task, the success rule, the protocol, the budget and what claim the evidence supports | Code, training runs, plots, README and report drafts | Every number against the raw files, the seeds, the failure causes by watching videos, and each claim in the report | Safety (what a failure would cost on hardware), quality (the acceptance criteria), legal (dataset and model terms) |
2. Key ideas
What to build
Deliver one learned policy for one simulated manipulation task, evaluated fairly. Choose one path.
- Path A, laptop CPU. Start from your lesson 5 folder. The task is PushT, with the public
lerobot/pushtdemonstrations. Change one thing in the small policy and evaluate both versions. Expect a weak policy. That is allowed: the marks are for the evidence, not the score. - Path B, one GPU. Train a standard LeRobot policy,
diffusionoract, on the same data withlerobot-train, and evaluate it. You can use a free cloud notebook if it gives you a GPU today. LeRobot’s hardware guide says its CPU-only machines are not meant for training these policies. On the CPU of a 28-core desktop, a diffusion policy took about six seconds per step at batch size 8, so a full run is not possible there.
Both paths use the same protocol and report. You may choose another public simulated task if your reviewer agrees in writing first. Do not use data you have no right to use.
Honest compute
- A GPU makes training faster and opens the larger policies. It does not make your evaluation fairer.
- On a free notebook, sessions end and plans change. Check what it offers today. Save checkpoints often and keep the training command in the README.
- If your policy is weak, say so, and spend your spare hours on the baselines, the stress test and the failure report. The rubric pays for those.
Acceptance criteria
A reviewer can check each of these. Number them in your README and say where each is met.
- Repository. One git repository. A fresh clone plus the README commands reproduces every number in the report. Versions of Python and each library are listed.
- Dataset note.
DATASET_NOTE.mdgives the source, licence, size, fields and three things the data does not show, copied from the dataset’s own page. - Policy. A trained policy, the exact training command, the seed and the training log. Large weights stay out of git: record where they are and their SHA-256 checksum.
- Protocol first.
PROTOCOL.mdnames the task, the success rule, the seed range, the number of episodes, the baselines and the stress test. Git history shows it was committed before the final evaluation. - Fair evaluation. At least 50 fixed-seed episodes. Report successes out of episodes with a 95 per cent Wilson interval, and mean best coverage or another stated second measure.
- Baselines. A policy that never moves and a random policy, on the same seeds.
- Stress test. At least one change you did not train for, such as noise on the sensor, with its own result and interval.
- One change. A second policy that differs from the first in one stated way (more training, a larger image, action chunks, data augmentation). Both evaluated on the same seeds. If it did not help, say so.
- Tests. Automated tests for your interval code and your failure sorting. Their output is saved in
results/tests.txtand all pass. - Failure report. At least 10 failures watched, each with a cause in your own words, grouped into 3 to 5 types with counts, and 3 videos linked.
- Show. An animated GIF or a screen recording under 30 seconds showing at least one best episode and one failure.
- Written report.
REPORT.mdwith the lesson 5 headings, a limits section and no claim beyond the evidence. - Disclosure. A line in the README on how an AI agent helped, and which files it touched.
- Peer run. One other person ran your README in a fresh environment, and their note is in the repository.
Repository layout
robot-learning-capstone/
README.md what, how to run, criteria checklist, versions, AI use
PROTOCOL.md committed before the final run
DATASET_NOTE.md
REPORT.md results, baselines, failure taxonomy, limits
REVIEW.md the peer's run note and rubric
requirements.txt pinned versions
src/pusht_bc/ code: prepare_data, policy, train, evaluate, stats, summarise
tests/test_stats.py
results/
summary.txt the table
tests.txt
<run>/results.csv, failures.csv, videos/
media/demo.gif
For Path B, add a commands.md with the exact lerobot-train and lerobot-eval commands you ran.
What to show
The README is the front door: two sentences on what the repository does, the GIF, the results table, the command that reproduces it, then the criteria checklist and the limits. A reviewer should know in 60 seconds what you built, how well it did and how sure you are.
3. Step by step
-
Decide the path and the one change. Write both in
PROTOCOL.mdbefore you train anything. For Path A choose one of: more epochs, a larger image than 48 by 48, action chunking (predict the next several actions and run them before looking again), or random shifts of the image as augmentation. Change only one. -
Set up the repository. Copy your
lesson-05/code into a new repositoryrobot-learning-capstone. Createrequirements.txtfrompip freeze, trimmed to what you use. WriteDATASET_NOTE.mdfrom your lesson 1 notes. -
Write the protocol. Reuse your lesson 5 protocol and choose the seed range now. Commit
PROTOCOL.md, then train. -
Train. For Path A, run
python -m pusht_bc.traintwice with two--outpaths, once for each version, and keep both logs. Your change to the code is part of the commit history.For Path B, this is the shape of the commands. Options change between LeRobot releases, so check
lerobot-train --helpfor your installed version, and check the LeRobot documentation. Measure the speed over the first hundred steps, then set--stepsto fit the time you have.lerobot-train \ --policy.type=diffusion \ --dataset.repo_id=lerobot/pusht \ --env.type=pusht \ --env_eval_freq=0 \ --policy.push_to_hub=false \ --policy.device=cuda \ --batch_size=64 \ --steps=50000 \ --output_dir=outputs/dp_pusht \ --job_name=dp_pushtThen evaluate the saved checkpoint with the same tool. It runs the policy in the PushT simulator, and its own seed, set with
--seed, decides the starting positions. Write that seed in the protocol.lerobot-eval \ --policy.path=outputs/dp_pusht/checkpoints/last/pretrained_model \ --env.type=pusht \ --eval.n_episodes=50 \ --eval.batch_size=10 \ --policy.device=cudaFor a Hub checkpoint of the same policy, LeRobot 0.7.0 stopped with a message about migrating the processor format when checked for this lesson. Train your own, as above.
-
Evaluate. Run the policy, both baselines and the stress test on the same seeds. For Path A, use the lesson 5 commands for each version. Put each
results.csvin its own folder underresults/, and runsummariseto makeresults/summary.txt. Save test output withpython -m pytest -q tests | tee results/tests.txt. -
Watch and write the failures. Watch at least 10 failure videos. Fill
failures.csvand group the causes. -
Make the GIF. This script joins MP4 files into one GIF. It keeps every second frame so the file stays small.
import sys from pathlib import Path import imageio.v2 as imageio out, *videos = sys.argv[1:] frames = [] for video in videos: reader = imageio.get_reader(video) frames.extend(frame for i, frame in enumerate(reader) if i % 2 == 0) reader.close() imageio.mimsave(out, frames, duration=100, loop=0) # milliseconds per frame print(f"wrote {out}: {len(frames)} frames, {Path(out).stat().st_size / 1e6:.1f} MB")python make_gif.py media/demo.gif results/best/videos/seed10000.mp4 results/best/videos/seed10001.mp4Do not cut frames to hide a failure. Say in the README which seeds are shown.
-
Write the report and README. Follow lesson 5’s headings. Open with the numbers, then the baselines, then the failures, then the limits. Run the README commands in a clean folder yourself before anyone else does.
-
Get a peer run. Ask a peer or a Sensei to follow the review steps below, and keep their note as
REVIEW.md.
4. Worked example: TrailMug
TrailMug is fictional. The arm, the rack and every number below are invented for teaching.
The TrailMug team’s capstone is their simulated arm that places a returned cup in a wash rack. Their protocol reads, in short: “Simulated task: the cup starts upright or tilted at one of 50 fixed positions. Success: the cup rests in the rack slot within 200 steps. Baselines: arm never moves, arm moves at random. Stress: 1 centimetre of noise on the camera position. Seeds 1 to 50.”
Their results table, with made-up numbers, reads:
| Run | Successes | 95% interval |
|---|---|---|
| Arm never moves | 0 of 50 | 0% to 7% |
| Random arm | 1 of 50 | 0% to 10% |
| Version A, 50 demonstrations | 31 of 50 | 48% to 74% |
| Version B, 80 demonstrations including tilted cups | 36 of 50 | 58% to 83% |
| Version B, camera noise | 27 of 50 | 40% to 67% |
B scored 5 more successes than A, but the intervals overlap heavily, so the team writes “B is not shown to be better than A in this test”. The taxonomy covers B’s 14 failures: 6 slipped on lifting, 5 missed the slot, 3 timed out. The GIF shows one slip, one miss and one success.
The limits section says: simulation only, one task, one training run per version, no real-world claim. That is what a reviewer most wants to find.
5. Working with AI agents on this
What the PDM decides. The task, the protocol, the one change, the claims the report may make and the point at which you stop training. You also decide the budget: how many GPU hours a trial may use.
What you delegate, and the exact brief. An agent can build the repository, run the evaluation, write tests and draft the README and report. This is the delegate rung. It becomes orchestrate when several agents train variants at once while you hold the results table. Adapt this brief:
Task: build the robot-learning capstone repository from my lesson-05 folder.
Task and protocol: see PROTOCOL.md. It is fixed. Do not edit it. Do not change seeds.
Deliver: the criteria checklist in README.md, results/summary.txt, results/tests.txt, failures.csv files, media/demo.gif and a draft REPORT.md.
Rules:
1. Every number in README.md and REPORT.md must come from a file under results/. Cite the file.
2. Report successes out of episodes with a 95% Wilson interval. Never a bare percentage.
3. Never re-run an evaluation to get a better number. If a run fails, report it and tell me.
4. Leave the failure-cause column blank. I will fill it after I watch the videos.
5. No claim about real robots. No word "works" without the number beside it.
6. Mark anything you could not verify as "unknown".
7. Do not download any data other than lerobot/pusht.
Output: a list of files you changed, the commands you ran with their output, and a list of assumptions.
How you verify. Recompute two numbers from the raw CSV. Check the seeds against the protocol. Check in git history that PROTOCOL.md came first. Watch the videos the report cites. Read the report against the table: agents make prose warmer than the numbers. Strike every claim with no number behind it.
Which disciplines are involved. Safety reads the taxonomy for failures that would be dangerous on hardware. Quality owns the acceptance criteria. Legal checks the dataset terms.
Where this sits on the ladder. You work at delegate. A pipeline that refuses to publish a report without a protocol, results files, tests and a peer note is govern. See Write a technical report and Team work with human review.
Disclose. Add to your README: “An AI agent drafted code and text from my brief. I ran every command, recomputed results from the raw files, watched the failure videos and wrote the failure causes.” See Disclose AI use and Reviewing and verifying AI output.
6. Common mistakes
- Changing the protocol after you see the results. Commit it first, and the history proves it.
- Reporting only the better version. Report both.
- Skipping the baselines. A score without them means nothing.
- Tuning on the test seeds. Keep evaluation seeds apart from anything used to train or tune.
- A GIF made of the best moments only. Show a failure too.
- Big claims from a small test. Fifty episodes with an interval show what they show. Say no more.
- A README nobody can run. Run it yourself in a clean folder.
- Committing large weights. Record a checksum and a location.
7. Practice: the capstone, with a rubric
Deliverable: the repository robot-learning-capstone in your own git hosting, with every item in the layout above.
Rubric
A reviewer scores each row from 0 to 4. A pass is 24 or more of 32, with no zero on rows 1, 2 or 7.
| # | What is scored | 0 | 2 | 4 |
|---|---|---|---|---|
| 1 | Reproducibility | Cannot be run | Runs with fixes | Fresh clone reproduces every number |
| 2 | Protocol and seeds | None | Written after results | Committed first, seeds fixed and kept |
| 3 | Statistics | Bare percentages | Interval for some runs | Every run has successes out of episodes and an interval |
| 4 | Baselines and stress test | None | Baselines only | Both baselines and a stress test, same seeds |
| 5 | One change, compared | None | Compared on different seeds | Compared on the same seeds, result stated even if no gain |
| 6 | Failure analysis | Counts only | Causes without videos | At least 10 watched failures, 3 to 5 types, 3 videos |
| 7 | Honesty of claims | Claims beyond the evidence | Some overreach | Every claim backed by a number, and a limits section |
| 8 | Presentation | No GIF, weak README | One of the two good | Clear README, GIF with a failure, tidy report |
The score does not depend on the success rate. A policy with 0 of 50, shown honestly with its baselines, can earn full marks.
How a Sensei or peer reviews
The review takes about an hour. Follow these steps, in order.
- Fresh run. Clone the repository into an empty folder. Follow the README exactly. Note every command that failed or needed a fix.
- History. Check in git that
PROTOCOL.mdis older than the final results. - Recompute. Pick one row of the table. Count successes in its
results.csvand compute the interval with the repository’s own code. It must match. - Watch. Watch two failure videos the author did not cite. Do you agree with the author’s cause for each?
- Claims. Read the report. Mark every claim that has no number or file behind it.
- Score and write. Fill the rubric. Write three comments: one strength, one risk and one question the author must answer. Save them as
REVIEW.mdin the author’s repository, with your name and the date, and say whether the run reproduced.
8. Checklist
- I chose a path and one change before training.
-
PROTOCOL.mdwas committed before the final evaluation. - I evaluated on at least 50 fixed seeds with baselines and a stress test.
- Every result has successes out of episodes and a 95 per cent interval.
- Tests for the interval and the failure sort pass, and their output is saved.
- I watched at least 10 failures and wrote the causes.
- The GIF shows a failure as well as a success.
- The report has a limits section and no claim beyond the evidence.
- A peer reproduced my README in a fresh environment.
- I disclosed how an AI agent helped.
9. Further learning
The LeRobot documentation and dataset page are the reference for both paths, and its hardware guide tells you what a GPU is for. The Diffusion Policy and ACT papers explain the methods behind the policies. The Wikipedia article on the binomial proportion confidence interval and the datasheet paper support your statistics and dataset note. Google’s technical writing course helps with the report. When you are done, go back to the Robot learning track overview and pick your next step: the Robot software track for the system a policy runs inside, or Safety reports and review for the habits that keep a learned system honest.
Finished this lesson?
This is only for you. It is saved in this browser, on this device, and nothing is sent anywhere. It is not a grade and not a certificate.
Go deeper
Further learning
Free or low-cost material from other publishers. We opened every link; check the cost label before you start.
Hugging Face
LeRobot (opens the publisher's site in a new tab)
- Free
- Docs
- Intermediate
The official documentation for LeRobot, Hugging Face's open toolkit for robot learning, released under the Apache 2.0 licence. It covers installation, recording and loading datasets, training policies and testing them in simulation. Use it as the reference for the commands in this track, and follow the install steps it shows on the day you work.
Hugging Face
LeRobotDataset v3.0 (opens the publisher's site in a new tab)
- Free
- Docs
- Intermediate
The format page for LeRobot datasets: Parquet files for numbers, MP4 files for video and a meta folder that describes them. It shows how to record a dataset, load it for training and stream it from the Hugging Face Hub. Read it before you collect or share demonstrations.
Hugging Face
Compute HW Guide for LeRobot Training (opens the publisher's site in a new tab)
- Free
- Docs
- Intermediate
Rough GPU memory needs and training times for LeRobot policies, from small action-chunking models to large vision-language-action models. The guide says its numbers are indicative, not exact. Read it before you rent or buy compute.
arXiv
Diffusion Policy: Visuomotor Policy Learning via Action Diffusion (opens the publisher's site in a new tab)
- Free
- Article
- Advanced
The paper that represents a robot policy as a denoising diffusion process. It reports results on 12 tasks from four robot manipulation benchmarks and explains why the method handles actions that have several good answers. Read the abstract first, then the evaluation section.
arXiv
Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware (opens the publisher's site in a new tab)
- Free
- Article
- Advanced
The paper that introduced Action Chunking with Transformers (ACT). It names the problem of errors that compound over time in imitation learning, and reports 80 to 90 per cent success on six real-world tasks from about 10 minutes of demonstrations each. Those results are the authors' own, on their own hardware.
Wikipedia
Binomial proportion confidence interval (opens the publisher's site in a new tab)
- Free
- Article
- Intermediate
An encyclopaedia article on confidence intervals for a success rate counted out of a number of trials. It gives the Wilson score interval used in this track and says why the simple normal approximation can mislead for small counts or rates near 0 or 1. Use it to check the formula in your code.
arXiv
Datasheets for Datasets (opens the publisher's site in a new tab)
- Free
- Article
- Intermediate
The paper by Gebru and colleagues, published in Communications of the ACM in 2021, that proposes a datasheet for every dataset. It lists questions on why the data was made, how it was collected and what it should not be used for. Use the questions as a template.
Google
Google Technical Writing courses (opens the publisher's site in a new tab)
- Free
- Course
- Beginner
Free short courses on writing clearly for engineers: short sentences, active voice, good lists. The habits make epics and issues easier to read. Start with the first course.



