- Advanced
- 3 to 4 hours
- 14 min read
- You marked this lesson done
- Updated
After this lesson you can
- Explain what a vision-language-action model is and how it differs from the small policy you trained in lesson 1
- Name five open models and four public datasets, and say what each one is for
- Read a licence label, a model card and a repository readme, and say when the label is not enough
- Work out whether a model can run on the hardware you have, before you download it
- Write a comparison card in git that cites a source for every cell
Read these first
Supports: the Robot learning track and the Master Developer Product Package, where a robot product may build on someone else’s model and must know its terms. Time: about three to four hours.
1. Why this matters
In lesson 1 you trained a tiny policy for one task. A vision-language-action model (VLA) is a large model that reads camera images and text and outputs robot actions. You give it a picture of a table and the sentence “put the cup in the rack”. It answers with the next motion. Labs train these on large collections of robot recordings, then publish them so others can fine-tune them for a new task or robot.
For a developer this is a buy-or-build call. A published model can save months. It can also arrive with a licence that blocks your use, a memory need that your laptop cannot meet, and training data that looks nothing like your task. The Product Development Manager’s job is to find that out before the team spends a week on it.
You cannot run most of these models on a laptop. You can still read their documents, compare them fairly and decide. That is what this lesson trains.
What the PDM decides here
| You decide | You may delegate to an agent | You verify | Other disciplines involved |
|---|---|---|---|
| Whether to build on a published model at all, and which one | Reading readmes and model cards, filling a comparison table, writing a Hub metadata script | Every licence and hardware claim against the official page, and who it was tested on | Legal (licences and terms), finance (GPU cost), safety (a large model is still not a safety guard) |
2. Key ideas
What a VLA is made of
A VLA joins three parts. A vision encoder turns camera images into numbers. A language model reads the instruction and those numbers. An action part, or head, turns the result into robot commands. Models differ in the action part. OpenVLA builds on the Llama 2 language model, per its readme. Octo and GR00T N1.7 use diffusion-style heads that remove noise step by step, the idea behind Diffusion Policy in lesson 1.
A parameter is one learned number inside the model. More parameters mean more memory.
Pretraining and fine-tuning
The makers pretrain the model on a large mix of robot data, so it learns general habits. You then fine-tune it: you train it a little more on your own demonstrations for your own robot and task. Fine-tuning needs a dataset in the right format, and the right camera names and action meaning. A model trained on one robot’s action format does not work on another until you adapt it. The openpi readme says so in plain words: its first model was developed for the team’s own robots, which differ from common platforms, and they do not expect every attempt to adapt it to succeed.
The open models, as their own pages state
| Model | Made by | Code licence | Weights | Hardware named by the makers |
|---|---|---|---|---|
| OpenVLA | The OpenVLA project | MIT | The readme says they may inherit limits from Llama 2 and follow the Llama Community License | LoRA fine-tuning of the 7B model on one 80 GB A100, or a GPU with at least about 27 GB; full fine-tuning on a node of 8 A100s |
| Octo | The Octo project | MIT | Octo-Small has 27 million parameters and Octo-Base 93 million, per the readme | The readme gives no memory figure. Check before you run it |
| openpi | Physical Intelligence | Apache 2.0 | Not verified for this lesson. Read the licence of each checkpoint | Inference over 8 GB; LoRA fine-tuning over 22.5 GB; full fine-tuning over 70 GB |
| SmolVLA | Hugging Face, in LeRobot | Apache 2.0 | The lerobot/smolvla_base card says Apache 2.0; about 450 million parameters |
20,000 fine-tuning steps take about 4 hours on one A100 per the LeRobot page; a Colab notebook is offered |
| GR00T N1.7 | NVIDIA | Apache 2.0 | NVIDIA Open Model License | Inference needs one GPU with 16 GB or more; fine-tuning 40 GB or more is recommended |
Treat the table as a map and not as a promise. These pages change. You will check each cell yourself. Octo and SmolVLA are the small ones. A larger model is not better by default: the evidence for your robot and task counts.
Licences: three layers
A model has at least three licences to read.
- The code. Usually permissive, such as MIT or Apache 2.0.
- The weights. The trained numbers. Their licence can be stricter than the code’s. NVIDIA names a separate Open Model License for GR00T’s weights. OpenVLA’s readme warns about the Llama licence.
- The training data. Models learn from datasets that carry their own terms.
The Hub’s licence label is a short tag. For GR00T N1.7 it says other, which means “go and read the licence”. For OpenVLA the label on the Hub says mit, while the code repository’s readme says the weights follow the Llama Community License. When two sources disagree, use the stricter one until you have read both. This lesson does not give legal advice. For a product, ask a lawyer.
The open datasets
| Dataset | What it is | What the sources say |
|---|---|---|
| Open X-Embodiment | A pooled collection of robot datasets from many labs | The abstract reports 22 robot types and 160,266 tasks. Each source dataset has its own terms: check them |
| DROID | Robot manipulation recorded in many real scenes | The abstract reports 76,000 trajectories, 350 hours, 564 scenes and 50 collectors over 12 months. The abstract says 84 tasks. The website figure differs, so this lesson quotes only the abstract |
| LIBERO | A simulated benchmark of manipulation tasks | The readme says 130 tasks in four suites, code under MIT and data under CC BY 4.0. LeRobot’s page says it needs Linux |
| lerobot/pusht | The small simulated task from lesson 1 | MIT label on the Hub |
Open X-Embodiment and DROID record real robots. LIBERO is a simulator, so you can evaluate a policy with no hardware.
What these models cannot do for you
- They do not know your robot. Action meaning, cameras and speed differ.
- A published success rate comes from the authors’ set-up, not yours. Lesson 5 shows how to measure your own.
- They are not a safety layer, and real hardware needs a trained supervisor.
- Open does not mean unrestricted. Read the terms.
3. Step by step
You need no GPU for steps 1 to 4. Step 5 is optional and needs one.
-
Make a card for each model and dataset. In your
robot-learningrepository, createlesson-04/and a filecomparison.md. For each of the five models and four datasets above, create a row with these columns: maker, task it was tested on, code licence, weights licence, data licence, hardware needed, source URL and the date you read it. Fill each cell from the official page. Writenot verifiedwhere you could not find it. An empty cell is better than a guess. -
Read the sources. Open the readme or model card for each row. Use the project pages in the resource list. Note any sentence that limits use, such as a non-commercial clause, a gated download or a “research only” note.
-
Read licence labels from the Hub with code. The Hub exposes each repository’s metadata. This script prints the licence label and the parameter count for any list of models and datasets, and says so when it cannot read one. Save it as
lesson-04/model_card.py. It needshuggingface_hub, which LeRobot already installs."""Read licence labels and parameter counts from the Hugging Face Hub, so a table cites a source.""" import sys from huggingface_hub import HfApi from huggingface_hub.utils import HfHubHTTPError api = HfApi() def describe(kind: str, repo_id: str) -> str: try: info = api.model_info(repo_id) if kind == "model" else api.dataset_info(repo_id) except HfHubHTTPError as error: return f"{kind:8} {repo_id}: could not read ({str(error).splitlines()[0]})" card = info.card_data.to_dict() if info.card_data else {} licence = card.get("license", "none stated") size = "" if kind == "model" and info.safetensors: size = f", {info.safetensors.total / 1e6:.0f}M parameters" return f"{kind:8} {repo_id}: licence label {licence}{size}, last modified {info.last_modified:%Y-%m-%d}" for argument in sys.argv[1:]: kind, repo_id = argument.split(":", 1) print(describe(kind, repo_id))python lesson-04/model_card.py model:openvla/openvla-7b model:rail-berkeley/octo-small-1.5 \ model:lerobot/smolvla_base model:nvidia/GR00T-N1.7-3B dataset:lerobot/pusht dataset:lerobot/liberoOn the day this lesson was checked, the labels read
mitfor OpenVLA and Octo,apache-2.0for SmolVLA,otherfor GR00T,mitfor PushT andapache-2.0for the LeRobot copy of LIBERO. The parameter counts were about 7.5 billion, 450 million and 3.1 billion for the three that report them. Your run may differ because repositories change. Paste the output intocomparison.mdwith the date.The
lerobot/liberolabel differs from the original LIBERO readme, which says CC BY 4.0 for the data. A re-upload can carry its own label. Write both and say which you rely on. -
Decide, with a reason, for one imagined project. TrailMug’s arm (see the worked example) is a single arm with one top camera. For your own invented project, pick the model you would try first and write three lines: what hardware it needs, what its licence allows, and what the data it was trained on does not cover. The smallest model that can do the job is a valid answer.
-
Optional, with a GPU: evaluate a published checkpoint in simulation. LeRobot ships LIBERO support on Linux. Follow the LeRobot LIBERO page for the install step, which is
pip install -e ".[libero]"from a source checkout of LeRobot, and set the rendering backend for a headless machine. Then evaluate a small number of episodes on one suite. The command below follows that page. Check it against the current page before you run it, because options change.export MUJOCO_GL=egl lerobot-eval \ --policy.path=lerobot/smolvla_libero \ --env.type=libero \ --env.task=libero_object \ --eval.batch_size=1 \ --eval.n_episodes=3The LeRobot page says
lerobot-evalprints a 95 per cent Wilson interval next to each success rate. Three episodes is far too few to say anything. It only proves your set-up runs. Notebook plans change, so check what a free one offers today. Thelerobot/smolvla_liberocheckpoint on the Hub shows an Apache 2.0 label. Read its card before you use it.Without a GPU, skip this step. Your
comparison.mdalready meets the lesson.
4. Worked example: TrailMug
TrailMug is fictional. The arm and every number below are invented for teaching.
The TrailMug team has a simulated arm that places a returned cup in a wash rack. Their lesson 1 policy, trained on 80 demonstrations, fails when the cup is tilted. A teammate suggests: “Just use a big VLA.” The product manager asks for a comparison card first.
The card shows three things. The largest model needs a GPU with more memory than anyone on the team has, so each run means renting a machine. One model’s weights come with a licence that the team’s lawyer has not read. And none of the models was trained on this arm: its action format differs, so adapting it is a project in itself.
The team decides on a smaller step. They will fine-tune the smallest model that fits a rented GPU, on the 80 demonstrations plus 40 with tilted cups, and test it on fixed starting positions in the simulator. They write down what would make them stop: if the fine-tuned model does not beat the lesson 1 policy on the same test, they keep the small policy.
The PDM did not pick a model for its fame. They read the terms, counted the hardware and set a test that can say no.
5. Working with AI agents on this
What the PDM decides. Whether the project may use a published model, which licence terms are acceptable, and what budget a trial may spend on GPUs. You also decide which claims in the comparison card are allowed to stand.
What you delegate, and the exact brief. An agent can read readmes, fill the table and run the metadata script. This is the delegate rung. Adapt this brief:
Task: fill lesson-04/comparison.md for five open robot models and four datasets.
Models: OpenVLA, Octo, openpi, SmolVLA, GR00T N1.7. Datasets: Open X-Embodiment, DROID, LIBERO, lerobot/pusht.
Columns: maker, task tested on, code licence, weights licence, data licence, hardware needed, source URL, date read.
Rules:
1. Use only official pages: the project readme, the Hugging Face model or dataset card, or the paper abstract.
2. Copy numbers exactly. Do not round or convert them.
3. If a cell cannot be found on an official page, write "not verified". Do not infer it from a similar model.
4. Where the Hub label and the readme disagree, write both and quote the sentence.
5. Run lesson-04/model_card.py and paste its output. Do not edit that output.
6. Do not run any model. Do not download weights.
Output: the table, a list of every "not verified" cell, and a list of every disagreement between sources.
How you verify. Open the source URL in at least five cells yourself and compare it with the table. Agents fill gaps with plausible numbers, and a model that read many readmes will blend them. Check every licence cell and every hardware cell. Ask for the quoted sentence behind any figure that looks surprising.
Which disciplines are involved. Legal reads the licence cells. Finance prices the GPU hours. Safety asks what the robot does when a large model is wrong. Operations asks who updates the card when a model is replaced.
Where this sits on the ladder. This is delegate. When many cards must be kept current, it moves toward orchestrate: an agent re-checks each source on a schedule, and a person reviews each change. A pipeline that refuses to publish a card with an unsourced cell is governing.
Disclose. Add to your README: “The comparison table was drafted by an AI agent. I checked each licence and hardware cell against its source.” See Disclose AI use and Reviewing and verifying AI output.
6. Common mistakes
- Reading only the label.
miton a model page can sit beside a stricter note in the readme.othertells you nothing until you read the licence. - Mixing up code and weights. A code licence does not give you the weights’ terms.
- Copying a headline result. A success rate on the authors’ robot says little about yours.
- Ignoring memory. A model that does not fit your GPU is not an option.
- Trusting an agent’s table. Models fill gaps. A cell without a source is a guess.
- Assuming a model fits your robot. Cameras, action meaning and control rate must match.
7. Practice: a comparison card
Deliverable: a folder lesson-04/ in your own git repository.
comparison.md: the table from step 1, every cell with a source URL and the date read, andnot verifiedwhere needed.model_card.pyand its pasted output.decision.md: one imagined project, the model you would try first and three reasons, written as a short note a manager could read.disagreements.md: each place where two sources differ, with both quotes.- If you ran step 5: the full command and the printed result, with a note that three episodes is only a set-up check.
- One line in the README on how an AI agent helped.
Check: ask a classmate or Sensei to pick three cells at random and check them against the sources.
8. Checklist
- I can explain a vision-language-action model in two sentences.
- I filled every cell from an official page, or wrote
not verified. - I separated the code, weights and data licences.
- I checked the hardware need before choosing a model.
- I noted where two sources disagree.
- I did not copy a published success rate as if it were mine.
- I recorded the date I read each source and my AI use.
9. Further learning
The OpenVLA, Octo, openpi and GR00T repositories are the primary sources for what each model is and needs, and their licence sections are short. The SmolVLA paper explains why a small model can be useful, and the LeRobot pages show how to fine-tune it. The Open X-Embodiment, DROID and LIBERO project pages describe the data and how it was collected. Read their abstracts and licence notes before you plan a run. Next, Evaluate a policy and report failures shows how to measure any of these policies fairly.
Finished this lesson?
This is only for you. It is saved in this browser, on this device, and nothing is sent anywhere. It is not a grade and not a certificate.
Go deeper
Further learning
Free or low-cost material from other publishers. We opened every link; check the cost label before you start.
OpenVLA project
OpenVLA: An Open-Source Vision-Language-Action Model (opens the publisher's site in a new tab)
- Free
- Article
- Advanced
The project page for OpenVLA, a 7-billion-parameter vision-language-action model trained on real robot demonstrations. It links the paper, the code and the model. Read the licence notes in the code repository before you use the weights.
Octo project
Octo: An Open-Source Generalist Robot Policy (opens the publisher's site in a new tab)
- Free
- Article
- Advanced
The project page for Octo, an open generalist robot policy trained on a large mix of robot datasets. It links the paper, the code and pretrained models. Use it as a contrast with the larger vision-language-action models.
Physical Intelligence
openpi (opens the publisher's site in a new tab)
- Free
- Docs
- Advanced
The open-source repository for the pi-zero family of vision-language-action models from Physical Intelligence, with code under the Apache 2.0 licence. Its readme gives GPU memory needs for running and fine-tuning. Read the licence notes for each model you download.
arXiv
SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics (opens the publisher's site in a new tab)
- Free
- Article
- Advanced
The paper for SmolVLA, a small vision-language-action model from Hugging Face that is built to be trained on a single GPU and run on consumer GPUs or even CPUs. It describes training on community-collected datasets. Read it with the LeRobot documentation, which has the code.
NVIDIA
Isaac-GR00T (opens the publisher's site in a new tab)
- Free
- Docs
- Advanced
The repository for NVIDIA Isaac GR00T, a family of open foundation models for generalist robots. Its readme states the code licence, the separate licence on the model weights and the GPU memory needed for inference and fine-tuning. Read those sections before you plan a run.
Open X-Embodiment Collaboration
Open X-Embodiment: Robotic Learning Datasets and RT-X Models (opens the publisher's site in a new tab)
- Free
- Article
- Advanced
The project page for Open X-Embodiment, a collaborative collection of robot datasets from many labs and robot types, and the RT-X models trained on it. It links the paper and the data. Check each source dataset's own licence before you reuse it.
DROID project
DROID: A Large-Scale In-the-Wild Robot Manipulation Dataset (opens the publisher's site in a new tab)
- Free
- Article
- Advanced
The project page for DROID, a large robot manipulation dataset recorded in many real settings. It describes how the data was collected and where to find it. Check the dataset's terms of use before you use it.
LIBERO project
LIBERO (opens the publisher's site in a new tab)
- Free
- Article
- Advanced
The project page for LIBERO, a simulated benchmark for robot manipulation and lifelong learning, with a code licence of MIT and a data licence of CC BY 4.0 stated in its repository. It is used to test vision-language-action policies in simulation. Read how its task suites are defined before you report a result.



