Physical AI

The language layer
your training data is missing.

MetaMule turns raw video into time-coded, natural-language action labels for physical AI — an automated first pass that feeds your annotators and fits your pipeline, not a platform to replace them.

Talk to us See the output
The bottleneck

Vision-Language-Action models need a per-step instruction.
The field is short on them.

Robot-learning and world-model training run on datasets where every step carries a natural-language instruction — the RLDS language annotation. It's demonstrably under-supplied: only about 70% of Open X-Embodiment episodes carry language annotations, and quality varies sharply across sub-datasets. The standard tooling even inserts a placeholder string when a dataset has none. Writing them by hand is slow and expensive — so they get skipped. That's exactly the work to automate.

The output

Time-coded action segments, at the granularity the field already uses.

MetaMule watches video and emits segments at the Ego4D activity → action → sub-action hierarchy. Every segment carries:

instruction

The language annotation

An imperative, one-line VLA / RLDS instruction — the highest-value, scarcest field.

description

Dense natural language

Exactly what happens in the segment, grounded in the frames.

contact_events

Approximate contact

Hand/actor–object contact, flagged for a human to confirm — never asserted as ground truth.

confidence · needs_review

On every label

So low-confidence segments route straight to your annotators.

"segments": [{
  "level": "action", "start_s": 83.0, "end_s": 96.8,
  "instruction": "Taxi the aircraft to the runway for takeoff",
  "description": "The aircraft rolls along the tarmac as the passenger raises a hand…",
  "contact_events": [{ "type": "grasp", "actor": "a1", "object": "o5" }],
  "confidence": 0.90, "needs_review": false
}]  // maps onto RLDS episode / step and LeRobot

Not geometry. No 3D boxes, pose, grasp, segmentation, or LiDAR — a model watching video can't produce those, and they're your strength. MetaMule fills only the language layer.

First pass, not a platform

It makes your annotators faster. It doesn't replace them.

Feeds, not competes

Every label carries a confidence and a review flag, so the output drops into your existing annotation stack as a pre-pass. Humans confirm or correct instead of labeling the language layer from scratch.

Honest by design

Uncertainty is surfaced, not hidden. Contact events are approximate; ambiguous segments are flagged. That triage is what makes an automated pass trustworthy at volume.

Fits your pipeline

Labeled to your action taxonomy, your granularity, your ingest format — RLDS, LeRobot, or your own. A specialized supplier for one layer, not a platform to adopt.

Proof

Real footage in, structured labels out.

A single 4K clip, labeled automatically into time-coded segments with instructions, descriptions, contact events and review flags — hand-verified against the footage, and it runs across a whole library.

21
segments from one 4-minute clip
100%
labels carry confidence + review
3
temporal levels: activity → action → sub-action

Bring us a clip and your output spec.

We'll label a representative clip to your schema so you see concrete output, not a promise. Then we adjust to your exact taxonomy and target format.

enterprise@metamule.ai

Grounded in the field's standards. Ego4D — 3,670 hours of egocentric video with timestamped narrations (CVPR 2022 / IEEE TPAMI 2025). Open X-Embodiment / RLDS — 1M+ real robot trajectories across 22 embodiments, episode→step structure with a language annotation per step (Google DeepMind + 21 institutions, 2023). MetaMule segments to the Ego4D granularity and structures output to map onto RLDS — conventions the field already uses.