Auto-labeling that starts with nothing

Offboard perception for construction sites, built from papers up. Classical geometry and learned models, implemented inMojo, assembled into a loop that begins with no labels at all and gets better each round.

The question

A pipeline that “starts unlabeled and improves” is only a meaningful claim if something can score it. No public dataset lets you: they either have no labels, so there is nothing to measure against, or they have them, so you are not really starting unlabeled.

So the data came first — a synthetic construction scene carrying its own held-out oracle, on one rule: only the scorer ever reads the truth. Then the components. Then the loop.

Three panels showing every labelled box centre over a 60-second scene, for round 0, round 1 and round 2. Round 0 has a dense cluster of phantom labels inside the dashed stockpile footprints; round 1 still has them; round 2's stockpiles are empty. — Every labelled box centre over one 60-second scene. Round 0 is pure geometry — it finds the machines and the workers, and also fits confident, vehicle-shaped boxes to slivers of stockpile. Round 2, trained on its own filtered output, leaves the piles alone.

Open the data in Foxglove →How to read it →

What is here

The data

A synthetic site with LiDAR, four cameras, proprioception and a held-out oracle. Reproducible from a seed, byte for byte.

The pipelines

Three named workflows: discover instances from geometry, name them from pixels, distil a detector and label again.

Numbers & learnings

What was measured, including the parts that went backwards and why they did.

Papers

Eleven implementations, each its own repository, and where each one sits in the pipeline.

The result worth reading twice

training labels   precision 0.865   recall 0.547
student output    precision 0.921   recall 0.656

The distilled model is better than its own supervision on both axes. It recovered objects the label filter had discarded and rejected false positives the filter had let through.

That only happens above a threshold. Trained on the unfilteredlabels the round before, the same code went backwards — it inherited the systematic errors and amplified them. Below some label-precision floor a self-training loop compounds its own mistakes; above it, it compounds its strengths. Finding roughly where that floor sits turned out to be more useful than any single score.

Everything on this site is measured on the machine, not estimated. Where a number contradicted what had been written down first — which happened five times — the number won and the write-up changed.