Auto-labeling that starts with nothing

Offboard perception for construction sites, built from papers up. Classical geometry and learned models, implemented in Mojo, assembled into a loop that begins with no labels at all and gets better each round.

The shape of it

Planning picks one stage, the pipeline runs it, and whatever rows it produces become a new immutable version — indexed by the registry that planning reads to decide again. train is the one stage with no arrow down: it writes a checkpoint, not a dataset.

Three things are worth taking from that picture. Planning is separate from doing — something decides which stage runs next, and that decision is written down alongside the alternatives it passed over. Every stage writes a versioned dataset rather than overwriting a file, so a result stays readable after the next round changes its inputs. And the arrow closes: what a round records is what the next round's planning reads. A loop whose output nothing reads is just a pipeline run twice.

The question

A pipeline that “starts unlabeled and improves” is only a meaningful claim if something can score it. No public dataset lets you: they either have no labels, so there is nothing to measure against, or they have them, so you are not really starting unlabeled. So the data came first — a synthetic construction scene carrying its own held-out oracle, on one rule: only the scorer ever reads the truth.

Three panels showing every labelled box centre over a 60-second scene, for round 0, round 1 and round 2. Round 0 has a dense cluster of phantom labels inside the dashed stockpile footprints; round 1 still has them; round 2's stockpiles are empty. — Every labelled box centre over one 60-second scene. Round 0 is pure geometry — it finds the machines and the workers, and also fits confident, vehicle-shaped boxes to slivers of stockpile. Round 2, trained on its own filtered output, leaves the piles alone.

Watch the loop run → How it works →

What each pipeline actually produces

The three workflows are a chain, not three alternatives. Each one reads the labels the previous one wrote and hands back a strictly richer set — same schema, more known about every object.

Pipeline	Reads	Writes	What is new
`geo_kinetic_discovery`	LiDAR sweeps, joint states	Unnamed instances — box, heading, trajectory	That a discrete something is there, and where it goes
`bootstrap_new_classes`	Those instances, plus the four camera streams	The same instances, named — each carrying `cls_conf` and `cls_source`	What each thing is, and how much to trust the name
`improve_offboard_model`	Named labels	A trained detector, and a fresh label set from its output	Objects the filter discarded; false positives it let through, removed

Only the third produces a model. The first two are search and naming — they never learn anything, which is what lets the whole thing start from zero. And because the third writes the same schema it reads, its output is a legal input to itself: that is the loop, and round 2 is simply round 1 run again.

Where Grounding DINO comes in

It is the naming step in pipeline 2, and only the naming step — it never proposes objects, because geometry has already decided what exists. Each instance is projected into the camera views where it appears largest, Grounding DINO is run on those with a text prompt, and the detections are matched back to the 3D box by IoU and voted across views.

That division of labour is the finding that shaped the pipeline. The detector names machines and cannot see people: over ten unobstructed views it returned the correct haul truck label nine times, and never once detected a worker plainly visible in three of them. Geometry finds people and cannot name machines — a worker is an easy cluster and an unnameable one. So each is asked only what it is good at, a size prior covers what the detector abstained on, and cls_source records which path produced every name. That column earns its place: detector-assigned names measured 100% correct, size-prior names 67%. A training step that cannot tell them apart is wrong about a third of the second group.

Nothing human-labelled enters anywhere in that chain. A person enters the loop later and deliberately, at the two gates marked on the diagram.

What is here

The live demo

The loop as a running service: every stage in the order it ran, what the router chose between, and the scene and labels to look at.

The data

A synthetic site with LiDAR, four cameras, proprioception and a held-out oracle. Reproducible from a seed, byte for byte.

The pipelines

Discover instances from geometry, name them from pixels, distil a detector and label again — driven by a durable loop that can stop and ask a human.

Datasets

Scenes, truth, prompts, labels and scores as one versioned, immutable kind of thing. Iceberg on magmalake, written natively from Mojo.

Numbers & learnings

What was measured, including the parts that went backwards and why they did.

Papers

Eleven implementations, each its own repository, and where each one sits in the pipeline.

Interesting tidbit

training labels   precision 0.865   recall 0.547   f1 0.670
student output    precision 0.903   recall 0.654   f1 0.759

The distilled model is better than its own supervision on both axes. It recovered objects the label filter had discarded and rejected false positives the filter had let through.

That only happens above a threshold. Trained on the unfiltered labels the round before, the same code went backwards — it inherited the systematic errors and amplified them. Below some label-precision floor a self-training loop compounds its own mistakes; above it, it compounds its strengths. Finding roughly where that floor sits turned out to be more useful than any single score.

Everything on this site is measured on the machine, not estimated. Where a number contradicted what had been written down first — which happened five times — the number won and the write-up changed.