The pipelines
Three named workflows, each resumable and each recording what it did. They are wiring, not algorithms — every stage is a component with its own repository. They also run as a durable service that can stop and ask a human.
1 · geo_kinetic_discovery
No model, no checkpoint, no class list — and no class names out either. What it produces is instances: a discrete something is here, this is its box, this is its trajectory.
proprioception self-mask forward kinematics from /ego/joint_states
→ terrain classification Stone.mojo — connectivity, not height
→ voxel clustering 26-neighbourhood connected components
→ oriented box fit PCA heading in the ground plane
→ gated association Hungarian.mojo — solve_gated
→ RTS smoothing Kalman.mojo — smooth_track
→ motion filter the stage that decides whether the loop compounds| Stage | Implementation | Paper |
|---|---|---|
| terrain classification | Stone.mojo | STONE — Park et al., ICRA 2026 |
| gated association | Hungarian.mojo | Jonker & Volgenant, Computing 38, 1987 |
| offline smoothing | Kalman.mojo | Rauch, Tung & Striebel, AIAA Journal 3(8), 1965 |
What geometry and motion can actually settle:
- Terrain against not-terrain — free and reliable.
- Instances — connected clusters standing on that terrain.
- Motion attributes — the strongest signal here by some distance, and the one the whole loop turns on.
- A size bucket — machine-sized and person-sized boxes differ by two and a half orders of magnitude, and a 5 m³ split separates them 97.5% of the time.
- The ego's own parts, exactly — kinematics knows which rigid body is the boom. The one place geometry hands over true named classes, free.
What it can never settle: excavator against dozer; worker against any person-shaped object; stockpile against spoil pile, where the discriminator is intent and simply is not present in a point cloud.
Why terrain is a connectivity question
Ground removal by height fails on a stockpile: a pile at its 34° angle of repose climbs 0.67 m across a one-metre cell, so the bottom is called ground and the top third survives as slivers that cluster into vehicle-shaped objects. Tightening the tolerance only trades those false positives for missed objects near slopes.
So the ground surface is grown, not thresholded — seeded from the cells the machine drove through, flooding outward one repose-limited step at a time. A pile is continuous with the grade beneath it; a truck is a three-metre jump. Height stops mattering, continuity starts mattering.
2 · bootstrap_new_classes
Gives those instances names, from two sources that fail in opposite directions.
select views where each instance projects largest and is in frame
→ detect Grounding DINO, prompt "excavator . haul truck . worker ."
→ associate detection ↔ projected 3D box, by IoU
→ vote aggregate across views, confidence-weighted
→ size prior for what the detector could not name| Stage | Implementation | Paper |
|---|---|---|
| open-vocabulary detection | GroundingDino.mojo | Grounding DINO — Liu et al., ECCV 2024 |
The detector names machines and cannot see people. Over ten
unobstructed views, nine returned haul truck with the correct
label; a worker visible in three of them was never detected once.
Geometry finds people and cannot name machines. So the
detector is asked only where it has a chance, and the size prior covers the
rest.
Every name carries cls_conf and cls_source, because
the two paths are not equally trustworthy — see
the numbers.
3 · improve_offboard_model
Distil a detector from existing labels, then use it to label better. Run it repeatedly and it is a self-training loop.
filter labels → the load-bearing parameter, not a knob
→ training set NPZ in the layout CenterPillars already reads
→ train student CenterPillars.py
→ infer CenterPillars.mojo
→ track the same association and smoothing as round 0
→ score against the held-out oracle| Component | Implementation | Paper |
|---|---|---|
| pillar encoder | CenterPillars.mojo · .py | PointPillars — Lang et al., CVPR 2019 |
| BEV backbone | SECOND — Yan et al., Sensors 18(10), 2018 | |
| centre head | CenterPoint — Yin et al., CVPR 2021 | |
| association & smoothing | Hungarian.mojo · Kalman.mojo | as round 0, above |
Two further stages are implemented and evaluated but not in the default
path — OfflinePoly.mojo
(Offline-Poly) and
LabelFormer.mojo
(LabelFormer). What holds each
back is on the papers page.
Nothing human-labelled enters at any point. The output is the same schema as the input, so feeding one round's labels into the next is the loop.
Whether that loop compounds or degrades is decided entirely by the filter feeding it. On unfiltered labels the first round went backwards. That is the single most important finding here, and it is on the numbers page.
The loop as a service
The three workflows above are the stages. What runs them is a Restate service, which is what lets the loop stop and wait for a person — for days if it needs to, holding no process while it waits — and survive being killed mid-stage.
train is the one stage with no
arrow down: it writes a checkpoint, not a dataset.
One call advances a run by one stage: ask the router what is runnable, record the decision, run it, record the outcome. A stage runs one of three ways, and the timeline says which:
| Executor | What runs there |
|---|---|
subprocess | sitegen and the training scripts, spawned with their output captured |
mojo | a handler on the executor service — in-process, no compile per call |
inproc | pure Mojo the orchestrator runs itself: the label filter, the planning stages, the Iceberg publish |
Deciding what to run next
Which stage runs next is a decision, and it is recorded. The router assembles the legal stages — those whose inputs exist and whose work the ledger has not already done — describes each one's inputs and outputs, and asks a model to name exactly one. The answer is checked against that list, never trusted: a model that invents a stage, or names a real one that cannot run yet, is overridden by a random legal pick.
Both the prompt and the verbatim answer are stored on the decision. That matters more than it sounds: a router quietly falling back to chance and a model that happens to agree with chance produce identical timelines unless you kept the exchange. No model transport is wired up yet, so today every decision is the fallback — and the timeline says so on each row, which is the honest version of a feature that is half-built.
Two planning stages
Choosing the next stage is the small decision. Two larger ones are stages in their own right, and both write their reasoning to the evaluations dataset rather than returning a number and forgetting it.
| Stage | Decides | On what signal |
|---|---|---|
select_scene | Which scene is worth labelling next | How unresolved it is — the fraction of tracks the detector could not corroborate, plus how weakly the rest were named. Not model uncertainty: that needs a model, and this has to work on round zero when there is not one. |
optimize_prompt | Which prompt the naming step should use | Class coverage over a gold set — what fraction of the classes actually present the prompt's phrases can even express. A prompt that cannot name a thing will never label it, so coverage is a ceiling, computable without running the detector at all. |
Every candidate's score is written, not just the winner's. A selection you cannot audit afterwards is indistinguishable from a coin toss.
Where a person comes in
Two stages block on a human, and blocking is the point — the run genuinely
stops until someone answers.
gold_labeling asks for ground truth on a scene
from scratch, producing the small trusted set that
optimize_prompt is scored against.
human_review shows the labels a round
produced and takes corrections. Either way the edits are materialised as a
new immutable version of the labels dataset with
cls_source = "human", so a correction is a citable artefact
rather than an edit that vanishes into a CSV.
Two guarantees that sound alike
Restate's journal makes a crashed run resume without redoing finished work within one invocation. The content-hash ledger skips work across separate runs. Hence the definition of runnable: inputs exist and the ledger lacks the key — not that the outputs are missing, which would make a stage whose inputs changed permanently ineligible and quietly serve stale results forever.