Approach
A detector, a tracker, one reference view and a rule per class
Nothing here is trained by us. The only learned parts are COCO-pretrained YOLO26 detectors (26m for Part A, 26s for Part B); everything after them is geometry and rules we wrote for this one junction. This page has enough detail to rebuild it.
Pipeline
The scene
A camcorder on a tripod, high on a building at one corner of the junction, looking straight up the avenue. North, south, east and west are our names for the arms as the camera sees them, not compass bearings. We describe everything in one 1920x1080 reference view; the 4K clips are mapped onto it.
Southbound (SB) traffic comes towards the camera on the left of the median, four lanes at the stop line. Northbound (NB) traffic drives away on the right, past a bus stop near the top of the frame.
The north crossing is a zebra across both carriageways just past the SB stop line, split by the median nose. The west crossing is a long diagonal zebra over the west arm at the bottom left, between three pink islands. Nothing else in view is a legal place to cross.
Things that are easy to get wrong here: cars parked along the left edge are off the carriageway; buses wait at the NB stop for 20 to 60 s; the SB queue at a red light stands still for 36 to 39 s; cyclists come out of the detector as a person plus a bike.
What is learned and what is a rule
| Part | Stage | Kind | What exactly |
|---|---|---|---|
| A | Detection | Learned | YOLO26m with COCO weights, not fine-tuned. 1280 px input, confidence 0.1. Keeps person, bicycle, car, motorcycle, bus, truck and six animal classes. |
| A | Tracking | Algorithm | ByteTrack (the Ultralytics implementation), one tracker per group so boxes of different kinds never get linked. High 0.3, low 0.1, new track 0.4, 2 s buffer. |
| A | Registration | Geometry | SIFT, 6000 features, on 960 px wide copies; homography by MAGSAC with 3 px tolerance against a midday and a dusk reference, both per-pixel median backgrounds. Done again on keyframes (every 2 s for 30 s, then every 10 s): the lamps use the latest, the rules one registration of the keyframes' median. |
| A | Signal phase | Rule | Lamp patches on the median-nose head, mapped from the reference view; the lamp with the strongest colour contrast relative to its threshold wins (20 on a 0 to 255 scale for red and yellow, 6 for green). |
| A | Rider or pedestrian | Rule | A person whose box is more than 30% covered by a bicycle or motorbike box in over 40% of its samples, or whose median speed is above 110 px/s or above 1.2 of its own height per second, is a rider. |
| A | Events | Rules | One function per class on trajectories, zones and phase. Thresholds were tuned on our dev labels, by hand and with a small grid search for jaywalking and failure to yield. |
| B | Detection | Learned | YOLO26s with COCO weights at 960 px, confidence 0.2, every 3rd frame (10 Hz). |
| B | Risk | Hand-built model | Constant-velocity look-ahead of 3 s from 0.8 s of history; pair hazard from closing speed, time to contact and footprint overlap in metres (one scale per pair, taken between the two users), plus hard braking; a pair counts after 0.6 s of danger; the worst pair sets the score, EMA 0.35. |
The Part B risk score
Every 0.1 s, for each pair of road users, with positions in metres (one scale per pair, taken between the two) and velocities fitted to the last 0.8 s of their tracks:
- Pair hazard
h = 1 / (1 + exp(2.5 (t - 0.9))) × (0.4 + 0.6 d) × min(1, c / 10)if they close in at c ≥ 4 m/s and, moving in straight lines, their footprint circles overlap within 3 s; otherwise 0. t is the time in seconds until the circles first touch, d the overlap at the closest approach divided by the sum of the radii (at most 1). Radii: person 0.35 m, bicycle 0.6, motorbike 0.7, car 1.3, truck 1.8, bus 2.0; both shrink to 40% when one of the two moves under 1 m/s. - Braking
risk = min(1, h × (1 + 0.5 max(b1, b2))), b = clip((a - 4) / 6, 0, 1)a is a user's deceleration in m/s² over the same 0.8 s, counted only if it was faster than 3 m/s. A pair counts once it has looked dangerous for 0.6 s without a break. - Score
score = 0.65 × previous score + 0.35 × the highest pair risk in the framePairs left out: two pedestrians, a pedestrian off the carriageway, users on opposite sides of the median, a moving vehicle and a standing one, users more than 30 m apart.
Code: pair_hazard and Anticipator._hazard in src/parivision/risk.py.
The rule for each class
Distances are in pixels of the reference view; near the north crossing one metre left to right is about 50 px, and fewer along the avenue. Score A averages over classes, so a class we predict that never happens in the test set adds a zero. Five classes have no rule. We have a U-turn rule, but we keep it off for this reason: nothing in view says the U-turns here are illegal, so the organisers may never label one.
- ACC
Accident
emittedTwo road users meet at speed, both velocities change at the contact, and both then stand together for 2 s. Checked on public CCTV crash clips; it never fires in our samples. Part B gives the per-frame probability that one starts within 5 s.
- NM
Near miss
no ruleNo rule yet. The Part B time-to-collision score is the closest signal we have, and we have not turned it into segments.
- RED
Red light
emittedA southbound vehicle's front crosses the stop line after the lamp has been red for at least 1 s, with at least 1.5 s of red still to go. Yellow does not count. The segment ends when its track ends, at most 60 s later.
- WW
Wrong way
emittedA vehicle faster than 40 px/s heads more than 120 degrees away from its carriageway's direction for at least 1.5 s, at least 6 px inside the carriageway (the service road along the far kerb does not count). Each carriageway has one legal direction in the reference view. On 1074 real tracks run backwards it finds 1057; it never fires on the samples.
- UT
Illegal U-turn
rule offWe find U-turns (a southbound vehicle turns round the median nose, within 130 px of it, and leaves northbound) but do not emit the class, because we have not confirmed which U-turns are illegal at this junction.
- SV
Stopped vehicle
emittedA vehicle stays under 12 px/s for 10 s or more on the northbound carriageway or in the junction box on the southbound path to the south exit. The bus stop is excluded, and so are the southbound approach, where cars queue at the red, and the northbound lanes east of x = 1720 px.
- JAY
Jaywalking
emittedA person who is not riding a bike has their feet on the carriageway, clearly outside every zebra, for at least 1 s. Riders are found by a two-wheeler box over the person, or by a median speed above 110 px/s or above 1.2 of their own height per second (walkers stay under about 0.9).
- FTY
Failure to yield
emittedA car, bus or truck moving at 20 px/s or more drives across a zebra while a walking pedestrian is out on it, not at the kerb, within 160 px of the vehicle. A car that stood before the zebra and drives on once the pedestrian is past its path and walking away has let them through, and does not count. Motorbikes are skipped. The segment runs from the front entering the zebra to the rear leaving it.
- IT
Illegal turn
no ruleNo rule yet. The plan is to compare the lane a vehicle holds at the stop line with the arm it leaves by.
- SLC
Solid line crossing
no ruleNo rule yet. The solid lane lines before the southbound stop line would be drawn once in the reference view and a lane change across them flagged.
- STOP
Stop line
emittedA southbound vehicle stands for at least 1 s with its front more than 8 px past the stop line while the lamp is red. The segment ends at green, or earlier if its track ends.
- CONG
Congestion
emittedSouthbound traffic stands still on green. From 8 s into the green, 8 or more vehicles under 15 px/s on roughly the last 40 m of the approach and past the stop line on the way to the south exit make a jam, and so do 5 or more past the stop line alone. A jam must start on green or yellow and last at least 6 s, and it carries on into the red while 5 or more still stand past the stop line.
- OBS
Road obstacle
no ruleNo rule yet. The detector already keeps six COCO animal classes, so an animal on the road could feed it.
- FIRE
Fire or smoke
no ruleNo rule yet, and nothing in the sample clips to test one on.
How we read the signal
The heads over the SB lanes face away from the camera. Two heads do face it. The one on the median nose is a three-lamp vehicle signal for the approach from the south, which runs in the same phase as SB; we read the phase from it, and it stays readable at dusk. The one on the pole at the left corner is a two-lamp pedestrian signal for the west crossing, and we do not use it.
For each of the three lamps we take a small patch around its centre, mapped from the reference view into the frame. A lit red lamp is much redder than its dark housing, a lit green one much greener, a lit yellow one much warmer. The lamp with the strongest contrast relative to its own threshold wins. Position decides the phase, not colour alone, so tail lights passing behind the head do not read as red.
When every lamp is dark the head is "off": the flashing part of green, or a car hiding the head. The cycle here is green, 3 s flashing green, about 3 s yellow, then red. We sample it every 0.1 s.
Why registration was needed
The clips are not framed identically: the afternoon clips are shifted by up to about 60 px and zoomed by 1 to 2% against the morning ones. And the camera is not quite still at the start of a recording: in C3896 the view drifts 9 px over the first 40 s while the camera settles, enough to put the signal lamps outside the patches we read. Every zone we test against (stop line, crossings, carriageways, bus stop) is drawn once, by hand, in the reference view. If we drew them per clip, a new test clip with slightly different framing would put the stop line in the wrong place.
So we match SIFT features of the clip's frames against a midday and a dusk reference, each a per-pixel median background with most of the traffic removed, and keep the homography with the most inliers: on the first frame, then again every 2 s for the first 30 s and every 10 s after. The signal lamps are read with the latest homography. At the end the keyframes' per-pixel median is registered once more, and that homography maps the tracks into the reference view for the rules.
Models, data and licences
| What | From | Licence | How we use it |
|---|---|---|---|
| YOLO26m, YOLO26s, YOLO26n weights | Ultralytics | AGPL-3.0 | YOLO26m for Part A, YOLO26s for Part B and, exported to ONNX, in the in-browser demo; YOLO26n for Part B in the CPU fallback |
| COCO 2017 | COCO Consortium | Annotations CC BY 4.0; images under their Flickr licences | What the detector weights were trained on. We did not train on it ourselves. |
| ByteTrack (Ultralytics implementation) | Zhang et al.; Ultralytics | MIT (original), AGPL-3.0 (Ultralytics) | Multi-object tracking |
| OpenCV | OpenCV team | Apache 2.0 | Video metadata, SIFT, homography, colour |
| PyAV | PyAV contributors | BSD-3-Clause | Decoding the clips for Part A |
| PyTorch | PyTorch Foundation | BSD-3-Clause | Runs the detectors |
| onnxruntime-web | Microsoft | MIT | Runs the YOLO26s export in the in-browser demo |
| ffmpeg.wasm | ffmpegwasm, FFmpeg developers | MIT; FFmpeg core GPL-2.0-or-later | Converts clips the browser cannot play, in the demo (source: github.com/ffmpegwasm/ffmpeg.wasm) |
| Sample clips C3896, C3897, C3902, C3905 | Hackathon organisers | No licence stated; given to participants for this hackathon | Scene analysis, our dev set and tuning. Not in the repository. |
| Our dev labels (labels/) | PariVision, drafted with Claude agents | AGPL-3.0, with the repository | Tuning and scoring on the sample clips |
Frame sampling and runtime budget
- Source
- 3840x2160, 29.97 fps, about 140 Mbit/s
- Part A frames
- every 3rd, 9.99 per second, resized to 1920 px, batches of 8
- Part A detector
- YOLO26m at 1280 px
- Part A time cap
- stops at 1.3x the clip length, and before that drops to every 2nd or 3rd sampled frame if it falls behind; a short clip gets up to 60 s but never more than 1.8x its length; the organisers' runner allows 3x for A and B together
- Part B frames
- every 3rd frame it receives, 1280 px wide, YOLO26s at 960 px; down to 1 per second if it falls behind, and none once the harness's own decoding of the frames still to come would take it past 95% of its budget (2.8x the clip length minus Part A's time)
- Measured runtime
- 3.28x clip length, Part A + Part B, Apple M5 laptop on MPS, time guards off. The harness allows 3x. On a Colab T4 the detectors take 0.34x (Part A) and 0.19x (Part B) of real time, but its 2 vCPUs decode these 4K 10-bit files at only 9 to 12 frames per second, so there the harness alone needs about 3x to read them. Part A first times the harness's own decoding loop on 40 frames and leaves it that time, with a 1.2x margin.
At 10 fps a walking pedestrian moves about 7 px of the reference view between samples near the north crossing, and a car at 40 km/h about 1.1 m. That is enough to know who is on a zebra and who crosses the stop line. The ablations on the Results page show what 5 fps and smaller detectors do to the score.