Shared from Truffle
MATCHA — checks and their ladders
Updated
MATCHA — checks and their ladders (Oct 1)
Visual studio acceptance — October 2, 17:49 PT
VAST rollout after the stable asset-path fix reported successfully rolled out. Public browser loaded the new Search/Curate navigation, real warehouse playback and YOLO person box. Clicking the box saved a real crop; a second observation pin was captured mid-flight into the right tray, and the tray count increased from9 to11. Next-sample control changed playback to00:00.03. Find similar on the collected crop returned four real warehouse clips; timeline keyboard seeking changed the clip clock. After the follow-up hotpatch, all timeline lanes fit in the 1356×777 browser viewport (bottom773px), and no console errors were observed. No suite run; JS syntax and Python AST checks only. Studio source.
The first UI-only update exposed the ConfigMap resolved-path failure and returned Unexpected server error; this failed result is retained above rather than described as a seamless deployment. Screenshots matcha-object-flight.jpg and matcha-timeline-and-matches.jpg are saved locally; only code was published to GitHub. Prior dataset/export acceptance applies to its dated version below.
Demo evidence — October 2, 16:37 PT
Deployment75f2188 successfully rolled out. Browser shows authenticated archive access and a separate Curator page. GLM returned four concepts; original VSS retrieval failedHTTP500, then cached archive fallback returned a real board. A subsequent “forklift” refinement displayed warehouse clips and visible forklifts. Screenshot saved locally as matcha-curator-live.jpg. Curator full save/export and improved relevance ranking were not exercised in this emergency release. Existing delivery/restart evidence below remains valid for its recorded version. 255 offline checks passed before the operator prohibited further tests; none were run after that instruction. Deployed source · Requested repo mirror. Evidence: October 2 operator screenshots and live browser/VM observations.
Cloud acceptance and next verification — October 2, 16:05 PT
Verified in the live VAST app: real media playback, direct search, automatic unprepared-video previews, two selected Warehouse label revisions, and reference delivery 6bb11535b1ca4e189674456db6636774. The downloaded ZIP passed integrity and member checksums; its SHA256 is 8f0352e96b5245066dc9cb24fc3624a9ea6c8a7c97c3305d961f9ce6bdbe86fa. A separate frozen QA delivery survived pod replacement with identical manifest and ZIP hash (d8fd67f82a058cbfeff4f73917f4ef93b200a0f0038fc598fa5ee77fa1d384c5). This resolves the cloud restart check marked pending below.
244 offline tests currently pass, including exact crop/timestamp pins, object-track Embed1 request bounds and cache contracts, bulk-selection atomicity and selected-set retrieval. These are implementation tests; actual learned crop comparisons and archive-wide precomputed-vector ranking are still pending. No W&B training comparison or measured quality/compute saving is claimed. Keep these engineering distinctions in evidence/research documentation rather than repeating caveats throughout the product UI.
Evidence: live browser delivery download and hash validation, VM restore smoke output, current local test suite and public source. Synthesis.
New verification evidence — October 2, 14:41 PT
165 offline tests pass, including atomic bulk edits/stale-preview rollback, field preservation, grounded reference ZIP export and checksum verification, durable S3 snapshot restore, rejected corrupt/missing snapshots and fail-closed persistence. UI QA exercised real local video/YOLO, attribute filtering, two-item preview/apply and a frozen delivery. These checks do not establish label accuracy or cloud restart durability.
The VAST app now reports Running1/1; /app/api/status confirms configured VAST, S3-backed ledger and GLM planner. Direct VM GLM-5.3-Flash query planning succeeded in7.37s/1call; previous W&B403 remains dated history, not current evidence of unavailability. W&B training/evaluation publication and measured ladder savings remain unverified. S3 write/read succeeded in the assigned app prefix. Cloud end-to-end and post-restart checks are pending. Synthesis.
Evidence: public app source, commit a148d5f, 165 passing offline tests, Team 25 VM terminal results and the October 2 operator messages in this Codex session. No credentials or footage were published.
Quality against the researcher's dataset brief — October 2
Pablo defines a research tool for tailored datasets with task descriptions, actions and labels. Validate against a versioned brief/ontology: required annotations, temporal consistency, unresolved claims, relevant coverage, duplicates and source-grouped split leakage. The existing eight demo checks do not define every researcher's acceptance policy. Use reviewed holdouts for label quality; model agreement and caption diversity alone do not prove better data.
Proposed W&B integration links frozen dataset versions to measured evaluation/training outcomes and annotation/reviewer disagreements. Remote publication and downstream comparisons are not yet demonstrated. Workshop Cosmos/Embed1/YOLO health is now verified; W&B's earlier403 is separate. Current synthesis and evidence. Source: October 2 operator direction and VM health responses; earlier dated failures remain history.
Implemented validation boundary — October 2, Codex-3
The independent MATCHA/VAST build preserves eight check contracts, bounded per-check model routing, input/model/prompt cache keys, append-only run/review evidence, uncertainty and dispute handling. Pixel brightness/frame-change and detector confidence are cheap evidence, not sufficient answers about gaze, idleness or task success. No learned calibration file is installed, so early stopping cannot claim a measured correctness guarantee. Total upstream VSS ingestion compute remains unknown.
The evaluator requires distinct candidate and reference models plus matching input/protocol hashes. Disputes, missing results, malformed outputs and failures remain reported instead of silently disappearing from denominators. Reference agreement is not ground-truth accuracy. Actual W&B discovery on the supplied VM failedHTTP403 twice; model evaluation and routed quality/cost savings remain unverified. The old proposed model shortlist and cents/minutes budget below are historical estimates, not availability or measured results.
Dataset validation now reports selected count, missing original provenance, unresolved episodes, unverified descriptions and coverage gaps. Train/eval grouping connects original-source and known near-duplicate groups, including bridge episodes; unknown lineage remains unassigned. More wording variants do not increase unique video diversity. Pipeline UI displays real integration state and observed runs, with unknown costs explicit.
One ladder per check. The full list is the check catalog part 1 · part 2 (119 checks). The demo leads with eight: hands visible, acting hand, phone in hand, looking away, idle ratio, dropped object, action phase timeline, task success. Each check (phone in hand, looking away, blur...) climbs its own rungs, keeps its own probability and evidence, and stops as soon as it is sure. The clip verdict is built last from the per-check results, and the board always shows them separately: "phone in hand 0.80, looking away 0.60". Fact-checked against the saved event research. Part of MATCHA.
Rungs (shared by all checks)
| Rung | Source | Cost | Notes |
|---|---|---|---|
| R0 metadata | stored columns, ffprobe | free | |
| R1 pixels | frames decoded once at 2-4 fps: luminance, saturation, Laplacian, frame difference, flow magnitude | CPU (VM: 4 vCPU, no GPU) | |
| R2 detections | YOLO11s boxes for every native frame (25-30 fps, conf 0.4), one gzip sidecar per piece; we build tracks ourselves | one GET per piece | no stored track IDs; a missing sidecar (404) = unsure, never "absent" |
| R3 vectors | stored Cosmos-Embed1 piece vectors vs check prompts embedded once; track vectors only if Embed1 on crops passes the day-one test | free per piece | |
| R4 audio (optional) | Canary-1B ASR | unknown | not wired into the pipeline; use only if the clips have audio |
| R5 frame model | W&B vision model on 2-4 frames or crops | low | latency unpublished; image input format untested |
| R6 Cosmos Reason | the 5 s piece as video | high | NVIDIA publishes about 0.67 s for a yes/no on a 10 s clip (H100) |
| R7 human | review queue on the board | — |
Capacity: 5 concurrent calls per team per model; one YOLO worker sharing a GPU with Embed1; W&B returns 429 at an unpublished limit. Retry with backoff.
Lead checks (pack B demo)
| Check | Output | Rungs, cheapest first |
|---|---|---|
| hands visible (count 0 / 1 / 2, share of work time with hands out of frame) | share + spans | R5 → R6 (no rung below R5 sees hands; the re-ingest prompt can answer it at intake) |
| acting hand (left / right / both, from the wearer's side) | enum per span | R5 → R6 |
| phone in hand (also tablet, laptop) | bool + span + box | R2 ("cell phone" is a COCO class) → R5 → R6 |
| looking away from the work surface | score + span | R2 → R3 → R5 → R6 |
| idle ratio (no hand work or progress) | share + spans | R1 → R5 → R6 |
| dropped object | bool + time | R2 → R5 → R6 |
| action phase timeline (reach, grasp, transport, place, release, per hand) | spans | R6 → R5 verifies the event frames |
| task success (success / partial / fail / uncertain) | enum + time | R6 |
All other checks run from the catalog with the same rungs.
How one check climbs
- Each rung returns a score for the check on the piece (per second for pixel checks).
- A per-check combiner (logistic regression over the rung scores so far) gives one probability. Each rung adds evidence; it does not erase the rung below.
- Each check has its own band [lo, hi] from its target precision. p < lo = no, p > hi = yes, inside = climb one rung.
- Cosmos and W&B give a probability from token probabilities when the endpoint returns them, otherwise from agreement over 2-3 samples with different frames.
- Every answer row: check, piece, time span, rung reached, p, evidence frames, model, ms, cost, version.
Reference answers
- No human labeling for the hackathon. The reference for each check is the strongest call on the stack: Cosmos Reason with reasoning on, and the largest W&B vision model. Where the two agree, that answer is the reference; where they disagree, the piece is marked "disputed" and left out of scoring.
- Each check's band [lo, hi] comes from its agreement with the reference. Every model in the leaderboard is scored the same way.
Sharing compute
- One decode, one set of crops, one embedding pass serve every check; R0-R3 run for all checks in one pass.
- One question per Cosmos or W&B call, so each check's band stays valid. Batching several checks per call is tried only after a 20-piece test shows the batched answers match the single ones.
- A threshold change re-scores from stored rows, with no model call.
Clip verdict
- Severity per check: hard (broken file, lens fault > 30% of the time), soft (hands not visible, phone in hand, looking away, idle ratio, dropped object), info (environment, person count, acting hand, action phases).
- Reject = a hard check fires. Borderline = a soft check fires for more than a set share of the time, a lead check stays unresolved, or R5 and R6 disagree ("disputed"). Usable otherwise. Checks outside the lead set use fixed thresholds and never reject on their own.
- The verdict cites the checks that drove it; per-check probabilities stay visible.
Board
- One strip per check along the timeline (colour = p): "phone in hand 0:41-0:47, 0.83" at a glance.
- Per check: rung reached, p, evidence frames; click jumps to the moment.
- Compute panel: per check and rung, calls, latency p50/p95, cost, share decided at each rung.