Harden tracking job isolation and metric integrity

[sudoax0n]

Summary

Hardens Flyt’s tracking pipeline against concurrent uploads, stale jobs, partial or crashed publication, uncertain process termination, disposable review state, and scientifically invalid event/metric output. The implementation lives in the canonical server.js and tracker.py; temporary patch automation and diagnostics are not part of the final diff.

Guarantees implemented

Job isolation, upload cancellation, and process termination

• Upload reservation occurs before Multer accepts a second file.
• Each run has a durable run ID, isolated workspace, cancellation controller, and tracked child-process set.
• Reset actively destroys an in-progress multipart upload, waits for request closure, removes its partial input, and reopens the upload gate only after cleanup succeeds.
• Reset aborts frame counting, tracking, and transcoding; termination escalates from SIGTERM to SIGKILL.
• Only an observed close event or an already-set exitCode/signalCode proves child termination.
• Signal-delivery exceptions, kill() returning false, and child error events without closure reject termination and keep the system gated.
• A failed stopping context is retained and retried by later resets rather than being discarded.
• If a child closes after an earlier termination failure, the server re-evaluates the retained stopping context and reopens uploads only after the child set is empty and run cleanup has completed.
• Stale runs cannot publish after reset or replacement.
• Startup removes abandoned input-* uploads and run-* workspaces left by a process crash.

Transactional publication and crash recovery

• The complete publication set is recorded in a transaction manifest before destination replacement.
• The manifest records every destination, stage file, backup file, and whether the destination previously existed.
• A crash during partial publication rolls back the entire incomplete transaction on startup; it cannot retain one new artifact while restoring the rest from the old run.
• An explicitly committed transaction is finalized rather than rolled back.
• Transaction-critical deletion is strict: deletion errors propagate and the path is verified absent. A failed rollback retains its manifest for a later recovery attempt rather than reporting success.
• Legacy orphan backups without a manifest fail closed instead of being used to guess a potentially mixed bundle.
• Current aliases and durable history data/events/verification/index are published as one transaction.

Frame integrity

• Exactly one TRACKER_SYNC marker is required.
• Independently decoded input frames, tracker frames, tracker-reported CSV rows, actual CSV rows, raw-video frames, and final-video frames must be positive integers and agree exactly.
• OpenCV CAP_PROP_FRAME_COUNT is diagnostic metadata (expectedMetadataMatches) and cannot reject an otherwise complete ffmpeg-verified decode.

Event, review, and metric integrity

• Courtship requires a valid two-fly observation, finite proximity, no occlusion, both fly areas present, acceptable identity confidence, and proximity below the configured threshold.
• Missing or merged detections are low-confidence and cannot independently become courtship bouts.
• The tracker emits tracking_valid and detection_count.
• Dropouts may retain coordinates for display continuity, but proximity and individual-fly velocity are stored as missing rather than fabricated or carried forward.
• Dashboard averages, history avgProximity, event means, and Prism exports ignore invalid/nonfinite/occluded proximity observations.
• Prism distance and individual-fly velocity fields are blank for invalid/merged observations.
• Run IDs are created before processing, and current and historical event IDs are scoped as <runId>:<eventId>.
• Verdict writes verify that the complete event ID actually occurs in that run’s events.json; unscoped and orphan event IDs are rejected.
• Current and historical views share the canonical history/<runId>/verification.json, so current-run verdicts survive later runs and historical reloads.

Regression coverage

Node utility tests

• strict tracker marker and frame-integrity validation;
• complete publication, in-process rollback, partial-publication crash rollback, and committed-transaction finalization;
• strict deletion-failure injection proving rollback remains failed and the manifest remains recoverable;
• fail-closed orphan backup handling;
• child error without close, failed termination tracking, and real SIGTERM → SIGKILL behavior;
• startup cleanup of abandoned runtime inputs/workspaces;
• event membership and run-scoped verification paths;
• validity-aware history and Prism metrics.

HTTP and lifecycle integration tests

• simultaneous uploads;
• active cancellation of a stalled multipart upload, partial-file removal, and gate reopening;
• reset waiting for active work to stop;
• reset during transcoding and stale-publication prevention;
• failed termination returning an error while uploads remain blocked;
• a second reset retrying the retained stopping context instead of falsely succeeding;
• a genuine later child close clearing the retained stopping state and allowing a subsequent run to complete;
• complete-count publication and mismatch preservation of the prior bundle;
• restricted browser origins;
• current-run verdict → second run → historical reload;
• rejection of unscoped and nonexistent run event IDs.

Tracker and frontend tests

• confident separate flies, occlusion, low confidence, missing flies, and invalid observations;
• initial zero velocity and reachable low assignment confidence;
• OpenCV frame-count mismatch treated as diagnostic;
• invalid proximity excluded from averages/event means/Prism output;
• invalid individual-fly velocity represented and exported as missing;
• historical verification and timestamp contracts.

CI

Permanent Critical fixes CI passed on clean head 8611885ad49cc7a0545897df4a6e3fa6dad819e6:

• Linux: backend/test lint, all Node utility + HTTP + lifecycle + metric tests, Vite production build, Python dependency/syntax checks, and tracker scientific regression tests.
• Windows: backend/test lint, all Node utility + HTTP + lifecycle + metric tests, and the Vite production build.

Review / merge notes

• Base: main at 61b9d424f777176f9aea84f4c4e04e5320cf1c18.
• Head: fix/critical-job-races at 8611885ad49cc7a0545897df4a6e3fa6dad819e6.
• main has not been modified or merged.
• Final diff: 14 purposeful files; the additional file is the focused termination-lifecycle integration test. No generated media, output CSVs, diagnostics, temporary workflows, or patch scripts.
• Keep this PR draft until a representative real-video smoke test succeeds on the actual Windows target environment, including MP4/H.264 plus the AVI/MOV formats Flyt intends to support, reset during a real tracker/ffmpeg stage, current-review persistence, and blank invalid proximity/velocity output.
• The development history contains 75 commits from iterative hardening; use Squash and merge only after the real-video smoke test passes.