diff --git a/CHANGELOG.md b/CHANGELOG.md index 78bc4ea..87411a3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,70 @@ All notable changes to Playproof are documented here. +## 0.10.0 + +### An arena: many profiles, many games, one definition + +`matrix.ts` turns one hand-written definition into cells, and a cell is one profile playing one game at one objective under one protocol and one sensor, repeated `reps` times. None of those has a default, because each changes what the number means. `matrix-run.ts` plays every cell, attests every replay, and reports mean pairwise Kendall tau-b between the per-game rankings. The runner accepts several definition files and pools them, because two games do not fit one definition: each names its own score channel and its own seed. + +### How an agent is asked for a decision changes its score more than the model does + +Three transports over the one `AgentDriver` seam, so a study can measure the choice instead of baking one in. + +| transport | one process per | measured, same program, native-2048 | +|---|---|---| +| `per-decision` | decision | 37.29 ms/decision, score **4** | +| `persistent` | episode | 1.28 ms/decision, score **1948** | +| `stream` | episode, asynchronous | see below | + +Spawning per decision silently forbids a program from keeping state. It still answers every decision and still attests clean, so the failure is invisible in every field the harness records. + +### The asynchronous transport, and what acts while the agent thinks + +`drivers/stream-sandbox.ts` writes each observation into a sandbox directory and reads moves back from a file, so the agent is never blocked and can run its own scripts and subagents. The game does not wait, which forces three declarations on the protocol: `queue`, `empty` (`noop` stops, `repeat-last` continues, and those are different games), and `pace`. + +- **Without a pace, 12 decisions complete in 2 ms.** An agent that must start a process has lost the episode before it printed a line, and the score measures the host. +- **The sandbox describes itself** in `brief.json`. An agent that guesses its vocabulary writes illegal words, and illegal words read as bad play. +- **An agent that rewrites its action file is still heard.** The file is read forward from an offset, which assumes appending. The first real agent to use this transport kept its last eight moves with `open(path, 'w')`; the offset then sat past the end of a shorter file and the driver was deaf for the rest of the episode, while the health record reported 112 starved decisions. A shorter file now means a rewrite, and its contents replace what was pending. + +### Authoring: measure what a profile builds, not how fast it types + +Under any live transport the agent's start-up time and typing rate count against its score. **Measured across nine cells: the rank correlation between moves delivered and score was 0.94.** One agent wrote a working expectimax search and scored 240, because the harness graded how fast its author typed. + +`author=` splits the cell. The agent gets a practice game at a different seed for `buildMin` minutes, scored by nobody, and must leave an executable `policy` behind. That program then runs cold against a fresh scored instance in its own process, and the replay is attested. Build cost lands in a `build` column and never touches the play score. An agent that leaves no policy is blocked, not scored zero. + +**Measured, native-2048, 6-minute build budget, two replicates each:** + +| profile | score | decisions played | build tokens | +|---|---|---|---| +| opus | 46,124 / 46,216 | 2000 / 2000 | 1.19M / 0.96M | +| sonnet | 8,944 / 3,888 | **609 / 317** | 5.14M / 2.76M | +| haiku | 4 / 60 | 2000 / 2000 | 2.23M / 1.96M | + +Opus's two runs differ by 92 points on 46,000. Sonnet spent 4.3x the tokens on a program that filled the board and lost in under a third of the horizon. The same model under a live transport scored 240. + +### Numbers that were not what they said + +- **The row scored a channel named `score`.** `goal=maximize:` was parsed, validated, sent to the agent, then discarded. CartPole publishes `steps`, so it scored null and could never be ranked, which is why a transfer statistic could not be computed over two games. Measured: null before, 17 after. +- **The rank correlation ignored direction.** A `minimize:` game was correlated raw against a maximize game, so a perfect transfer would have reported as a perfect inversion. +- **An unmetered arm reported zero.** A streamed episode ends while the agent is still working, so an end-of-run cost report never arrives. Usage now accrues from the agent's own stream, and an unmetered arm reports null. A control that declares itself free still totals zero. +- **`authMode` says which credential path was billed.** `'api-key' | 'oauth'`, the union `@tangle-network/agent-interface` uses, read from `CLAUDE_CODE_AUTH_MODE` as the CLI agent registry sets it. Under `oauth` an absent cost means unbilled this way, not free. +- **The transport note substituted one fact for another.** The fate of the agent returned early and hid the starvation count, so the one arm whose agent finished was the one arm whose starvation could not be read. +- **A practice game had no bound.** `pace=0` is correct for a scored episode played by a program. Applied to practice it produced 480,000 turns, 449,821 observation files and 1.7 GB in one cell. Practice now has a 250 ms pace floor and a 20,000-turn cap that depends on no declared number. +- **A cell that takes hours printed nothing for hours.** One authoring cell spent 2h13m on 2000 decisions with a compiled search and reported nothing throughout, so a slow run could not be told from a hung one. A cell is now announced before it runs and prints an elapsed heartbeat. + +### A frame-advance barrier that fails must say so + +`gap()` sends one run-loop iteration between RetroArch frame advances. `FRAMEADVANCE` returns as soon as RetroArch accepts it, so without that iteration the next advance can be issued while the previous frame is still settling. `command` returns null when it gives up and `gap` discarded it, so a barrier that did not happen looked exactly like one that did. The same-process replay gate failed four times in one day with a specific shape: two replays of one boot state and one input log agreed byte for byte to emuFrame 811, then differed on channel values at identical frame numbers. + +### Composition, not reimplementation + +`hillclimb.ts` makes a played game a `Verifier` that a shot loop can climb. `improve` in `@tangle-network/agent-runtime` already owns the loop over ten profile surfaces, and `compareOptimizationMethods` in `@tangle-network/agent-eval` already owns disjoint train, selection and final-test scenarios, bootstrap resamples and Bonferroni-adjusted simultaneous confidence. A game score is not a pass: a green suite has nothing left to win, so `keepGoing` says the program works and another shot is still worth spending. `examples/agent-eval-fit.mts` exists to be compiled, so a change on either side that breaks the join fails the typecheck instead of failing a study three hours in. + +### Documentation + +The README was 1042 lines across 17 sections, so a reader met install, one example, then twenty-five sections of calibration theory before learning an arena existed. It is now 231 lines with a links table, and 700 lines moved into `docs/` as reference pages. The first code block did not run: it referenced `target`, `chooseAction` and `privateKey`, none of which it declared. It now uses the bundled 2048 target and was executed to confirm its output. + ## 0.9.0 ### A tag cannot publish unless ci passed on the commit it points at diff --git a/examples/author-2048.matrix b/examples/author-2048.matrix index a191467..64fe14b 100644 --- a/examples/author-2048.matrix +++ b/examples/author-2048.matrix @@ -13,9 +13,9 @@ game.puzzle adapter=native-2048 target=2048 # A real horizon. A program does not need the clock a typist needs, so the # episode can be as long as the game deserves. -objective.score goal=maximize:score horizon=2000 budgetUsd=4 +objective.score goal=maximize:score horizon=20000 budgetUsd=4 protocol.det frameskip=1 sticky=0 seeds=1 queue=8 empty=noop pace=0 sensor.ascii pixels=off channels=- -reps 2 +reps 1 diff --git a/package.json b/package.json index fc10f4e..b6ebd05 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@tangle-network/playproof", - "version": "0.9.0", + "version": "0.10.0", "description": "Turn games into verifiable, cost-aware benchmarks for any agent.", "license": "Apache-2.0", "type": "module",