Skip to content

Replay upstream state_machine_tests.zig scenarios against the OCaml core - #8

Open
devin-ai-integration[bot] wants to merge 2 commits into
trunkfrom
devin/1791238954-zig-corpus-parity
Open

devin-ai-integration[bot] wants to merge 2 commits into
trunkfrom
devin/1791238954-zig-corpus-parity

Conversation

@devin-ai-integration

Copy link
Copy Markdown
Contributor

Summary

Adds the first automated OCaml-vs-TigerBeetle comparison (follow-up to closing #7). ocam/test/zig_corpus.ml parses every check(...) table in the retained ocam/src/state_machine_tests.zig (byte-identical to the pinned upstream copy) and runs it against State_machine. It mirrors check_version in the Zig file:

  • the table grammar from testing/table.zig (_ = default, leading letter ignored, -n = maxInt - n);
  • expected-reply construction (timestamps, exists echoing the original or rolled-back timestamp, pending-id fill-ins);
  • the prepare/pulse clock, with batch_max.create_transfers = 30 from the test_min config.

Status codes are read from src/tigerbeetle.zig at runtime and matched against Result_code.*_codes, so coarse OCaml statuses still count as matches.

The report is diffed against test/zig_corpus.expected on dune runtest. Parity progress and regressions therefore show up as a diff, and dune promote accepts an intended change. Current baseline:

41 scenarios: 18 pass, 18 fail, 5 skip
create results: 1 status mismatches, 79 timestamp-only mismatches

The failures point at two real divergences:

  1. Error results return timestamp = 0. Upstream returns the event's commit timestamp for every non-exists result, including linked_event_failed. In OCaml this comes from error status = { timestamp = 0L; ... } in state_machine.ml. This causes almost all of the failures.
  2. imported events: imported batch. OCaml rejects an imported transfer with imported_event_timestamp_must_not_regress because it checks against the global commit_timestamp. Zig checks against the transfers groove's key_max, i.e. per object type.

A SKIP means the scenario uses something the OCaml types can't express yet: CDC (get_change_events), reserved/padding bits, or statuses with no OCaml equivalent (exists_with_different_user_data_128, imported_event_timestamp_must_postdate_debit_account).

Supporting changes:

  • State_machine.unsafe_replace_account_for_testing implements the harness's setup action, which overwrites balances directly.
  • The tests stanza now lists modules explicitly so the corpus runner stays a separate executable.
  • README and OCAML_REWRITE are updated.

Checked locally on OCaml 5.2.1 with ocamlformat 0.26.2: dune build, dune build @fmt, dune runtest and opam lint all pass.

Link to Devin session: https://app.devin.ai/sessions/1ffda120da2640bc8996571fcff0a87c
Open in Devin Desktop: https://app.devin.ai/desktop/session/1ffda120da2640bc8996571fcff0a87c?variant=devin
Requested by: @gpu004

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant