Benchmark serialization options#1525
Draft
gonzalocasas wants to merge 13 commits into
Draft
Conversation
Formats: add optimized compas_pb (protobuf) plus compas_pb_zip/zstd, and compas_msgpack (+ _zstd) — MessagePack over the JSON-shape tree (the Kumiki approach) applied to COMPAS types via an msgspec enc_hook on __jsondump__. All optional; skipped if the dep is missing. Corpus: add `graph` and collections of primitives/shapes (points, vectors, lines, frames, planes, boxes, spheres, circles). The harness now measures a list of Data or a single object transparently; metrics gains max/RMS coordinate error for lossy profiles. Report: executive summary is rendered client-side with a three-pill Uncompressed/Compressed/All filter that hides non-matching formats, re-normalizes the bars, and re-bases the summary; tiles report median ratios + range across subjects + win-counts. Samples: results/samples/ holds tiny encoded artifacts per subject — .json, .pb + .pb.json (protobuf bytes rendered back to JSON), .msgpack + .msgpack.json — to inspect each wire shape. Fairness: serialize a fresh fixture per format. Dumping to JSON accesses .guid (forcing it); a shared object let compas_pb re-serialize those forced guids, overstating its size on primitive/shape lists by ~30%. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds polylines, polygons, beziers, polyhedrons, and transformations (17 subjects total). The compound types each hold several points, exercising compas_pb's flat repeated-double point arrays; all round-trip losslessly and compas_pb is smallest and faster to load than JSON on them (e.g. polylines@10k: pb 2.4 MB / 269 ms vs json 5.4 MB / 381 ms). Also caps the `full` preset to runnable sizes (mesh 1e6, pointcloud 1e7) so a full run completes in reasonable time/memory; the N1 scaling trend is clear by there. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Captures results/baseline_full.{csv,html} for the bulk-numeric subjects (mesh, mesh_attrs,
pointcloud, graph) swept across 1e3 -> 1e6 elements. At 1e6, compas_pb vs JSON: wire
2.1-3.0x smaller, load 1.2-3.2x faster, peak memory 1.2-1.9x lower. N1 met with a proposed
acceptance bar (>=1e5 elements): wire >=2x, load >=1.2x, peak-mem >=1.2x.
Caps the full-preset pointcloud at 1e6 (was 1e7): beyond ~1e6 the tracemalloc peak-memory
probe dominates runtime.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds the remaining geometry types so the corpus benchmarks all 31 of compas_pb's native serializable types: conics (arc/ellipse/parabola/hyperbola), solids (cylinder/cone/capsule/ torus), quaternion, and the transformation subtypes (translation/rotation/scale/shear/ reflection/projection). 32 subjects; all round-trip losslessly except the ~1e-16 quirks the fidelity check surfaces (frames/planes re-normalize axes — JSON too; rotation round-trips through axis+angle rather than the exact matrix). The report's executive summary gains a coverage banner (31/31 serializable types benchmarked), computed from compas_pb's SerializerRegistry vs the corpus fixture types. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Re-runs the 32-subject baseline after compas_pb's Rotation-as-matrix fix: `rotations` is now lossless, leaving only `frames`/`planes` (the ~1e-16 axis re-normalization quirk that affects JSON too). Drops the resolved Rotation item from the pending list. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
One-off spot check (pointcloud at 5e7 points, size+time only, no tracemalloc): wire 2.42x smaller, dump 8.6x faster, load 1.21x faster than JSON, no OOM (RSS ~9 GB). The wire/load ratios are flat vs the 1e6 baseline, confirming 1e6 was sufficient for N1. Reframes the PRD-scale item as a productization stress test (only peak-memory-at-scale remains, needing an RSS probe). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Manual (workflow_dispatch) job that installs compas + the optional formats (msgspec, zstandard) + the optimized compas_pb from a configurable git ref (default the benchmark/double-precision branch; its generated _pb2 is committed, so no protoc needed), runs the benchmark at the chosen preset, and uploads results/ (CSV + HTML report + samples) as a downloadable artifact. Manual-only because runner noise makes timings unreliable on push. Both repos are public, so no auth is required. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds push (exploration/serialization) and pull_request triggers alongside workflow_dispatch.
Automatic runs use the quick preset via `${{ inputs.x || 'default' }}` fallbacks (inputs are
empty outside workflow_dispatch). A concurrency group cancels superseded runs so a push and
its PR event don't duplicate work.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1525 +/- ##
==========================================
+ Coverage 62.38% 62.40% +0.02%
==========================================
Files 208 208
Lines 22485 22498 +13
==========================================
+ Hits 14027 14040 +13
Misses 8458 8458 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
|
Yay, please check also the RobotModel and Trajectory classes and make their serialization fast too. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is an on-going effort to benchmark a few different serialization options. The baseline is COMPAS' json serialization, and on top of that, I added
compas_pbserialization. I've been trying to addmsgspecserialization based on the implementations that the Kumiki project has, but it's not quite there yet (the current implementation is a bit slower and bigger than compas_pb, but I assume it's because it is relying on the json shape dict)What type of change is this?
Checklist
Put an
xin the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.CHANGELOG.mdfile in theUnreleasedsection under the most fitting heading (e.g.Added,Changed,Removed).invoke test).invoke lint).compas.datastructures.Mesh.