DRIVER-153: negotiate and implement SCYLLA_USE_METADATA_ID extension by nikagra · Pull Request #770 · scylladb/python-driver

nikagra · 2026-03-26T19:06:42Z

Summary

Implements the SCYLLA_USE_METADATA_ID Scylla CQL protocol extension (DRIVER-153), which backports the prepared-statement metadata-ID mechanism from CQL v5 to earlier protocol versions.

When the extension is negotiated:

The server includes a result metadata hash in the PREPARE response
The driver sends that hash back with every EXECUTE request, allowing the server to skip sending full result metadata on every response (skip_meta=True)
If the result schema has changed, the server sets the METADATA_CHANGED flag and includes the new metadata ID + new column metadata in the response — the driver picks this up and updates its cached metadata automatically

Rebased on #934 (merged groundwork that threads each connection's negotiated ProtocolFeatures into message serialization) and reworked to resolve the review discussion below.

Design change from the previous version of this PR

Previously, ExecuteMessage carried a use_metadata_id flag and ResponseFuture._query() mutated skip_meta/result_metadata_id on the shared message after borrowing a connection. Per @Lorak-mmk's review (#770 (comment) and the surrounding thread), that design is gone:

ExecuteMessage is immutable once constructed — skip_meta and result_metadata_id are set from the prepared statement's cached metadata at construction time (in _create_response_future), same as before this extension existed.
send_body decides what actually reaches the wire from (protocol_version, protocol_features) — the connection's negotiated features, supplied by protocol: pass negotiated ProtocolFeatures to message serialization #934's plumbing. The metadata-id field is written whenever the serving connection speaks CQL v5+ or negotiated the extension — always, not conditionally — with an empty b'' sentinel when the statement has no id yet (mixed cluster / rolling upgrade). _SKIP_METADATA_FLAG is only set when that same condition holds, so a statement executed against a connection without the extension never asks the server to skip metadata it has no way to recover.

This fixes two problems the old design had:

Speculative-execution race: the old _query() mutated the one shared ExecuteMessage per connection borrowed; with speculative retries or a mixed-feature cluster, two connections could interleave mutation and encoding of the same message. Immutability removes the race entirely.
Control-connection fallback gap (@dkropachev, DRIVER-153: negotiate and implement SCYLLA_USE_METADATA_ID extension #770 (comment)): _query_control_connection never went through the mutation block and would omit the id field on an extension-enabled control connection. Every send path now serializes correctly with zero extra code, because the decision is made in send_body from the connection actually being used.

Other review threads resolved

Torn metadata/id pair (@Lorak-mmk DRIVER-153: negotiate and implement SCYLLA_USE_METADATA_ID extension #770 (comment), @dkropachev DRIVER-153: negotiate and implement SCYLLA_USE_METADATA_ID extension #770 (comment)): PreparedStatement now stores (result_metadata, result_metadata_id) as one tuple replaced in a single attribute assignment (update_result_metadata()), with result_metadata/result_metadata_id as compatibility properties over it. A reader can no longer observe the metadata of one schema version paired with the id of another — the previous per-field write ordering (and its GIL-dependent correctness comment) is gone. The compatibility setters are documented as non-atomic relative to each other, pointing callers at update_result_metadata().
Behavioral fix beyond what was asked: when the server sends a new metadata id without column metadata (_set_result), the old code adopted the id anyway. That manufactures the exact unrecoverable state the pair-atomicity fix is meant to prevent: the server would then match the id and stop sending metadata, while the driver decodes against stale cached columns forever. Now that response is logged as a warning and nothing is cached — the next EXECUTE resends the old id, the server detects the mismatch, and the driver recovers with full metadata.
Rolling-upgrade docs (@Lorak-mmk DRIVER-153: negotiate and implement SCYLLA_USE_METADATA_ID extension #770 (comment)): reworded to describe the actual self-healing behavior — a statement prepared before the extension was negotiated sends the empty sentinel on its first execute over an extension-enabled connection and acquires an id from the resulting METADATA_CHANGED response; no client restart needed. Also reworded the section's opening paragraph so skip_meta reads as conditional on the extension, not pre-existing default behavior.
Wrong spec link (@Lorak-mmk DRIVER-153: negotiate and implement SCYLLA_USE_METADATA_ID extension #770 (comment)): replaced the opensource-docs CQL-language-extensions link with the scylladb repo's docs/dev/protocol-extensions.md, which actually documents this extension.
bool(result_metadata) question (@Lorak-mmk DRIVER-153: negotiate and implement SCYLLA_USE_METADATA_ID extension #770 (comment)): result_metadata is None for statements with NO_METADATA in their PREPARE response (LWT/conditional statements) and [] for statements returning zero columns (plain INSERT/UPDATE/DELETE). Both are falsy, and both correctly keep skip_meta off — there's nothing to decode against in either case. Documented in a code comment at the construction site.

Changes

cassandra/protocol_features.py

Add USE_METADATA_ID = "SCYLLA_USE_METADATA_ID" constant and use_metadata_id field to ProtocolFeatures
Parse the extension from the SUPPORTED frame; include it in STARTUP when present

cassandra/protocol.py

Bug fix: _SKIP_METADATA_FLAG is now actually written to the wire — it was stored on _QueryMessage but never sent (effectively dead code upstream)
recv_results_prepared: read result_metadata_id for the Scylla extension (pre-v5) in addition to standard CQL v5+
ExecuteMessage.send_body: decides both the metadata-id field and the skip-metadata flag from (protocol_version, protocol_features) at serialization time; writes the b'' sentinel when the statement has no id

cassandra/query.py

PreparedStatement.result_metadata/result_metadata_id become properties over a single (result_metadata, result_metadata_id) tuple; add update_result_metadata() for atomic pair replacement

cassandra/cluster.py

_create_response_future: snapshot the metadata pair once, construct ExecuteMessage immutably with skip_meta/result_metadata_id set from that snapshot
_set_result: on METADATA_CHANGED, replace the cached pair atomically; ignore (with a warning) a response that carries a new id without column metadata
_execute_after_prepare: same atomic update on reprepare, keeping the previous id when the response carries none

docs/scylla-specific.rst

Document the extension and its behaviour; corrected rolling-upgrade description and spec reference link

CHANGELOG.rst

Added a Features entry for the extension

Live-server verification of the `b''` sentinel

The one open question from review was whether Scylla actually treats the empty b'' metadata-id sentinel — sent by a statement that was prepared before the extension was negotiated, e.g. mid rolling-upgrade — as a mismatch, versus rejecting it as a malformed frame. Resolved two ways:

Cross-driver precedent. The identical scenario is already covered by a merged, live-server-tested case in the Java driver: scylladb/java-driver#758 adds PreparedStatementIT.should_handle_empty_metadata_id_when_executing_statement_when_supported, which nulls a prepared statement's cached id, executes, and asserts the id comes back non-null against a real Scylla node. The gocql implementation (scylladb/gocql#590) independently reaches the identical wire convention — its Test_framer_writeExecuteFrame unit test asserts a nil id serializes to a zero-length short-bytes field, the same convention as this PR's b''. All three drivers trace back to the same server-side fix (scylladb/scylladb#23292).

Live run against this driver. Added tests/integration/standard/test_scylla_metadata_id.py (3 tests) and ran it against a real Scylla node via CCM:

source ~/Envs/scylla/bin/activate   # venv with ccm + a Scylla build already cached locally, no download needed
cd python-driver
SCYLLA_VERSION=release:2026.1.9 PROTOCOL_VERSION=4 pytest tests/integration/standard/test_scylla_metadata_id.py -v

tests/integration/standard/test_scylla_metadata_id.py::ScyllaMetadataIdTests::test_empty_sentinel_id_triggers_metadata_changed PASSED [ 33%]
tests/integration/standard/test_scylla_metadata_id.py::ScyllaMetadataIdTests::test_extension_is_negotiated PASSED [ 66%]
tests/integration/standard/test_scylla_metadata_id.py::ScyllaMetadataIdTests::test_metadata_changed_recovers_after_schema_change PASSED [100%]
======================== 3 passed, 3 warnings in 25.09s ========================

test_empty_sentinel_id_triggers_metadata_changed is the direct check: prepare a statement, force its result_metadata_id back to None via update_result_metadata() (simulating a statement prepared before the extension was known), execute it, and confirm both that no error is raised and that a fresh result_metadata_id comes back afterward — proving Scylla treats the sentinel as METADATA_CHANGED, not a protocol error.

Test plan

Unit tests across test_protocol_features.py, test_protocol.py, test_query.py, test_response_future.py covering: feature negotiation, STARTUP options, wire-format assertions for the metadata-id field and skip-metadata flag (present/suppressed, sentinel for None, v4/v5), PREPARE response decoding with/without the extension, atomic metadata-pair replacement (constructor, update_result_metadata, compatibility setters), _create_response_future construction gating (id present/absent, LWT/NO_METADATA, zero-column), a regression proving _query sends the message unmutated (the old speculative-execution race), _set_result METADATA_CHANGED and the anomalous-response warning path (nothing cached), and _execute_after_prepare reprepare handling
Full unit test suite passes (693 passed, 111 skipped)
Cython build (build_ext --inplace) compiles the reworked signatures; tests pass against the compiled modules
Integration tests against a Scylla node with the extension: verified live via CCM (Scylla 2026.1.9) that schema changes after PREPARE are detected and metadata is updated without re-preparation, and that the empty-id sentinel is accepted as a mismatch, not rejected — see "Live-server verification" above

Pre-review checklist

I have split my patch into logically separate commits.
All commit messages clearly explain what they change and why.
I added relevant tests for new features and bug fixes.
All commits compile, pass static checks and pass test.
PR description sums up the changes and reasons why they should be introduced.
I have provided docstrings for the public items that I want to introduce.
I have adjusted the documentation in ./docs/.
I added appropriate Fixes: annotations to PR description.

Copilot

Pull request overview

Implements negotiation and support for Scylla’s SCYLLA_USE_METADATA_ID protocol extension to enable metadata-id based skip_meta behavior (backporting CQL v5 prepared-statement metadata-id semantics to earlier protocol versions).

Changes:

Adds SCYLLA_USE_METADATA_ID parsing from SUPPORTED and includes it in STARTUP when negotiated.
Extends protocol encode/decode to read/write result_metadata_id for PREPARE/EXECUTE on pre-v5 when the extension is used, and fixes on-wire encoding of _SKIP_METADATA_FLAG.
Updates execution/result handling to conditionally use skip_meta and to refresh cached prepared metadata when the server reports metadata changes.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
`cassandra/protocol_features.py`	Adds the `SCYLLA_USE_METADATA_ID` feature flag and includes it in negotiated STARTUP options.
`cassandra/protocol.py`	Writes `_SKIP_METADATA_FLAG` in query params; adds pre-v5 extension handling for `result_metadata_id` in PREPARE/EXECUTE.
`cassandra/cluster.py`	Adjusts when `skip_meta` is enabled and updates cached prepared metadata/id on `METADATA_CHANGED` responses.
`tests/unit/test_protocol_features.py`	Adds unit tests for feature parsing and STARTUP option inclusion.
`tests/unit/test_protocol.py`	Adds unit tests for skip-meta flag encoding and metadata-id handling in pre-v5 PREPARE/EXECUTE paths.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

mykaul · 2026-03-29T16:42:19Z

I'm not sure where, but we should document this - with reference mainly to the scylladb docs about this feature.

nikagra · 2026-03-30T17:00:35Z

@mykaul Documentation I'm aware of is MetadataId extension in CQLv4 Requirement Document

dkropachev

One blocking correctness issue below: skip_meta is being enabled for prepared statements that can still have empty/absent cached result metadata.

coderabbitai · 2026-05-28T20:32:14Z

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

@coderabbitai resume to resume automatic reviews.
@coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

▶️ Resume reviews
🔍 Trigger review

📝 Walkthrough

Walkthrough

This PR adds SCYLLA_USE_METADATA_ID negotiation, protocol metadata-id serialization and parsing, atomic prepared-statement metadata caching, and cache refresh handling for changed schemas. Execute messages conditionally send skip_meta and result_metadata_id; responses update cached metadata when column metadata is returned. Documentation and unit tests cover protocol versions, feature negotiation, reprepare behavior, cache consistency, and warning paths.

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant ResponseFuture
  participant Server
  participant PreparedStatement
  Client->>ResponseFuture: execute prepared statement
  ResponseFuture->>PreparedStatement: snapshot metadata and id
  ResponseFuture->>Server: send ExecuteMessage
  Server-->>ResponseFuture: return result metadata id and optional columns
  ResponseFuture->>PreparedStatement: update cached metadata pair
  ResponseFuture-->>Client: return rows

Possibly related PRs

scylladb/python-driver#913: Changes protocol message encoding based on negotiated protocol features in cassandra/protocol.py.

Suggested labels: area/Driver_-_python-driver

Suggested reviewers: dkropachev, sylwiaszunejko

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 59.26% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly names the main change: negotiating and implementing the SCYLLA_USE_METADATA_ID extension.
Description check	✅ Passed	The description closely matches the template and covers summary, tests, docs, and checklist items, with only the Fixes annotation left unchecked.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai · 2026-05-28T20:36:00Z

Actionable comments posted: 0

nikagra · 2026-06-02T20:42:42Z

🤖: All review issues (1–9) have been addressed and the branch has been squashed to the required 2-commit shape. Requesting re-review from @Lorak-mmk and @dkropachev.

What changed since last review:

Production commit (8accdb5a) — no functional changes, two cleanups squashed in:

Issue 8 (docs scope claim): corrected docs/scylla-specific.rst to say the extension applies to EXECUTE requests, not PREPARE
Issue 9 (GIL comment): added a note in _set_result explaining why the metadata update write ordering is safe under the GIL

Test commit (bfc97602) — all 6 fix/addition commits squashed in, commit message fully replaced to enumerate every test:

Issue 1: test_query_no_skip_meta_without_extension fixture corrected (result_metadata=[] was falsy, defeating the assertion)
Issue 2: test_execute_after_prepare_updates_result_metadata_id and test_execute_after_prepare_no_metadata_id_in_response added to cover the _execute_after_prepare reprepare path
Issue 3: test_recv_results_metadata_no_metadata_flag_skips_metadata_id tightened — now asserts not hasattr(result, 'result_metadata_id') rather than is None, and checks column_metadata not result_metadata
Issue 4: test_recv_results_prepared_v5_reads_metadata_id added — covers the v5 native uses_prepared_metadata() decode path
Issue 5: test_execute_message_v5_skip_meta_sets_flag added — confirms _SKIP_METADATA_FLAG is correctly written into the 4-byte v5 flags word (this flag was dead code in upstream before this PR)
Issue 6: test_repeat_orig_query_after_succesful_reprepare fixed — result_metadata_id value changed from str to bytes; assertion that value is stored on prepared_statement added
Issue 7: test_set_result_warns_when_metadata_id_but_column_metadata_is_none added — covers the column_metadata=None (absent) variant of the METADATA_CHANGED warning path

CI on the 8-commit pre-squash branch: 18/19 passed; test libev (3.14t) was cancelled after a 6-hour runner stall (infrastructure timeout — test asyncio (3.14t) on the same commit passed cleanly).

Connection.send_msg now passes the connection's negotiated ProtocolFeatures to the encoder; _ProtocolHandler.encode_message accepts it as a new required protocol_features argument (passed by keyword from send_msg) and forwards it to every message's send_body, which gains the same parameter. Messages carry connection-independent request data; send_body decides the wire format from (protocol_version, protocol_features), so fields belonging to a negotiated protocol extension are emitted exactly on the connections that negotiated it — on every send path, including the control-connection fallback, and without mutating shared message objects per attempt. This is pure plumbing: no message consumes the parameter yet, so no bytes on the wire change. It is groundwork for the SCYLLA_USE_METADATA_ID (scylladb#770) and TABLETS_ROUTING_V2 (scylladb#913) extensions, which must serialize extension fields based on what the serving connection negotiated. The encode side becomes symmetric with decode_message, which already receives protocol_features. This changes the contracted signature of encode_message: custom protocol handlers overriding it must accept the protocol_features keyword argument. The argument is deliberately required, with no default and no fallback for old-style encoders: extensions are negotiated per connection at STARTUP before the per-request handler is known, so an encoder unaware of protocol_features could silently omit fields a negotiated extension requires; omitting it fails fast with TypeError instead. Tests: send_msg hands the connection's features to the encoder; encode_message forwards them into send_body (plain and compressed paths) and raises TypeError when the argument is omitted; a byte-identity suite pins frames for representative messages (v3/v4/v5) to the exact bytes produced before this change, both without features and with all-default features. Co-authored-by: Dawid Mędrek <dawid.medrek@scylladb.com>

Connection.send_msg now passes the connection's negotiated ProtocolFeatures to the encoder; _ProtocolHandler.encode_message accepts it as a new required protocol_features argument (passed by keyword from send_msg) and forwards it to every message's send_body, which gains the same parameter. Messages carry connection-independent request data; send_body decides the wire format from (protocol_version, protocol_features), so fields belonging to a negotiated protocol extension are emitted exactly on the connections that negotiated it — on every send path, including the control-connection fallback, and without mutating shared message objects per attempt. This is pure plumbing: no message consumes the parameter yet, so no bytes on the wire change. It is groundwork for the SCYLLA_USE_METADATA_ID (#770) and TABLETS_ROUTING_V2 (#913) extensions, which must serialize extension fields based on what the serving connection negotiated. The encode side becomes symmetric with decode_message, which already receives protocol_features. This changes the contracted signature of encode_message: custom protocol handlers overriding it must accept the protocol_features keyword argument. The argument is deliberately required, with no default and no fallback for old-style encoders: extensions are negotiated per connection at STARTUP before the per-request handler is known, so an encoder unaware of protocol_features could silently omit fields a negotiated extension requires; omitting it fails fast with TypeError instead. Tests: send_msg hands the connection's features to the encoder; encode_message forwards them into send_body (plain and compressed paths) and raises TypeError when the argument is omitted; a byte-identity suite pins frames for representative messages (v3/v4/v5) to the exact bytes produced before this change, both without features and with all-default features. Co-authored-by: Dawid Mędrek <dawid.medrek@scylladb.com>

coderabbitai

Actionable comments posted: 1

🧹 Nitpick comments (1)

tests/unit/test_protocol.py (1)
473-489: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Consider ClassVar annotation for the read-only frame-hex lookup table.

Ruff RUF012 flags EXPECTED_FRAMES as a mutable class attribute. It's only read here, so risk is low, but annotating with typing.ClassVar[dict] (or moving it to module scope) documents the intent and silences the lint.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/unit/test_protocol.py` around lines 473 - 489, Annotate the read-only
EXPECTED_FRAMES class attribute with typing.ClassVar[dict] so Ruff RUF012
recognizes it as intentional class-level state. Preserve the existing frame
mappings and lookup behavior.
Source: Linters/SAST tools

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/unit/test_protocol.py`:
- Around line 260-282: Add a standalone test method definition before the
`_METADATA_ID_FLAG` ROWS-decoding test body, using the surrounding test class’s
naming and `self` conventions. Keep its existing docstring, buffer setup,
`recv_results_metadata` call, and assertions inside the new method so pytest
discovers it independently from
`test_recv_results_prepared_v5_reads_metadata_id`.

---

Nitpick comments:
In `@tests/unit/test_protocol.py`:
- Around line 473-489: Annotate the read-only EXPECTED_FRAMES class attribute
with typing.ClassVar[dict] so Ruff RUF012 recognizes it as intentional
class-level state. Preserve the existing frame mappings and lookup behavior.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: edb6a82d-1220-476f-9fe6-b25c6d7b238e

📥 Commits

Reviewing files that changed from the base of the PR and between d3300e2 and 8aecb92.

📒 Files selected for processing (9)

cassandra/cluster.py
cassandra/protocol.py
cassandra/protocol_features.py
cassandra/query.py
docs/scylla-specific.rst
tests/unit/test_protocol.py
tests/unit/test_protocol_features.py
tests/unit/test_query.py
tests/unit/test_response_future.py

🚧 Files skipped from review as they are similar to previous changes (3)

tests/unit/test_protocol_features.py
cassandra/protocol_features.py
docs/scylla-specific.rst

Copilot

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

Copilot

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.

Copilot

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

@dkropachev

Implement the SCYLLA_USE_METADATA_ID protocol extension, which backports the CQL v5 prepared-statement metadata-id mechanism to earlier protocol versions. When negotiated, the server includes a hash of the result metadata in the PREPARE response; the driver sends it back with every EXECUTE, allowing the server to omit result metadata from responses (skip_meta) and to report schema changes with METADATA_CHANGED plus fresh metadata, which the driver adopts automatically. protocol_features.py: parse the extension from SUPPORTED, echo it in STARTUP, expose it as ProtocolFeatures.use_metadata_id. protocol.py: ExecuteMessage carries connection-independent request data (skip_meta, result_metadata_id) fixed at construction; serialization decides the wire format from the (protocol_version, protocol_features) that Connection.send_msg supplies for the serving connection: - The metadata-id field is written iff the connection speaks CQL v5+ or negotiated the extension - always, on such connections. An empty sentinel (b'') is written when the statement has no id (prepared before the extension was active, e.g. during a rolling upgrade, or an LWT statement): the sentinel mismatch makes the server respond with METADATA_CHANGED plus the current id and metadata, so such statements acquire an id on their first execution. This also fixes a TypeError on v5 when result_metadata_id was None. - _SKIP_METADATA_FLAG is written only when the SCYLLA_USE_METADATA_ID extension is negotiated on the connection; without the metadata-id mechanism a schema change after PREPARE would leave the driver decoding rows with stale cached metadata. This is deliberately narrower than the metadata-id field above: on native CQL v5 the field is part of the frame layout, but the driver does not request skip there. Upstream never emitted _SKIP_METADATA_FLAG on any version (_write_query_params never wrote it), and enabling the skip optimization for native v5 is a separate change kept out of scope for this Scylla extension. Because messages are immutable after construction, every send path is correct without per-path setup - including the control-connection fallback - and concurrent sends of the same message (speculative executions) cannot race on per-connection state. query.py: PreparedStatement stores (result_metadata, result_metadata_id) as one tuple replaced in a single attribute assignment, read through compatibility properties and updated via update_result_metadata(). Response callbacks update statements while request threads read them; a torn pair (fresh id + stale metadata) would make the server skip sending metadata while rows are decoded against the wrong columns, with no recovery. The compatibility setters are documented as non-atomic relative to each other - update_result_metadata() is the atomic path; the setters exist only for callers assigning the old individual attributes. cluster.py: _create_response_future snapshots the pair once and requests skip_meta only when the statement has both an id and usable cached metadata (result_metadata is None for NO_METADATA/LWT statements and [] for zero-column statements; neither can nor needs to skip metadata). The same snapshot is handed to the ResponseFuture, so a skip_meta response is decoded against the metadata that pairs with the id the message sent - not a later re-read of the statement cache, which a concurrent METADATA_CHANGED could have replaced between construction and send (and which also keeps speculative sends of one message internally consistent). _set_result adopts a METADATA_CHANGED response by replacing the pair atomically; a response carrying a new id without column metadata is ignored with a warning, since adopting the id alone would create the unrecoverable stale-decode state. skip_meta additionally stays off for continuous paging (@dkropachev): Connection.process_msg hardcodes result_metadata=None for every page after the first, since it isn't threaded through the paging session - a skip_meta response has nothing to decode page 2+ against, and would crash on it. _execute_after_prepare refreshes the pair from exactly what the reprepare response carries, including the id (@dkropachev): falling back to the previously cached id when the response has none risks pairing it with metadata from a different schema version than the one that id was computed for - e.g. if the schema changed and then reverted between the two PREPAREs, the old id can become valid again for the current schema while paired locally with an intermediate version's metadata, with no server-side mismatch to catch it. Dropping it instead lets the next id-aware execute re-acquire a correctly paired id through the same b'' sentinel self-healing path a never-prepared statement uses. docs/scylla-specific.rst: documents the extension and its behaviour, worded so the skip_meta optimization reads as conditional on the extension being negotiated rather than pre-existing default behaviour. CHANGELOG.rst: add a Features entry for the extension.

@dkropachev

Unit tests for the extension across its layers: test_protocol_features.py: SCYLLA_USE_METADATA_ID parsed from SUPPORTED and echoed in STARTUP options; absent by default. test_protocol.py (wire format): - metadata-id field written on v4 iff the connection negotiated the extension, with the exact bytes asserted; empty sentinel (b'') when the statement has no id, on both the extension path (v4) and the v5 native path (previously a TypeError); - _SKIP_METADATA_FLAG written when skip_meta is requested and the SCYLLA_USE_METADATA_ID extension is negotiated (v4 or v5), and NOT set on a native v5 connection without the extension (the id field is still written there, but the driver does not request skip); also suppressed - together with the id field - on a v4 connection without the extension, even when the statement carries an id; - PREPARED response decoding reads result_metadata_id iff the extension was negotiated (or v5); METADATA_CHANGED/NO_METADATA flag handling. test_query.py: PreparedStatement stores the (result_metadata, result_metadata_id) pair atomically - constructor, update_result_metadata, and the backwards-compatible single-attribute setters all replace the pair as one unit, and previously-taken snapshots stay internally consistent. test_response_future.py: - _create_response_future builds ExecuteMessage from a single pair snapshot: skip_meta only with both an id and usable cached metadata; disabled for id-less statements, NO_METADATA/LWT statements (result_metadata None) and zero-column statements (result_metadata []), while the id still rides on the message; - _query sends the message exactly as constructed (no per-connection mutation - regression test for the speculative-execution race) and decodes a skip_meta response against the metadata snapshotted when the message was built, not a later read of the statement cache (regression for a concurrent METADATA_CHANGED racing the send); - _set_result METADATA_CHANGED path replaces the cached pair atomically; a response with a new id but no column metadata (empty or absent) is ignored with a warning, leaving the cached pair unchanged - adopting the id alone would poison the cache with a stale-metadata/current-id pair the server would never refresh; - _execute_after_prepare refreshes the pair from exactly what the reprepare response carries, including the id, and no longer keeps the previous id when the response has none (@dkropachev: doing so risked pairing a stale id with metadata from a different schema version - test_execute_after_prepare_no_metadata_id_in_response_clears_id); - a statement with valid cached metadata+id must still get skip_meta=False when continuous_paging_options is set (@dkropachev: Connection.process_msg hardcodes result_metadata=None for paging-session pages after the first, so a skip_meta response would crash decoding them - test_create_execute_message_continuous_paging_disables_skip_meta). tests/integration/standard/test_scylla_metadata_id.py: live-server coverage against a real Scylla node via CCM, closing the one gap unit tests can't - whether Scylla actually treats the empty result_metadata_id sentinel as a mismatch rather than a protocol error. Confirms extension negotiation, the normal METADATA_CHANGED-after-ALTER-TABLE path, and the sentinel round trip: a statement forced back to result_metadata_id=None (simulating one prepared before the extension was known, e.g. mid rolling-upgrade) executes without error and comes back with a fresh id. Mirrors the equivalent live test already merged in the Java driver (scylladb/java-driver#758, should_handle_empty_metadata_id_when_executing_statement_when_supported). Run locally against Scylla 2026.1.9 via CCM; see PR description for setup and log excerpt.

nikagra requested review from Copilot and dkropachev March 26, 2026 19:08

Copilot started reviewing on behalf of nikagra March 26, 2026 19:08 View session

Copilot AI reviewed Mar 26, 2026

View reviewed changes

Comment thread cassandra/cluster.py Outdated

Comment thread cassandra/protocol.py Outdated

nikagra requested a review from Copilot March 27, 2026 12:03

Copilot started reviewing on behalf of nikagra March 27, 2026 12:03 View session

Copilot AI reviewed Mar 27, 2026

View reviewed changes

Comment thread cassandra/cluster.py Outdated

nikagra force-pushed the driver-153-scylla-use-metadata-id branch from ade35d8 to f42e225 Compare March 27, 2026 12:32

andrzej-jackowski-scylladb mentioned this pull request Apr 2, 2026

Add support for SCYLLA_USE_METADATA_ID and skip metadata #457

Closed

8 tasks

nikagra requested a review from sylwiaszunejko April 9, 2026 11:12

nikagra marked this pull request as ready for review April 9, 2026 21:30

dkropachev requested changes Apr 14, 2026

View reviewed changes

Comment thread cassandra/cluster.py Outdated

nikagra force-pushed the driver-153-scylla-use-metadata-id branch from 6eea397 to a86fd53 Compare April 15, 2026 09:09

nikagra requested a review from dkropachev April 15, 2026 09:12

nikagra force-pushed the driver-153-scylla-use-metadata-id branch from 7ba5835 to a86fd53 Compare April 15, 2026 11:12

nikagra mentioned this pull request Apr 15, 2026

CI: fix id-token permission for "Test wheels building" #820

Merged

nikagra force-pushed the driver-153-scylla-use-metadata-id branch from a86fd53 to 8880f03 Compare April 22, 2026 12:34

nikagra force-pushed the driver-153-scylla-use-metadata-id branch from 170fd31 to 5fe1902 Compare May 14, 2026 14:45

nikagra requested a review from Lorak-mmk May 14, 2026 14:45

nikagra force-pushed the driver-153-scylla-use-metadata-id branch from fcd3eba to 5fe1902 Compare May 15, 2026 09:22

nikagra force-pushed the driver-153-scylla-use-metadata-id branch from 5fe1902 to 251b1a8 Compare May 28, 2026 20:32

nikagra force-pushed the driver-153-scylla-use-metadata-id branch 2 times, most recently from de8d3fc to bfc9760 Compare June 2, 2026 20:42

Lorak-mmk force-pushed the master branch from f2a9e87 to 763af09 Compare June 15, 2026 10:57

nikagra force-pushed the driver-153-scylla-use-metadata-id branch from bfc9760 to d3300e2 Compare June 19, 2026 08:54

nikagra mentioned this pull request Jul 14, 2026

protocol: pass negotiated ProtocolFeatures to message serialization #934

Merged

12 tasks

nikagra force-pushed the driver-153-scylla-use-metadata-id branch from d3300e2 to 8aecb92 Compare July 20, 2026 13:54

nikagra marked this pull request as draft July 20, 2026 13:55

coderabbitai Bot requested a review from dkropachev July 20, 2026 13:55

coderabbitai Bot added the area/Driver_-_python-driver label Jul 20, 2026

coderabbitai Bot reviewed Jul 20, 2026

View reviewed changes

Comment thread tests/unit/test_protocol.py

nikagra force-pushed the driver-153-scylla-use-metadata-id branch from 8aecb92 to a620293 Compare July 20, 2026 20:47

nikagra requested a review from Lorak-mmk July 20, 2026 20:56

nikagra force-pushed the driver-153-scylla-use-metadata-id branch from a620293 to e636095 Compare July 20, 2026 21:50

nikagra marked this pull request as ready for review July 20, 2026 21:52

Copilot AI review requested due to automatic review settings July 21, 2026 12:00

nikagra force-pushed the driver-153-scylla-use-metadata-id branch from e636095 to 7208531 Compare July 21, 2026 12:00

Copilot started reviewing on behalf of nikagra July 21, 2026 12:00 View session

Copilot AI reviewed Jul 21, 2026

View reviewed changes

Comment thread tests/unit/test_protocol.py Outdated

Copilot AI review requested due to automatic review settings July 21, 2026 12:50

nikagra force-pushed the driver-153-scylla-use-metadata-id branch from 7208531 to cea992c Compare July 21, 2026 12:50

Copilot started reviewing on behalf of nikagra July 21, 2026 12:51 View session

Copilot AI reviewed Jul 21, 2026

View reviewed changes

Comment thread tests/unit/test_protocol.py Outdated

Comment thread cassandra/cluster.py Outdated

Copilot AI review requested due to automatic review settings July 21, 2026 13:26

nikagra force-pushed the driver-153-scylla-use-metadata-id branch from cea992c to 2295d2d Compare July 21, 2026 13:26

Copilot started reviewing on behalf of nikagra July 21, 2026 13:26 View session

Copilot AI reviewed Jul 21, 2026

View reviewed changes

dkropachev reviewed Jul 21, 2026

View reviewed changes

Comment thread cassandra/cluster.py Outdated

Comment thread cassandra/cluster.py Outdated

nikagra force-pushed the driver-153-scylla-use-metadata-id branch from 2295d2d to 7a49fbc Compare July 22, 2026 22:09

nikagra added 2 commits July 23, 2026 00:30

nikagra force-pushed the driver-153-scylla-use-metadata-id branch from 7a49fbc to 4292f0d Compare July 22, 2026 22:34

Uh oh!

Conversation

nikagra commented Mar 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Design change from the previous version of this PR

Other review threads resolved

Changes

Live-server verification of the b'' sentinel

Test plan

Pre-review checklist

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

mykaul commented Mar 29, 2026

Uh oh!

nikagra commented Mar 30, 2026

Uh oh!

dkropachev left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

coderabbitai Bot commented May 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reviews paused

Walkthrough

Sequence Diagram(s)

❌ Failed checks (1 warning)

Uh oh!

coderabbitai Bot commented May 28, 2026

Uh oh!

nikagra commented Jun 2, 2026

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

nikagra commented Mar 26, 2026 •

edited

Loading

Live-server verification of the `b''` sentinel

coderabbitai Bot commented May 28, 2026 •

edited

Loading