Merge instrumentation declarative config with pyprotobuf HTTP exporter (no-requests)#9
Draft
ocelotl wants to merge 67 commits into
Draft
Merge instrumentation declarative config with pyprotobuf HTTP exporter (no-requests)#9ocelotl wants to merge 67 commits into
ocelotl wants to merge 67 commits into
Conversation
Hand-written message classes with SerializeToString() backed by python-protobuf wire primitives instead of google.protobuf. Covers common, resource, trace, logs, metrics, and collector service request types — a full equivalent of opentelemetry-proto. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Mirrors opentelemetry-exporter-otlp-proto-common but imports from opentelemetry.pyproto.* instead of google.protobuf-backed stubs. Covers trace, metrics (with OTLPMetricExporterMixin), and log encoding. Metrics encoding is restructured to use constructive object building instead of protobuf in-place mutation. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…google.protobuf) Mirrors opentelemetry-exporter-otlp-proto-http but uses pyproto-common encoding. Includes OTLPSpanExporter, OTLPLogExporter, OTLPMetricExporter with full retry logic, compression support, and metrics batch splitting. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…package Embeds the python-protobuf wire encoding primitives (varint, tag, scalars, enum) as opentelemetry.pyproto._pyproto, a private subpackage that cannot be discovered or depended upon by end-user code on PYTHONPATH. Removes the python-protobuf external dependency from pyproject.toml and updates all internal imports (_wire.py, common_pyproto2.py, metrics_pyproto2.py) to use the new location. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
One test file per module:
- test__varint.py — boundary values, multi-byte encoding, negative guard
- test__tag.py — formula verification, hand-computed literals for all wire types
- test__scalars.py — hand-computed literals + struct oracle for fixed-width types,
ZigZag arithmetic oracle for sint32/sint64, UTF-8 byte-count
correctness for encode_string, verbatim pass-through for encode_bytes
- test__enum.py — non-negative and negative values, identity with encode_int
Also fixes _varint.py: adds ValueError guard for negative inputs (without it,
the encoding loop would be infinite for any negative value in Python).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Source files (_varint, _tag, _scalars, _enum):
- Add module docstrings explaining wire types, bit layout, and examples
- Add function docstrings with worked examples, valid ranges, and spec refs
- Add line-by-line inline comments explaining each step of the encoding logic
- Change 'import struct' to 'from struct import pack' and update all call sites
- Align _scalars implementation with documented behavior: explicit ZigZag
masking steps in encode_sint32/sint64, encode_int using named variable,
encode_uint32/uint64 not masking (relying on encode_varint's guard)
Test files:
- Change 'import math' to 'from math import e, inf, nan, pi, tau'
- Change 'from struct import pack, unpack' everywhere
- Move all inline function-body imports to module level
- Remove unused 'raises' import from test__scalars.py
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace every `import x` with `from x import y` in all three pyproto packages (opentelemetry-pyproto, pyproto-common, pyproto-http). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Documents the from-import rule: always use `from x import y`, never bare `import x`. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add a full module docstring explaining: - why the module exists (proto3 field layer above the _pyproto kernel) - why it is named _wire (protobuf wire format) - why it lives outside _pyproto (layering: kernel must not know proto3 semantics) - the four wire-type constants and which proto3 types use each Expand every function docstring to explain the proto3 default-omission rule, the on-wire layout, which OTel schema fields use it, and the distinction between _dbl (omit on 0.0) and _opt_dbl (omit only on None). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Rename _pyproto/ to _pyprotobuf/ - Remove underscores from all module filenames inside the package: _varint.py → varint.py, _tag.py → tag.py, _scalars.py → scalars.py, _enum.py → enum.py - Move _wire.py into _pyprotobuf/ and rename it to fields.py - Update __init__.py to use relative imports from the new filenames - fields.py uses relative imports from siblings (scalars, tag, varint) - Update all message files: _pyproto → _pyprotobuf, _wire → _pyprotobuf.fields - Rename test files: test__X.py → test_X.py and update their imports - Update fields.py docstring to reflect new name and location Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Function names: msg, string, byt, u64, bool_field, fix32, fix64, dbl, opt_dbl, sint32, packed_uint64, packed_fix64, packed_double. Constants: WT_VARINT, WT_64BIT, WT_LEN, WT_32BIT. _str renamed to string to avoid shadowing the str builtin. All callers in message files updated accordingly. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Tests for varint, tag, scalars, and enum are now under tests/_pyprotobuf/ to mirror the source package structure. New test_fields.py covers all 13 helpers and 4 wire-type constants in _pyprotobuf/fields.py (385 tests total pass). Assisted-by: Claude Sonnet 4.6
The _pypb2 suffix mirrors _pb2 directly — py + pb2 — making the parallel with the standard protobuf-generated stack immediately legible. Updated all import sites in the exporter packages. Assisted-by: Claude Sonnet 4.6
Require a separate commit after every discrete change, keeping each commit self-contained and the repository in a working state. Assisted-by: Claude Sonnet 4.6
The rule belongs in opentelemetry-pyproto/AGENTS.md instead, not in the upstream project's guidance file. Assisted-by: Claude Sonnet 4.6
Moves the import convention into the standard AGENTS.md location and adds the rule to commit after every discrete change. Assisted-by: Claude Sonnet 4.6
Used for oracle tests that verify our pure-Python encoding produces byte-for-byte identical output to google.protobuf. Assisted-by: Claude Sonnet 4.6
Each existing test file gains a section that builds a dynamic proto2
message class and asserts byte-for-byte equality between our encoder
and google.protobuf's SerializeToString() output:
test_varint.py — encode_varint against a uint64 field
test_tag.py — encode_tag prefix across all four wire types and
field numbers spanning 1-byte to 4-byte tag varints
test_enum.py — encode_enum against a Color enum field (0/1/2)
test_scalars.py — all 14 scalar encoders, one proto2 field each
503 tests total pass.
Assisted-by: Claude Sonnet 4.6
…es in workspace Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…lemetry-proto Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…emetry-proto Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…telemetry-proto Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…roto Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…-proto Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…proto Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
flags is proto field 16 but was written after parent_span_id (field 4). Proto3 serializes fields in ascending field number order, so flags must come after status (field 15). The encoder always sets flags to at least 0x100, so every encoded span produced byte-mismatched output vs proto. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Four classes wrote their high-numbered repeated fields first, violating proto3's ascending field number order for deterministic serialization: - Exemplar: filtered_attributes (field 7) written before time (2), span_id (4), trace_id (5), as_int (6) - NumberDataPoint: attributes (field 7) written before time fields (2,3), as_double (4), exemplars (5), as_int (6) - HistogramDataPoint: attributes (field 9) written before all other fields - SummaryDataPoint: attributes (field 7) written before time fields and quantile_values (6) All now emit fields in ascending field number order, producing identical bytes to google.protobuf serialization. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds opentelemetry-proto, opentelemetry-exporter-otlp-proto-common, opentelemetry-test-utils, protobuf, and pytest as dev dependencies so the package can run tests that compare pyproto and proto encoder output. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…trace_path) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…gs_path) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…t_metrics_data) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ed filenames Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Allows the pyproto exporter packages to work with stable PyPI releases (e.g. opentelemetry-api 1.42.x) so the demo docker-compose can install alongside instrumentation packages that require older semantic-conventions. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… exporters - Copy _exporter_metrics.py (no protobuf deps) into pyproto-common - All three exporters: add meter_provider param, create_exporter_metrics(), wrap export loop in self._metrics.export_operation() context manager, record result.error / result.error_attrs on failure paths - Metric exporter: add set_meter_provider() method - Log exporter: add DuplicateFilter to prevent log spam on repeated failures Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add FromString classmethod to ExportTrace/Logs/MetricsServiceResponse in pyproto (required by gRPC stubs as response_deserializer) - Add handwritten gRPC stub modules (*_pypb2_grpc.py) for trace, logs, and metrics collector services in opentelemetry-pyproto - Create new exporter/opentelemetry-exporter-otlp-pyproto-grpc package with: - OTLPExporterMixin (exporter.py): gRPC export core without google.rpc RetryInfo - OTLPSpanExporter, OTLPMetricExporter, OTLPLogExporter backed by pyproto stubs - Full retry/reconnect logic, credentials, compression, internal metrics support - Entry points for traces, metrics, and logs exporters Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
78 tests covering OTLPExporterMixin (channel creation, env vars, retry logic, shutdown, compression) and all three signal exporters (trace, metric, log), including metric batch splitting via max_export_batch_size. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Benchmarks a generic Record proto message (string, uint64, double, bool, bytes, fixed64, fixed32, embedded message, packed repeated uint64 and double) serialised by _pyprotobuf pure-Python helpers vs google.protobuf C extension. Includes a correctness assertion that both produce identical bytes. Run with: uv run pytest tests/_pyprotobuf/test_benchmark.py -v Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds three new benchmark categories to tests/_pyprotobuf/test_benchmark.py: - Per-field-type (11 field types × 2): each _pyprotobuf helper vs the equivalent google.protobuf single-field encode. Includes a correctness parametrize that asserts identical bytes for every field type. - Scaling (string / bytes / packed at 4 B / 256 B / 16 KB and 10 / 100 / 1000 elements): measures how encoding time grows with payload size for the three field types whose cost is proportional to data length. - Varint bit-width (1 / 2 / 3 / 5 bytes): benchmarks encode_varint at representative values for each continuation-byte count — the hottest code path since every tag and every varint-typed value goes through it. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…it benchmarks Three new benchmark categories: 5. Concatenation strategy — test_encode_concat_pyproto vs test_encode_join_pyproto: `+` and b"".join() are statistically indistinguishable (~58 µs mean each). The intermediate-object overhead of `+` is negligible at this message size. 6. All-default fields — test_all_defaults_pyproto vs test_all_defaults_protobuf: Measures the minimum SerializeToString() cost when every helper returns b"". pyproto: ~3.9 µs (9 function calls returning b""). google.protobuf: ~2.7 µs (empty Record() construction + serialize). 7. Construction vs serialization split — test_encode_protobuf_construct, test_encode_protobuf_serialize, and per-field test_field_serialize_only_protobuf: Construction alone: ~19.5 µs. Serialization alone: ~2.1 µs. google.protobuf's cost is almost entirely in construction (~90%). Fair encoding-only comparison: pyproto ~57 µs vs protobuf serialize ~2 µs. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Encodes a Container with 1, 5, and 20 Item sub-messages (each with string, uint64, and double fields). Tests whether per-element msg() wrapping cost compounds linearly. Includes correctness parametrize asserting identical bytes at every N. Results show the gap grows with N: at N=1 pyproto is ~1.5× slower; at N=20 it is ~2.3× slower. Cost scales linearly on both sides, but pyproto's per-element overhead (encode_varint + string.encode + b"".join) accumulates faster than google.protobuf's C-extension per-element path. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…g support Add configure_instrumentation() to activate instrumentors listed under instrumentation/development.python in a declarative config file. Each key is resolved via the opentelemetry_instrumentor entry point group and instrument(**opts) is called with the supplied options. An enabled: false value suppresses a library without raising; unknown entry points and instrumentor failures are logged without stopping the others. Wire configure_instrumentation() into configure_sdk() so the python section is applied alongside providers on the declarative config path. Fixes open-telemetry#5361 Assisted-by: Claude Sonnet 4.6
…ass conversion Schema keys like "otlp_file/development" use "/" as a namespace separator for development/experimental features. Python dataclass field names cannot contain "/" so they use "_" (e.g. "otlp_file_development"). Without normalization, the key falls through to additional_properties and triggers an entry-point lookup for "otlp_file/development" (which doesn't exist), rather than using the dedicated _create_otlp_file_development_span_exporter factory function.
…n no propagators configured When _propagators is empty the for-loop never runs and context remains None. Returning None causes callers (e.g. flask instrumentation) to call context.attach(None), which sets the ContextVar explicitly to None and corrupts all subsequent get_current() calls. Return Context() as a safe default instead.
…g support Add configure_instrumentation() to activate instrumentors listed under instrumentation/development.python in a declarative config file. Each key is resolved via the opentelemetry_instrumentor entry point group and instrument(**opts) is called with the supplied options. An enabled: false value suppresses a library without raising; unknown entry points and instrumentor failures are logged without stopping the others. Wire configure_instrumentation() into configure_sdk() so the python section is applied alongside providers on the declarative config path. Fixes open-telemetry#5361 Assisted-by: Claude Sonnet 4.6
…ass conversion Schema keys like "otlp_file/development" use "/" as a namespace separator for development/experimental features. Python dataclass field names cannot contain "/" so they use "_" (e.g. "otlp_file_development"). Without normalization, the key falls through to additional_properties and triggers an entry-point lookup for "otlp_file/development" (which doesn't exist), rather than using the dedicated _create_otlp_file_development_span_exporter factory function.
If an instrumentor class exposes a config_dataclass attribute, the raw YAML options are run through _dict_to_dataclass before instrument() is called. This reuses the same type-coercion pipeline used for SDK component configuration so instrumentors declare a typed dataclass schema and the SDK handles the rest. Instrumentors without config_dataclass continue to receive options unchanged, preserving backwards compatibility.
…s, expand slash-normalization comment
- Rename walrus variable `v` to `value` (pylint C0103) - Add pylint disable=no-self-use at class level in test file (pylint R6301) - Move `instrumentation` import after `_tracer_provider` in _sdk.py (ruff I001) - Convert `import logging/dataclasses/unittest` to from-imports - Apply ruff format to reformatted lines
- Add inspect.isclass + is_dataclass guard before using the configuration attribute, so non-dataclass values are silently ignored - Skip instrument() when the instrumentor is already active to avoid duplicate-instrumentation warnings when opentelemetry-instrument and declarative config are both in play - Add Instrumentation section to docs/sdk/configuration.rst with instrumentation/development.python YAML example - Add instrumentation/development example to the declarative-config example otel-config.yaml and README
Replace requests.Session-based transport with stdlib urllib.request/ urllib.error across the trace, log and metric HTTP exporters, sharing a small _post/_build_ssl_context helper in _common.py. The credential- provider entry point's requests.Session type check is commented out (rather than removed) since there is no longer a requests.Session type to validate against; the session= constructor param and its wiring are dropped since there is no longer a Session object for it to populate.
…_http_exporter # Conflicts: # opentelemetry-sdk/src/opentelemetry/sdk/_configuration/_conversion.py # opentelemetry-sdk/tests/_configuration/test_instrumentation.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
issue_5361(instrumentation.python declarative config support) withno-requests(pure-Python protobuf implementation + OTLP pyproto HTTP exporter using urllib instead of requests) into one branch.issue_5361andno-requests's ancestor (pyproto) independently implemented the same instrumentation-config feature (both closing Declarative configuration: support for the language-specific declarative instrumentation configuration open-telemetry/opentelemetry-python#5361); keptissue_5361's more complete, review-addressed version (_configuration/instrumentation.py) and dropped the earlier draft (_configuration/_instrumentation.py), fixing the resulting duplicate import in_sdk.py._configuration/_conversion.py.Test plan
configure_sdk/configure_instrumentationimport cleanlyopentelemetry-sdk/tests/_configuration/test_instrumentation.py— 14/14 passCompressionconfirmed pre-existing on standaloneno-requestsbranch (missing dependency on ad-hoc PYTHONPATH, not caused by this merge)