Skip to content

pb/otlp registers the upstream OTLP descriptors, panicking any binary that also links the otel SDK #117

Description

@matthewdevenny

pkg/otlp (added in v1.3.4) is backed by a buf-generated copy of opentelemetry-proto vendored under github.com/code-cargo/cargowall/pb/otlp (proto/buf.gen.otlp.yaml). Generated Go descriptors register themselves into the process-global protoregistry.GlobalFiles keyed by their source path, and managed mode rewrites only go_package — not the file path or the proto package. So pb/otlp/.../common/v1/common.pb.go claims opentelemetry/proto/common/v1/common.proto, the exact key the canonical go.opentelemetry.io/proto/otlp/common/v1 claims.

Any binary that links both panics at package init, before main:

panic: proto: file "opentelemetry/proto/common/v1/common.proto" is already registered
		previously from: "github.com/code-cargo/cargowall/pb/otlp/opentelemetry/proto/common/v1"
		currently from:  "go.opentelemetry.io/proto/otlp/common/v1"

go.opentelemetry.io/proto/otlp/common/v1.file_opentelemetry_proto_common_v1_common_proto_init()
	/go/pkg/mod/go.opentelemetry.io/proto/otlp@v1.11.0/common/v1/common.pb.go:803

Demonstrated, not hypothetical: this took down the CodeCargo app's cargowall init container on a routine v1.3.2 → v1.3.6 bump. The container is an eBPF firewall that gates job egress, so every job in the cluster failed at Init:Error with no network path. v1.3.3 is clean; v1.3.4 is the first affected release.

Why it matters

  • It fails on every invocation, not just when OTLP is configured. cmd/start.go:54 and cmd/api.go:32 import pkg/otlp unconditionally, so the descriptors link regardless of whether otlp.NewFromEnv returns nil. cargowall --version panics.
  • There is no fix available through the OSS API. Registration happens in init() of a transitively imported package, so no flag, env var, or call ordering by the embedder can avoid it. The only lever is a link-time one.
  • It targets the primary embedder. Anyone wrapping cmd.StartCargoWall to add hooks is likely already exporting their own logs and traces via the otel SDK — which is precisely what pulls in go.opentelemetry.io/proto/otlp. Note that the trace exporters are enough on their own: otlptrace pulls common/v1 and resource/v1.

The collision is 64 registrations across all four vendored files:

vendored file also registered by
opentelemetry/proto/common/v1/common.proto otlplog, otlptrace
opentelemetry/proto/resource/v1/resource.proto otlplog, otlptrace
opentelemetry/proto/logs/v1/logs.proto otlplog
opentelemetry/proto/collector/logs/v1/logs_service.proto otlplog

Current downstream workaround

The app builds its cargowall binary with -ldflags "-X google.golang.org/protobuf/reflect/protoregistry.conflictPolicy=ignore". It works — both exporters marshal concrete generated structs, and protoimpl reads the descriptor embedded in the type rather than the global registry — but it is explicitly outside protobuf-go's compatibility promise, it silences any future duplicate registration in that binary, and it leaves by-name lookups (FindMessageByName, anypb, reflection) resolving to whichever copy won the race. warn instead of ignore is not viable in practice: it writes ~320 lines of stderr into every job's init container.

Every embedder has to discover this independently, from a panic in production.

Proposal

option dep cost maintenance resolves
A. Drop pb/otlp, import go.opentelemetry.io/proto/otlp adds grpc + grpc-gateway/v2 none by construction
B. Hand-roll the encoding with protobuf/encoding/protowire none ~250 LOC to own by construction — nothing registers
C. Re-vendor under a private proto package and file path none a renamed fork of 4 upstream .proto files yes, but fork drifts

A is the smallest diff: delete pb/otlp and proto/buf.gen.otlp.yaml, retarget four import lines in pkg/otlp/{exporter,mapper}.go. The cost is real though — go.opentelemetry.io/proto/otlp@v1.11.0 requires google.golang.org/grpc and grpc-ecosystem/grpc-gateway/v2, which roughly doubles the dependency surface of a security tool that currently holds a deliberately tight list.

B preserves the zero-dependency posture that motivated the vendoring in the first place, and is the only option that puts nothing in the global registry at all. The payload is small and stable — ExportLogsServiceRequest → ResourceLogs → ScopeLogs → LogRecord — with AnyValue's recursive oneof as the only fiddly part. mapper.go already does the structural work; this replaces generated marshalling with explicit field emission.

C is wire-safe but not recommended. It leaves four renamed upstream .proto files that have to be re-forked by hand every time opentelemetry-proto adds a field, and buys nothing over B.

For the record, a rename or reimplementation is wire-compatible: the exporter POSTs a proto.Marshaled body as application/x-protobuf (pkg/otlp/exporter.go:192,236), which carries no type names, and none of these messages contain google.protobuf.Any. Confirmed empirically — marshalling a fully populated ExportLogsServiceRequest with the vendored types and unmarshalling it with go.opentelemetry.io/proto/otlp round-trips byte-for-byte.

Whichever lands, it should go out as a patch release so embedders can drop the conflictPolicy flag.

Out of scope

The trace and metrics halves of opentelemetry-proto — pb/otlp only vendors what the logs exporter needs, and that is the whole conflict surface today.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions