🖼️ fix: Describe Inline Media in Langfuse Spans Instead of Exporting It

[usnavy13]

Closes #578

What breaks

Every span that serializes the conversation exports its own copy of each inline attachment (#578). A 2.86 MiB PDF produced a 15.3 MiB trace: four copies of the 3.8 MiB base64 payload, spread across the agent root span, the agent span and the llm generation. Runs through AgentModelCall add that chain and its prompt span, and each later turn that replays the file exports it again.

The Langfuse SDK extracts only data: URIs, and only while media upload is enabled. So raw base64, Bedrock byte buffers and, with upload off, data: URIs all go out as text.

Change

prepareLangfuseSpanForExport gains a step between tool-output redaction and shaping. omitLangfuseSpanInlineMedia, in the new src/langfuseInlineMedia.ts, parses langfuse.observation.input and langfuse.observation.output and replaces each payload with a descriptor. The rest of the part is exported unchanged:

 {
   "type": "input_file",
   "filename": "report.pdf",
-  "file_data": "data:application/pdf;base64,JVBERi0xLjQK… (3.8 MiB)"
+  "file_data": "[inline media omitted from trace: application/pdf, 3001128 bytes]"
 }

Payloads are detected by content rather than by provider part shape, so current and future layouts are both covered:

Payload (4 KiB or more)	Replaced
Raw base64 string: Google media and inlineData, Anthropic source.data, input_audio	Always, since the SDK can never upload it
data:<mime>;base64, URI: image_url, input_file, file.file_data, video_url	While Langfuse media upload is disabled; otherwise left for the SDK to upload
Serialized Buffer ({ type: 'Buffer', data: number[] }): Bedrock document and image bytes	Always

The MIME type comes from the data: URI, or from a sibling mimeType, mime_type, media_type or mediaType key. The byte count is the decoded size.

Cost on the export path:

• Attributes shorter than 4 KiB are skipped without parsing.
• Longer attributes are parsed once: about 0.7 ms for 200 KB, and about 18 ms for 27 MiB.
• An attribute is re-serialized only when something was replaced.
• Base64 detection samples the first and last 256 characters. A full anchored regex test took about 100 ms per 27 MiB string.
• Attributes that are no longer valid JSON, such as ones OpenTelemetry has truncated, are left as they are.

AGENTS.md lists the new module and adds the behavior to the trace invariants.

Configuration

• Omission is the default.
• inlineMediaTracing: { enabled: true } on the run or agent LangfuseConfig, or LANGFUSE_TRACE_INLINE_MEDIA=true, restores verbatim export.
• Precedence is agent, then run, then env, as with toolOutputTracing.
• inlineMediaTracing joins the processor cache key.
• Whether media upload is enabled is resolved the same way LangfuseSpanProcessor resolves it: the mediaUploadEnabled param first, then LANGFUSE_MEDIA_UPLOAD_ENABLED, where only false and 0 disable.

What changes for hosts

• By default, traces no longer contain attachment bytes. A host that read them back from Langfuse needs to opt in.
• Other opaque base64 strings of 4 KiB or more in traced messages, such as encrypted reasoning content, are also replaced with a descriptor.
• What the model receives is unchanged. Only exported span attributes are rewritten.

Tests

• src/specs/langfuse-inline-media.test.ts runs through createLangfuseSpanProcessor with a real LangfuseSpanProcessor, startObservation and an in-memory exporter. It covers:

  • a Google media part, with its text kept
  • inputs and outputs
  • data: URIs with filenames while upload is disabled, and left alone while it is enabled
  • an Anthropic base64 source and a Bedrock buffer
  • ordinary content exported byte for byte
  • an attribute that is itself a payload
  • truncated JSON left unchanged
  • run opt-in, env opt-in and agent override
  • media-upload resolution that matches the SDK

  With the new step disabled, the six omission cases fail.

• langfuse-instrumentation.test.ts: processors are not reused across inline media policies.

• npx jest langfuse deterministic-trace-id ToolNode.langfuse passes: 12 suites, 216 tests. npx tsc --noEmit, ESLint, sort-imports --check and check:circular-deps are clean.

• Live export to a self-hosted Langfuse, with this build applied to 3.9.7:

  • A 2.86 MiB PDF dropped from 15.3 MiB per trace to about 20 KB, both on a Google model and on an OpenAI Responses model.
  • An 11.45 MiB PDF, and a follow-up turn that replayed it, each exported about 20 KB.
  • Usage and cost details were unchanged.