USDT probes for client↔server link events — external tracing (eBPF) without modifying traffic

[MoloF]

Problem

With pool_mode = transaction, the client connection ↔ server connection mapping
lives for a single transaction and exists only in PgCat's memory. External
observability systems (distributed tracing, eBPF agents) therefore cannot
deterministically correlate a query issued by a specific application with the
specific Postgres backend process that executed it.

Existing workarounds either modify traffic or are unreliable:

• SQL comments (sqlcommenter) alter the query text; content-based matching from
  the outside is further complicated by PgCat renaming prepared statements on
  the server leg;
• SET application_name on checkout adds an extra round-trip per transaction
  and noise in the Postgres logs;
• uprobes on internal functions are fragile for a Rust binary (inlining, no
  stable ABI), and Tokio task migration across worker threads makes
  syscall-level timing heuristics unreliable.

Proposal

Add static USDT tracepoints (via the usdt crate) to the client↔server link
lifecycle:

• pgcat:client_server_linked(client_addr, client_port, server_addr, server_port, pool, user)
  — fired when a server connection is assigned to a client transaction (checkout);
• pgcat:client_server_unlinked(client_addr, client_port, server_addr, server_port, pool, user)
  — fired when the connection is returned to the pool (release).

Properties of this approach:

• zero cost when disabled: a probe compiles to a NOP plus an ELF note; overhead
  appears only when a consumer actually attaches;
• a stable contract based on probe names instead of fragile offsets — survives
  recompilation and version upgrades;
• not a single byte is added to the stream towards Postgres; pooler behavior is
  unchanged;
• precedent: PostgreSQL itself ships DTrace/USDT probes (--enable-dtrace),
  and the usdt crate (by Oxide Computer) is used in production.

Sketch

#[usdt::provider]
mod pgcat {
    fn client_server_linked(client_addr: &str, client_port: u16,
                            server_addr: &str, server_port: u16,
                            pool: &str, user: &str) {}
    fn client_server_unlinked(client_addr: &str, client_port: u16,
                              server_addr: &str, server_port: u16,
                              pool: &str, user: &str) {}
}

Consuming the probes with no extra code, for verification:

bpftrace -e 'usdt:/usr/local/bin/pgcat:pgcat:client_server_linked
  { printf("%s:%d -> %s:%d [%s]\n", str(arg0), arg1, str(arg2), arg3, str(arg4)); }'

Questions for the maintainers

1. Is the usdt dependency acceptable? I can gate it behind a cargo feature
   (usdt-probes, enabled or disabled by default — your call).
2. Could you point me to the canonical places in the codebase where the
   client↔server link is assigned and released, so the probes land in the
   right functions?
3. Any preferences regarding probe naming and the argument set?
4. Is a Linux-only implementation acceptable (no-op on other platforms)?

Willingness to contribute

I'm ready to implement this in a PR: the probes, documentation, a bpftrace
consumption example, and build checks with the feature enabled and disabled.
Opening this issue first to confirm the feature fits the project's direction
and to agree on scope before starting the work.

Usage context

We are building end-to-end tracing across Node.js → PgCat → PostgreSQL: eBPF
agents already run on the application and database hosts; the missing piece is
an authoritative source of the client↔server leg mapping inside the pooler.
USDT probes close that gap without touching the traffic.