Skip to content

feat: enforce otel-only tracing in service and lambda scaffolding [DEV-590] - #205

Merged
negarciacamilo merged 6 commits into
mainfrom
feat/update-scaffolding
Jul 23, 2026
Merged

feat: enforce otel-only tracing in service and lambda scaffolding [DEV-590]#205
negarciacamilo merged 6 commits into
mainfrom
feat/update-scaffolding

Conversation

@negarciacamilo

@negarciacamilo negarciacamilo commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Description

Removes the X-Ray tracing fallback from lambda and service scaffolding so every generated lambda and service is wired for OpenTelemetry only, including new lambdas added to existing legacy X-Ray services — this forces migration instead of letting new code perpetuate X-Ray. Also fixes stale/hardcoded otel-layer ARNs and a missing serverless.yml block, aligns the generated OTel spans and logging with the monorepo's observability.md conventions, removes a chunk of dead domain-scaffolding code found along the way, and drops a broken/unused plugin while adding one that was missing.

Task Context

What is the current behavior?

  • new:lambda detected whether the target service was already on OTel or X-Ray and generated the matching tracing code, meaning a legacy X-Ray service kept getting new X-Ray-instrumented lambdas indefinitely.
  • The worker layer (resources.go, worker.go, idempotency.go) always used the legacy X-Ray tracer regardless of the service's tracing mode — this branch was never actually wired to OTel.
  • serverless.yml.tmpl hardcoded the OTel collector and Datadog extension layer ARNs directly instead of referencing the shared config/sls/layers.yml, and those hardcoded ARNs were already stale. It also lacked the custom.group metadata block and serverless-plugin-lambda-deployments, both present in nearly every real service.
  • Draft embedded and generated its own copy of collector.yaml and duplicated OTel-managed env vars in environment.yml, both of which can drift from the monorepo's SSoT.
  • Generated lambda code added spans to DI/Provide functions and private idempotency helpers, and worker spans in HTTP lambdas discarded their context — all against the monorepo's observability.md conventions. The SQS/SNS+SQS idempotency template also referenced an undeclared base package, so generated code didn't compile (only caught by manual review — the test suite only checks template syntax, not full compilation).
  • Idempotency error/panic paths used the legacy logger.Get() instead of log.Get(), breaking log↔trace correlation; the same legacy logger was used in domain provider files.
  • One domain service span used a snake_case operation name instead of the required kebab-case.
  • Draft's new:domain computed and unit-tested a whole providers/ package (generator, service, postgres/repository) that was never actually written to disk — pure dead code.
  • new:service offered a "warmup" plugin option that 0 of 41 real services use, and package.json.tmpl always installed serverless-plugin-datadog, which observability.md says is legacy-X-Ray-only and should not be present for OTel services.

What is the new behavior?

  • All lambda and service scaffolding (handler bootstrap and worker layers, across all lambda types) generates OTel instrumentation only — the X-Ray branch and the UseOtel detection/flag have been removed entirely.
  • serverless.yml.tmpl now references the shared config/sls/layers.yml for ARNs, adds the custom.group/deploymentSettings blocks and serverless-plugin-lambda-deployments, matching real services; new:service runs mage otel:fix post-create so the collector config and environment section stay synced with the monorepo's source of truth.
  • Removed spans from DI/Provide functions and idempotency helpers (handler/worker spans remain, since those are real entry points); HTTP worker spans now propagate their context onto the echo request so downstream calls nest correctly; fixed the missing base package import so SQS/SNS+SQS lambdas actually compile.
  • Replaced logger.Get() with log.Get() (trace-correlated *Context variants) everywhere the legacy logger was used, in both lambda and domain templates; fixed the log package import path and a stray broken variable reference along the way; fixed a snake_case span operation name to kebab-case.
  • Deleted the dead providers/ package templates and their loaders/tests — verified against a real domain (betsettlements) that the actually-generated service/provide.go + repository/provide.go already match the standardized DI pattern used across the monorepo.
  • Removed the warmup plugin end-to-end (flag, prompt, template, package.json) and removed serverless-plugin-datadog from package.json.tmpl.
  • go.mod/go.sum dependencies bumped (aws-sdk-go-v2, sentry-go, testify, x/crypto, huh, and related transitive deps) to go 1.25, with the pre-commit hook's pinned go.mod version updated to match.

Additional Context

Two other legacy X-Ray branches exist (feat/otel-support, feat/otel-scaffolding-support) that appear superseded by work already merged into this branch.

Removes the X-Ray/OTel branch entirely from lambda templates (handler and
worker layers) so every generated lambda, including ones added to legacy
X-Ray services, uses OTel — forcing eventual migration instead of letting
new code perpetuate X-Ray. Also fixes serverless.yml layer ARN references
to point at the shared config/sls/layers.yml SSoT instead of hardcoded,
stale ARNs, and runs `mage otel:fix` after scaffolding a new service so its
otel-layer collector config and environment.yml otel section stay in sync
with the monorepo's source of truth.

Also bumps go.mod/go.sum dependencies (aws-sdk-go-v2, sentry-go, testify,
x/crypto, huh, and related transitive deps) to go 1.25, and updates the
pre-commit hook's pinned go.mod version to match.
@negarciacamilo
negarciacamilo marked this pull request as ready for review July 23, 2026 15:08
…template

OPENTELEMETRY_COLLECTOR_CONFIG_URI is one of the vars mage otel:fix injects
into the otel:begin/otel:end section from the monorepo's SSoT
(config/sls/otel-environment.yml). Hardcoding it in the template outside
that section was redundant and would just get relocated on the first
mage otel:fix run.
…ventions

- Remove spans from ProvideResources and idempotency helpers (GenerateIdempotencyKey,
  SetIdempotency, DelIdempotency, deleteIdempotency) across all lambda types — per
  agents/observability.md §4/§4.5/§8, DI providers and private helpers must never be
  traced (orphaned root traces, cached singletons, no observability value), and >1
  StartSpan per file is a red flag.
- http worker.go.tmpl: propagate the span-carrying context back onto the echo request
  (ec.SetRequest) so any downstream call the developer adds inherits the handler span
  instead of getting a parentless one.
- sqs worker.go.tmpl: keep the span-derived context available (was discarded) for
  consistency with the other lambda types.
- Replace logger.Get() with log.Get().*Context() in idempotency error/panic paths so
  logs correlate with the active trace (dd.trace_id/dd.span_id), per §6. Also fixes
  the log package import path (pkg/log doesn't exist; the real package is
  projects/framev2/pkg/log) and a stray unresolved newCtx reference.
- Remove a pre-existing unused log import in sqs worker.go.tmpl.
- domains providers/service.go.tmpl and providers/postgres/repository.go.tmpl:
  replace legacy logger.Get() with log.Get() (per §6, no *Context variant
  available here since these Provide functions don't receive a ctx).
- service/postgres/search_one.go.tmpl: fix span operation name from
  snake_case "search_one" to kebab-case "search-one" per §4.3.

Verified against real domains (searchsynonyms, betsettlements): service layer
tracing (ServiceSpanName + LayerService) and repository layer being untraced
(driver auto-instruments) already matched the established pattern — no
further changes needed there. Dynamo domain scaffolding intentionally stays
a bare skeleton (no generated CRUD methods), consistent with existing docs.
providers/generator.go.tmpl, providers/service.go.tmpl, and
providers/postgres/repository.go.tmpl were loaded and unit-tested but never
actually written to disk by newdomain/exec.go — the real DI files
(service/provide.go, repository/provide.go) already follow the standardized
sync.Once + providers.MustAs pattern used across real domains (verified
against domains/betsettlements). Only providers/generators/nanoid/tableid,
which is genuinely written to pkg/providers/generators/nanoid/tableid/,
is kept.
…ml with real services

- sqs/snssqs idempotency.go.tmpl: GenerateIdempotencyKey referenced
  base.CtxKeyRootID without importing pkg/consumer/base — generated code
  didn't compile. assertGoSyntax only parses syntax, so this slipped through.
- Remove serverless-plugin-warmup end to end (dto field, form prompt,
  serverless.yml.tmpl, package.json.tmpl, tests) — 0 of 41 real services use
  it, it was dead scaffolding.
- serverless.yml.tmpl: add the custom.group ([service, otel]) block, present
  in every real OTel service and absent from every X-Ray one; add
  serverless-plugin-lambda-deployments + custom.deploymentSettings, used by
  38/41 services regardless of tracing mode.
- package.json.tmpl: drop serverless-plugin-datadog — per observability.md
  rule 3, OTel services use the Datadog Extension layer directly and this
  plugin is legacy-X-Ray-only; its presence in many real services' package.json
  is migration debt, not the target state for a newly scaffolded service.
@negarciacamilo
negarciacamilo merged commit 735fbec into main Jul 23, 2026
2 checks passed
@negarciacamilo
negarciacamilo deleted the feat/update-scaffolding branch July 23, 2026 16:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants