Skip to content

Wire the Temporal engine's minikube profile so bridged agents actually run, and prove it - #205

Open
imaustink wants to merge 2 commits into
mainfrom
test/temporal-minikube-e2e
Open

Wire the Temporal engine's minikube profile so bridged agents actually run, and prove it#205
imaustink wants to merge 2 commits into
mainfrom
test/temporal-minikube-e2e

Conversation

@imaustink

Copy link
Copy Markdown
Owner

Summary

Follow-up to #204 (which fixed the missing durable-agents.dev/bridged chart annotation). That fix alone was necessary but not sufficient to actually exercise a bridged agent end-to-end in this suite's minikube profile: no Temporal server exists there at all, and several of the Temporal engine's own config knobs had no default suited to a hermetic profile, each independently causing every turn to silently degrade to a bare conversational answer or an unresolvable identity-link park — never reaching a real bridged episode.

Found and fixed all of them, empirically, by driving real turns through the deployed cluster and reading Temporal's own workflow history rather than inferring from logs:

  1. No Temporal server in minikube. e2e/manifests/temporal-dev-server.yaml adds a single-node dev-mode server (temporalio/admin-tools's temporal server start-dev), applied by e2e/scripts/up.sh before the release deploys.
  2. temporal-engine.nats.url unset → bridged pod agents disabled entirely at the worker.
  3. temporal-engine.qdrant.host unset → all retrieval activities disabled, so no Agent could ever be found.
  4. temporal-engine.gateway.identity.defaultSubject/defaultRoles unset → every internal call from agent-orchestrator arrives at the Go gateway with no bearer token at all (its own AGENT_TEMPORAL_ENGINE_TOKEN secret key doesn't exist), resolving to an empty Caller.Subject — and the workflow treats an empty subject as "skip the catalog, answer bare," unconditionally.
  5. temporal-engine.identityLink.gatewayUrl unset → the worker uses an in-memory dev fake identity-link store, completely disconnected from this suite's real credential seeding (e2e/support/credential-store.ts, which writes real Kubernetes Secrets the real integration-gateway API reads).
  6. A Go bug (engines/temporal/internal/identitylink/identitylink.go): the claude-remote provider's /claude-auth/api/token?mode=login response shape uses a credentialsJson field, but the client only ever read token — so a real, retrievable credential was silently treated as absent, permanently parking any agent requiring the claude-remote provider.
  7. A core-controller bug (controllers/core-controller/internal/controller/agentrun_controller.go): the Temporal engine names AgentRun CRs agentrun-<agentId>-<uuid> itself. The reconciler's Job name template re-prefixed agentrun- onto that, producing a Job name — and therefore the API server's auto-added job-name pod-template label, which reuses it verbatim — over Kubernetes' 63-byte label limit. This broke Job creation for any AgentRun the Temporal engine launches, not just bridged ones.

The actual proof point

e2e/specs/bridged-agent-workflow.e2e.ts drives the same webhook path happy-path.e2e.ts does, then asks Temporal itself (via the temporal CLI against the dev server, not a log line or an inferred side effect) whether a BridgedAgentWorkflow execution actually completed for the route's target agent. Verified passing, confirmed via direct testing that a real AgentRun → Job → pod → completed episode → posted GitHub comment chain runs end-to-end.

Test plan

  • bridged-agent-workflow.e2e.ts passes on its own against a fully-wired minikube cluster
  • Manually verified, via direct /invoke and signed webhook probes, that the fix chain (annotation → NATS → Qdrant → identity → identity-link → Job naming) produces a real, WORKFLOW_EXECUTION_STATUS_COMPLETED BridgedAgentWorkflow execution, a real AgentRun/Job/pod completing, and the agent's reply posted back through the (fake) GitHub API
  • go build ./... clean in both engines/temporal and controllers/core-controller
  • Full e2e suite regression run did not complete within the time spent on this change — the suite has legitimately long real-time paced specs (resilience.e2e.ts) and a full run was still in progress after ~25 minutes with no clear sign of hanging vs. just being slow; it was cut off rather than left to fabricate a result. Worth a follow-up run before merge.

🤖 Generated with Claude Code

https://claude.ai/code/session_01HctRMtdZixptEZeMebhADq

imaustink and others added 2 commits August 8, 2026 08:32
The chart annotation fix (PR #204) was necessary but not sufficient: no
Temporal server exists in the e2e minikube profile, and the engine's own
config had several unset wires that silently degrade every turn to a
conversational bare-answer or a link-required park, never reaching a real
bridged episode. Fixed all of them and added the e2e spec that proves it:

- e2e/manifests/temporal-dev-server.yaml: a single-node dev-mode Temporal
  server (temporalio/admin-tools' `temporal server start-dev`), applied by
  e2e/scripts/up.sh before the release deploys.
- charts/agent-controller/values-e2e.yaml: wires temporal-engine.nats.url,
  .qdrant.host (retrieval was otherwise disabled entirely),
  .gateway.identity.defaultSubject/defaultRoles (every internal call from
  agent-orchestrator arrives tokenless, so an empty Caller.Subject made
  every turn bare-answer regardless of capability/retrieval/forcedAgentId),
  and .identityLink.gatewayUrl (without it the worker uses an in-memory dev
  fake, completely disconnected from this suite's real credential seeding).
- engines/temporal/internal/identitylink/identitylink.go: the claude-remote
  provider's `/claude-auth/api/token?mode=login` response uses a
  `credentialsJson` field, not `token` -- the client only ever read `token`,
  so a real, retrievable credential was silently treated as absent.
- controllers/core-controller/internal/controller/agentrun_controller.go:
  the Temporal engine names AgentRun CRs "agentrun-<agentId>-<uuid>" itself;
  prefixing "agentrun-" onto that again produced a Job name (and the API
  server's auto-added job-name pod-template label, which reuses it
  verbatim) over the 63-byte label limit, so the reconciler could never
  create a Job for ANY AgentRun this engine launches.
- e2e/specs/bridged-agent-workflow.e2e.ts: drives the same webhook path
  happy-path.e2e.ts does, then asks Temporal itself (not a log line) for a
  completed BridgedAgentWorkflow execution -- verified passing.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HctRMtdZixptEZeMebhADq
It was reusing 18092, the same local port identity-keying.e2e.ts
already binds for its own gateway port-forward -- a collision waiting
to bite the full suite even though file-level serial execution mostly
hides it today.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HctRMtdZixptEZeMebhADq
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant