diff --git a/CHANGELOG.md b/CHANGELOG.md index b2dd4383a..c6b30c72f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,37 @@ All notable changes to this project are documented in this file. Format follows For narrative release notes written for operators and product owners, see [RELEASE_NOTES.md](RELEASE_NOTES.md). +## [1.9.0] - 2026-07-20 + +Feature release making **Bedrock prompt-cache economics stable and measurable**. Three cache-busting defects in the model-call path are fixed (per-turn history mutation, nondeterministic skill ordering, single-cachePoint fragility), and a new observability layer makes every model call's cache behavior diagnosable: prefix fingerprints and a `cacheStatus` classification on each cost row, an admin Session Cost Anatomy drill-down page, CloudWatch EMF metrics, and a dashboard with alarms. Also fixes chat-input textarea sizing and points the deployed runtime's Word-document tools at the real user-files bucket. Requires a CDK deploy (new dashboard construct + one runtime env var); no data migration. + +### πŸš€ Added + +- Prompt-cache observability per model call β€” `PrefixFingerprintHook` hashes the three cacheable prefix components (toolConfig, effective system prompt, message history) onto each cost row; write-time `cacheStatus` classification (`first_write | hit | miss_ttl_expired | miss_avoidable | uncached`) with `wastedUsd` for avoidable misses priced from the row's own pricing snapshot; session-row cache rollups (`totalCacheReadTokens`, `totalCacheWriteTokens`, `avoidableMissCount`, `wastedUsd`); `GET /admin/costs/sessions/{sessionId}/calls` admin endpoint; and per-call CloudWatch EMF metrics. Kill switch `PROMPT_CACHE_OBSERVABILITY_ENABLED=false` (default ON) (#697) +- Admin **Session Cost Anatomy** page at `/admin/costs/sessions/:id` β€” chronological per-call table with color-coded `cacheStatus` badges and prefix-fingerprint diffing that names which hash (tools/system/history) flipped versus the previous call, plus session-level cache summary and a session-id lookup form on the Cost Analytics dashboard (#700) + +### ⚑ Performance + +- Restored conversation history is now byte-stable between turns β€” tool-content truncation is driven by a persisted `truncation_anchor` in compaction state instead of a sliding window that re-mutated history every turn, which forced a full prompt-cache prefix re-write (~$2.50/MTok on a 35k–150k-token prefix) nearly every turn (#697) +- Skill records reach the `` system-prompt block in deterministic order (sorted at the repository, RBAC-union, and injection layers), removing a per-turn prompt-cache invalidator (#697) +- Model calls now carry 3 of Bedrock's max-4 cachePoints (toolConfig tail, system tail, last-user-message) so a message-level cache miss still reads the stable tools+system prefix from cache instead of re-writing everything; the added points are gated to models that support them (#697) + +### πŸ› Fixed + +- Chat input textarea is scrollable once content exceeds its max height, clamps its growth, and resets to its base height after sending (#696) +- The first cache write after a run of below-threshold (uncached) calls is classified `first_write`, not `miss_avoidable` β€” it no longer inflates the AvoidableMiss/WastedUsd metrics or the Session Cost Anatomy page (#701) +- Word-document tools on the deployed runtime save to the real user-files bucket β€” `S3_USER_FILES_BUCKET_NAME` is now set on the inference-api Runtime, where its absence made the tools fall back to a literal `user-files` bucket and fail with `AccessDenied` (#702) +- Word-document tools fail fast with a clear "storage is not configured" message when the bucket env var is unset, instead of surfacing a confusing S3 `PermanentRedirect`/`AccessDenied` mid-run (#706) + +### πŸ—οΈ Infrastructure + +- `PromptCacheObservabilityConstruct` (`lib/constructs/observability/`) β€” CloudWatch dashboard over the `AgentCoreStack/PromptCache` EMF metrics (cache read/write tokens, cache-efficiency expression, AvoidableMiss, WastedUsd) with a Logs Insights widget grouped by `cacheStatus`, plus console-only alarms on AvoidableMiss and WastedUsd (stricter in prod; `NOT_BREACHING` on missing data so the kill switch stays quiet) (#699) +- `S3_USER_FILES_BUCKET_NAME` added to the inference-api AgentCore Runtime environment (#702) + +### πŸ“š Docs + +- CLAUDE.md β€” prompt-cache determinism/byte-stability contract, the fingerprint-based cache-miss debugging recipe, and a token-cost-effectiveness design tenet (#697, #698) + ## [1.8.0] - 2026-07-19 Feature release delivering **Skills v2** β€” skills are redesigned from tool-binding containers into pure, portable knowledge bundles that Agents load on demand, and for the first time **any signed-in user can author their own**. Skills ship enabled by default. Also completes the session-metadata static-sort-key migration (issue #175 Phases 2–3) with an operator backfill and a GSI-only read contraction, and consolidates the two artifact tool-catalog rows into a single "Artifacts" toggle. **Three backfill scripts must be run per environment** β€” see [RELEASE_NOTES.md](RELEASE_NOTES.md) deployment notes. No new AWS resources; a CDK deploy is needed only to set the `SKILLS_ENABLED` env var explicitly. diff --git a/CLAUDE.MD b/CLAUDE.MD index 8e1af5c83..7d74fa792 100644 --- a/CLAUDE.MD +++ b/CLAUDE.MD @@ -39,6 +39,8 @@ npx cdk deploy {prefix}-PlatformStack - **Admin endpoints** go under `/admin//`, user-facing under `//` - **Errors stream as assistant messages** via SSE (not HTTP error codes) - **Signal-based state** throughout frontend (`signal()`, `computed()`) +- **Prompt-cache stability is a contract.** Bedrock prompt caching is exact-prefix-match: any list that reaches the system prompt or `toolConfig` (skills, tools, models, MCP tool listings) must be deterministically ordered at its source, and restored conversation history must be byte-stable between compaction-state changes (see the truncation anchor in `TurnBasedSessionManager`). An order flip or history mutation between turns silently re-writes a 30k–150k-token prefix at the $2.50/MTok cache-write premium. `PROMPT_CACHE_OBSERVABILITY_ENABLED=false` disables the observability layer (fingerprint hook, cacheStatus derivation, EMF metrics) β€” the caching itself stays on +- **Token cost effectiveness is a design tenet β€” engineer against waste, not against context.** Before merging a change that touches the model call path, answer: what does this add to the prompt, on every turn, for the life of every session? (1) Anything in the cacheable prefix (system prompt, `toolConfig`, restored history) must be deterministic and append-only β€” see the prompt-cache contract above. (2) Per-turn payloads (tool results, MCP responses, retrieved documents) should be bounded or offloaded, never unbounded pass-through. (3) Don't guess β€” verify: `cacheStatus` + fingerprint hashes on the session's `C#` rows, `GET /admin/costs/sessions/{id}/calls`, and the `AgentCoreStack/PromptCache` EMF metrics exist to prove a change's cost impact. The balance: "cost effective" means eliminating waste (avoidable cache re-writes, duplicated context, oversized payloads) β€” it never means stripping context the model needs for a quality answer. When cost and answer quality genuinely conflict, quality wins; look for the cheaper path to the *same* quality, not a cheaper answer - **All dependencies use exact version pins** β€” no `^`, `~`, or `>=` - **Never install packages without explicit user approval** @@ -118,6 +120,7 @@ Do not reintroduce a Bearer-only `Depends(...)` on any user-facing route. If you - **Tool not appearing:** Check `__init__.py` export, RBAC permissions, `enabled_tools`, ToolRegistry - **Session not persisting:** Check AgentCore Memory config, session_id, TurnBasedSessionManager flush - **SSE stream disconnecting:** Check 600s timeout, client connection, quota exceeded events +- **Cost spike / cache miss:** Check `cacheStatus` + `toolConfigHash`/`systemPromptHash`/`historyHash` on the session's `C#` rows in sessions-metadata (or `GET /admin/costs/sessions/{id}/calls`) β€” the hash that changed between consecutive calls names the cache-buster ## Coding Standards diff --git a/README.md b/README.md index 5b9906bb8..c014c9a75 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ **An open-source, production-ready Generative AI platform for institutions** *Built by Boise State University, designed for everyone.* -[![Release](https://img.shields.io/badge/Release-v1.8.0-6366f1?style=flat&logo=github&logoColor=white)](RELEASE_NOTES.md) +[![Release](https://img.shields.io/badge/Release-v1.9.0-6366f1?style=flat&logo=github&logoColor=white)](RELEASE_NOTES.md) [![Nightly](https://github.com/Boise-State-Development/agentcore-public-stack/actions/workflows/nightly.yml/badge.svg)](https://github.com/Boise-State-Development/agentcore-public-stack/actions/workflows/nightly.yml) ![Python](https://img.shields.io/badge/Python-3.13+-3776AB?style=flat&logo=python&logoColor=white) @@ -296,7 +296,7 @@ agentcore-public-stack/ See [RELEASE_NOTES.md](RELEASE_NOTES.md) for the full changelog, including new features, bug fixes, platform upgrades, and deployment notes for each release. -**Current release:** v1.8.0 +**Current release:** v1.9.0 --- diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md index 5c9db597f..dd55d3682 100644 --- a/RELEASE_NOTES.md +++ b/RELEASE_NOTES.md @@ -1,3 +1,69 @@ +# Release Notes β€” v1.9.0 + +**Release Date:** July 20, 2026 +**Previous Release:** v1.8.0 (July 19, 2026) + +--- + +> πŸ—οΈ **CDK deploy required this release** β€” a new CloudWatch dashboard construct and one new env var on the inference-api Runtime. No data migration, no backfills, no dependency changes. Standard order: `platform.yml` β†’ `backend.yml` β†’ `frontend-deploy.yml`. + +--- + +## Highlights + +v1.9.0 makes **Bedrock prompt-cache spend stable and measurable**. A production conversation audit found 75% of one session's cost was avoidable cache re-writes; this release fixes the three defects behind that waste β€” restored history that mutated every turn, skills injected in nondeterministic order, and a single fragile cachePoint β€” and adds the observability to prove it and catch regressions: every model call now records prefix fingerprints and a `cacheStatus` verdict, admins get a per-session **Cost Anatomy** drill-down page, and operators get a CloudWatch dashboard with alarms on avoidable waste. Smaller fixes: the chat input textarea scrolls and resets correctly, and Word-document saves work on the deployed runtime (the container wasn't told which S3 bucket to use). + +## Prompt-cache stability β€” stop paying for avoidable re-writes + +Bedrock prompt caching is exact-prefix-match: if any byte of the cached prefix changes between turns, the whole prefix re-writes at the $2.50/MTok cache-write premium instead of reading at the ~$0.30/MTok cache-read rate. On a typical 35k–150k-token session prefix, one silent cache-buster costs more per turn than most turns' actual work. Three were found and fixed (#697): + +- **Restored history is now byte-stable.** Tool-content truncation ran on every session restore behind a sliding protected-turns window, so each new turn re-mutated the turn that had just aged past the window β€” invalidating the cache nearly every turn. Truncation is now driven by a persisted `truncation_anchor` in the session's compaction state: it advances only when the compaction checkpoint advances (where the slice already pays a single re-write) or opportunistically when the cache TTL (default 300s, `AGENTCORE_MEMORY_COMPACTION_CACHE_TTL_SECONDS`) has lapsed since the previous turn and the cache entry is dead anyway β€” so pending truncations apply for free. +- **Skills inject in deterministic order.** Skill records reached the `` system-prompt block in whatever order DynamoDB `batch_get_item` and Python set iteration produced, changing the system prompt between turns of the same session. Ordering is now sorted at three layers: the skills repository, the RBAC grant-union resolver (which returned `list(set)` β€” order varies per process via hash randomization), and the injection point itself. +- **Three cachePoints instead of one.** The auto strategy places a single message-level cachePoint; when its lookup misses (one proven mode: a wide parallel tool fan-out pushes the previous checkpoint past Anthropic's ~20-block cache lookback), *nothing* was read and the entire prefix re-wrote. Requests now carry 3 of Bedrock's max-4 cachePoints β€” toolConfig tail, system-prompt tail, and the existing last-user-message point β€” so a message-level miss still reads the stable tools+system prefix from cache. The added points are gated on `ModelConfig.bedrock_cache_points_supported()` since non-Anthropic models reject them. + +## Prompt-cache observability β€” every model call explains its cache behavior + +Diagnosing the waste above originally took hours of manual forensics against raw DynamoDB cost rows. That whole class of investigation is now a column diff (#697, #700, #699, #701). + +### Backend + +- `PrefixFingerprintHook` (a Strands `BeforeModelCallEvent` hook) hashes the three cacheable prefix components per model call β€” toolConfig (order-sensitive canonical JSON), the effective system prompt captured *after* AgentSkills injection, and message history excluding the newest message β€” and the stream coordinator persists them on the turn's cost rows. When a cache miss happens, the hash that changed between consecutive calls names the cache-buster. +- Each cost row gets a write-time `cacheStatus` β€” `first_write`, `hit`, `miss_ttl_expired`, `miss_avoidable`, or `uncached` β€” derived against the session's previous cost row, plus `wastedUsd` for avoidable misses priced at the cache-write premium over cache-read from the row's own pricing snapshot. Turn rows now write sequentially so each call classifies against its true predecessor. A follow-up fix (#701) classifies the first write after a run of below-threshold calls as `first_write` rather than `miss_avoidable`, so short-prompt sessions don't inflate the waste metrics. +- Session rows carry rollups next to `totalCost` β€” `totalCacheReadTokens`, `totalCacheWriteTokens`, `avoidableMissCount`, `wastedUsd` β€” so lists and admin views get a cache-efficiency ratio without scanning cost rows. +- `GET /admin/costs/sessions/{sessionId}/calls` (admin-only) returns the chronological per-call rows with token splits, cost, `cacheStatus`, and fingerprints, plus a session-level cache summary. +- Everything derived is behind `PROMPT_CACHE_OBSERVABILITY_ENABLED` (default ON, `=false` to disable the hook, the classification's extra GSI read, and EMF emission). Raw cache read/write token rollups are unaffected β€” they're usage passthrough, not derived. + +### Frontend + +- New **Session Cost Anatomy** page at `/admin/costs/sessions/:id`: summary tiles (total cost, cache efficiency, avoidable misses, wasted USD, cache read/write tokens) over a chronological calls table with color-coded `cacheStatus` badges. Fingerprint diffing flags which hash β€” tools, system, or history β€” flipped versus the previous fingerprinted call, which is the diagnosis on any `miss_avoidable` row. Expandable rows show full hashes and message counts; a session-id lookup form on the Cost Analytics dashboard is the entry point. + +### Infrastructure + +- `PromptCacheObservabilityConstruct` (new `lib/constructs/observability/` area, composed into `PlatformStack`) builds a CloudWatch dashboard over the `AgentCoreStack/PromptCache` EMF namespace both APIs emit into: cache read/write token trends, a cache-efficiency MathExpression, AvoidableMiss and WastedUsd, and a Logs Insights widget grouped by `cacheStatus`. Console-only alarms on AvoidableMiss and WastedUsd Sums (stricter thresholds in prod, `NOT_BREACHING` on missing data so the kill switch keeps them quiet). Deliberately no SNS β€” alerting infra remains out of scope, matching kb-sync and scheduled runs. + +### Test Coverage + +~1,800 lines of new tests: fingerprint/classification unit tests (including the below-threshold regression), cachePoint position and budget assertions, forced-order skill-sorting regressions, CDK construct assertions, and Vitest specs for the anatomy page, diff util, and HTTP service. + +## πŸ› Bug fixes + +- **The chat input became unusable on long prompts.** The textarea carried `overflow-hidden` with unclamped height growth, so past its 200px cap the content could neither be seen nor scrolled β€” and after sending, the box stayed expanded. Growth is now clamped with scrolling enabled past the cap, and the input resets to its base height on submit (#696) +- **Word-document saves failed on the deployed runtime with `AccessDenied`.** The AgentCore Runtime env set the user-files *table* name but not `S3_USER_FILES_BUCKET_NAME`, so the Word tools fell back to a literal `user-files` bucket the role has no access to. The env var is now set on the Runtime (#702), and the tools fail fast with a clear "storage is not configured" message β€” before spending a Code Interpreter run β€” if the variable is ever missing again (#706) + +## πŸ—οΈ Infrastructure + +- New CloudWatch dashboard + alarms construct (see spotlight above) β€” CloudWatch-console resources only, no SNS, no new IAM of note. +- `S3_USER_FILES_BUCKET_NAME` on the inference-api Runtime environment; the role's existing `UserFilesBucketAccess` grant already covers the bucket (#702) +- New env var `PROMPT_CACHE_OBSERVABILITY_ENABLED` on app-api and inference-api (default ON; set `=false` per environment to disable the observability layer β€” caching itself stays on). + +## πŸš€ Deployment notes + +Standard order, and this release uses all three: **`platform.yml` first** (the dashboard construct and the Runtime env var are CDK changes; the runtime picks up its current image via SSM, so the infra deploy is safe on its own) β†’ `backend.yml` β†’ `frontend-deploy.yml`. No backfills, no data migration, no dependency changes. + +After deploy, the **PromptCache dashboard** appears in the CloudWatch console. Expect `first_write` rows at the start of sessions and after idle gaps β€” only `miss_avoidable` indicates regression. The observability layer is per-call metadata; if it ever needs to be silenced in an environment, set `PROMPT_CACHE_OBSERVABILITY_ENABLED=false` and redeploy that service β€” the alarms go quiet on missing data by design. + +--- + # Release Notes β€” v1.8.0 **Release Date:** July 19, 2026 diff --git a/VERSION b/VERSION index 27f9cd322..f8e233b27 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.8.0 +1.9.0 diff --git a/backend/pyproject.toml b/backend/pyproject.toml index 57f66059b..5ccd017eb 100644 --- a/backend/pyproject.toml +++ b/backend/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "agentcore-stack" -version = "1.8.0" +version = "1.9.0" requires-python = ">=3.10" description = "Multi-agent conversational AI system with AWS Bedrock AgentCore" readme = "README.md" diff --git a/backend/src/agents/builtin_tools/word_document_tool.py b/backend/src/agents/builtin_tools/word_document_tool.py index 3e517bb4d..e336fb7ad 100644 --- a/backend/src/agents/builtin_tools/word_document_tool.py +++ b/backend/src/agents/builtin_tools/word_document_tool.py @@ -69,6 +69,10 @@ class _DocGenError(Exception): """Raised when Code Interpreter fails to run the document code.""" +class _StorageNotConfiguredError(Exception): + """Raised when the user-files S3 bucket is not configured for the runtime.""" + + def _region() -> str: return ( os.environ.get("AWS_REGION") @@ -190,7 +194,18 @@ def _s3(): def _user_files_bucket() -> str: - return os.environ.get("S3_USER_FILES_BUCKET_NAME", "user-files") + # Fail loudly rather than silently targeting a literal "user-files" bucket + # the runtime has no access to. That default misreported a missing env var + # as an S3 PermanentRedirect / AccessDenied and cost real debugging time. + # The runtime env is wired in infrastructure's + # inference-agentcore-construct.ts (S3_USER_FILES_BUCKET_NAME). + bucket = os.environ.get("S3_USER_FILES_BUCKET_NAME") + if not bucket: + raise _StorageNotConfiguredError( + "S3_USER_FILES_BUCKET_NAME is not set; the runtime cannot store or " + "retrieve Word documents." + ) + return bucket # --------------------------------------------------------------------------- @@ -512,6 +527,16 @@ def _error(text: str) -> Dict[str, Any]: "not found in the environment or Parameter Store." ) +_NO_STORAGE_MESSAGE = ( + "❌ Word document storage is not configured " + "(S3_USER_FILES_BUCKET_NAME is not set on the runtime)." +) + + +def _storage_configured() -> bool: + """True when the user-files bucket env var is set.""" + return bool(os.environ.get("S3_USER_FILES_BUCKET_NAME")) + # --------------------------------------------------------------------------- # Tool factories @@ -577,6 +602,8 @@ async def create_word_document( code_interpreter_id = _get_code_interpreter_id() if not code_interpreter_id: return _error(_NO_CI_MESSAGE) + if not _storage_configured(): + return _error(_NO_STORAGE_MESSAGE) try: file_bytes = await asyncio.to_thread( @@ -642,6 +669,8 @@ async def modify_word_document( code_interpreter_id = _get_code_interpreter_id() if not code_interpreter_id: return _error(_NO_CI_MESSAGE) + if not _storage_configured(): + return _error(_NO_STORAGE_MESSAGE) source = await _find_word_document(user_id, session_id, document_name) if source is None: @@ -754,6 +783,8 @@ async def read_word_document(document_name: str) -> Dict[str, Any]: code_interpreter_id = _get_code_interpreter_id() if not code_interpreter_id: return _error(_NO_CI_MESSAGE) + if not _storage_configured(): + return _error(_NO_STORAGE_MESSAGE) source = await _find_word_document(user_id, session_id, document_name) if source is None: diff --git a/backend/src/agents/main_agent/base_agent.py b/backend/src/agents/main_agent/base_agent.py index acdc7b133..58210b783 100644 --- a/backend/src/agents/main_agent/base_agent.py +++ b/backend/src/agents/main_agent/base_agent.py @@ -17,6 +17,7 @@ OAuthConsentHook, MCPExternalApprovalHook, ContextAttributionHook, + PrefixFingerprintHook, ) from agents.main_agent.tools import ( create_default_registry, @@ -297,6 +298,16 @@ def _create_hooks(self) -> List: # final metadata SSE event. hooks.append(ContextAttributionHook()) + # Per-model-call prompt-cache prefix fingerprints (toolConfig / + # system prompt / history hashes). Best-effort; the stream + # coordinator persists them on each call's metadata row so avoidable + # cache misses are diagnosable from stored data. + # PROMPT_CACHE_OBSERVABILITY_ENABLED=false disables the layer. + from apis.shared.observability import prompt_cache_observability_enabled + + if prompt_cache_observability_enabled(): + hooks.append(PrefixFingerprintHook()) + return hooks def _build_mcp_external_approval_hook(self) -> MCPExternalApprovalHook: diff --git a/backend/src/agents/main_agent/config/constants.py b/backend/src/agents/main_agent/config/constants.py index c817f463f..48ceeb481 100644 --- a/backend/src/agents/main_agent/config/constants.py +++ b/backend/src/agents/main_agent/config/constants.py @@ -27,6 +27,7 @@ class EnvVars: COMPACTION_TOKEN_THRESHOLD = "AGENTCORE_MEMORY_COMPACTION_TOKEN_THRESHOLD" COMPACTION_PROTECTED_TURNS = "AGENTCORE_MEMORY_COMPACTION_PROTECTED_TURNS" COMPACTION_MAX_TOOL_CONTENT_LENGTH = "AGENTCORE_MEMORY_COMPACTION_MAX_TOOL_CONTENT_LENGTH" + COMPACTION_CACHE_TTL_SECONDS = "AGENTCORE_MEMORY_COMPACTION_CACHE_TTL_SECONDS" # --- Restored-history repair --- # Kill switch for the restore-time tool-pairing/alternation repair @@ -98,6 +99,8 @@ class Defaults: COMPACTION_TOKEN_THRESHOLD = 100_000 COMPACTION_PROTECTED_TURNS = 3 COMPACTION_MAX_TOOL_CONTENT_LENGTH = 500 + # Bedrock prompt-cache TTL (seconds); see CompactionConfig.cache_ttl_seconds + COMPACTION_CACHE_TTL_SECONDS = 300 # --- DynamoDB Tables --- DYNAMODB_QUOTA_TABLE = "UserQuotas" diff --git a/backend/src/agents/main_agent/core/agent_factory.py b/backend/src/agents/main_agent/core/agent_factory.py index 6d9e99852..95a57b89b 100644 --- a/backend/src/agents/main_agent/core/agent_factory.py +++ b/backend/src/agents/main_agent/core/agent_factory.py @@ -196,12 +196,29 @@ def create_agent( f"({model_config.retry_config.sdk_initial_delay}s-{model_config.retry_config.sdk_max_delay}s backoff)" ) + # Bedrock prompt caching: give the system prompt its own cachePoint by + # passing it as a SystemContentBlock list with a trailing cachePoint + # (the cache_prompt model-config key is deprecated). Together with + # cache_tools (set in to_bedrock_config) this keeps the stable + # system+tools prefix readable from cache even when the auto-placed + # message-level cache point misses β€” see the cachePoint budget comment + # in ModelConfig.to_bedrock_config. Strands' auto strategy strips only + # message-level cachePoints, never system ones. Agent.system_prompt + # remains the plain string (split_system_prompt concatenates the text + # blocks), so hashing/attribution/voice consumers are unaffected. + agent_system_prompt: Any = system_prompt + if system_prompt and model_config.bedrock_cache_points_supported(): + agent_system_prompt = [ + {"text": system_prompt}, + {"cachePoint": {"type": "default"}}, + ] + # Create agent with session manager, hooks, and system prompt # Use SequentialToolExecutor to prevent concurrent browser operations # This prevents "Failed to start and initialize Playwright" errors with NovaAct agent = Agent( model=model, - system_prompt=system_prompt, + system_prompt=agent_system_prompt, tools=tools, tool_executor=SequentialToolExecutor(), session_manager=session_manager, diff --git a/backend/src/agents/main_agent/core/model_config.py b/backend/src/agents/main_agent/core/model_config.py index af99f9acf..422ea57f5 100644 --- a/backend/src/agents/main_agent/core/model_config.py +++ b/backend/src/agents/main_agent/core/model_config.py @@ -311,6 +311,23 @@ def get_provider(self) -> ModelProvider: # Default to configured provider return self.provider + def bedrock_cache_points_supported(self) -> bool: + """Whether explicit Bedrock cachePoints (tools/system) may be sent. + + Mirrors Strands' ``BedrockModel._cache_strategy`` predicate: Anthropic + models are the only Bedrock family with prompt-cache support. Auto + (message-level) caching no-ops safely on other models, but explicit + tools/system cachePoints would be sent verbatim and rejected with a + ValidationException β€” so both are gated here, alongside + ``caching_enabled``, on the Bedrock provider path. + """ + model_lower = self.model_id.lower() + return ( + self.caching_enabled + and self.get_provider() == ModelProvider.BEDROCK + and ("claude" in model_lower or "anthropic" in model_lower) + ) + def to_bedrock_config(self) -> Dict[str, Any]: """Convert to BedrockModel kwargs, translating canonical inference params.""" config: Dict[str, Any] = {"model_id": self.model_id} @@ -330,11 +347,34 @@ def to_bedrock_config(self) -> Dict[str, Any]: # unconditionally on the Bedrock path. config["use_native_token_count"] = True - # Bedrock prompt caching. CacheConfig(strategy="auto") lets Strands - # place cache points per-model: for a model that supports automatic - # caching it injects a cachePoint on the system/tools/last-user blocks; - # for one that doesn't it logs a warning and no-ops, so this is safe to - # set whenever caching is enabled. Requires strands-agents>=1.48.0: a + # Bedrock prompt caching β€” three cachePoints per request (Bedrock + # allows max 4; nothing else in this codebase adds one, see the + # position test in tests/agents/main_agent/core/test_bedrock_cache_points.py): + # + # 1. toolConfig tail β€” cache_tools="default" (_build_tools_cache_point) + # 2. system tail β€” SystemContentBlock list built by + # AgentFactory.create_agent (the deprecated + # cache_prompt config key is NOT used) + # 3. last user message β€” CacheConfig(strategy="auto"), which places + # exactly ONE message-level point and strips + # any others (_inject_cache_point). It does + # not touch the system/tools points. + # + # The tools+system points make a message-level lookup miss cost a + # cache READ of the stable prefix instead of a full re-write at + # $2.5/MTok (write premium). One proven miss mode is structural: + # Anthropic's cache lookback checks only ~20 content blocks behind + # the breakpoint, so a wide parallel tool fan-out (e.g. 18 parallel + # calls = ~38 new blocks) pushes the previous checkpoint out of range + # and forces a full re-write (prod session aecd387d: cacheRead=0, + # cacheWrite=134k mid-turn). With separate tools/system points the + # ~28k-token static prefix still reads from cache on those turns. + # + # For a model whose id Strands doesn't recognize as cache-capable, + # auto strategy logs a warning and no-ops β€” but cache_tools and a + # system cachePoint are sent unconditionally once configured, so both + # are gated on bedrock_cache_points_supported() (the same predicate + # Strands' auto mode uses). Requires strands-agents>=1.48.0: a # cachePoint trailing a non-PDF `document` attachment is rejected by # Bedrock's Anthropic adapter with "ValidationException ... # content.N.type: Field required" (agent force-stop on any turn with a @@ -347,6 +387,8 @@ def to_bedrock_config(self) -> Dict[str, Any]: if self.caching_enabled: from strands.models import CacheConfig config["cache_config"] = CacheConfig(strategy="auto") + if self.bedrock_cache_points_supported(): + config["cache_tools"] = "default" if self.retry_config: from botocore.config import Config as BotocoreConfig diff --git a/backend/src/agents/main_agent/session/compaction_models.py b/backend/src/agents/main_agent/session/compaction_models.py index a0fbc506c..3758094f3 100644 --- a/backend/src/agents/main_agent/session/compaction_models.py +++ b/backend/src/agents/main_agent/session/compaction_models.py @@ -29,6 +29,14 @@ class CompactionState: # compaction event in this session. Surfaced on session-metadata GET so # the frontend's end-of-conversation indicator survives a refresh. total_summarized_turns: int = 0 + # Absolute message index below which tool contents are truncated on + # restore. Bedrock prompt caching requires an exact prefix match, so + # truncation must be a pure function of persisted state β€” this anchor + # only moves when the checkpoint advances (the slice already forces a + # cache re-write) or when the prompt cache has already expired between + # turns (the re-write is free then). It must never be derived from a + # per-restore sliding window. + truncation_anchor: int = 0 def to_dict(self) -> Dict[str, Any]: """Convert to dictionary for DynamoDB storage.""" @@ -38,6 +46,7 @@ def to_dict(self) -> Dict[str, Any]: "lastInputTokens": self.last_input_tokens, "updatedAt": self.updated_at, "totalSummarizedTurns": self.total_summarized_turns, + "truncationAnchor": self.truncation_anchor, } @classmethod @@ -45,12 +54,17 @@ def from_dict(cls, data: Optional[Dict[str, Any]]) -> "CompactionState": """Create from DynamoDB item dictionary.""" if not data: return cls() + checkpoint = int(data.get("checkpoint", 0)) return cls( - checkpoint=int(data.get("checkpoint", 0)), + checkpoint=checkpoint, summary=data.get("summary"), last_input_tokens=int(data.get("lastInputTokens", 0)), updated_at=data.get("updatedAt"), total_summarized_turns=int(data.get("totalSummarizedTurns", 0)), + # Legacy records predate the anchor: default it to the checkpoint + # so nothing retained by the slice is truncated (byte-stable from + # the first restore under the anchor design). + truncation_anchor=int(data.get("truncationAnchor", checkpoint)), ) @@ -83,6 +97,11 @@ class CompactionConfig: token_threshold: int = 100_000 # Trigger checkpoint when exceeded protected_turns: int = 3 # Recent turns to protect from truncation max_tool_content_length: int = 500 # Max chars before truncating tool output + # Bedrock prompt-cache TTL. When more than this many seconds have passed + # since the previous turn, the cache entry has already expired, so pending + # truncations can be applied without forcing an otherwise-avoidable + # prefix re-write. + cache_ttl_seconds: int = 300 @classmethod def from_env(cls) -> "CompactionConfig": @@ -92,4 +111,5 @@ def from_env(cls) -> "CompactionConfig": token_threshold=int(os.environ.get(EnvVars.COMPACTION_TOKEN_THRESHOLD, str(Defaults.COMPACTION_TOKEN_THRESHOLD))), protected_turns=int(os.environ.get(EnvVars.COMPACTION_PROTECTED_TURNS, str(Defaults.COMPACTION_PROTECTED_TURNS))), max_tool_content_length=int(os.environ.get(EnvVars.COMPACTION_MAX_TOOL_CONTENT_LENGTH, str(Defaults.COMPACTION_MAX_TOOL_CONTENT_LENGTH))), + cache_ttl_seconds=int(os.environ.get(EnvVars.COMPACTION_CACHE_TTL_SECONDS, str(Defaults.COMPACTION_CACHE_TTL_SECONDS))), ) diff --git a/backend/src/agents/main_agent/session/hooks/__init__.py b/backend/src/agents/main_agent/session/hooks/__init__.py index 9b9748444..017112a59 100644 --- a/backend/src/agents/main_agent/session/hooks/__init__.py +++ b/backend/src/agents/main_agent/session/hooks/__init__.py @@ -2,12 +2,14 @@ from agents.main_agent.session.hooks.context_attribution import ContextAttributionHook from agents.main_agent.session.hooks.oauth_consent import OAuthConsentHook +from agents.main_agent.session.hooks.prefix_fingerprint import PrefixFingerprintHook from agents.main_agent.session.hooks.stop import StopHook from agents.main_agent.session.hooks.tool_approval import MCPExternalApprovalHook __all__ = [ "ContextAttributionHook", "OAuthConsentHook", + "PrefixFingerprintHook", "StopHook", "MCPExternalApprovalHook", ] diff --git a/backend/src/agents/main_agent/session/hooks/prefix_fingerprint.py b/backend/src/agents/main_agent/session/hooks/prefix_fingerprint.py new file mode 100644 index 000000000..2d2b27155 --- /dev/null +++ b/backend/src/agents/main_agent/session/hooks/prefix_fingerprint.py @@ -0,0 +1,103 @@ +"""Hook that fingerprints the cacheable request prefix before each model call. + +Bedrock prompt caching is exact-prefix-match: toolConfig, then system prompt, +then message history must be byte-identical to the previous call for cached +tokens to be read. When a session shows ``cacheStatus=miss_avoidable`` on its +metadata rows, these three hashes β€” persisted per call alongside tokenUsage β€” +show *which* component diverged between consecutive calls, replacing hours of +raw-row forensics with a column diff. + +On every ``BeforeModelCallEvent`` the hook computes: + +- ``toolConfigHash``: canonical JSON of the tool specs the model will see, in + registration order (order-sensitive on purpose β€” order flips are a real + cache-buster). +- ``systemPromptHash``: the effective system prompt. Fires after plugin + injection (e.g. AgentSkills' ```` block lands on + ``BeforeInvocationEvent``), so nondeterministic skill ordering is visible + here. +- ``historyHash``: ``agent.messages`` excluding the newest message β€” the + prior-history prefix that must match the previous call's full history for + a cache hit. + +Fingerprints accumulate in a per-turn list on the agent (one entry per model +call; a tool-use turn makes several). The stream coordinator resets the list +at turn start and attaches entry N to the Nth assistant message's metadata +row. Best-effort: any failure is swallowed so fingerprinting can never break +a model call. +""" + +import logging +from typing import Any, Dict, List, Optional + +from strands.hooks import BeforeModelCallEvent, HookProvider, HookRegistry + +from apis.shared.observability import fingerprint_canonical_json, fingerprint_text + +logger = logging.getLogger(__name__) + +# Per-turn list of fingerprint dicts, stashed on the Strands agent instance. +_FINGERPRINTS_ATTR = "_prefix_fingerprints" + + +def reset_prefix_fingerprints(agent: Any) -> None: + """Clear the per-turn fingerprint list. Called at turn start.""" + setattr(agent, _FINGERPRINTS_ATTR, []) + + +def get_prefix_fingerprint(agent: Any, index: Optional[int] = None) -> Optional[Dict[str, Any]]: + """Return the fingerprint for the ``index``-th model call of this turn. + + ``index=None`` returns the latest entry (used by single-call persistence + paths like the interrupted-turn writer). Returns None when the hook never + fired (non-Bedrock paths, hook disabled, or index out of range). + """ + fingerprints: List[Dict[str, Any]] = getattr(agent, _FINGERPRINTS_ATTR, None) or [] + if not fingerprints: + return None + if index is None: + return fingerprints[-1] + if 0 <= index < len(fingerprints): + return fingerprints[index] + # More assistant messages than observed model calls (shouldn't happen) β€” + # better to attach nothing than a wrong fingerprint. + return None + + +class PrefixFingerprintHook(HookProvider): + """Compute toolConfig / system-prompt / history hashes per model call.""" + + def register_hooks(self, registry: HookRegistry, **kwargs: Any) -> None: + registry.add_callback(BeforeModelCallEvent, self._on_before_model_call) + + def _on_before_model_call(self, event: BeforeModelCallEvent) -> None: + try: + agent = event.agent + + tool_specs = agent.tool_registry.get_all_tool_specs() + system_prompt = getattr(agent, "system_prompt", None) + messages = list(getattr(agent, "messages", None) or []) + + # System prompt may be a plain string or structured content blocks + # (SystemContentBlock list, used to place cache points). + if isinstance(system_prompt, str) or system_prompt is None: + system_hash = fingerprint_text(system_prompt) + else: + system_hash = fingerprint_canonical_json(system_prompt) + + fingerprint = { + "toolConfigHash": fingerprint_canonical_json(tool_specs), + "systemPromptHash": system_hash, + # History *prefix*: everything except the newest message. For + # a cache hit this must equal the previous call's full history. + "historyHash": fingerprint_canonical_json(messages[:-1]), + "messageCount": len(messages), + } + + fingerprints = getattr(agent, _FINGERPRINTS_ATTR, None) + if fingerprints is None: + fingerprints = [] + setattr(agent, _FINGERPRINTS_ATTR, fingerprints) + fingerprints.append(fingerprint) + except Exception as e: # noqa: BLE001 - observability must never break a turn + logger.debug("Prefix fingerprinting skipped: %s", e) diff --git a/backend/src/agents/main_agent/session/tests/test_compaction.py b/backend/src/agents/main_agent/session/tests/test_compaction.py index 740369847..eaf7c5af9 100644 --- a/backend/src/agents/main_agent/session/tests/test_compaction.py +++ b/backend/src/agents/main_agent/session/tests/test_compaction.py @@ -19,32 +19,44 @@ def test_default_state(self): assert state.summary is None assert state.last_input_tokens == 0 assert state.updated_at is None + assert state.truncation_anchor == 0 def test_to_dict(self): state = CompactionState( checkpoint=10, summary="Test summary", last_input_tokens=50000, - updated_at="2025-01-15T10:00:00Z" + updated_at="2025-01-15T10:00:00Z", + truncation_anchor=14, ) d = state.to_dict() assert d["checkpoint"] == 10 assert d["summary"] == "Test summary" assert d["lastInputTokens"] == 50000 assert d["updatedAt"] == "2025-01-15T10:00:00Z" + assert d["truncationAnchor"] == 14 def test_from_dict(self): data = { "checkpoint": 5, "summary": "Previous context", "lastInputTokens": 75000, - "updatedAt": "2025-01-15T12:00:00Z" + "updatedAt": "2025-01-15T12:00:00Z", + "truncationAnchor": 9, } state = CompactionState.from_dict(data) assert state.checkpoint == 5 assert state.summary == "Previous context" assert state.last_input_tokens == 75000 assert state.updated_at == "2025-01-15T12:00:00Z" + assert state.truncation_anchor == 9 + + def test_from_dict_legacy_record_defaults_anchor_to_checkpoint(self): + """Records written before the anchor existed must not re-truncate + retained history: the anchor defaults to the checkpoint, so nothing + the slice keeps is mutated.""" + state = CompactionState.from_dict({"checkpoint": 7}) + assert state.truncation_anchor == 7 def test_from_dict_handles_none(self): state = CompactionState.from_dict(None) @@ -66,18 +78,21 @@ def test_default_config(self): assert config.token_threshold == 100_000 assert config.protected_turns == 3 assert config.max_tool_content_length == 500 + assert config.cache_ttl_seconds == 300 def test_from_env(self, monkeypatch): monkeypatch.setenv("AGENTCORE_MEMORY_COMPACTION_ENABLED", "true") monkeypatch.setenv("AGENTCORE_MEMORY_COMPACTION_TOKEN_THRESHOLD", "50000") monkeypatch.setenv("AGENTCORE_MEMORY_COMPACTION_PROTECTED_TURNS", "3") monkeypatch.setenv("AGENTCORE_MEMORY_COMPACTION_MAX_TOOL_CONTENT_LENGTH", "1000") + monkeypatch.setenv("AGENTCORE_MEMORY_COMPACTION_CACHE_TTL_SECONDS", "600") config = CompactionConfig.from_env() assert config.enabled is True assert config.token_threshold == 50000 assert config.protected_turns == 3 assert config.max_tool_content_length == 1000 + assert config.cache_ttl_seconds == 600 class TestCompactionHelpers: @@ -159,29 +174,6 @@ def test_find_valid_cutoff_indices(self, mock_messages): # Should find indices 0, 4, 8 (the actual user questions, not tool results) assert valid_indices == [0, 4, 8] - def test_find_protected_indices(self, mock_messages): - """Test finding indices that should be protected from truncation""" - protected_turns = 2 - - # Find valid cutoff indices first - turn_start_indices = [] - for i, msg in enumerate(mock_messages): - if msg.get('role') == 'user': - content = msg.get('content', []) - is_tool_result = any( - isinstance(block, dict) and 'toolResult' in block - for block in content if isinstance(content, list) - ) - if not is_tool_result: - turn_start_indices.append(i) - - # With 3 turns at [0, 4, 8] and protected_turns=2, protect from index 4 onwards - turns_to_protect = min(protected_turns, len(turn_start_indices)) - protected_start_idx = turn_start_indices[-turns_to_protect] # Index 4 - - protected_indices = set(range(protected_start_idx, len(mock_messages))) - assert protected_indices == {4, 5, 6, 7, 8, 9} - def test_checkpoint_calculation(self, mock_messages): """Test checkpoint is set at oldest protected turn boundary""" protected_turns = 2 diff --git a/backend/src/agents/main_agent/session/tests/test_compaction_integration.py b/backend/src/agents/main_agent/session/tests/test_compaction_integration.py index 1bb2a93e9..de17cb9a8 100644 --- a/backend/src/agents/main_agent/session/tests/test_compaction_integration.py +++ b/backend/src/agents/main_agent/session/tests/test_compaction_integration.py @@ -179,13 +179,6 @@ def test_compaction_with_messages(): logger.info(f"Added {len(test_messages)} test messages") logger.info(f"Total messages: {len(agent.messages)}") - # Test truncation - protected_indices = session_manager._find_protected_indices( - agent.messages, - session_manager.compaction_config.protected_turns - ) - logger.info(f"Protected indices: {protected_indices}") - # Test valid cutoff indices valid_indices = session_manager._find_valid_cutoff_indices(agent.messages) logger.info(f"Valid cutoff indices: {valid_indices}") diff --git a/backend/src/agents/main_agent/session/turn_based_session_manager.py b/backend/src/agents/main_agent/session/turn_based_session_manager.py index e1c98f799..8293b6019 100644 --- a/backend/src/agents/main_agent/session/turn_based_session_manager.py +++ b/backend/src/agents/main_agent/session/turn_based_session_manager.py @@ -7,9 +7,18 @@ after the SDK finishes its standard session restore. Compaction Strategy (two-feature approach): -- Stage 1: Tool content truncation β€” applied every turn, reduces verbose tool I/O +- Stage 1: Tool content truncation β€” applied only below the persisted + truncation anchor, which moves at checkpoint advances or when the Bedrock + prompt cache has already expired between turns - Stage 2: Checkpoint + Summary β€” triggered when token threshold exceeded +Byte-stability contract: between compaction-state changes, restoring the same +stored history must produce byte-identical ``agent.messages``. Bedrock prompt +caching requires an exact prefix match, so any per-restore mutation of older +turns (e.g. a sliding truncation window) breaks the cached prefix and forces a +full re-write (~$2.5/MTok on a 35k–150k prefix) nearly every turn β€” far more +expensive than the read tokens truncation saves. + Based on: https://github.com/aws-samples/sample-strands-agent-with-agentcore """ @@ -43,7 +52,8 @@ class TurnBasedSessionManager(AgentCoreMemorySessionManager): Features: - Checkpoint-based message loading (skip old messages, prepend summary) - - Tool content truncation (reduce verbose tool I/O in older turns) + - Tool content truncation below a persisted anchor (cache-safe: history + is byte-stable between compaction-state changes) - Session cancellation support (via cancelled flag) - Compaction state persisted in DynamoDB session metadata """ @@ -235,6 +245,9 @@ def initialize(self, agent: "Agent", **kwargs: Any) -> None: except Exception as e: logger.error(f"Compaction failed, using full history: {e}", exc_info=True) self.compaction_state = CompactionState() + # Force update_after_turn to re-load persisted state rather than + # saving these defaults over the real checkpoint/anchor. + self._compaction_state_loaded = False self._valid_cutoff_indices = [] self._all_messages_for_summary = [] @@ -257,7 +270,16 @@ def _apply_compaction(self, agent: "Agent") -> None: Modifies agent.messages in-place to: 1. Skip old messages (checkpoint-based) 2. Prepend conversation summary - 3. Truncate verbose tool content in older turns + 3. Truncate tool content strictly below the persisted truncation anchor + + Byte-stability contract: this derivation must be a pure function of + (stored history, persisted compaction state). Truncation is therefore + driven ONLY by ``compaction_state.truncation_anchor`` β€” never by a + window computed from the current message count, which would re-mutate + an older turn on every restore and break Bedrock's exact-prefix cache + match. The anchor moves at checkpoint advances (``update_after_turn``, + where the slice already pays the one cache re-write) and + opportunistically here when the prompt cache has already expired. """ all_messages = agent.messages @@ -278,12 +300,20 @@ def _apply_compaction(self, agent: "Agent") -> None: # if update_after_turn actually advances the checkpoint) self._all_messages_for_summary = all_messages[:] + # Advance the truncation anchor only while the prompt cache is + # already cold β€” the prefix re-write is unavoidable then, so pending + # truncations are free. Persisted before use so subsequent restores + # derive the identical history. + self._maybe_advance_truncation_anchor() + # Apply checkpoint: skip old messages, prepend summary checkpoint = self.compaction_state.checkpoint stage = "none" + offset = 0 if checkpoint > 0 and checkpoint < len(all_messages): messages_to_process = all_messages[checkpoint:] + offset = checkpoint summary = self.compaction_state.summary if summary and messages_to_process: @@ -294,26 +324,79 @@ def _apply_compaction(self, agent: "Agent") -> None: else: messages_to_process = all_messages - # Apply truncation (always when compaction enabled) - protected_indices = self._find_protected_indices( - messages_to_process, self.compaction_config.protected_turns - ) - truncated_messages, truncation_count, _ = self._truncate_tool_contents( - messages_to_process, protected_indices=protected_indices - ) - - if truncation_count > 0: - stage = "checkpoint+truncation" if stage == "checkpoint" else "truncation" + # Truncate only messages strictly below the anchor (absolute index), + # translated into post-slice coordinates via the checkpoint offset. + truncation_count = 0 + anchor = min(max(self.compaction_state.truncation_anchor, offset), len(all_messages)) + if anchor > offset: + protected_indices = set(range(anchor - offset, len(messages_to_process))) + messages_to_process, truncation_count, _ = self._truncate_tool_contents( + messages_to_process, protected_indices=protected_indices + ) + if truncation_count > 0: + stage = "checkpoint+truncation" if stage == "checkpoint" else "truncation" - agent.messages = truncated_messages + agent.messages = messages_to_process logger.info( f"Compaction initialized: stage={stage}, " f"original={self._total_message_count_at_init}, " f"final={len(agent.messages)}, " + f"anchor={self.compaction_state.truncation_anchor}, " f"truncations={truncation_count}" ) + def _maybe_advance_truncation_anchor(self) -> None: + """Advance the truncation anchor while the prompt cache is already cold. + + The anchor normally moves only when the checkpoint advances (that + slice already forces one full prefix re-write, so folding truncation + into it costs nothing extra). But when more than + ``cache_ttl_seconds`` have passed since the previous turn, the + Bedrock prompt-cache entry has expired anyway β€” the next call + re-writes the prefix regardless β€” so the anchor can slide up to the + protected-turns boundary for free. Persisted immediately so every + subsequent restore derives the identical truncated history. + """ + state = self.compaction_state + config = self.compaction_config + if not self._cache_window_expired(state.updated_at, config.cache_ttl_seconds): + return + + cutoffs = self._valid_cutoff_indices + if len(cutoffs) <= config.protected_turns: + return + + sliding_anchor = cutoffs[-config.protected_turns] + if sliding_anchor <= max(state.truncation_anchor, state.checkpoint): + return + + logger.info( + "Truncation anchor advance (cache expired): %d -> %d", + state.truncation_anchor, sliding_anchor, + ) + state.truncation_anchor = sliding_anchor + self._save_compaction_state(state) + + @staticmethod + def _cache_window_expired(updated_at: Optional[str], ttl_seconds: int) -> bool: + """True when the previous turn is older than the prompt-cache TTL. + + ``updated_at`` is stamped by ``_save_compaction_state`` on every turn, + so it is a faithful proxy for the previous model call. Unparseable or + missing timestamps return False (conservative: assume the cache may + still be warm and leave history untouched). + """ + if not updated_at: + return False + try: + last = datetime.fromisoformat(updated_at.replace("Z", "+00:00")) + except (ValueError, TypeError): + return False + if last.tzinfo is None: + last = last.replace(tzinfo=timezone.utc) + return (datetime.now(timezone.utc) - last).total_seconds() > ttl_seconds + # ========================================================================= # Compaction State Persistence # ========================================================================= @@ -642,6 +725,12 @@ async def update_after_turn( summary = self._generate_fallback_summary(messages_to_summarize) self.compaction_state.checkpoint = new_checkpoint + # The anchor rides the checkpoint: everything the slice retains stays + # byte-identical until the next compaction-state change, so the single + # mutation (slice + summary) is paid with exactly one cache re-write. + self.compaction_state.truncation_anchor = max( + self.compaction_state.truncation_anchor, new_checkpoint + ) self.compaction_state.summary = summary # Running total persisted alongside the rest of the compaction state # so a refresh can rehydrate the end-of-conversation summary indicator. @@ -737,19 +826,6 @@ def _find_valid_cutoff_indices(self, messages: List[Dict]) -> List[int]: valid_indices.append(i) return valid_indices - def _find_protected_indices(self, messages: List[Dict], protected_turns: int) -> set: - """Find message indices that should be protected from truncation.""" - if protected_turns <= 0: - return set() - - turn_start_indices = self._find_valid_cutoff_indices(messages) - if not turn_start_indices: - return set() - - turns_to_protect = min(protected_turns, len(turn_start_indices)) - protected_start_idx = turn_start_indices[-turns_to_protect] - return set(range(protected_start_idx, len(messages))) - # ========================================================================= # Truncation (Stage 1 Compaction) # ========================================================================= diff --git a/backend/src/agents/main_agent/skills/strands_mapping.py b/backend/src/agents/main_agent/skills/strands_mapping.py index 2f04c4cad..2b395fe71 100644 --- a/backend/src/agents/main_agent/skills/strands_mapping.py +++ b/backend/src/agents/main_agent/skills/strands_mapping.py @@ -262,6 +262,11 @@ def build_skills_runtime( if not records: return None, None + # Deterministic order regardless of fetch order: these records become the + # system-prompt block, and any order flip between turns + # of a session invalidates the Bedrock prompt cache (exact-prefix match). + records = sorted(records, key=lambda r: getattr(r, "skill_id", "") or "") + skills = [record_to_strands_skill(r) for r in records] plugin = AgentSkills(skills=skills) read_tool = make_read_skill_file_tool(records) diff --git a/backend/src/agents/main_agent/streaming/stream_coordinator.py b/backend/src/agents/main_agent/streaming/stream_coordinator.py index 4f4174f9b..0b4731935 100644 --- a/backend/src/agents/main_agent/streaming/stream_coordinator.py +++ b/backend/src/agents/main_agent/streaming/stream_coordinator.py @@ -11,6 +11,10 @@ from typing import Any, AsyncGenerator, Dict, List, Optional, Union from agents.main_agent.config.constants import EnvVars +from agents.main_agent.session.hooks.prefix_fingerprint import ( + get_prefix_fingerprint, + reset_prefix_fingerprints, +) from apis.shared.errors import ( ConversationalErrorEvent, ErrorCode, @@ -81,6 +85,11 @@ async def stream_response( os.environ[EnvVars.SESSION_ID] = session_id os.environ[EnvVars.USER_ID] = user_id + # Per-turn prompt-cache prefix fingerprints: clear the previous + # turn's entries so entry N of this turn maps to the turn's Nth + # model call (the agent instance is cached across turns). + reset_prefix_fingerprints(agent) + # Track timing for latency metrics stream_start_time = time.time() # Wall-clock turn start as a tz-aware datetime. Used post-turn to @@ -933,19 +942,24 @@ async def stream_response( first_token_time=first_token_for_message, agent=main_agent_wrapper, # Use wrapper instead of internal agent citations=citations_for_message, # Pass citations for persistence + call_index=idx, # Nth model call of this turn (prefix fingerprint lookup) ) ) - # Execute all metadata storage tasks in parallel - # Use return_exceptions=True to prevent one failure from cancelling others - if metadata_tasks: - results = await asyncio.gather(*metadata_tasks, return_exceptions=True) - # Log any failures (but don't raise - metadata failures shouldn't break streaming) - for idx, result in enumerate(results): - if isinstance(result, Exception): - logger.error(f"Failed to store metadata for message {message_ids_to_store[idx]}: {result}") + # Execute metadata storage tasks SEQUENTIALLY, in call order. + # The write path derives each call's cacheStatus from the + # session's previous cost row; parallel writes would race the + # turn's own rows and misclassify calls 2..N. These are + # post-stream background writes, so the latency cost is + # invisible to the user. + for idx, task in enumerate(metadata_tasks): + try: + await task + except Exception as task_error: + # Log but don't raise - metadata failures shouldn't break streaming + logger.error(f"Failed to store metadata for message {message_ids_to_store[idx]}: {task_error}") - logger.info(f"βœ… Message metadata stored for {len(message_ids_to_store)} assistant messages (parallel)") + logger.info(f"βœ… Message metadata stored for {len(message_ids_to_store)} assistant messages (sequential)") # Store displayText for user message if original_message differs from augmented if original_message: @@ -2072,6 +2086,7 @@ async def _store_message_metadata( first_token_time: Optional[float], agent: Any = None, citations: Optional[List] = None, + call_index: Optional[int] = None, ) -> None: """ Store message-level metadata (token usage, latency, model info, citations) @@ -2086,6 +2101,10 @@ async def _store_message_metadata( first_token_time: Timestamp of first token received agent: Agent instance for extracting model info citations: Optional list of citation dicts from RAG retrieval + call_index: Index of this model call within the turn, used to + look up the matching prompt-cache prefix fingerprint stashed + by PrefixFingerprintHook. None β†’ latest fingerprint (single- + call paths like interrupted-turn persistence). """ try: from apis.shared.sessions.models import Attribution, LatencyMetrics, MessageMetadata, ModelInfo, TokenUsage @@ -2219,6 +2238,16 @@ async def _store_message_metadata( ) if context_window is not None: metadata_kwargs["contextWindow"] = context_window + + # Prompt-cache prefix fingerprints for this model call + # (extra field via extra="allow"; persisted on the cost row + # so cache misses are diagnosable component-by-component). + strands_agent = getattr(agent, "agent", None) if agent is not None else None + if strands_agent is not None: + prefix_fingerprint = get_prefix_fingerprint(strands_agent, call_index) + if prefix_fingerprint: + metadata_kwargs["prefixFingerprints"] = prefix_fingerprint + message_metadata = MessageMetadata(**metadata_kwargs) # Store metadata diff --git a/backend/src/apis/app_api/admin/costs/models.py b/backend/src/apis/app_api/admin/costs/models.py index 6791b4596..b8c7cbaa7 100644 --- a/backend/src/apis/app_api/admin/costs/models.py +++ b/backend/src/apis/app_api/admin/costs/models.py @@ -96,6 +96,55 @@ class CostTrend(BaseModel): active_users: int = Field(..., alias="activeUsers") +class PrefixFingerprints(BaseModel): + """Prompt-cache prefix hashes for one model call (see PrefixFingerprintHook).""" + model_config = ConfigDict(populate_by_name=True) + + tool_config_hash: Optional[str] = Field(None, alias="toolConfigHash") + system_prompt_hash: Optional[str] = Field(None, alias="systemPromptHash") + history_hash: Optional[str] = Field(None, alias="historyHash") + message_count: Optional[int] = Field(None, alias="messageCount") + + +class SessionCallRow(BaseModel): + """One model call within a session's cost anatomy.""" + model_config = ConfigDict(populate_by_name=True) + + timestamp: str + message_id: Optional[int] = Field(None, alias="messageId") + model_id: Optional[str] = Field(None, alias="modelId") + + input_tokens: int = Field(0, alias="inputTokens") + output_tokens: int = Field(0, alias="outputTokens") + cache_read_tokens: int = Field(0, alias="cacheReadTokens") + cache_write_tokens: int = Field(0, alias="cacheWriteTokens") + + cost: float = 0.0 + cache_status: Optional[str] = Field(None, alias="cacheStatus") + cache_gap_seconds: Optional[int] = Field(None, alias="cacheGapSeconds") + wasted_usd: float = Field(0.0, alias="wastedUsd") + prefix_fingerprints: Optional[PrefixFingerprints] = Field( + None, alias="prefixFingerprints" + ) + + +class SessionCostAnatomy(BaseModel): + """Per-call cost anatomy for one session (admin cache-miss forensics).""" + model_config = ConfigDict(populate_by_name=True) + + session_id: str = Field(..., alias="sessionId") + calls: List[SessionCallRow] + + total_cost: float = Field(0.0, alias="totalCost") + total_cache_read_tokens: int = Field(0, alias="totalCacheReadTokens") + total_cache_write_tokens: int = Field(0, alias="totalCacheWriteTokens") + avoidable_miss_count: int = Field(0, alias="avoidableMissCount") + wasted_usd: float = Field(0.0, alias="wastedUsd") + # cacheRead / (cacheRead + cacheWrite) over the session; None until + # there has been any cache activity. + cache_efficiency: Optional[float] = Field(None, alias="cacheEfficiency") + + class AdminCostDashboard(BaseModel): """Complete admin cost dashboard response combining all metrics.""" model_config = ConfigDict(populate_by_name=True) diff --git a/backend/src/apis/app_api/admin/costs/routes.py b/backend/src/apis/app_api/admin/costs/routes.py index 8c451751f..5d8600432 100644 --- a/backend/src/apis/app_api/admin/costs/routes.py +++ b/backend/src/apis/app_api/admin/costs/routes.py @@ -22,6 +22,7 @@ TierUsageSummary, CostTrend, AdminCostDashboard, + SessionCostAnatomy, ) from .service import AdminCostService @@ -109,6 +110,60 @@ async def get_cost_dashboard( ) +@router.get("/sessions/{session_id}/calls", response_model=SessionCostAnatomy) +async def get_session_cost_anatomy( + session_id: str, + admin_user: User = Depends(require_admin), + service: AdminCostService = Depends(get_cost_service) +): + """ + Get the per-model-call "cost anatomy" of a session. + + Returns one row per model call β€” timestamp, input/cacheRead/cacheWrite/ + output tokens, cost, derived cacheStatus (first_write | hit | + miss_ttl_expired | miss_avoidable | uncached), and the prompt-cache + prefix fingerprint hashes (toolConfig / system prompt / history) β€” plus + session-level rollups (cache efficiency, avoidable-miss count, wastedUsd). + + This is the forensic view for diagnosing avoidable prompt-cache + re-writes: on a miss_avoidable row, diff its fingerprint hashes against + the previous row's to see which prefix component changed. + + Args: + session_id: Session identifier (any user's session β€” admin scope) + admin_user: Authenticated admin user (injected) + service: Admin cost service (injected) + + Returns: + SessionCostAnatomy with chronological per-call rows + + Raises: + HTTPException: + - 401 if not authenticated + - 403 if user lacks admin role + - 404 if the session has no cost records + - 500 if server error + """ + logger.info("Admin requesting session cost anatomy") + + try: + anatomy = await service.get_session_cost_anatomy(session_id) + except Exception as e: + logger.error(f"Error getting session cost anatomy: {e}", exc_info=True) + raise HTTPException( + status_code=500, + detail="Failed to retrieve session cost anatomy" + ) + + if not anatomy.calls: + raise HTTPException( + status_code=404, + detail=f"No cost records found for session {session_id}" + ) + + return anatomy + + @router.get("/top-users", response_model=List[TopUserCost]) async def get_top_users( period: Optional[str] = Query( diff --git a/backend/src/apis/app_api/admin/costs/service.py b/backend/src/apis/app_api/admin/costs/service.py index 08d67922b..def19ba56 100644 --- a/backend/src/apis/app_api/admin/costs/service.py +++ b/backend/src/apis/app_api/admin/costs/service.py @@ -16,6 +16,9 @@ TierUsageSummary, CostTrend, AdminCostDashboard, + PrefixFingerprints, + SessionCallRow, + SessionCostAnatomy, ) logger = logging.getLogger(__name__) @@ -313,6 +316,99 @@ async def get_daily_trends( logger.error(f"Error getting daily trends: {e}") raise + async def get_session_cost_anatomy(self, session_id: str) -> SessionCostAnatomy: + """ + Get the per-model-call cost anatomy for one session. + + Reads every C# cost record for the session (chronological) and maps + each to a SessionCallRow with token splits, cost, derived cacheStatus, + and the prompt-cache prefix fingerprints β€” the data needed to see + where a session's spend went and which prefix component broke the + cache on a miss. Rows written before this feature shipped simply lack + cacheStatus/fingerprints and render as nulls. + + Args: + session_id: Session identifier (any user's β€” admin scope). + + Returns: + SessionCostAnatomy with per-call rows and session-level rollups. + """ + records = await self.storage.get_session_cost_records(session_id) + + calls: List[SessionCallRow] = [] + total_cost = 0.0 + total_cache_read = 0 + total_cache_write = 0 + avoidable_misses = 0 + wasted_usd = 0.0 + + for record in records: + token_usage = record.get("tokenUsage") or {} + model_info = record.get("modelInfo") or {} + fingerprints_raw = record.get("prefixFingerprints") + + # cost is a breakdown dict ({"total": ...}) on the streaming path + # or a bare float on the legacy path. + cost_raw = record.get("cost") + if isinstance(cost_raw, dict): + cost_raw = cost_raw.get("total") + try: + cost = float(cost_raw) if cost_raw is not None else 0.0 + except (TypeError, ValueError): + cost = 0.0 + + cache_read = int(token_usage.get("cacheReadInputTokens") or 0) + cache_write = int(token_usage.get("cacheWriteInputTokens") or 0) + cache_status = record.get("cacheStatus") + row_wasted = float(record.get("wastedUsd") or 0.0) + + total_cost += cost + total_cache_read += cache_read + total_cache_write += cache_write + if cache_status == "miss_avoidable": + avoidable_misses += 1 + wasted_usd += row_wasted + + gap_raw = record.get("cacheGapSeconds") + calls.append(SessionCallRow( + timestamp=record.get("timestamp", ""), + message_id=record.get("messageId"), + model_id=model_info.get("modelId"), + input_tokens=int(token_usage.get("inputTokens") or 0), + output_tokens=int(token_usage.get("outputTokens") or 0), + cache_read_tokens=cache_read, + cache_write_tokens=cache_write, + cost=cost, + cache_status=cache_status, + cache_gap_seconds=int(gap_raw) if gap_raw is not None else None, + wasted_usd=row_wasted, + prefix_fingerprints=( + PrefixFingerprints(**fingerprints_raw) + if isinstance(fingerprints_raw, dict) else None + ), + )) + + cache_traffic = total_cache_read + total_cache_write + cache_efficiency = ( + total_cache_read / cache_traffic if cache_traffic > 0 else None + ) + + logger.info( + f"Session cost anatomy: {len(calls)} calls, " + f"{avoidable_misses} avoidable misses, wasted=${wasted_usd:.4f}" + ) + + return SessionCostAnatomy( + session_id=session_id, + calls=calls, + total_cost=round(total_cost, 6), + total_cache_read_tokens=total_cache_read, + total_cache_write_tokens=total_cache_write, + avoidable_miss_count=avoidable_misses, + wasted_usd=round(wasted_usd, 6), + cache_efficiency=cache_efficiency, + ) + async def get_dashboard( self, period: Optional[str] = None, diff --git a/backend/src/apis/shared/observability/__init__.py b/backend/src/apis/shared/observability/__init__.py new file mode 100644 index 000000000..1acf860c6 --- /dev/null +++ b/backend/src/apis/shared/observability/__init__.py @@ -0,0 +1,25 @@ +"""Shared observability helpers (prompt-cache economics, EMF metrics).""" + +from apis.shared.observability.prompt_cache import ( + CACHE_TTL_SECONDS, + PROMPT_CACHE_OBSERVABILITY_ENABLED_ENV, + prompt_cache_observability_enabled, + CacheStatus, + classify_cache_status, + compute_wasted_usd, + fingerprint_canonical_json, + fingerprint_text, +) +from apis.shared.observability.emf import emit_prompt_cache_metrics + +__all__ = [ + "CACHE_TTL_SECONDS", + "PROMPT_CACHE_OBSERVABILITY_ENABLED_ENV", + "prompt_cache_observability_enabled", + "CacheStatus", + "classify_cache_status", + "compute_wasted_usd", + "fingerprint_canonical_json", + "fingerprint_text", + "emit_prompt_cache_metrics", +] diff --git a/backend/src/apis/shared/observability/emf.py b/backend/src/apis/shared/observability/emf.py new file mode 100644 index 000000000..ab17944bf --- /dev/null +++ b/backend/src/apis/shared/observability/emf.py @@ -0,0 +1,89 @@ +"""CloudWatch Embedded Metric Format (EMF) emission for prompt-cache metrics. + +EMF turns a structured log line into CloudWatch metrics with no SDK calls, +no batching agent, and no extra IAM: CloudWatch Logs extracts any log event +whose message is a JSON object carrying the ``_aws.CloudWatchMetrics`` +directive. Both compute surfaces already ship stdout to CloudWatch Logs +(inference-api via its AgentCore Runtime log group, app-api via ECS awslogs), +so a raw JSON line on stdout is all that's needed. + +The line must be *exactly* the JSON object β€” a ``[INFO] logger-name:`` prefix +from the app's standard formatter would break extraction β€” so this module +uses a dedicated non-propagating logger with a message-only formatter. +""" + +import json +import logging +import os +import sys +import time +from typing import Optional + +_EMF_NAMESPACE = os.environ.get("EMF_NAMESPACE", "AgentCoreStack/PromptCache") + +# Dedicated raw-JSON stdout logger. propagate=False keeps the app-level +# formatter (and its non-JSON prefixes) away from these lines. +_emf_logger = logging.getLogger("apis.shared.observability.emf.raw") +if not _emf_logger.handlers: + _handler = logging.StreamHandler(sys.stdout) + _handler.setFormatter(logging.Formatter("%(message)s")) + _emf_logger.addHandler(_handler) + _emf_logger.setLevel(logging.INFO) + _emf_logger.propagate = False + +logger = logging.getLogger(__name__) + + +def emit_prompt_cache_metrics( + cache_read_tokens: int, + cache_write_tokens: int, + avoidable_miss: bool, + wasted_usd: float = 0.0, + model_id: Optional[str] = None, + session_id: Optional[str] = None, + cache_status: Optional[str] = None, +) -> None: + """Emit one EMF record for a completed model call. + + Metrics (no dimensions β€” fleet-wide sums are the alerting target; the + per-model/per-session detail rides along as queryable log properties): + + - ``CacheReadTokens`` / ``CacheWriteTokens``: fleet cache traffic; their + ratio is the cache-efficiency dashboard line. + - ``AvoidableMiss``: count of calls classified ``miss_avoidable`` β€” the + alarm target (a prefix-stability regression shows up as a step change). + - ``WastedUsd``: dollars attributed to avoidable re-writes. + + Best-effort: never raises. + """ + try: + record = { + "_aws": { + "Timestamp": int(time.time() * 1000), + "CloudWatchMetrics": [ + { + "Namespace": _EMF_NAMESPACE, + "Dimensions": [[]], + "Metrics": [ + {"Name": "CacheReadTokens", "Unit": "Count"}, + {"Name": "CacheWriteTokens", "Unit": "Count"}, + {"Name": "AvoidableMiss", "Unit": "Count"}, + {"Name": "WastedUsd", "Unit": "None"}, + ], + } + ], + }, + "CacheReadTokens": int(cache_read_tokens or 0), + "CacheWriteTokens": int(cache_write_tokens or 0), + "AvoidableMiss": 1 if avoidable_miss else 0, + "WastedUsd": round(float(wasted_usd or 0.0), 6), + } + if model_id: + record["modelId"] = model_id + if session_id: + record["sessionId"] = session_id + if cache_status: + record["cacheStatus"] = cache_status + _emf_logger.info(json.dumps(record, separators=(",", ":"))) + except Exception as e: # noqa: BLE001 - metrics must never break a request + logger.debug("EMF emission skipped: %s", e) diff --git a/backend/src/apis/shared/observability/prompt_cache.py b/backend/src/apis/shared/observability/prompt_cache.py new file mode 100644 index 000000000..712ddfbfb --- /dev/null +++ b/backend/src/apis/shared/observability/prompt_cache.py @@ -0,0 +1,169 @@ +"""Prompt-cache economics: prefix fingerprints and per-call cache classification. + +Bedrock prompt caching is exact-prefix-match with a ~5-minute sliding TTL. +When the request prefix (toolConfig + system prompt + prior message history) +is byte-identical to the previous call's, tokens are read from cache at a +steep discount; any divergence forces a full cache re-write at a premium. +A production audit (session aecd387d, $1.60) showed 75% of spend was +avoidable re-writes caused by nondeterministic prefix assembly β€” and proving +that took hours of manual forensics. This module makes the whole class +measurable: + +- ``fingerprint_*`` produce short stable hashes of the three prefix + components so consecutive metadata rows show *which* component changed + when a cache miss happens. +- ``classify_cache_status`` labels each model call from its token usage and + the previous call's row. +- ``compute_wasted_usd`` prices the avoidable portion of a re-write. + +Pure functions only β€” no AWS calls β€” so they are unit-testable and safe to +import from any package (agents/, app_api, inference_api all may import +``apis.shared``). +""" + +import hashlib +import json +import os +from enum import Enum +from typing import Any, Mapping, Optional + +# Kill switch for the whole prompt-cache observability layer (fingerprint +# hook, per-call cacheStatus derivation + session rollups, EMF emission). +# Default ON; only the literal string "false" disables it β€” an empty or +# unset value stays enabled (workflow env vars can materialize as ""). +PROMPT_CACHE_OBSERVABILITY_ENABLED_ENV = "PROMPT_CACHE_OBSERVABILITY_ENABLED" + + +def prompt_cache_observability_enabled() -> bool: + """Whether prompt-cache observability is enabled (env kill switch). + + Read per call (no module-level caching) so tests and live config changes + behave predictably; the env read is negligible next to the DynamoDB + lookup and hash work it gates. + """ + return os.environ.get(PROMPT_CACHE_OBSERVABILITY_ENABLED_ENV, "").lower() != "false" + +# Bedrock prompt-cache TTL (sliding, seconds). A gap between consecutive +# model calls larger than this means the cache entry legitimately expired β€” +# the re-write was unavoidable. +CACHE_TTL_SECONDS = 300 + + +class CacheStatus(str, Enum): + """Derived per-call prompt-cache outcome. + + - ``first_write``: the initial, expected cache population β€” either no + previous call row for the session, or every activity-bearing signal + says no cache existed yet (the previous call was ``uncached``, e.g. + the prompt was below the model's minimum cacheable prefix, so there + was nothing to read from). + - ``hit``: tokens were read from cache (``cacheReadInputTokens > 0``). + Partial re-writes of a changed suffix still count as hits. + - ``miss_ttl_expired``: nothing read, cache re-written, and the gap since + the previous call exceeded the cache TTL β€” unavoidable. + - ``miss_avoidable``: nothing read, cache re-written, previous call had a + live cache entry within the TTL β€” the prefix must have changed. This is + the bug class the fingerprints exist to diagnose. + - ``uncached``: no cache activity at all (caching disabled, non-Bedrock + provider, or prompt below the minimum cacheable length). + """ + + FIRST_WRITE = "first_write" + HIT = "hit" + MISS_TTL_EXPIRED = "miss_ttl_expired" + MISS_AVOIDABLE = "miss_avoidable" + UNCACHED = "uncached" + + +def fingerprint_text(text: Optional[str]) -> str: + """Short stable hash of a text blob (e.g. the system prompt).""" + return hashlib.sha256((text or "").encode("utf-8")).hexdigest()[:16] + + +def fingerprint_canonical_json(obj: Any) -> str: + """Short stable hash of a JSON-serializable structure. + + Dict keys are sorted so key insertion order never changes the hash, but + list order is preserved β€” deliberately, because list order (tool specs, + message history, content blocks) is exactly what Bedrock's exact-prefix + match is sensitive to. + """ + canonical = json.dumps(obj, sort_keys=True, separators=(",", ":"), default=str) + return hashlib.sha256(canonical.encode("utf-8")).hexdigest()[:16] + + +def classify_cache_status( + cache_read_tokens: int, + cache_write_tokens: int, + previous_call_exists: bool, + gap_seconds: Optional[float], + previous_cached_prefix_tokens: Optional[int] = None, +) -> CacheStatus: + """Classify one model call's cache outcome. + + Args: + cache_read_tokens: ``cacheReadInputTokens`` for this call. + cache_write_tokens: ``cacheWriteInputTokens`` for this call. + previous_call_exists: Whether the session has an earlier call row. + gap_seconds: Seconds since the previous call row's timestamp; None + when unknown (treated conservatively as expired, not avoidable). + previous_cached_prefix_tokens: Previous call's cacheRead + cacheWrite + token total, or None when unknown. Zero means the previous call + was uncached (e.g. prompt below the model's minimum cacheable + prefix), so no cache entry existed for this call to read. + """ + if cache_read_tokens > 0: + return CacheStatus.HIT + if cache_write_tokens <= 0: + return CacheStatus.UNCACHED + if not previous_call_exists: + return CacheStatus.FIRST_WRITE + if previous_cached_prefix_tokens is not None and previous_cached_prefix_tokens <= 0: + # The previous call wrote nothing to the cache, so there was no entry + # to read from β€” this write is the session's first real population + # (typically the first prompt to cross the minimum cacheable length), + # not a miss of any kind. + return CacheStatus.FIRST_WRITE + if gap_seconds is None or gap_seconds > CACHE_TTL_SECONDS: + return CacheStatus.MISS_TTL_EXPIRED + return CacheStatus.MISS_AVOIDABLE + + +def compute_wasted_usd( + cache_status: CacheStatus, + cache_write_tokens: int, + previous_cached_prefix_tokens: Optional[int], + pricing_snapshot: Optional[Mapping[str, Any]], +) -> float: + """USD wasted by an avoidable cache re-write; 0.0 for every other status. + + The waste is the re-written portion of the prefix that was already cached + on the previous call (``min(cacheWrite, previous cacheRead + cacheWrite)``; + falls back to the full cacheWrite when the previous split is unknown), + priced at the cache-write premium over the cache-read rate the tokens + *should* have cost. + + Args: + cache_status: Result of :func:`classify_cache_status`. + cache_write_tokens: ``cacheWriteInputTokens`` for this call. + previous_cached_prefix_tokens: Previous call's cacheRead + cacheWrite + token total, or None when unavailable. + pricing_snapshot: The row's ``pricingSnapshot`` dict (camelCase keys). + """ + if cache_status is not CacheStatus.MISS_AVOIDABLE: + return 0.0 + if not pricing_snapshot or cache_write_tokens <= 0: + return 0.0 + + # `or 0` (not `.get(..., 0)`) β€” rows can store an explicit None. + write_price = pricing_snapshot.get("cacheWritePricePerMtok") or 0 + read_price = pricing_snapshot.get("cacheReadPricePerMtok") or 0 + premium_per_mtok = write_price - read_price + if premium_per_mtok <= 0: + return 0.0 + + rewritten = cache_write_tokens + if previous_cached_prefix_tokens is not None and previous_cached_prefix_tokens > 0: + rewritten = min(cache_write_tokens, previous_cached_prefix_tokens) + + return (rewritten / 1_000_000) * premium_per_mtok diff --git a/backend/src/apis/shared/rbac/service.py b/backend/src/apis/shared/rbac/service.py index 4c1f93e80..8b937c22f 100644 --- a/backend/src/apis/shared/rbac/service.py +++ b/backend/src/apis/shared/rbac/service.py @@ -192,9 +192,13 @@ def _merge_permissions( return UserEffectivePermissions( user_id=user_id, app_roles=[r.role_id for r in roles], - tools=list(all_tools), - models=list(all_models), - skills=list(all_skills), + # Sorted for a deterministic order across processes (set iteration + # varies with hash randomization). These lists reach the model's + # system prompt / tool config, where an order flip between turns + # invalidates the Bedrock prompt cache. + tools=sorted(all_tools), + models=sorted(all_models), + skills=sorted(all_skills), quota_tier=quota_tier, resolved_at=datetime.now(timezone.utc).isoformat() + "Z", ) diff --git a/backend/src/apis/shared/sessions/metadata.py b/backend/src/apis/shared/sessions/metadata.py index bc75d0c9e..7fc1f5225 100644 --- a/backend/src/apis/shared/sessions/metadata.py +++ b/backend/src/apis/shared/sessions/metadata.py @@ -237,6 +237,16 @@ async def _store_message_metadata_cloud( # Extract timestamp for SK and GSI timestamp = metadata_dict.get("attribution", {}).get("timestamp", datetime.now(timezone.utc).isoformat()) + # Derive prompt-cache observability (cacheStatus / wastedUsd) from + # this call's usage + the session's previous cost row. Best-effort: + # {} on any failure so it can never block the critical write. + cache_observability = _derive_cache_observability( + session_id=session_id, + table=table, + timestamp=timestamp, + message_metadata=message_metadata, + ) + # Generate unique ID for SK to prevent collisions unique_id = str(uuid_lib.uuid4()) @@ -270,7 +280,11 @@ async def _store_message_metadata_cloud( "ttl": ttl, # Cost and usage metadata - **metadata_decimal + **metadata_decimal, + + # Derived prompt-cache observability (cacheStatus, cacheGapSeconds, + # wastedUsd) β€” {} when derivation was skipped or failed + **_convert_floats_to_decimal(cache_observability), } # Store in DynamoDB @@ -279,14 +293,21 @@ async def _store_message_metadata_cloud( logger.info(f"πŸ’Ύ Stored cost record in DynamoDB table {table_name}") logger.info(f" Session: {session_id}, Message: {message_id}, SK: C#{timestamp}#{unique_id[:8]}...") + # Per-call EMF metrics (CacheReadTokens / CacheWriteTokens / + # AvoidableMiss / WastedUsd) for the fleet cache-efficiency + # dashboard + alarm. Best-effort, never raises. + _emit_cache_metrics(session_id, message_metadata, cache_observability) + # Bump session-level aggregates (totalCost, lastContextTokens, - # contextWindow) for the session-cost badge. Best-effort β€” drift is - # repaired by lazy backfill on the next metadata read. + # contextWindow, cache-efficiency counters) for the session-cost + # badge and admin lists. Best-effort β€” drift is repaired by lazy + # backfill on the next metadata read. await _bump_session_aggregates( session_id=session_id, user_id=user_id, message_metadata=message_metadata, table=table, + cache_observability=cache_observability, ) # Update pre-aggregated cost summary for fast quota checks @@ -313,6 +334,174 @@ async def _store_message_metadata_cloud( +def _extract_cache_usage(message_metadata: MessageMetadata) -> Tuple[int, int]: + """Return (cache_read_tokens, cache_write_tokens) from a metadata object.""" + token_usage = message_metadata.token_usage + if not token_usage: + return 0, 0 + return ( + token_usage.cache_read_input_tokens or 0, + token_usage.cache_write_input_tokens or 0, + ) + + +def _extract_pricing_dict(message_metadata: MessageMetadata) -> Optional[Dict[str, Any]]: + """Return the pricingSnapshot as a camelCase dict, or None.""" + if not message_metadata.model_info: + return None + pricing = message_metadata.model_info.pricing_snapshot + if pricing is None: + return None + if hasattr(pricing, "model_dump"): + return pricing.model_dump(by_alias=True) + return pricing + + +def _derive_cache_observability( + session_id: str, + table, + timestamp: str, + message_metadata: MessageMetadata, +) -> Dict[str, Any]: + """Classify this model call's prompt-cache outcome against the previous call. + + Queries the session's most recent existing ``C#`` cost row (one GSI read, + Limit=1) and derives: + + - ``cacheStatus``: first_write | hit | miss_ttl_expired | miss_avoidable + | uncached (see ``apis.shared.observability.CacheStatus``). + - ``cacheGapSeconds``: whole seconds since the previous call, when known. + - ``wastedUsd``: for avoidable misses, the re-written previously-cached + prefix priced at the cache-write premium over the cache-read rate, + using this row's own pricingSnapshot. + + The stream coordinator writes a turn's rows sequentially in call order, + so within a multi-call turn each call sees its predecessor. Returns {} + when there is no token usage to classify or on any failure β€” derivation + must never block the critical cost-record write. + """ + try: + from boto3.dynamodb.conditions import Key + from datetime import datetime + + from apis.shared.observability import ( + CacheStatus, + classify_cache_status, + compute_wasted_usd, + prompt_cache_observability_enabled, + ) + + # Kill switch: also skips the session cache rollups and (via the + # empty dict) the EMF emission downstream. + if not prompt_cache_observability_enabled(): + return {} + + if not message_metadata.token_usage: + return {} + + cache_read, cache_write = _extract_cache_usage(message_metadata) + + # Most recent existing cost row for this session (GSI_SK = C# + # sorts chronologically; descending scan + Limit=1 = the previous call). + response = table.query( + IndexName="SessionLookupIndex", + KeyConditionExpression=( + Key("GSI_PK").eq(f"SESSION#{session_id}") + & Key("GSI_SK").begins_with("C#") + ), + ScanIndexForward=False, + Limit=1, + ) + prev_items = response.get("Items", []) + prev_row = _convert_decimal_to_float(prev_items[0]) if prev_items else None + + gap_seconds: Optional[float] = None + prev_cached_prefix: Optional[int] = None + if prev_row: + prev_ts = prev_row.get("timestamp") + try: + current_dt = datetime.fromisoformat(timestamp.replace("Z", "+00:00")) + prev_dt = datetime.fromisoformat(str(prev_ts).replace("Z", "+00:00")) + gap_seconds = (current_dt - prev_dt).total_seconds() + except (ValueError, AttributeError, TypeError): + gap_seconds = None + + prev_usage = prev_row.get("tokenUsage") or {} + prev_cached_prefix = int( + (prev_usage.get("cacheReadInputTokens") or 0) + + (prev_usage.get("cacheWriteInputTokens") or 0) + ) + + status = classify_cache_status( + cache_read_tokens=cache_read, + cache_write_tokens=cache_write, + previous_call_exists=prev_row is not None, + gap_seconds=gap_seconds, + previous_cached_prefix_tokens=prev_cached_prefix, + ) + wasted_usd = compute_wasted_usd( + cache_status=status, + cache_write_tokens=cache_write, + previous_cached_prefix_tokens=prev_cached_prefix, + pricing_snapshot=_extract_pricing_dict(message_metadata), + ) + + result: Dict[str, Any] = { + "cacheStatus": status.value, + "wastedUsd": round(wasted_usd, 6), + } + if gap_seconds is not None and gap_seconds >= 0: + result["cacheGapSeconds"] = int(gap_seconds) + + if status is CacheStatus.MISS_AVOIDABLE: + logger.warning( + "πŸ”₯ Avoidable prompt-cache miss: session=%s gap=%ss cacheWrite=%d wasted=$%.6f", + session_id, result.get("cacheGapSeconds"), cache_write, wasted_usd, + ) + return result + + except Exception as e: + # JUSTIFICATION: cache classification is derived observability; the + # authoritative usage numbers are already on the row. Never block the + # cost-record write over it. + logger.debug("Cache observability derivation skipped: %s", e) + return {} + + +def _emit_cache_metrics( + session_id: str, + message_metadata: MessageMetadata, + cache_observability: Dict[str, Any], +) -> None: + """Emit per-call EMF metrics for the fleet cache dashboard. Never raises.""" + try: + from apis.shared.observability import ( + CacheStatus, + emit_prompt_cache_metrics, + prompt_cache_observability_enabled, + ) + + if not prompt_cache_observability_enabled(): + return + + if not message_metadata.token_usage: + return + + cache_read, cache_write = _extract_cache_usage(message_metadata) + status = cache_observability.get("cacheStatus") + emit_prompt_cache_metrics( + cache_read_tokens=cache_read, + cache_write_tokens=cache_write, + avoidable_miss=status == CacheStatus.MISS_AVOIDABLE.value, + wasted_usd=cache_observability.get("wastedUsd") or 0.0, + model_id=message_metadata.model_info.model_id if message_metadata.model_info else None, + session_id=session_id, + cache_status=status, + ) + except Exception as e: # noqa: BLE001 - metrics must never break the write path + logger.debug("Cache EMF emission skipped: %s", e) + + async def _update_cost_summary_async( user_id: str, timestamp: str, @@ -1240,6 +1429,7 @@ async def _bump_session_aggregates( user_id: str, message_metadata: MessageMetadata, table, + cache_observability: Optional[Dict[str, Any]] = None, ) -> None: """Atomically update the session row's denormalized cost + context fields. @@ -1247,6 +1437,10 @@ async def _bump_session_aggregates( ``update_item`` call: - ``ADD totalCost :c`` β€” concurrent-safe across overlapping turns. + - ``ADD totalCacheReadTokens / totalCacheWriteTokens / avoidableMissCount + / wastedUsd`` β€” per-session cache-efficiency rollups so lists and + admin views can show a cache-efficiency ratio without scanning the + session's cost rows. - ``SET lastContextTokens :t, contextWindow :w`` β€” last-write-wins, which is the right behavior for "most recent turn." @@ -1293,7 +1487,33 @@ async def _bump_session_aggregates( update_parts_set.append("contextWindow = :w") values[":w"] = int(context_window) - update_expression = "ADD totalCost :c SET " + ", ".join(update_parts_set) + # Per-session cache-efficiency rollups (next to totalCost). Zero + # deltas are added unconditionally so the attributes exist (as 0) + # from the session's first call β€” simpler consumers, no sparse-field + # handling. + cache_read = token_usage.cache_read_input_tokens or 0 if token_usage else 0 + cache_write = token_usage.cache_write_input_tokens or 0 if token_usage else 0 + observability = cache_observability or {} + is_avoidable_miss = observability.get("cacheStatus") == "miss_avoidable" + wasted_usd = observability.get("wastedUsd") or 0.0 + + update_parts_add = [ + "totalCost :c", + "totalCacheReadTokens :cacheRead", + "totalCacheWriteTokens :cacheWrite", + "avoidableMissCount :avoidableMiss", + "wastedUsd :wasted", + ] + values[":cacheRead"] = int(cache_read) + values[":cacheWrite"] = int(cache_write) + values[":avoidableMiss"] = 1 if is_avoidable_miss else 0 + # wastedUsd comes from our own compute_wasted_usd (finite, rounded), + # but coerce defensively β€” a bad value must not break the bump. + values[":wasted"] = Decimal(str(_coerce_cost_total(wasted_usd))) + + update_expression = ( + "ADD " + ", ".join(update_parts_add) + " SET " + ", ".join(update_parts_set) + ) table.update_item( Key={"PK": f"USER#{user_id}", "SK": sk}, diff --git a/backend/src/apis/shared/skills/repository.py b/backend/src/apis/shared/skills/repository.py index 6a83bb3f7..66bb9d688 100644 --- a/backend/src/apis/shared/skills/repository.py +++ b/backend/src/apis/shared/skills/repository.py @@ -319,7 +319,11 @@ async def batch_get_skills( skill_ids: List of skill identifiers Returns: - List of SkillDefinition objects (may be shorter if some not found) + List of SkillDefinition objects (may be shorter if some not found), + sorted by skill_id. DynamoDB batch_get_item response order is + arbitrary; these records feed the system-prompt + block, and an order flip between turns invalidates the Bedrock + prompt cache (exact-prefix match). """ if not skill_ids: return [] @@ -342,7 +346,7 @@ async def batch_get_skills( [SkillDefinition.from_dynamo_item(item) for item in items] ) - return skills + return sorted(skills, key=lambda s: s.skill_id) except ClientError as e: logger.error(f"Error batch getting skills: {e}") diff --git a/backend/src/apis/shared/storage/dynamodb_storage.py b/backend/src/apis/shared/storage/dynamodb_storage.py index 3206c7227..077a8dffd 100644 --- a/backend/src/apis/shared/storage/dynamodb_storage.py +++ b/backend/src/apis/shared/storage/dynamodb_storage.py @@ -276,6 +276,54 @@ async def get_session_metadata( except ClientError as e: raise Exception(f"Failed to get session metadata: {e}") + async def get_session_cost_records( + self, + session_id: str + ) -> List[Dict[str, Any]]: + """ + Get ALL cost records for a session, chronologically, without a user + filter β€” the admin cost-anatomy view inspects any user's session. + Non-admin surfaces must keep using ``get_session_metadata`` (which + filters by userId). + + Schema: + GSI: SessionLookupIndex + GSI_PK: SESSION# + GSI_SK: begins_with C# (C# β€” chronological sort) + """ + from boto3.dynamodb.conditions import Key + + try: + items: List[Dict[str, Any]] = [] + last_evaluated_key = None + while True: + query_kwargs = { + "IndexName": "SessionLookupIndex", + "KeyConditionExpression": ( + Key("GSI_PK").eq(f"SESSION#{session_id}") + & Key("GSI_SK").begins_with("C#") + ), + "ScanIndexForward": True, + } + if last_evaluated_key: + query_kwargs["ExclusiveStartKey"] = last_evaluated_key + response = self.sessions_metadata_table.query(**query_kwargs) + items.extend(response.get("Items", [])) + last_evaluated_key = response.get("LastEvaluatedKey") + if not last_evaluated_key: + break + + results = [] + for item in items: + item_float = self._convert_decimal_to_float(item) + for key in ["PK", "SK", "GSI_PK", "GSI_SK", "GSI1PK", "GSI1SK", "ttl"]: + item_float.pop(key, None) + results.append(item_float) + return results + + except ClientError as e: + raise Exception(f"Failed to get session cost records: {e}") + async def get_user_cost_summary( self, user_id: str, diff --git a/backend/tests/agents/main_agent/core/test_bedrock_cache_points.py b/backend/tests/agents/main_agent/core/test_bedrock_cache_points.py new file mode 100644 index 000000000..4295cd3ac --- /dev/null +++ b/backend/tests/agents/main_agent/core/test_bedrock_cache_points.py @@ -0,0 +1,217 @@ +""" +Bedrock prompt-cache resilience β€” three cachePoints per request. + +A message-level cache lookup can miss structurally: Anthropic's cache lookback +checks only ~20 content blocks behind the breakpoint, so a wide parallel tool +fan-out pushes the previous checkpoint out of range (prod session aecd387d: +cacheRead=0, cacheWrite=134k mid-turn). Dedicated cachePoints on toolConfig and +the system prompt keep the stable prefix readable from cache on those turns. + +Contract under test (see ModelConfig.to_bedrock_config comment): + 1. toolConfig.tools tail β€” via cache_tools="default" + 2. system tail β€” via SystemContentBlock list from AgentFactory + 3. last user message tail β€” via CacheConfig(strategy="auto") +Bedrock allows max 4 cachePoints per request; nothing else may add one, so the +formatted request must contain exactly 3. +""" + +import json +from unittest.mock import MagicMock, patch + +import pytest + +from agents.main_agent.core.model_config import ModelConfig, ModelProvider + +CLAUDE_MODEL_ID = "us.anthropic.claude-haiku-4-5-20251001-v1:0" + + +def _count_cache_points(node) -> int: + """Count every cachePoint block anywhere in a formatted request.""" + if isinstance(node, dict): + return sum(_count_cache_points(v) for v in node.values()) + ( + 1 if "cachePoint" in node else 0 + ) + if isinstance(node, list): + return sum(_count_cache_points(item) for item in node) + return 0 + + +# --------------------------------------------------------------------------- +# ModelConfig: cache_tools + support predicate +# --------------------------------------------------------------------------- +class TestCacheToolsConfig: + def test_cache_tools_set_for_claude_with_caching(self): + config = ModelConfig(model_id=CLAUDE_MODEL_ID, caching_enabled=True) + assert config.to_bedrock_config()["cache_tools"] == "default" + + def test_no_cache_tools_when_caching_disabled(self): + config = ModelConfig(model_id=CLAUDE_MODEL_ID, caching_enabled=False) + assert "cache_tools" not in config.to_bedrock_config() + + def test_no_cache_tools_for_non_anthropic_bedrock_model(self): + """A model Strands' auto strategy would no-op on must not get explicit + cachePoints either β€” Bedrock would reject them with ValidationException.""" + config = ModelConfig(model_id="amazon.nova-pro-v1:0", caching_enabled=True) + assert "cache_tools" not in config.to_bedrock_config() + + def test_support_predicate_false_for_non_bedrock_provider(self): + config = ModelConfig( + model_id="gpt-4o", provider=ModelProvider.OPENAI, caching_enabled=True + ) + assert config.bedrock_cache_points_supported() is False + + +# --------------------------------------------------------------------------- +# AgentFactory: system prompt wrapped as SystemContentBlock list +# --------------------------------------------------------------------------- +class TestFactorySystemPromptCachePoint: + @patch("agents.main_agent.core.agent_factory.Agent") + @patch("agents.main_agent.core.agent_factory.CountTokensBedrockModel") + def test_system_prompt_gets_trailing_cache_point(self, _mock_model, mock_agent_cls): + from agents.main_agent.core.agent_factory import AgentFactory + + AgentFactory.create_agent( + model_config=ModelConfig(model_id=CLAUDE_MODEL_ID, caching_enabled=True), + system_prompt="You are a helpful assistant.", + tools=[], + session_manager=MagicMock(), + ) + + assert mock_agent_cls.call_args.kwargs["system_prompt"] == [ + {"text": "You are a helpful assistant."}, + {"cachePoint": {"type": "default"}}, + ] + + @patch("agents.main_agent.core.agent_factory.Agent") + @patch("agents.main_agent.core.agent_factory.CountTokensBedrockModel") + def test_plain_string_when_caching_disabled(self, _mock_model, mock_agent_cls): + from agents.main_agent.core.agent_factory import AgentFactory + + AgentFactory.create_agent( + model_config=ModelConfig(model_id=CLAUDE_MODEL_ID, caching_enabled=False), + system_prompt="You are a helpful assistant.", + tools=[], + session_manager=MagicMock(), + ) + + assert ( + mock_agent_cls.call_args.kwargs["system_prompt"] + == "You are a helpful assistant." + ) + + @patch("agents.main_agent.core.agent_factory.Agent") + @patch("agents.main_agent.core.agent_factory.CountTokensBedrockModel") + def test_empty_prompt_never_wrapped(self, _mock_model, mock_agent_cls): + """Bedrock rejects a cachePoint with no preceding content.""" + from agents.main_agent.core.agent_factory import AgentFactory + + AgentFactory.create_agent( + model_config=ModelConfig(model_id=CLAUDE_MODEL_ID, caching_enabled=True), + system_prompt="", + tools=[], + session_manager=MagicMock(), + ) + + assert mock_agent_cls.call_args.kwargs["system_prompt"] == "" + + +# --------------------------------------------------------------------------- +# End-to-end: the formatted ConverseStream request +# --------------------------------------------------------------------------- +class TestFormattedRequestCachePoints: + @pytest.fixture + def model(self, monkeypatch): + """Real Strands BedrockModel built from our production config path. + + boto3 client construction needs a region but no credentials, and + format_request never touches the network. + """ + monkeypatch.setenv("AWS_REGION", "us-west-2") + from agents.main_agent.core.bedrock_count_tokens import CountTokensBedrockModel + + config = ModelConfig(model_id=CLAUDE_MODEL_ID, caching_enabled=True) + return CountTokensBedrockModel(**config.to_bedrock_config()) + + @pytest.fixture + def request_parts(self, model): + """Format a fan-out-shaped conversation and return the request.""" + system_prompt_content = [ + {"text": "You are a helpful assistant."}, + {"cachePoint": {"type": "default"}}, + ] + tool_specs = [ + { + "name": "get_thread", + "description": "Fetch a thread", + "inputSchema": {"json": {"type": "object", "properties": {}}}, + } + ] + messages = [ + { + "role": "user", + # Stale message-level point from the previous turn β€” auto + # strategy must strip it (it manages message points itself). + "content": [{"text": "first turn"}, {"cachePoint": {"type": "default"}}], + }, + { + "role": "assistant", + "content": [ + {"toolUse": {"toolUseId": "t1", "name": "get_thread", "input": {}}} + ], + }, + { + "role": "user", + "content": [ + { + "toolResult": { + "toolUseId": "t1", + "content": [{"text": "thread body"}], + "status": "success", + } + } + ], + }, + ] + return model.format_request( + messages, tool_specs, system_prompt_content=system_prompt_content + ) + + def test_tool_config_tail_is_cache_point(self, request_parts): + assert request_parts["toolConfig"]["tools"][-1] == { + "cachePoint": {"type": "default"} + } + + def test_system_tail_is_cache_point(self, request_parts): + assert request_parts["system"][-1] == {"cachePoint": {"type": "default"}} + # Strands' auto strategy strips only message-level points β€” the + # system point must survive. + assert request_parts["system"][0] == {"text": "You are a helpful assistant."} + + def test_last_user_message_tail_is_cache_point(self, request_parts): + last_user = [m for m in request_parts["messages"] if m["role"] == "user"][-1] + assert last_user["content"][-1] == {"cachePoint": {"type": "default"}} + + def test_stale_message_cache_point_stripped(self, request_parts): + first_msg_blocks = request_parts["messages"][0]["content"] + assert all("cachePoint" not in block for block in first_msg_blocks) + + def test_exactly_three_cache_points_total(self, request_parts): + """Bedrock's hard limit is 4 cachePoints per request; we budget 3 + (tools, system, auto message point). If this fails at >3, something + new started adding cachePoints β€” rebalance the budget before shipping.""" + assert _count_cache_points(request_parts) == 3, json.dumps( + request_parts, default=str, indent=2 + ) + + def test_caching_disabled_yields_zero_cache_points(self, monkeypatch): + monkeypatch.setenv("AWS_REGION", "us-west-2") + from agents.main_agent.core.bedrock_count_tokens import CountTokensBedrockModel + + config = ModelConfig(model_id=CLAUDE_MODEL_ID, caching_enabled=False) + model = CountTokensBedrockModel(**config.to_bedrock_config()) + request = model.format_request( + [{"role": "user", "content": [{"text": "hi"}]}], + None, + system_prompt_content=[{"text": "You are a helpful assistant."}], + ) + assert _count_cache_points(request) == 0 diff --git a/backend/tests/agents/main_agent/session/test_compaction_models.py b/backend/tests/agents/main_agent/session/test_compaction_models.py index 20f6ce634..898233625 100644 --- a/backend/tests/agents/main_agent/session/test_compaction_models.py +++ b/backend/tests/agents/main_agent/session/test_compaction_models.py @@ -49,6 +49,7 @@ def test_to_dict_has_camel_case_keys(self): "lastInputTokens", "updatedAt", "totalSummarizedTurns", + "truncationAnchor", } def test_to_dict_values_match(self): @@ -99,6 +100,17 @@ def test_from_dict_partial_data_fills_defaults(self): assert state.last_input_tokens == 0 assert state.updated_at is None + def test_from_dict_roundtrips_truncation_anchor(self): + state = CompactionState.from_dict({"checkpoint": 3, "truncationAnchor": 9}) + assert state.truncation_anchor == 9 + + def test_from_dict_legacy_record_defaults_anchor_to_checkpoint(self): + """Records written before the anchor existed must not re-truncate + retained history: the anchor defaults to the checkpoint, so nothing + the slice keeps is mutated on restore.""" + state = CompactionState.from_dict({"checkpoint": 7}) + assert state.truncation_anchor == 7 + # --------------------------------------------------------------------------- # CompactionState.from_dict with None or empty dict (Req 16.4) diff --git a/backend/tests/agents/main_agent/session/test_compaction_stability.py b/backend/tests/agents/main_agent/session/test_compaction_stability.py new file mode 100644 index 000000000..bc69080bd --- /dev/null +++ b/backend/tests/agents/main_agent/session/test_compaction_stability.py @@ -0,0 +1,342 @@ +"""Byte-stability tests for restore-time compaction. + +Bedrock prompt caching requires an exact prefix match, so +``TurnBasedSessionManager.initialize()`` must derive the restored history as +a pure function of (stored messages, persisted compaction state). The old +design truncated tool contents behind a sliding protected-turns window, +which re-mutated the turn that just aged past the window on every restore β€” +breaking the cached prefix and forcing a full prefix re-write (~$2.5/MTok on +a 35k–150k prefix) nearly every turn (observed in prod session aecd387d: +-382/-1035/-1513 inter-turn prefix-token shrinkages with cacheRead=0 well +inside the cache TTL). + +These tests pin the redesign: truncation is driven only by the persisted +``truncation_anchor``, which moves at checkpoint advances (where the slice +already pays the one cache re-write) or opportunistically when the prompt +cache has already expired between turns. +""" + +import copy +import json +from datetime import datetime, timedelta, timezone +from unittest.mock import MagicMock + +import pytest + +from agents.main_agent.session.compaction_models import CompactionConfig, CompactionState +from agents.main_agent.session.turn_based_session_manager import TurnBasedSessionManager + +from .conftest import ( + make_user_message, + make_assistant_message, + make_tool_use_message, + make_tool_result_message, +) + + +LONG_RESULT = "R" * 400 # well above the fixture's max_tool_content_length=50 + + +def make_tool_turn(i: int) -> list: + """One 4-message turn with a tool result long enough to be truncatable.""" + return [ + make_user_message(f"Question {i}"), + make_tool_use_message(f"t{i}", "search", {"q": f"query {i}"}), + make_tool_result_message(f"t{i}", LONG_RESULT), + make_assistant_message(f"Answer {i}"), + ] + + +def make_tool_conversation(turns: int) -> list: + messages = [] + for i in range(turns): + messages.extend(make_tool_turn(i)) + return messages + + +def _now() -> datetime: + return datetime.now(timezone.utc) + + +def _iso(dt: datetime) -> str: + return dt.isoformat() + + +def _dump(messages) -> str: + return json.dumps(messages, sort_keys=True) + + +def _fresh_state(**overrides) -> dict: + """Persisted state stamped 'just now' β€” the prompt cache is still warm.""" + return {"compaction": CompactionState(updated_at=_iso(_now()), **overrides).to_dict()} + + +def _stale_state(age_seconds: int = 600, **overrides) -> dict: + """Persisted state older than the cache TTL β€” the cache has expired.""" + return { + "compaction": CompactionState( + updated_at=_iso(_now() - timedelta(seconds=age_seconds)), **overrides + ).to_dict() + } + + +@pytest.fixture +def restore_session(make_session_manager, compaction_config): + """Simulate one full session restore against an in-memory state store. + + Each call builds a fresh manager (as a new container invocation would), + wires ``_load/_save_compaction_state`` to ``state_store`` so persistence + behaves like the sessions-metadata record, and runs the real + ``initialize()`` restore path (sanitize + compaction + repair). Returns + the derived ``agent.messages``. + """ + + def _restore(state_store: dict, stored_messages: list): + mgr = make_session_manager(compaction_config=compaction_config) + mgr._load_compaction_state = lambda: CompactionState.from_dict( + state_store.get("compaction") + ) + + def _save(state: CompactionState) -> None: + state.updated_at = _iso(_now()) + state_store["compaction"] = state.to_dict() + + mgr._save_compaction_state = _save + mgr._retrieve_session_summaries = lambda: [] + + session_agent = MagicMock() + session_agent.state = {} + session_agent.conversation_manager_state = {} + session_messages = [] + for msg in copy.deepcopy(stored_messages): + sm = MagicMock() + sm.to_message.return_value = msg + session_messages.append(sm) + + mgr.read_agent = MagicMock(return_value=session_agent) + mgr.list_messages = MagicMock(return_value=session_messages) + mgr._is_new_session = False + + agent = MagicMock() + agent.agent_id = "default" + agent.messages = [] + agent.conversation_manager.restore_from_session.return_value = [] + agent.conversation_manager.removed_message_count = 0 + + mgr.initialize(agent) + return agent.messages + + return _restore + + +@pytest.fixture +def make_wired_manager(make_session_manager, compaction_config): + """Manager wired to an in-memory state store, for update_after_turn tests.""" + + def _make(state_store: dict): + mgr = make_session_manager(compaction_config=compaction_config) + mgr._load_compaction_state = lambda: CompactionState.from_dict( + state_store.get("compaction") + ) + + def _save(state: CompactionState) -> None: + state.updated_at = _iso(_now()) + state_store["compaction"] = state.to_dict() + + mgr._save_compaction_state = _save + mgr._retrieve_session_summaries = lambda: [] + return mgr + + return _make + + +class TestByteStability: + """agent.messages must be byte-identical across consecutive restores + when no compaction-state change occurs.""" + + def test_consecutive_restores_identical_no_state(self, restore_session): + stored = make_tool_conversation(6) + store = {} + first = restore_session(store, stored) + second = restore_session(store, stored) + assert _dump(first) == _dump(second) + # Nothing may be truncated: no anchor has ever been set. + assert "[truncated" not in _dump(first) + + def test_consecutive_restores_identical_warm_cache(self, restore_session): + stored = make_tool_conversation(8) + store = _fresh_state() + first = restore_session(store, stored) + second = restore_session(store, stored) + assert _dump(first) == _dump(second) + assert "[truncated" not in _dump(first) + + def test_prefix_stable_as_turns_accumulate(self, restore_session): + """The regression the sliding window caused: appending a new turn must + not mutate any earlier message. Under the old design the turn that + aged past protected_turns was newly truncated here, breaking the + cached prefix on every turn.""" + stored = make_tool_conversation(6) + before = restore_session(_fresh_state(), stored) + + stored_next = stored + make_tool_turn(6) + # A real turn refreshes updated_at within the TTL. + after = restore_session(_fresh_state(), stored_next) + + assert _dump(after[: len(before)]) == _dump(before) + + def test_checkpointed_restore_is_stable(self, restore_session): + """With a persisted checkpoint + summary, consecutive restores still + derive the identical history (summary prepend is deterministic).""" + stored = make_tool_conversation(8) + checkpoint = 5 * 4 # start of turn 5 + store = _fresh_state( + checkpoint=checkpoint, + truncation_anchor=checkpoint, + summary="Earlier discussion about queries 0-4.", + ) + first = restore_session(store, stored) + second = restore_session(store, stored) + assert _dump(first) == _dump(second) + assert len(first) == len(stored) - checkpoint + assert "" in first[0]["content"][0]["text"] + assert "[truncated" not in _dump(first) + + def test_legacy_state_without_anchor_is_stable_and_untruncated(self, restore_session): + """Records written before truncationAnchor existed default the anchor + to the checkpoint: retained history is never truncated.""" + stored = make_tool_conversation(8) + legacy = { + "checkpoint": 20, + "summary": "Legacy summary", + "lastInputTokens": 120_000, + "updatedAt": _iso(_now()), + "totalSummarizedTurns": 5, + } + store = {"compaction": legacy} + first = restore_session(store, stored) + second = restore_session(store, stored) + assert _dump(first) == _dump(second) + assert "[truncated" not in _dump(first) + + +class TestAnchorTruncation: + def test_truncation_applies_only_below_anchor(self, restore_session): + stored = make_tool_conversation(6) + anchor = 3 * 4 # start of turn 3 + store = _fresh_state(truncation_anchor=anchor) + derived = restore_session(store, stored) + + assert "[truncated" in _dump(derived[:anchor]) + assert "[truncated" not in _dump(derived[anchor:]) + # And the derivation is stable. + assert _dump(restore_session(store, stored)) == _dump(derived) + + def test_anchor_beyond_history_is_clamped(self, restore_session): + stored = make_tool_conversation(2) + store = _fresh_state(truncation_anchor=999) + derived = restore_session(store, stored) + assert len(derived) == len(stored) + assert "[truncated" in _dump(derived) + + +class TestOpportunisticAnchorAdvance: + def test_expired_cache_advances_and_persists_anchor(self, restore_session): + """When the prompt cache has already expired (>TTL since the previous + turn) the anchor slides to the protected-turns boundary for free and + is persisted, so the very next restore derives identical history.""" + stored = make_tool_conversation(8) + store = _stale_state(age_seconds=600) + first = restore_session(store, stored) + + expected_anchor = (8 - 3) * 4 # cutoffs[-3] = start of turn 5 + assert store["compaction"]["truncationAnchor"] == expected_anchor + assert "[truncated" in _dump(first[:expected_anchor]) + assert "[truncated" not in _dump(first[expected_anchor:]) + + # The save stamped updated_at=now, so the follow-up restore is inside + # the TTL: no further movement, byte-identical output. + second = restore_session(store, stored) + assert store["compaction"]["truncationAnchor"] == expected_anchor + assert _dump(first) == _dump(second) + + def test_warm_cache_never_advances_anchor(self, restore_session): + stored = make_tool_conversation(8) + store = _fresh_state() + restore_session(store, stored) + assert store["compaction"]["truncationAnchor"] == 0 + + def test_no_advance_when_too_few_turns(self, restore_session): + stored = make_tool_conversation(3) + store = _stale_state(age_seconds=600) + derived = restore_session(store, stored) + assert store["compaction"]["truncationAnchor"] == 0 + assert "[truncated" not in _dump(derived) + + def test_missing_updated_at_treated_as_warm(self, restore_session): + stored = make_tool_conversation(8) + store = {"compaction": CompactionState().to_dict()} # updated_at=None + restore_session(store, stored) + assert store["compaction"]["truncationAnchor"] == 0 + + +class TestCacheWindowExpired: + def test_none_and_garbage_are_not_expired(self): + assert TurnBasedSessionManager._cache_window_expired(None, 300) is False + assert TurnBasedSessionManager._cache_window_expired("not-a-timestamp", 300) is False + + def test_z_suffix_and_naive_timestamps_parse(self): + old = (_now() - timedelta(seconds=900)).strftime("%Y-%m-%dT%H:%M:%S") + assert TurnBasedSessionManager._cache_window_expired(old + "Z", 300) is True + assert TurnBasedSessionManager._cache_window_expired(old, 300) is True # naive β†’ UTC + + def test_recent_timestamp_not_expired(self): + assert TurnBasedSessionManager._cache_window_expired(_iso(_now()), 300) is False + + +class TestCheckpointAdvanceMovesAnchor: + @pytest.mark.asyncio + async def test_update_after_turn_sets_anchor_with_checkpoint( + self, make_wired_manager, restore_session + ): + stored = make_tool_conversation(8) + store = _fresh_state() + mgr = make_wired_manager(store) + + result = await mgr.update_after_turn(input_tokens=5000, current_messages=stored) + + expected_checkpoint = (8 - 3) * 4 + assert result is not None + assert result.new_checkpoint == expected_checkpoint + assert mgr.compaction_state.checkpoint == expected_checkpoint + assert mgr.compaction_state.truncation_anchor == expected_checkpoint + assert store["compaction"]["truncationAnchor"] == expected_checkpoint + + # Post-advance restores are byte-stable and truncate nothing the + # slice retained (anchor == checkpoint). + first = restore_session(store, stored) + second = restore_session(store, stored) + assert _dump(first) == _dump(second) + assert "[truncated" not in _dump(first) + + @pytest.mark.asyncio + async def test_anchor_never_regresses_below_prior_anchor(self, make_wired_manager): + stored = make_tool_conversation(8) + anchor = (8 - 3) * 4 + 4 # ahead of where the checkpoint will land + store = _fresh_state(truncation_anchor=anchor) + mgr = make_wired_manager(store) + + result = await mgr.update_after_turn(input_tokens=5000, current_messages=stored) + assert result is not None + assert mgr.compaction_state.truncation_anchor == anchor + + @pytest.mark.asyncio + async def test_below_threshold_saves_state_without_movement(self, make_wired_manager): + stored = make_tool_conversation(8) + store = _fresh_state(truncation_anchor=8, checkpoint=4) + mgr = make_wired_manager(store) + + result = await mgr.update_after_turn(input_tokens=10, current_messages=stored) + assert result is None + assert store["compaction"]["checkpoint"] == 4 + assert store["compaction"]["truncationAnchor"] == 8 diff --git a/backend/tests/agents/main_agent/session/test_prefix_fingerprint_hook.py b/backend/tests/agents/main_agent/session/test_prefix_fingerprint_hook.py new file mode 100644 index 000000000..f4e1471c3 --- /dev/null +++ b/backend/tests/agents/main_agent/session/test_prefix_fingerprint_hook.py @@ -0,0 +1,126 @@ +"""PrefixFingerprintHook β€” per-model-call prompt-cache prefix hashes.""" + +from types import SimpleNamespace + +from agents.main_agent.session.hooks.prefix_fingerprint import ( + PrefixFingerprintHook, + get_prefix_fingerprint, + reset_prefix_fingerprints, +) + + +def _agent(tool_specs=None, system_prompt="You are helpful.", messages=None): + return SimpleNamespace( + tool_registry=SimpleNamespace( + get_all_tool_specs=lambda: tool_specs if tool_specs is not None else [] + ), + system_prompt=system_prompt, + messages=messages if messages is not None else [], + ) + + +def _fire(agent): + PrefixFingerprintHook()._on_before_model_call(SimpleNamespace(agent=agent)) + + +def _msg(role, text): + return {"role": role, "content": [{"text": text}]} + + +class TestFingerprintCapture: + def test_appends_one_entry_per_model_call(self): + agent = _agent(messages=[_msg("user", "hi")]) + reset_prefix_fingerprints(agent) + _fire(agent) + agent.messages.append(_msg("assistant", "hello")) + agent.messages.append(_msg("user", "more")) + _fire(agent) + + first = get_prefix_fingerprint(agent, 0) + second = get_prefix_fingerprint(agent, 1) + assert first is not None and second is not None + assert first["messageCount"] == 1 + assert second["messageCount"] == 3 + # Latest-entry fallback used by single-call persistence paths. + assert get_prefix_fingerprint(agent, None) == second + + def test_fingerprint_shape(self): + agent = _agent(tool_specs=[{"name": "t1"}], messages=[_msg("user", "hi")]) + reset_prefix_fingerprints(agent) + _fire(agent) + fp = get_prefix_fingerprint(agent, 0) + assert set(fp) == { + "toolConfigHash", + "systemPromptHash", + "historyHash", + "messageCount", + } + assert all(len(fp[k]) == 16 for k in ("toolConfigHash", "systemPromptHash", "historyHash")) + + def test_history_hash_excludes_newest_message(self): + # Two calls whose histories share the same prefix but different + # newest message must produce the SAME historyHash: for a cache hit + # the prior prefix is what must match. + shared_prefix = [_msg("user", "hi"), _msg("assistant", "hello")] + a = _agent(messages=shared_prefix + [_msg("user", "question A")]) + b = _agent(messages=shared_prefix + [_msg("user", "question B")]) + reset_prefix_fingerprints(a) + reset_prefix_fingerprints(b) + _fire(a) + _fire(b) + assert ( + get_prefix_fingerprint(a, 0)["historyHash"] + == get_prefix_fingerprint(b, 0)["historyHash"] + ) + + def test_tool_order_flip_changes_tool_config_hash(self): + specs = [{"name": "alpha"}, {"name": "beta"}] + a = _agent(tool_specs=list(specs)) + b = _agent(tool_specs=list(reversed(specs))) + reset_prefix_fingerprints(a) + reset_prefix_fingerprints(b) + _fire(a) + _fire(b) + assert ( + get_prefix_fingerprint(a, 0)["toolConfigHash"] + != get_prefix_fingerprint(b, 0)["toolConfigHash"] + ) + + def test_structured_system_prompt_supported(self): + # AgentSkills' block-level injection can turn system_prompt into a + # list of SystemContentBlock dicts. + agent = _agent(system_prompt=[{"text": "base"}, {"text": ""}]) + reset_prefix_fingerprints(agent) + _fire(agent) + assert get_prefix_fingerprint(agent, 0)["systemPromptHash"] + + +class TestLifecycleAndSafety: + def test_reset_clears_previous_turn(self): + agent = _agent() + reset_prefix_fingerprints(agent) + _fire(agent) + reset_prefix_fingerprints(agent) + assert get_prefix_fingerprint(agent, 0) is None + assert get_prefix_fingerprint(agent, None) is None + + def test_out_of_range_index_returns_none(self): + agent = _agent() + reset_prefix_fingerprints(agent) + _fire(agent) + assert get_prefix_fingerprint(agent, 5) is None + + def test_never_raises_on_broken_agent(self): + class _Broken: + @property + def tool_registry(self): + raise RuntimeError("boom") + + broken = _Broken() + _fire(broken) # must swallow + assert get_prefix_fingerprint(broken, None) is None + + def test_works_without_prior_reset(self): + agent = _agent() + _fire(agent) + assert get_prefix_fingerprint(agent, 0) is not None diff --git a/backend/tests/agents/main_agent/session/test_turn_based_session_manager.py b/backend/tests/agents/main_agent/session/test_turn_based_session_manager.py index efd908e57..77e111550 100644 --- a/backend/tests/agents/main_agent/session/test_turn_based_session_manager.py +++ b/backend/tests/agents/main_agent/session/test_turn_based_session_manager.py @@ -130,33 +130,6 @@ def test_only_assistant_messages(self, make_session_manager): assert mgr._find_valid_cutoff_indices(messages) == [] -class TestFindProtectedIndices: - - def test_protect_last_2_turns(self, make_session_manager): - mgr = make_session_manager() - messages = make_conversation(4) # indices 0-7, turns at 0,2,4,6 - protected = mgr._find_protected_indices(messages, 2) - # Last 2 turn starts are at index 4 and 6, so protect 4..7 - assert protected == set(range(4, 8)) - - def test_zero_protected_turns(self, make_session_manager): - mgr = make_session_manager() - messages = make_conversation(3) - assert mgr._find_protected_indices(messages, 0) == set() - - def test_more_protected_than_available(self, make_session_manager): - mgr = make_session_manager() - messages = make_conversation(2) # 4 messages, 2 turns - protected = mgr._find_protected_indices(messages, 10) - # All messages protected since we only have 2 turns - assert protected == set(range(0, 4)) - - def test_no_valid_cutoffs(self, make_session_manager): - mgr = make_session_manager() - messages = [make_assistant_message("a1")] - assert mgr._find_protected_indices(messages, 2) == set() - - # =========================================================================== # Task 3 β€” Tool content truncation (Stage 1) # =========================================================================== @@ -589,19 +562,21 @@ def test_existing_agent_no_compaction_loads_all_messages(self, make_session_mana assert mgr.message_count == 6 def test_existing_agent_compaction_no_checkpoint(self, make_session_manager, compaction_config): - """Compaction enabled with checkpoint=0 should load all messages with truncation.""" + """Compaction enabled with checkpoint=0 loads all messages untouched: + with no truncation anchor set, even a long tool result is preserved + byte-for-byte (truncation is anchor-driven, never per-restore).""" mgr = make_session_manager(compaction_config=compaction_config) mgr._load_compaction_state = MagicMock(return_value=CompactionState()) - # A valid Converse history: the truncatable toolResult is preceded by - # its matching toolUse turn, so the restore-time pairing repair no-ops - # and this test exercises compaction/truncation in isolation. + # A valid Converse history: the toolResult is preceded by its matching + # toolUse turn, so the restore-time pairing repair no-ops and this + # test exercises compaction in isolation. messages = [ make_user_message("q1"), make_assistant_message("a1"), make_user_message("q2"), make_tool_use_message("t1", "search", {"q": "x"}), - make_tool_result_message("t1", "x" * 200), # will be truncated + make_tool_result_message("t1", "x" * 200), ] session_agent = self._make_mock_session_agent() mgr.read_agent = MagicMock(return_value=session_agent) @@ -611,8 +586,9 @@ def test_existing_agent_compaction_no_checkpoint(self, make_session_manager, com agent = self._make_mock_agent() mgr.initialize(agent) - # All 5 messages kept (checkpoint=0), but truncation applied + # All 5 messages kept (checkpoint=0) with no truncation (anchor=0) assert len(agent.messages) == 5 + assert agent.messages[4]["content"][0]["toolResult"]["content"][0]["text"] == "x" * 200 # Valid cutoffs cached for user text messages (indices 0, 2); the # toolResult user turn at index 4 is not a valid cutoff. assert mgr._valid_cutoff_indices == [0, 2] @@ -641,10 +617,12 @@ def test_existing_agent_compaction_with_checkpoint_slices_messages(self, make_se assert "" in first_text def test_existing_agent_compaction_checkpoint_plus_truncation(self, make_session_manager, compaction_config): - """Both checkpoint slicing and truncation should apply.""" + """Checkpoint slicing plus anchor-driven truncation: messages between + the checkpoint and the persisted truncation anchor are truncated; + messages at or after the anchor are preserved byte-for-byte.""" mgr = make_session_manager(compaction_config=compaction_config) mgr._load_compaction_state = MagicMock( - return_value=CompactionState(checkpoint=2) + return_value=CompactionState(checkpoint=2, truncation_anchor=5) ) # Valid Converse history: the truncatable toolResult follows its @@ -655,7 +633,11 @@ def test_existing_agent_compaction_checkpoint_plus_truncation(self, make_session make_assistant_message("old2"), make_user_message("new1"), make_tool_use_message("t1", "search", {"q": "r"}), - make_tool_result_message("t1", "r" * 200), # truncatable + make_tool_result_message("t1", "r" * 200), # below anchor: truncated + make_assistant_message("answer1"), + make_user_message("new2"), + make_tool_use_message("t2", "search", {"q": "s"}), + make_tool_result_message("t2", "s" * 200), # at/after anchor: preserved ] session_agent = self._make_mock_session_agent() mgr.read_agent = MagicMock(return_value=session_agent) @@ -665,8 +647,13 @@ def test_existing_agent_compaction_checkpoint_plus_truncation(self, make_session agent = self._make_mock_agent() mgr.initialize(agent) - # Sliced from index 2: [new1, toolUse, toolResult] = 3 messages remain - assert len(agent.messages) == 3 + # Sliced from index 2: 7 messages remain + assert len(agent.messages) == 7 + # Absolute index 4 (below anchor 5) truncated to max_tool_content_length=50 + truncated_text = agent.messages[2]["content"][0]["toolResult"]["content"][0]["text"] + assert "[truncated" in truncated_text + # Absolute index 8 (after anchor) untouched + assert agent.messages[6]["content"][0]["toolResult"]["content"][0]["text"] == "s" * 200 def test_duplicate_agent_id_raises(self, make_session_manager): """Second initialize with same agent_id should raise SessionException.""" diff --git a/backend/tests/agents/main_agent/skills/test_strands_mapping.py b/backend/tests/agents/main_agent/skills/test_strands_mapping.py index 1e2b427d9..85b7ff978 100644 --- a/backend/tests/agents/main_agent/skills/test_strands_mapping.py +++ b/backend/tests/agents/main_agent/skills/test_strands_mapping.py @@ -130,6 +130,28 @@ def test_returns_plugin_and_tool(self, monkeypatch): assert isinstance(plugin, AgentSkills) assert read_tool.tool_name == "read_skill_file" + def test_prompt_order_stable_across_fetch_order(self, monkeypatch): + # The plugin renders in the order it receives the + # skills; a between-turn order flip changes the system prompt and + # invalidates the Bedrock prompt cache. Regardless of the order the + # fetch returns records, the runtime must produce one canonical order. + recs = [ + _record(skill_id="zeta_skill"), + _record(skill_id="alpha_skill"), + _record(skill_id="midway_skill"), + ] + + def names_for(fetch_order): + monkeypatch.setattr( + sm, "fetch_active_skill_records", lambda ids: list(fetch_order) + ) + plugin, _ = sm.build_skills_runtime(["any"]) + return [s.name for s in plugin.get_available_skills()] + + forward = names_for(recs) + reversed_order = names_for(list(reversed(recs))) + assert forward == reversed_order == ["alpha-skill", "midway-skill", "zeta-skill"] + class TestReadSkillFile: def _tool(self, monkeypatch, records, store): diff --git a/backend/tests/agents/main_agent/test_prompt_cache_determinism.py b/backend/tests/agents/main_agent/test_prompt_cache_determinism.py new file mode 100644 index 000000000..b4f09e326 --- /dev/null +++ b/backend/tests/agents/main_agent/test_prompt_cache_determinism.py @@ -0,0 +1,173 @@ +"""CI determinism guard for the Bedrock prompt-cache prefix. + +Bedrock prompt caching is exact-prefix-match: if the system prompt or +toolConfig differs between two turns of the same session, the cache is +re-written at a premium. A prod audit (session aecd387d, $1.60) traced 75% +of spend to exactly that β€” skill records reaching the ```` +block in nondeterministic order, and RBAC grant unions materialized from +sets (hash-randomized iteration order). + +This guard builds the chat-agent surface twice from *shuffled* skill and +tool record orders and asserts the production prefix fingerprints +(``PrefixFingerprintHook`` β€” the same hashes persisted on every cost row) +are identical. It complements the targeted sorting fixes in +``apis.shared.skills.repository``, ``apis.shared.rbac.service``, and +``agents.main_agent.skills.strands_mapping``: any future assembly stage +that reintroduces order instability fails here. +""" + +import random +from types import SimpleNamespace +from unittest.mock import MagicMock + +import pytest +from strands import Agent, tool +from strands.hooks import BeforeInvocationEvent + +from agents.main_agent.session.hooks.prefix_fingerprint import ( + PrefixFingerprintHook, + get_prefix_fingerprint, + reset_prefix_fingerprints, +) +from agents.main_agent.skills import strands_mapping as sm +from apis.shared.rbac.service import AppRoleService + + +# --------------------------------------------------------------------------- +# Fixture data: skill records + local tools +# --------------------------------------------------------------------------- + +def _skill_record(skill_id: str) -> SimpleNamespace: + return SimpleNamespace( + skill_id=skill_id, + display_name=skill_id.replace("_", " ").title(), + description=f"Description for {skill_id}.", + instructions=f"# {skill_id}\nInstructions body.", + allowed_tools=[], + skill_metadata={}, + resources=[], + status="active", + ) + + +SKILL_RECORDS = [ + _skill_record("zeta_skill"), + _skill_record("alpha_skill"), + _skill_record("midway_skill"), + _skill_record("beta_skill"), +] + + +@tool +def alpha_tool(query: str) -> str: + """Alpha tool.""" + return query + + +@tool +def beta_tool(query: str) -> str: + """Beta tool.""" + return query + + +@tool +def gamma_tool(query: str) -> str: + """Gamma tool.""" + return query + + +TOOLS_BY_ID = {"alpha_tool": alpha_tool, "beta_tool": beta_tool, "gamma_tool": gamma_tool} + + +def _role(role_id: str, tools: list, skills: list, priority: int = 10) -> SimpleNamespace: + return SimpleNamespace( + role_id=role_id, + priority=priority, + effective_permissions=SimpleNamespace( + tools=tools, models=[], skills=skills, quota_tier=None + ), + ) + + +# --------------------------------------------------------------------------- +# Agent assembly mirroring the production pipeline stages under test: +# RBAC merge (grant union β†’ enabled ids) β†’ skills runtime (records β†’ +# AgentSkills plugin) β†’ Agent construction β†’ plugin system-prompt injection +# β†’ PrefixFingerprintHook (the persisted hashes). +# --------------------------------------------------------------------------- + +async def _build_and_fingerprint(monkeypatch, rng: random.Random) -> dict: + # RBAC seam: roles and their granted lists in shuffled order β€” the union + # used to be materialized from a set, so order leaked hash randomization. + role_a_tools = ["alpha_tool", "beta_tool"] + role_b_tools = ["gamma_tool", "beta_tool"] + role_skills = [r.skill_id for r in SKILL_RECORDS] + rng.shuffle(role_a_tools) + rng.shuffle(role_b_tools) + rng.shuffle(role_skills) + roles = [ + _role("role-a", role_a_tools, role_skills[:2]), + _role("role-b", role_b_tools, role_skills[2:]), + ] + rng.shuffle(roles) + merged = AppRoleService._merge_permissions(None, user_id="user-1", roles=roles) + + # Skills seam: fetch returns records in shuffled (nondeterministic) order. + shuffled_records = list(SKILL_RECORDS) + rng.shuffle(shuffled_records) + monkeypatch.setattr(sm, "fetch_active_skill_records", lambda ids: shuffled_records) + plugin, read_tool = sm.build_skills_runtime(merged.skills) + + tools = [TOOLS_BY_ID[tool_id] for tool_id in merged.tools if tool_id in TOOLS_BY_ID] + tools.append(read_tool) + + model = MagicMock() + model.config = {} + agent = Agent( + model=model, + system_prompt="You are the campus assistant.", + tools=tools, + plugins=[plugin], + ) + + # Drive the plugin's system-prompt injection the way a real invocation + # does, then capture the same fingerprints production persists per call. + await plugin._on_before_invocation(BeforeInvocationEvent(agent=agent)) + reset_prefix_fingerprints(agent) + PrefixFingerprintHook()._on_before_model_call(SimpleNamespace(agent=agent)) + + fingerprint = get_prefix_fingerprint(agent, 0) + assert fingerprint is not None, "fingerprint hook failed to capture" + return fingerprint + + +@pytest.mark.asyncio +async def test_agent_prefix_identical_across_shuffled_record_orders(monkeypatch): + fp_a = await _build_and_fingerprint(monkeypatch, random.Random(1234)) + fp_b = await _build_and_fingerprint(monkeypatch, random.Random(987654)) + + assert fp_a["systemPromptHash"] == fp_b["systemPromptHash"], ( + "System prompt differs between two builds from shuffled skill/role " + "record orders β€” a nondeterministic assembly stage is back and will " + "invalidate the Bedrock prompt cache every turn." + ) + assert fp_a["toolConfigHash"] == fp_b["toolConfigHash"], ( + "toolConfig differs between two builds from shuffled tool-grant " + "orders β€” a nondeterministic assembly stage is back and will " + "invalidate the Bedrock prompt cache every turn." + ) + + +@pytest.mark.asyncio +async def test_skills_xml_actually_present_in_fingerprinted_prompt(monkeypatch): + # Guard the guard: if AgentSkills ever stops injecting before the hook + # captures, the system-prompt assertion above would trivially pass. + monkeypatch.setattr(sm, "fetch_active_skill_records", lambda ids: list(SKILL_RECORDS)) + plugin, read_tool = sm.build_skills_runtime([r.skill_id for r in SKILL_RECORDS]) + model = MagicMock() + model.config = {} + agent = Agent(model=model, system_prompt="Base.", tools=[read_tool], plugins=[plugin]) + await plugin._on_before_invocation(BeforeInvocationEvent(agent=agent)) + prompt_text = str(agent.system_prompt) + assert "available_skills" in prompt_text + assert "alpha-skill" in prompt_text diff --git a/backend/tests/shared/test_metadata_cache_derivation.py b/backend/tests/shared/test_metadata_cache_derivation.py new file mode 100644 index 000000000..7f8da4f63 --- /dev/null +++ b/backend/tests/shared/test_metadata_cache_derivation.py @@ -0,0 +1,245 @@ +"""Write-time cache derivation in apis.shared.sessions.metadata: +``_derive_cache_observability`` (previous-row classification) and the +cache-efficiency counters ``_bump_session_aggregates`` adds to the session row. +""" + +from datetime import datetime, timedelta, timezone +from decimal import Decimal + +import pytest + +from apis.shared.sessions import metadata as md +from apis.shared.sessions.models import ( + MessageMetadata, + ModelInfo, + PricingSnapshot, + TokenUsage, +) + + +NOW = datetime(2026, 7, 19, 12, 0, 0, tzinfo=timezone.utc) + + +def _metadata(input_t=100, output_t=50, cache_read=0, cache_write=0, with_pricing=True): + pricing = None + if with_pricing: + pricing = PricingSnapshot( + input_price_per_mtok=3.0, + output_price_per_mtok=15.0, + cache_write_price_per_mtok=3.75, + cache_read_price_per_mtok=0.30, + snapshot_at=NOW.isoformat(), + ) + return MessageMetadata( + token_usage=TokenUsage( + input_tokens=input_t, + output_tokens=output_t, + total_tokens=input_t + output_t, + cache_read_input_tokens=cache_read, + cache_write_input_tokens=cache_write, + ), + model_info=ModelInfo( + model_id="us.anthropic.claude-sonnet-4-5-20250929-v1:0", + model_name="Claude Sonnet 4.5", + pricing_snapshot=pricing, + ), + ) + + +class _FakeTable: + """Fake DynamoDB table: query() serves the previous-cost-row lookup.""" + + def __init__(self, prev_items=None): + self._prev_items = prev_items or [] + self.query_kwargs = None + self.update_kwargs = None + + def query(self, **kwargs): + self.query_kwargs = kwargs + return {"Items": self._prev_items} + + def update_item(self, **kwargs): + self.update_kwargs = kwargs + return {} + + +def _prev_row(seconds_ago, cache_read=0, cache_write=0): + ts = (NOW - timedelta(seconds=seconds_ago)).isoformat() + return { + "timestamp": ts, + "tokenUsage": { + "cacheReadInputTokens": Decimal(cache_read), + "cacheWriteInputTokens": Decimal(cache_write), + }, + } + + +class TestDeriveCacheObservability: + def _derive(self, table, meta): + return md._derive_cache_observability( + session_id="sess-1", + table=table, + timestamp=NOW.isoformat(), + message_metadata=meta, + ) + + def test_no_token_usage_returns_empty(self): + meta = MessageMetadata(token_usage=None) + assert self._derive(_FakeTable(), meta) == {} + + def test_first_write_without_previous_row(self): + result = self._derive(_FakeTable(), _metadata(cache_write=5000)) + assert result["cacheStatus"] == "first_write" + assert result["wastedUsd"] == 0.0 + assert "cacheGapSeconds" not in result + + def test_hit_with_previous_row(self): + table = _FakeTable([_prev_row(seconds_ago=30, cache_write=5000)]) + result = self._derive(table, _metadata(cache_read=5000, cache_write=100)) + assert result["cacheStatus"] == "hit" + assert result["cacheGapSeconds"] == 30 + assert result["wastedUsd"] == 0.0 + + def test_avoidable_miss_within_ttl_prices_waste(self): + # Previous call cached 4000 + 1000 tokens; this call re-writes 6000 + # with zero read, 60s later β†’ avoidable. Waste = min(6000, 5000) + # tokens at the (3.75 - 0.30)/Mtok premium. + table = _FakeTable([_prev_row(seconds_ago=60, cache_read=4000, cache_write=1000)]) + result = self._derive(table, _metadata(cache_write=6000)) + assert result["cacheStatus"] == "miss_avoidable" + assert result["cacheGapSeconds"] == 60 + assert result["wastedUsd"] == round((5000 / 1_000_000) * 3.45, 6) + + def test_ttl_expired_miss_beyond_gap(self): + table = _FakeTable([_prev_row(seconds_ago=1200, cache_write=5000)]) + result = self._derive(table, _metadata(cache_write=6000)) + assert result["cacheStatus"] == "miss_ttl_expired" + assert result["wastedUsd"] == 0.0 + + def test_uncached_when_no_cache_activity(self): + table = _FakeTable([_prev_row(seconds_ago=30)]) + result = self._derive(table, _metadata()) + assert result["cacheStatus"] == "uncached" + + def test_first_write_after_below_threshold_uncached_call(self): + # Session whose prior calls were all below the minimum cacheable + # prefix: the previous row shows zero cache activity, so this call's + # write (crossing the threshold) is the expected first population β€” + # not miss_avoidable, and never priced as waste. + table = _FakeTable([_prev_row(seconds_ago=30)]) + result = self._derive(table, _metadata(cache_write=4122)) + assert result["cacheStatus"] == "first_write" + assert result["wastedUsd"] == 0.0 + assert result["cacheGapSeconds"] == 30 + + def test_query_targets_previous_row_descending_limit_1(self): + table = _FakeTable() + self._derive(table, _metadata(cache_write=100)) + assert table.query_kwargs["IndexName"] == "SessionLookupIndex" + assert table.query_kwargs["ScanIndexForward"] is False + assert table.query_kwargs["Limit"] == 1 + + def test_failure_returns_empty_never_raises(self): + class _Boom: + def query(self, **kwargs): + raise RuntimeError("dynamo down") + + assert self._derive(_Boom(), _metadata(cache_write=100)) == {} + + +class TestBumpSessionAggregatesCacheCounters: + @pytest.fixture + def session_lookup(self, monkeypatch): + async def _fake_get_session(session_id, user_id, table): + return {"SK": f"S#{session_id}"} + + monkeypatch.setattr(md, "_get_session_by_gsi", _fake_get_session) + + async def _bump(self, table, meta, observability): + await md._bump_session_aggregates( + session_id="sess-1", + user_id="user-1", + message_metadata=meta, + table=table, + cache_observability=observability, + ) + + @pytest.mark.asyncio + async def test_adds_cache_counters_next_to_total_cost(self, session_lookup): + table = _FakeTable() + meta = _metadata(cache_read=4000, cache_write=1000) + await self._bump( + table, meta, {"cacheStatus": "miss_avoidable", "wastedUsd": 0.01725} + ) + + expr = table.update_kwargs["UpdateExpression"] + values = table.update_kwargs["ExpressionAttributeValues"] + for attr in ( + "totalCost :c", + "totalCacheReadTokens :cacheRead", + "totalCacheWriteTokens :cacheWrite", + "avoidableMissCount :avoidableMiss", + "wastedUsd :wasted", + ): + assert attr in expr + assert values[":cacheRead"] == 4000 + assert values[":cacheWrite"] == 1000 + assert values[":avoidableMiss"] == 1 + assert values[":wasted"] == Decimal("0.01725") + + @pytest.mark.asyncio + async def test_zero_deltas_when_not_avoidable(self, session_lookup): + table = _FakeTable() + await self._bump(table, _metadata(cache_read=4000), {"cacheStatus": "hit", "wastedUsd": 0.0}) + values = table.update_kwargs["ExpressionAttributeValues"] + assert values[":avoidableMiss"] == 0 + assert values[":wasted"] == Decimal("0") + + @pytest.mark.asyncio + async def test_missing_observability_defaults_to_zeroes(self, session_lookup): + table = _FakeTable() + await self._bump(table, _metadata(), None) + values = table.update_kwargs["ExpressionAttributeValues"] + assert values[":cacheRead"] == 0 + assert values[":cacheWrite"] == 0 + assert values[":avoidableMiss"] == 0 + + +class TestKillSwitch: + """PROMPT_CACHE_OBSERVABILITY_ENABLED=false disables derivation and EMF.""" + + def test_flag_default_and_parsing(self, monkeypatch): + from apis.shared.observability import ( + PROMPT_CACHE_OBSERVABILITY_ENABLED_ENV, + prompt_cache_observability_enabled, + ) + + monkeypatch.delenv(PROMPT_CACHE_OBSERVABILITY_ENABLED_ENV, raising=False) + assert prompt_cache_observability_enabled() is True + # Empty string (workflow env vars can materialize as "") stays enabled + monkeypatch.setenv(PROMPT_CACHE_OBSERVABILITY_ENABLED_ENV, "") + assert prompt_cache_observability_enabled() is True + monkeypatch.setenv(PROMPT_CACHE_OBSERVABILITY_ENABLED_ENV, "FALSE") + assert prompt_cache_observability_enabled() is False + monkeypatch.setenv(PROMPT_CACHE_OBSERVABILITY_ENABLED_ENV, "true") + assert prompt_cache_observability_enabled() is True + + def test_derivation_disabled_returns_empty(self, monkeypatch): + monkeypatch.setenv("PROMPT_CACHE_OBSERVABILITY_ENABLED", "false") + table = _FakeTable([_prev_row(seconds_ago=30, cache_write=5000)]) + result = md._derive_cache_observability( + session_id="sess-1", + table=table, + timestamp=NOW.isoformat(), + message_metadata=_metadata(cache_read=5000, cache_write=100), + ) + assert result == {} + + def test_emf_disabled_emits_nothing(self, monkeypatch, capsys): + monkeypatch.setenv("PROMPT_CACHE_OBSERVABILITY_ENABLED", "false") + md._emit_cache_metrics( + session_id="sess-1", + message_metadata=_metadata(cache_read=1000, cache_write=200), + cache_observability={"cacheStatus": "miss_avoidable", "wastedUsd": 0.01}, + ) + assert capsys.readouterr().out == "" diff --git a/backend/tests/shared/test_prompt_cache_observability.py b/backend/tests/shared/test_prompt_cache_observability.py new file mode 100644 index 000000000..43523276e --- /dev/null +++ b/backend/tests/shared/test_prompt_cache_observability.py @@ -0,0 +1,235 @@ +"""Prompt-cache observability primitives: classification, waste pricing, +prefix fingerprints, and EMF record shape.""" + +import io +import json +import logging + +from apis.shared.observability import ( + CACHE_TTL_SECONDS, + CacheStatus, + classify_cache_status, + compute_wasted_usd, + emit_prompt_cache_metrics, + fingerprint_canonical_json, + fingerprint_text, +) + + +PRICING = { + "inputPricePerMtok": 3.0, + "outputPricePerMtok": 15.0, + "cacheWritePricePerMtok": 3.75, + "cacheReadPricePerMtok": 0.30, +} + + +class TestClassifyCacheStatus: + def test_first_write(self): + assert ( + classify_cache_status(0, 5000, previous_call_exists=False, gap_seconds=None) + is CacheStatus.FIRST_WRITE + ) + + def test_hit_when_any_cache_read(self): + assert ( + classify_cache_status(4000, 200, previous_call_exists=True, gap_seconds=10) + is CacheStatus.HIT + ) + + def test_hit_wins_even_without_previous_row(self): + # A read implies the cache was warm regardless of what rows we kept. + assert ( + classify_cache_status(4000, 0, previous_call_exists=False, gap_seconds=None) + is CacheStatus.HIT + ) + + def test_miss_avoidable_within_ttl(self): + assert ( + classify_cache_status(0, 5000, previous_call_exists=True, gap_seconds=45) + is CacheStatus.MISS_AVOIDABLE + ) + + def test_miss_ttl_expired_beyond_ttl(self): + assert ( + classify_cache_status( + 0, 5000, previous_call_exists=True, gap_seconds=CACHE_TTL_SECONDS + 1 + ) + is CacheStatus.MISS_TTL_EXPIRED + ) + + def test_boundary_gap_exactly_ttl_is_avoidable(self): + assert ( + classify_cache_status( + 0, 5000, previous_call_exists=True, gap_seconds=CACHE_TTL_SECONDS + ) + is CacheStatus.MISS_AVOIDABLE + ) + + def test_unknown_gap_is_conservatively_expired(self): + assert ( + classify_cache_status(0, 5000, previous_call_exists=True, gap_seconds=None) + is CacheStatus.MISS_TTL_EXPIRED + ) + + def test_uncached_when_no_cache_activity(self): + assert ( + classify_cache_status(0, 0, previous_call_exists=True, gap_seconds=10) + is CacheStatus.UNCACHED + ) + + def test_below_threshold_then_crossing_is_first_write(self): + # Prior calls were below the minimum cacheable prefix (uncached), so + # no cache entry existed β€” the first call that crosses the threshold + # is the expected initial population, not an avoidable miss. + assert ( + classify_cache_status( + 0, + 4122, + previous_call_exists=True, + gap_seconds=45, + previous_cached_prefix_tokens=0, + ) + is CacheStatus.FIRST_WRITE + ) + + def test_unknown_previous_prefix_stays_avoidable(self): + # None means we couldn't see the previous call's cache split β€” keep + # the pre-existing (avoidable) classification rather than masking. + assert ( + classify_cache_status( + 0, + 5000, + previous_call_exists=True, + gap_seconds=45, + previous_cached_prefix_tokens=None, + ) + is CacheStatus.MISS_AVOIDABLE + ) + + +class TestComputeWastedUsd: + def test_avoidable_miss_priced_at_write_read_premium(self): + # 1M re-written tokens, all previously cached β†’ full premium. + wasted = compute_wasted_usd( + CacheStatus.MISS_AVOIDABLE, + cache_write_tokens=1_000_000, + previous_cached_prefix_tokens=2_000_000, + pricing_snapshot=PRICING, + ) + assert wasted == (3.75 - 0.30) + + def test_rewritten_capped_at_previous_cached_prefix(self): + # Only 100k of the 1M written were cached before β€” the new suffix + # would have been written under a hit too, so it isn't waste. + wasted = compute_wasted_usd( + CacheStatus.MISS_AVOIDABLE, + cache_write_tokens=1_000_000, + previous_cached_prefix_tokens=100_000, + pricing_snapshot=PRICING, + ) + assert wasted == (100_000 / 1_000_000) * (3.75 - 0.30) + + def test_unknown_previous_prefix_uses_full_write(self): + wasted = compute_wasted_usd( + CacheStatus.MISS_AVOIDABLE, + cache_write_tokens=500_000, + previous_cached_prefix_tokens=None, + pricing_snapshot=PRICING, + ) + assert wasted == (500_000 / 1_000_000) * (3.75 - 0.30) + + def test_zero_for_non_avoidable_statuses(self): + for status in ( + CacheStatus.FIRST_WRITE, + CacheStatus.HIT, + CacheStatus.MISS_TTL_EXPIRED, + CacheStatus.UNCACHED, + ): + assert ( + compute_wasted_usd(status, 1_000_000, 1_000_000, PRICING) == 0.0 + ) + + def test_zero_without_pricing_or_with_none_prices(self): + assert compute_wasted_usd(CacheStatus.MISS_AVOIDABLE, 1000, 1000, None) == 0.0 + # Rows can store explicit None prices (managed models). + assert ( + compute_wasted_usd( + CacheStatus.MISS_AVOIDABLE, + 1000, + 1000, + {"cacheWritePricePerMtok": None, "cacheReadPricePerMtok": None}, + ) + == 0.0 + ) + + +class TestFingerprints: + def test_text_fingerprint_stable_and_short(self): + assert fingerprint_text("hello") == fingerprint_text("hello") + assert fingerprint_text("hello") != fingerprint_text("hello!") + assert len(fingerprint_text("hello")) == 16 + assert fingerprint_text(None) == fingerprint_text("") + + def test_dict_key_order_does_not_matter(self): + a = {"b": 1, "a": [1, 2]} + b = {"a": [1, 2], "b": 1} + assert fingerprint_canonical_json(a) == fingerprint_canonical_json(b) + + def test_list_order_matters(self): + # Deliberate: tool-spec / message order is what Bedrock's + # exact-prefix match is sensitive to. + assert fingerprint_canonical_json([1, 2]) != fingerprint_canonical_json([2, 1]) + + def test_non_json_values_do_not_raise(self): + assert isinstance(fingerprint_canonical_json({"x": object()}), str) + + +class TestEmfEmission: + def _capture(self, **kwargs): + from apis.shared.observability import emf + + stream = io.StringIO() + handler = logging.StreamHandler(stream) + handler.setFormatter(logging.Formatter("%(message)s")) + emf._emf_logger.addHandler(handler) + try: + emit_prompt_cache_metrics(**kwargs) + finally: + emf._emf_logger.removeHandler(handler) + return stream.getvalue().strip() + + def test_record_is_raw_json_with_emf_directive(self): + line = self._capture( + cache_read_tokens=1000, + cache_write_tokens=200, + avoidable_miss=True, + wasted_usd=0.0123456789, + model_id="us.anthropic.claude-sonnet-4-5-20250929-v1:0", + session_id="sess-1", + cache_status="miss_avoidable", + ) + record = json.loads(line) # the whole line must be the JSON object + directive = record["_aws"]["CloudWatchMetrics"][0] + metric_names = {m["Name"] for m in directive["Metrics"]} + assert metric_names == { + "CacheReadTokens", + "CacheWriteTokens", + "AvoidableMiss", + "WastedUsd", + } + assert record["CacheReadTokens"] == 1000 + assert record["CacheWriteTokens"] == 200 + assert record["AvoidableMiss"] == 1 + assert record["WastedUsd"] == 0.012346 # rounded to 6 places + assert record["modelId"].startswith("us.anthropic") + assert record["cacheStatus"] == "miss_avoidable" + + def test_no_miss_and_defaults(self): + line = self._capture( + cache_read_tokens=0, cache_write_tokens=0, avoidable_miss=False + ) + record = json.loads(line) + assert record["AvoidableMiss"] == 0 + assert record["WastedUsd"] == 0.0 + assert "modelId" not in record diff --git a/backend/tests/shared/test_skills_repository.py b/backend/tests/shared/test_skills_repository.py index 9c156b5fe..3e9a1e5d9 100644 --- a/backend/tests/shared/test_skills_repository.py +++ b/backend/tests/shared/test_skills_repository.py @@ -169,6 +169,31 @@ async def test_batch_get_skills(self, skill_repository): skills = await skill_repository.batch_get_skills(["skill_one", "skill_two", "ghost"]) assert {s.skill_id for s in skills} == {"skill_one", "skill_two"} + @pytest.mark.asyncio + async def test_batch_get_skills_sorted_regardless_of_response_order( + self, skill_repository, monkeypatch + ): + # DynamoDB batch_get_item response order is arbitrary; these records + # feed the system-prompt block, where an order flip + # between turns invalidates the Bedrock prompt cache. Force a + # descending response order and assert the repo still returns sorted. + for sid in ("zeta", "alpha", "midway"): + await skill_repository.create_skill(_make_skill(sid)) + + client = skill_repository._dynamodb.meta.client + real_batch_get = client.batch_get_item + + def descending_batch_get(**kwargs): + response = real_batch_get(**kwargs) + for items in response.get("Responses", {}).values(): + items.sort(key=lambda item: item["PK"], reverse=True) + return response + + monkeypatch.setattr(client, "batch_get_item", descending_batch_get) + + skills = await skill_repository.batch_get_skills(["zeta", "alpha", "midway"]) + assert [s.skill_id for s in skills] == ["alpha", "midway", "zeta"] + @pytest.mark.asyncio async def test_batch_get_empty(self, skill_repository): assert await skill_repository.batch_get_skills([]) == [] diff --git a/backend/uv.lock b/backend/uv.lock index 1c575a2fc..48a5cada1 100644 --- a/backend/uv.lock +++ b/backend/uv.lock @@ -12,7 +12,7 @@ resolution-markers = [ [[package]] name = "agentcore-stack" -version = "1.8.0" +version = "1.9.0" source = { editable = "." } dependencies = [ { name = "aiofiles" }, diff --git a/docs/kaizen/review-queue.md b/docs/kaizen/review-queue.md index 5f7eeecc9..c1dff5f71 100644 --- a/docs/kaizen/review-queue.md +++ b/docs/kaizen/review-queue.md @@ -5,6 +5,20 @@ Items added by `kaizen-research`, consumed by `kaizen-review-prep`. ## Open +### [2026-07-19] Track harness-sdk#3348 (rolling pair of message cachePoints) β€” local workaround gated on dashboard evidence +- **Source**: Phil-initiated (PR #697 follow-up) β€” https://github.com/strands-agents/harness-sdk/issues/3348 (filed by philmerrell, open, no maintainer response yet); prod session aecd387d (18-way parallel tool fan-out β†’ cacheRead=0 / cacheWrite=134k mid-turn, the ~20-block Anthropic lookback miss mode documented in `model_config.py:366`) +- **Surface**: backend (`agents/main_agent/core/model_config.py` 3-cachePoint budget). If built locally: strands 1.48's `_inject_cache_point` **strips any pre-existing message-level cachePoints**, so a rolling pair requires dropping `CacheConfig(strategy="auto")` and hand-placing both points via a hook β€” the 4th Bedrock cachePoint slot is free. Position tests in `tests/agents/main_agent/core/test_bedrock_cache_points.py` are the safety net. +- **Effort Γ— Impact**: (track) free Γ— M; (local workaround) M Γ— M β€” but high regression risk in the highest cost-of-regression area +- **Subtracts**: no β€” upstream-native fix preferred; a local workaround would later be deleted in favor of it +- **Status**: open β€” check #3348 each scan. Do NOT build the local workaround until the `AvoidableMiss`/`WastedUsd` dashboard (PR #697 follow-up, in flight) shows the fan-out miss mode fires often enough to earn the risk. + +### [2026-07-19] Spike: ContextOffloader adoption (ingestion-time tool-result offload β†’ S3) +- **Source**: Phil-initiated (PR #697 follow-up) β€” strands 1.48.0 ships `ContextOffloader` as a vended plugin (`strands/vended_plugins/context_offloader/`): hooks `AfterToolCallEvent`, token-gates results (default 2,500 via `model.count_tokens`), stores oversized blocks, rewrites to preview + reference, registers `retrieve_offloaded_content`. Relates to long-open issue #266 (large tool-result offload). +- **Surface**: backend (`agent_factory.py` β€” `plugins=` already plumbed; a custom S3 `Storage` backend is the real build β€” the plugin ships InMemory/File only, and references must survive AgentCore Runtime restores across turns) +- **Effort Γ— Impact**: M–H Γ— M–H (payoff directly measurable by the PR #697 cache/cost metrics) +- **Subtracts**: partial β€” bounds MCP/tool payload growth at the source instead of relying solely on reactive below-anchor truncation in `TurnBasedSessionManager` (which stays for legacy history) +- **Status**: open β€” spike before commitment; four known gotchas: (1) `evict_after_cycles=20` runs on `BeforeModelCallEvent` and touches *prior* messages β€” potential byte-stability cache-buster, verify semantics or set `None` and expire via S3 lifecycle; (2) `model.count_tokens` per tool result adds latency, and Bedrock CountTokens rejects `us.*` inference-profile ids (de-prefix precedent in context attribution); (3) adoption flips `toolConfigHash` once (expected; the new tool must land in the deterministically-ordered tool list); (4) check SPA tool-result rendering against placeholder content. + ### [2026-07-17] Bump `bedrock-agentcore` 1.9.1 β†’ 1.18.0 (closes #571 cross-process Memory reorder + #482 SSE deadlock) - **Source**: research/2026-07-17.md β–Έ Top 5 #1 β€” bedrock-agentcore 1.18.0 (Jul 10, PR #572/#573 close #571; #482 fix in 1.17.0/PR #563) β€” https://github.com/aws/bedrock-agentcore-sdk-python/releases - **Surface**: backend (`backend/pyproject.toml` + coupled `boto3`; inference-api chat router + `TurnBasedSessionManager` flush ordering; full local pytest suite) diff --git a/frontend/ai.client/package-lock.json b/frontend/ai.client/package-lock.json index 92fcfcfe6..0e6b860e5 100644 --- a/frontend/ai.client/package-lock.json +++ b/frontend/ai.client/package-lock.json @@ -1,12 +1,12 @@ { "name": "ai.client", - "version": "1.8.0", + "version": "1.9.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "ai.client", - "version": "1.8.0", + "version": "1.9.0", "dependencies": { "@angular/cdk": "21.2.14", "@angular/common": "21.2.17", diff --git a/frontend/ai.client/package.json b/frontend/ai.client/package.json index 359bc4d20..051419903 100644 --- a/frontend/ai.client/package.json +++ b/frontend/ai.client/package.json @@ -1,6 +1,6 @@ { "name": "ai.client", - "version": "1.8.0", + "version": "1.9.0", "scripts": { "ng": "ng", "start": "ng serve", diff --git a/frontend/ai.client/src/app/admin/admin.routes.ts b/frontend/ai.client/src/app/admin/admin.routes.ts index 8b95ca798..1596493d4 100644 --- a/frontend/ai.client/src/app/admin/admin.routes.ts +++ b/frontend/ai.client/src/app/admin/admin.routes.ts @@ -11,6 +11,10 @@ export const adminRoutes: Routes = [ path: 'costs', loadComponent: () => import('./costs/admin-costs.page').then(m => m.AdminCostsPage), }, + { + path: 'costs/sessions/:id', + loadComponent: () => import('./costs/pages/session-cost-anatomy.page').then(m => m.SessionCostAnatomyPage), + }, { path: 'quota', loadChildren: () => import('./quota-tiers/quota-routing.module').then(m => m.quotaRoutes), diff --git a/frontend/ai.client/src/app/admin/costs/admin-costs.page.ts b/frontend/ai.client/src/app/admin/costs/admin-costs.page.ts index 37acd5c97..3dc31f88d 100644 --- a/frontend/ai.client/src/app/admin/costs/admin-costs.page.ts +++ b/frontend/ai.client/src/app/admin/costs/admin-costs.page.ts @@ -7,10 +7,12 @@ import { signal, } from '@angular/core'; import { Router } from '@angular/router'; +import { FormsModule } from '@angular/forms'; import { NgIcon, provideIcons } from '@ng-icons/core'; import { heroArrowLeft, heroArrowDownTray, + heroMagnifyingGlass, } from '@ng-icons/heroicons/outline'; import { AdminCostStateService } from './services'; import { PeriodSelectorComponent } from './components/period-selector.component'; @@ -28,6 +30,7 @@ import { ModelBreakdownComponent } from './components/model-breakdown.component' @Component({ selector: 'app-admin-costs', imports: [ + FormsModule, NgIcon, PeriodSelectorComponent, SystemSummaryCardComponent, @@ -35,7 +38,7 @@ import { ModelBreakdownComponent } from './components/model-breakdown.component' CostTrendsChartComponent, ModelBreakdownComponent, ], - providers: [provideIcons({ heroArrowLeft, heroArrowDownTray })], + providers: [provideIcons({ heroArrowLeft, heroArrowDownTray, heroMagnifyingGlass })], changeDetection: ChangeDetectionStrategy.OnPush, template: `
@@ -66,6 +69,46 @@ import { ModelBreakdownComponent } from './components/model-breakdown.component'
+ +
+
+ +

+ Look up per-call cache diagnostics for a session ID. +

+
+
+
+
+ +
+
+ @if (loading()) {
@@ -177,6 +220,9 @@ export class AdminCostsPage implements OnInit { trends = this.stateService.trends; modelUsage = this.stateService.modelUsage; + // Session-id lookup for the cost-anatomy drill-down + sessionLookupId = signal(''); + // Track pagination state for top users private topUsersLimit = signal(20); hasMoreUsers = computed( @@ -238,6 +284,13 @@ export class AdminCostsPage implements OnInit { this.router.navigate(['/admin/users', userId]); } + onInspectSession(): void { + const sessionId = this.sessionLookupId().trim(); + if (sessionId) { + this.router.navigate(['/admin/costs/sessions', sessionId]); + } + } + async onLoadMoreUsers(): Promise { const newLimit = this.topUsersLimit() + 20; this.topUsersLimit.set(newLimit); diff --git a/frontend/ai.client/src/app/admin/costs/models/admin-cost.models.ts b/frontend/ai.client/src/app/admin/costs/models/admin-cost.models.ts index 25935db24..677654228 100644 --- a/frontend/ai.client/src/app/admin/costs/models/admin-cost.models.ts +++ b/frontend/ai.client/src/app/admin/costs/models/admin-cost.models.ts @@ -91,6 +91,59 @@ export interface AdminCostDashboard { dailyTrends?: CostTrend[]; } +// ========== Session Cost Anatomy ========== + +/** + * Derived prompt-cache status for one model call. + * Null on rows persisted before the cache-observability feature. + */ +export type CacheStatus = + | 'first_write' + | 'hit' + | 'miss_ttl_expired' + | 'miss_avoidable' + | 'uncached'; + +/** Prompt-cache prefix hashes for one model call. */ +export interface PrefixFingerprints { + toolConfigHash?: string | null; + systemPromptHash?: string | null; + historyHash?: string | null; + messageCount?: number | null; +} + +/** One model call within a session's cost anatomy. */ +export interface SessionCallRow { + timestamp: string; + messageId?: number | null; + modelId?: string | null; + + inputTokens: number; + outputTokens: number; + cacheReadTokens: number; + cacheWriteTokens: number; + + cost: number; + cacheStatus?: CacheStatus | null; + cacheGapSeconds?: number | null; + wastedUsd: number; + prefixFingerprints?: PrefixFingerprints | null; +} + +/** Per-call cost anatomy for one session (admin cache-miss forensics). */ +export interface SessionCostAnatomy { + sessionId: string; + calls: SessionCallRow[]; + + totalCost: number; + totalCacheReadTokens: number; + totalCacheWriteTokens: number; + avoidableMissCount: number; + wastedUsd: number; + /** cacheRead / (cacheRead + cacheWrite); null until any cache activity. */ + cacheEfficiency: number | null; +} + // ========== API Request Options ========== export interface DashboardRequestOptions { diff --git a/frontend/ai.client/src/app/admin/costs/pages/session-cost-anatomy.page.spec.ts b/frontend/ai.client/src/app/admin/costs/pages/session-cost-anatomy.page.spec.ts new file mode 100644 index 000000000..836485687 --- /dev/null +++ b/frontend/ai.client/src/app/admin/costs/pages/session-cost-anatomy.page.spec.ts @@ -0,0 +1,160 @@ +import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest'; +import { TestBed } from '@angular/core/testing'; +import { provideRouter } from '@angular/router'; +import { HttpErrorResponse } from '@angular/common/http'; +import { of, throwError } from 'rxjs'; +import { SessionCostAnatomyPage } from './session-cost-anatomy.page'; +import { AdminCostHttpService } from '../services/admin-cost-http.service'; +import { SessionCostAnatomy } from '../models'; + +const MOCK_ANATOMY: SessionCostAnatomy = { + sessionId: 'sess-1', + totalCost: 0.42, + totalCacheReadTokens: 20_000, + totalCacheWriteTokens: 5_000, + avoidableMissCount: 1, + wastedUsd: 0.03, + cacheEfficiency: 0.8, + calls: [ + { + timestamp: '2026-07-19T10:00:00Z', + messageId: 1, + modelId: 'us.anthropic.claude-sonnet-5', + inputTokens: 100, + outputTokens: 50, + cacheReadTokens: 0, + cacheWriteTokens: 5_000, + cost: 0.1, + cacheStatus: 'first_write', + cacheGapSeconds: null, + wastedUsd: 0, + prefixFingerprints: { toolConfigHash: 'aaaa1111', systemPromptHash: 'bbbb2222', historyHash: 'cccc3333', messageCount: 2 }, + }, + { + timestamp: '2026-07-19T10:01:00Z', + messageId: 2, + modelId: 'us.anthropic.claude-sonnet-5', + inputTokens: 120, + outputTokens: 60, + cacheReadTokens: 0, + cacheWriteTokens: 5_200, + cost: 0.12, + cacheStatus: 'miss_avoidable', + cacheGapSeconds: 60, + wastedUsd: 0.03, + prefixFingerprints: { toolConfigHash: 'DIFFERENT', systemPromptHash: 'bbbb2222', historyHash: 'cccc3333', messageCount: 4 }, + }, + ], +}; + +describe('SessionCostAnatomyPage', () => { + let getSessionCostAnatomy: ReturnType; + + function setup(mock: ReturnType) { + getSessionCostAnatomy = mock; + TestBed.resetTestingModule(); + TestBed.configureTestingModule({ + providers: [ + provideRouter([]), + { provide: AdminCostHttpService, useValue: { getSessionCostAnatomy } }, + ], + }); + TestBed.overrideComponent(SessionCostAnatomyPage, { + set: { template: '
' }, + }); + const fixture = TestBed.createComponent(SessionCostAnatomyPage); + fixture.componentRef.setInput('id', 'sess-1'); + fixture.detectChanges(); + return fixture; + } + + afterEach(() => { + TestBed.resetTestingModule(); + }); + + it('loads the anatomy for the routed session id and annotates fingerprint diffs', async () => { + const fixture = setup(vi.fn().mockReturnValue(of(MOCK_ANATOMY))); + const page = fixture.componentInstance; + + await vi.waitFor(() => { + expect(page.anatomyResource.hasValue()).toBe(true); + }); + + expect(getSessionCostAnatomy).toHaveBeenCalledWith('sess-1'); + const rows = page.rows(); + expect(rows).toHaveLength(2); + expect(rows[0].changed).toEqual([]); + // The flipped toolConfigHash is the diagnosis on the miss_avoidable row. + expect(rows[1].changed).toEqual(['toolConfigHash']); + expect(page.notFound()).toBe(false); + }); + + it('treats a 404 as "session has no cost rows"', async () => { + const fixture = setup( + vi.fn().mockReturnValue( + throwError(() => new HttpErrorResponse({ status: 404, statusText: 'Not Found' })) + ) + ); + const page = fixture.componentInstance; + + await vi.waitFor(() => { + expect(page.anatomyResource.error()).toBeTruthy(); + }); + expect(page.notFound()).toBe(true); + }); + + it('does not report notFound for other errors', async () => { + const fixture = setup( + vi.fn().mockReturnValue( + throwError(() => new HttpErrorResponse({ status: 500, statusText: 'Server Error' })) + ) + ); + const page = fixture.componentInstance; + + await vi.waitFor(() => { + expect(page.anatomyResource.error()).toBeTruthy(); + }); + expect(page.notFound()).toBe(false); + }); + + it('toggles row expansion', () => { + const fixture = setup(vi.fn().mockReturnValue(of(MOCK_ANATOMY))); + const page = fixture.componentInstance; + + expect(page.isExpanded(0)).toBe(false); + page.toggleExpand(0); + expect(page.isExpanded(0)).toBe(true); + page.toggleExpand(0); + expect(page.isExpanded(0)).toBe(false); + }); + + it('formats cache efficiency, handling null', () => { + const fixture = setup(vi.fn().mockReturnValue(of(MOCK_ANATOMY))); + const page = fixture.componentInstance; + + expect(page.formatEfficiency(null)).toBe('β€”'); + expect(page.formatEfficiency(0.8)).toBe('80.0%'); + }); + + it('formats cache gaps, handling null', () => { + const fixture = setup(vi.fn().mockReturnValue(of(MOCK_ANATOMY))); + const page = fixture.componentInstance; + + expect(page.formatGap(null)).toBe('β€”'); + expect(page.formatGap(undefined)).toBe('β€”'); + expect(page.formatGap(42)).toBe('42s'); + expect(page.formatGap(60)).toBe('1m'); + expect(page.formatGap(312)).toBe('5m 12s'); + }); + + it('maps cache statuses to color-coded badge classes', () => { + const fixture = setup(vi.fn().mockReturnValue(of(MOCK_ANATOMY))); + const page = fixture.componentInstance; + + expect(page.getStatusClass('hit')).toContain('bg-green-100'); + expect(page.getStatusClass('first_write')).toContain('bg-blue-100'); + expect(page.getStatusClass('miss_ttl_expired')).toContain('bg-yellow-100'); + expect(page.getStatusClass('miss_avoidable')).toContain('bg-red-100'); + expect(page.getStatusClass('uncached')).toContain('bg-gray-100'); + }); +}); diff --git a/frontend/ai.client/src/app/admin/costs/pages/session-cost-anatomy.page.ts b/frontend/ai.client/src/app/admin/costs/pages/session-cost-anatomy.page.ts new file mode 100644 index 000000000..96197b730 --- /dev/null +++ b/frontend/ai.client/src/app/admin/costs/pages/session-cost-anatomy.page.ts @@ -0,0 +1,469 @@ +import { + ChangeDetectionStrategy, + Component, + computed, + inject, + input, + resource, + signal, +} from '@angular/core'; +import { DatePipe } from '@angular/common'; +import { HttpErrorResponse } from '@angular/common/http'; +import { RouterLink } from '@angular/router'; +import { firstValueFrom } from 'rxjs'; +import { NgIcon, provideIcons } from '@ng-icons/core'; +import { heroArrowLeft, heroChevronDown } from '@ng-icons/heroicons/outline'; +import { AdminCostHttpService } from '../services/admin-cost-http.service'; +import { CacheStatus, SessionCallRow } from '../models'; +import { + AnatomyRow, + FINGERPRINT_KEYS, + FINGERPRINT_LABELS, + FingerprintKey, + buildAnatomyRows, + truncateHash, +} from './session-cost-anatomy.util'; + +/** + * Admin drill-down: per-model-call cost anatomy for one session. + * + * The forensic view for prompt-cache diagnostics β€” each call row carries its + * cache status and prefix-fingerprint hashes, and the hash that flipped + * between consecutive calls names the cache-buster (the diagnosis on + * `miss_avoidable` rows). + */ +@Component({ + selector: 'app-session-cost-anatomy', + imports: [RouterLink, NgIcon, DatePipe], + providers: [provideIcons({ heroArrowLeft, heroChevronDown })], + changeDetection: ChangeDetectionStrategy.OnPush, + template: ` +
+ + + + + +
+

Session Cost Anatomy

+

+ {{ id() }} +

+
+ + @if (anatomyResource.isLoading()) { + +
+
+
+

Loading session cost anatomy…

+
+
+ } @else if (notFound()) { + +
+

No cost rows for this session

+

+ Nothing has been recorded under this session ID β€” check the ID, or the session may predate + cost tracking. +

+
+ } @else if (anatomyResource.error()) { + +
+

Failed to load session cost anatomy. Please try again.

+ +
+ } @else if (anatomyResource.hasValue()) { + +
+
+

+ Total Cost +

+

+ {{ formatCurrency(anatomyResource.value().totalCost) }} +

+
+
+

+ Cache Efficiency +

+

+ {{ formatEfficiency(anatomyResource.value().cacheEfficiency) }} +

+
+
+

+ Avoidable Misses +

+

+ {{ anatomyResource.value().avoidableMissCount }} +

+
+
+

+ Wasted +

+

+ {{ formatCurrency(anatomyResource.value().wastedUsd, 4) }} +

+
+
+

+ Cache Read +

+

+ {{ formatTokens(anatomyResource.value().totalCacheReadTokens) }} +

+
+
+

+ Cache Write +

+

+ {{ formatTokens(anatomyResource.value().totalCacheWriteTokens) }} +

+
+
+ + +
+
+ + + + + + + + + + + + + + + + + + + @for (row of rows(); track row.index) { + + + + + + + + + + + + + + + @if (isExpanded(row.index)) { + + + + } + } @empty { + + + + } + +
ExpandTimeModelInReadWriteOutCostCache StatusGapWastedFingerprints
+ + + {{ row.call.timestamp | date: 'MMM d, HH:mm:ss' }} + + {{ row.call.modelId ?? 'β€”' }} + + {{ formatTokens(row.call.inputTokens) }} + + {{ formatTokens(row.call.cacheReadTokens) }} + + {{ formatTokens(row.call.cacheWriteTokens) }} + + {{ formatTokens(row.call.outputTokens) }} + + {{ formatCurrency(row.call.cost, 4) }} + + @if (row.call.cacheStatus; as status) { + {{ getStatusLabel(status) }} + } @else { + β€” + } + + {{ formatGap(row.call.cacheGapSeconds) }} + + {{ row.call.wastedUsd > 0 ? formatCurrency(row.call.wastedUsd, 4) : 'β€”' }} + + @if (row.call.prefixFingerprints; as fp) { +
+ @for (key of fingerprintKeys; track key) { + + {{ fingerprintLabels[key] }} {{ truncateHash(fp[key]) }} + + } +
+ } @else { + β€” + } +
+
+
+
+ Timestamp +
+
+ {{ row.call.timestamp }} +
+
+
+
+ Message ID +
+
+ {{ row.call.messageId ?? 'β€”' }} +
+
+
+
+ Message Count +
+
+ {{ row.call.prefixFingerprints?.messageCount ?? 'β€”' }} +
+
+ @for (key of fingerprintKeys; track key) { +
+
+ {{ fingerprintLabels[key] }} Hash + @if (isChanged(row, key)) { + (changed) + } +
+
+ {{ row.call.prefixFingerprints?.[key] ?? 'β€”' }} +
+
+ } +
+
+ No model calls recorded for this session. +
+
+
+ } +
+ `, +}) +export class SessionCostAnatomyPage { + private costHttp = inject(AdminCostHttpService); + + /** Session ID from the `costs/sessions/:id` route (component input binding). */ + readonly id = input.required(); + + readonly fingerprintKeys = FINGERPRINT_KEYS; + readonly fingerprintLabels = FINGERPRINT_LABELS; + readonly truncateHash = truncateHash; + + readonly anatomyResource = resource({ + params: () => ({ id: this.id() }), + loader: ({ params }) => firstValueFrom(this.costHttp.getSessionCostAnatomy(params.id)), + }); + + /** Chronological call rows annotated with fingerprint diffs. */ + readonly rows = computed(() => + this.anatomyResource.hasValue() ? buildAnatomyRows(this.anatomyResource.value().calls) : [] + ); + + /** True when the backend returned 404 β€” the session has no cost rows. */ + readonly notFound = computed(() => this.errorStatus(this.anatomyResource.error()) === 404); + + private expandedRows = signal>(new Set()); + + isExpanded(index: number): boolean { + return this.expandedRows().has(index); + } + + toggleExpand(index: number): void { + this.expandedRows.update((current) => { + const next = new Set(current); + if (next.has(index)) { + next.delete(index); + } else { + next.add(index); + } + return next; + }); + } + + isChanged(row: AnatomyRow, key: FingerprintKey): boolean { + return row.changed.includes(key); + } + + getFingerprintClass(row: AnatomyRow, key: FingerprintKey): string { + const base = 'inline-flex items-center rounded-2xl px-2 py-0.5 font-mono text-xs/5'; + if (this.isChanged(row, key)) { + return `${base} bg-red-100 font-semibold text-red-700 ring-1 ring-inset ring-red-300 dark:bg-red-900/40 dark:text-red-300 dark:ring-red-700`; + } + return `${base} bg-gray-100 text-gray-600 dark:bg-gray-700 dark:text-gray-400`; + } + + getStatusClass(status: CacheStatus): string { + const base = 'inline-flex items-center rounded-2xl px-2.5 py-0.5 text-xs/5 font-medium'; + switch (status) { + case 'hit': + return `${base} bg-green-100 text-green-800 dark:bg-green-900/30 dark:text-green-300`; + case 'first_write': + return `${base} bg-blue-100 text-blue-800 dark:bg-blue-900/30 dark:text-blue-300`; + case 'miss_ttl_expired': + return `${base} bg-yellow-100 text-yellow-800 dark:bg-yellow-900/30 dark:text-yellow-300`; + case 'miss_avoidable': + return `${base} bg-red-100 text-red-800 dark:bg-red-900/30 dark:text-red-300`; + case 'uncached': + default: + return `${base} bg-gray-100 text-gray-800 dark:bg-gray-700 dark:text-gray-300`; + } + } + + getStatusLabel(status: CacheStatus): string { + switch (status) { + case 'hit': + return 'Hit'; + case 'first_write': + return 'First Write'; + case 'miss_ttl_expired': + return 'Miss (TTL)'; + case 'miss_avoidable': + return 'Miss (Avoidable)'; + case 'uncached': + return 'Uncached'; + default: + return status; + } + } + + formatCurrency(value: number, maxFractionDigits = 2): string { + return new Intl.NumberFormat('en-US', { + style: 'currency', + currency: 'USD', + minimumFractionDigits: 2, + maximumFractionDigits: maxFractionDigits, + }).format(value); + } + + formatTokens(tokens: number): string { + if (tokens >= 1_000_000) { + return `${(tokens / 1_000_000).toFixed(1)}M`; + } else if (tokens >= 1_000) { + return `${(tokens / 1_000).toFixed(1)}K`; + } + return tokens.toString(); + } + + formatEfficiency(efficiency: number | null): string { + if (efficiency === null) { + return 'β€”'; + } + return `${(efficiency * 100).toFixed(1)}%`; + } + + formatGap(seconds: number | null | undefined): string { + if (seconds == null) { + return 'β€”'; + } + if (seconds >= 60) { + const minutes = Math.floor(seconds / 60); + const rest = seconds % 60; + return rest > 0 ? `${minutes}m ${rest}s` : `${minutes}m`; + } + return `${seconds}s`; + } + + private errorStatus(error: unknown): number | undefined { + if (error instanceof HttpErrorResponse) { + return error.status; + } + // resource() may wrap loader errors; check the cause chain. + const cause = (error as { cause?: unknown } | null | undefined)?.cause; + if (cause instanceof HttpErrorResponse) { + return cause.status; + } + return undefined; + } +} diff --git a/frontend/ai.client/src/app/admin/costs/pages/session-cost-anatomy.util.spec.ts b/frontend/ai.client/src/app/admin/costs/pages/session-cost-anatomy.util.spec.ts new file mode 100644 index 000000000..8d77e6373 --- /dev/null +++ b/frontend/ai.client/src/app/admin/costs/pages/session-cost-anatomy.util.spec.ts @@ -0,0 +1,98 @@ +import { describe, it, expect } from 'vitest'; +import { buildAnatomyRows, truncateHash } from './session-cost-anatomy.util'; +import { SessionCallRow } from '../models'; + +function makeCall(overrides: Partial = {}): SessionCallRow { + return { + timestamp: '2026-07-19T10:00:00Z', + inputTokens: 100, + outputTokens: 50, + cacheReadTokens: 0, + cacheWriteTokens: 0, + cost: 0.01, + wastedUsd: 0, + ...overrides, + }; +} + +describe('buildAnatomyRows', () => { + it('never flags the first call', () => { + const rows = buildAnatomyRows([ + makeCall({ + prefixFingerprints: { toolConfigHash: 'aaa', systemPromptHash: 'bbb', historyHash: 'ccc' }, + }), + ]); + expect(rows).toHaveLength(1); + expect(rows[0].changed).toEqual([]); + }); + + it('flags exactly the hash that flipped between consecutive calls', () => { + const rows = buildAnatomyRows([ + makeCall({ + prefixFingerprints: { toolConfigHash: 'aaa', systemPromptHash: 'bbb', historyHash: 'ccc' }, + }), + makeCall({ + prefixFingerprints: { toolConfigHash: 'XXX', systemPromptHash: 'bbb', historyHash: 'ccc' }, + }), + ]); + expect(rows[1].changed).toEqual(['toolConfigHash']); + }); + + it('flags multiple flipped hashes', () => { + const rows = buildAnatomyRows([ + makeCall({ + prefixFingerprints: { toolConfigHash: 'aaa', systemPromptHash: 'bbb', historyHash: 'ccc' }, + }), + makeCall({ + prefixFingerprints: { toolConfigHash: 'aaa', systemPromptHash: 'YYY', historyHash: 'ZZZ' }, + }), + ]); + expect(rows[1].changed).toEqual(['systemPromptHash', 'historyHash']); + }); + + it('skips rows without fingerprints as comparison baselines', () => { + const rows = buildAnatomyRows([ + makeCall({ + prefixFingerprints: { toolConfigHash: 'aaa', systemPromptHash: 'bbb', historyHash: 'ccc' }, + }), + makeCall({ prefixFingerprints: null }), + makeCall({ + prefixFingerprints: { toolConfigHash: 'XXX', systemPromptHash: 'bbb', historyHash: 'ccc' }, + }), + ]); + // Row 1 has nothing to diff; row 2 diffs against row 0, not the null row. + expect(rows[1].changed).toEqual([]); + expect(rows[2].changed).toEqual(['toolConfigHash']); + }); + + it('does not flag a hash when either side is missing', () => { + const rows = buildAnatomyRows([ + makeCall({ prefixFingerprints: { toolConfigHash: 'aaa', historyHash: 'ccc' } }), + makeCall({ + prefixFingerprints: { toolConfigHash: 'aaa', systemPromptHash: 'now-present', historyHash: 'ccc' }, + }), + ]); + expect(rows[1].changed).toEqual([]); + }); + + it('does not flag identical fingerprints', () => { + const fp = { toolConfigHash: 'aaa', systemPromptHash: 'bbb', historyHash: 'ccc' }; + const rows = buildAnatomyRows([ + makeCall({ prefixFingerprints: fp }), + makeCall({ prefixFingerprints: { ...fp } }), + ]); + expect(rows[1].changed).toEqual([]); + }); +}); + +describe('truncateHash', () => { + it('truncates to the first 8 chars', () => { + expect(truncateHash('0123456789abcdef')).toBe('01234567'); + }); + + it('returns an em dash for missing hashes', () => { + expect(truncateHash(null)).toBe('β€”'); + expect(truncateHash(undefined)).toBe('β€”'); + expect(truncateHash('')).toBe('β€”'); + }); +}); diff --git a/frontend/ai.client/src/app/admin/costs/pages/session-cost-anatomy.util.ts b/frontend/ai.client/src/app/admin/costs/pages/session-cost-anatomy.util.ts new file mode 100644 index 000000000..76907398a --- /dev/null +++ b/frontend/ai.client/src/app/admin/costs/pages/session-cost-anatomy.util.ts @@ -0,0 +1,62 @@ +import { PrefixFingerprints, SessionCallRow } from '../models'; + +export type FingerprintKey = 'toolConfigHash' | 'systemPromptHash' | 'historyHash'; + +export const FINGERPRINT_KEYS: readonly FingerprintKey[] = [ + 'toolConfigHash', + 'systemPromptHash', + 'historyHash', +]; + +export const FINGERPRINT_LABELS: Record = { + toolConfigHash: 'Tools', + systemPromptHash: 'System', + historyHash: 'History', +}; + +/** One call row annotated with fingerprint-diff results against the previous fingerprinted call. */ +export interface AnatomyRow { + call: SessionCallRow; + index: number; + /** Fingerprint hashes that flipped vs. the nearest previous call that has fingerprints. */ + changed: FingerprintKey[]; +} + +/** + * Annotate chronological call rows with which prefix-fingerprint hashes + * changed since the previous fingerprinted call β€” on a `miss_avoidable` + * row, the flipped hash names the cache-buster. + * + * A hash is flagged only when both rows carry a value for it and the values + * differ; rows predating the fingerprint feature (null fingerprints) are + * skipped as comparison baselines. + */ +export function buildAnatomyRows(calls: SessionCallRow[]): AnatomyRow[] { + let previous: PrefixFingerprints | undefined; + + return calls.map((call, index) => { + const fingerprints = call.prefixFingerprints ?? undefined; + const changed: FingerprintKey[] = []; + + if (fingerprints && previous) { + for (const key of FINGERPRINT_KEYS) { + const current = fingerprints[key]; + const before = previous[key]; + if (current != null && before != null && current !== before) { + changed.push(key); + } + } + } + + if (fingerprints) { + previous = fingerprints; + } + + return { call, index, changed }; + }); +} + +/** First 8 chars of a fingerprint hash, or an em dash when absent. */ +export function truncateHash(hash: string | null | undefined): string { + return hash ? hash.slice(0, 8) : 'β€”'; +} diff --git a/frontend/ai.client/src/app/admin/costs/services/admin-cost-http.service.spec.ts b/frontend/ai.client/src/app/admin/costs/services/admin-cost-http.service.spec.ts index 67f5a9e09..82ad514d2 100644 --- a/frontend/ai.client/src/app/admin/costs/services/admin-cost-http.service.spec.ts +++ b/frontend/ai.client/src/app/admin/costs/services/admin-cost-http.service.spec.ts @@ -65,6 +65,35 @@ describe('AdminCostHttpService', () => { req.flush(mockSummary); }); + it('should get session cost anatomy', () => { + const mockAnatomy = { + sessionId: 'sess-1', + calls: [], + totalCost: 1.23, + totalCacheReadTokens: 1000, + totalCacheWriteTokens: 500, + avoidableMissCount: 0, + wastedUsd: 0, + cacheEfficiency: 2 / 3, + }; + + service.getSessionCostAnatomy('sess-1').subscribe(anatomy => { + expect(anatomy).toEqual(mockAnatomy); + }); + + const req = httpMock.expectOne('http://localhost:8000/admin/costs/sessions/sess-1/calls'); + expect(req.request.method).toBe('GET'); + req.flush(mockAnatomy); + }); + + it('should URL-encode the session id in the anatomy request', () => { + service.getSessionCostAnatomy('a/b c').subscribe(); + + const req = httpMock.expectOne('http://localhost:8000/admin/costs/sessions/a%2Fb%20c/calls'); + expect(req.request.method).toBe('GET'); + req.flush({ sessionId: 'a/b c', calls: [] }); + }); + it('should export data', () => { const mockBlob = new Blob(['csv data'], { type: 'text/csv' }); diff --git a/frontend/ai.client/src/app/admin/costs/services/admin-cost-http.service.ts b/frontend/ai.client/src/app/admin/costs/services/admin-cost-http.service.ts index 5f79bac74..b83793ffe 100644 --- a/frontend/ai.client/src/app/admin/costs/services/admin-cost-http.service.ts +++ b/frontend/ai.client/src/app/admin/costs/services/admin-cost-http.service.ts @@ -12,6 +12,7 @@ import { DashboardRequestOptions, TopUsersRequestOptions, TrendsRequestOptions, + SessionCostAnatomy, } from '../models'; /** @@ -128,6 +129,17 @@ export class AdminCostHttpService { return this.http.get(`${this.baseUrl()}/trends`, { params }); } + /** + * Get the per-model-call cost anatomy for one session. + * Returns chronological call rows with cache status, prefix fingerprints, + * and session-level cache rollups. 404 when the session has no cost rows. + */ + getSessionCostAnatomy(sessionId: string): Observable { + return this.http.get( + `${this.baseUrl()}/sessions/${encodeURIComponent(sessionId)}/calls` + ); + } + /** * Export cost data for a period. * Returns blob for download. diff --git a/frontend/ai.client/src/app/session/components/chat-input/chat-input.component.css b/frontend/ai.client/src/app/session/components/chat-input/chat-input.component.css index 2bcb6cfb9..e0ccd0cfb 100644 --- a/frontend/ai.client/src/app/session/components/chat-input/chat-input.component.css +++ b/frontend/ai.client/src/app/session/components/chat-input/chat-input.component.css @@ -6,26 +6,4 @@ transition: border-color 0.2s ease-in-out, box-shadow 0.2s ease-in-out; } -.chat-textarea { - min-height: 3rem; - max-height: 12rem; - transition: border-color 0.2s ease-in-out, box-shadow 0.2s ease-in-out; - font-family: inherit; - font-size: 0.875rem; - line-height: 1.5rem; -} - -/* Auto-resize textarea */ -.chat-textarea { - field-sizing: content; -} - -@supports not (field-sizing: content) { - .chat-textarea { - height: auto; - } -} - - - diff --git a/frontend/ai.client/src/app/session/components/chat-input/chat-input.component.html b/frontend/ai.client/src/app/session/components/chat-input/chat-input.component.html index 6428a4681..1e0193146 100644 --- a/frontend/ai.client/src/app/session/components/chat-input/chat-input.component.html +++ b/frontend/ai.client/src/app/session/components/chat-input/chat-input.component.html @@ -60,7 +60,7 @@ (focus)="onFocus()" (blur)="onBlur()" placeholder="How can I help you today?" - class="block w-full resize-none bg-transparent px-6 py-4 text-base text-gray-900 placeholder:text-gray-500 dark:text-gray-100 dark:placeholder:text-gray-400 focus:outline-hidden overflow-hidden" + class="block w-full resize-none bg-transparent px-6 py-4 text-base text-gray-900 placeholder:text-gray-500 dark:text-gray-100 dark:placeholder:text-gray-400 focus:outline-hidden overflow-y-auto" style="min-height: 60px; max-height: 200px;" > diff --git a/frontend/ai.client/src/app/session/components/chat-input/chat-input.component.ts b/frontend/ai.client/src/app/session/components/chat-input/chat-input.component.ts index 91853a6d7..010f3beed 100644 --- a/frontend/ai.client/src/app/session/components/chat-input/chat-input.component.ts +++ b/frontend/ai.client/src/app/session/components/chat-input/chat-input.component.ts @@ -38,6 +38,11 @@ import { ToolService } from '../../../services/tool/tool.service'; import { VoiceChatService, type VoiceStatus } from '../../services/voice'; import { SystemPromptsService } from '../../../services/system-prompts/system-prompts.service'; +// Must stay in sync with the inline min-height/max-height on the textarea in +// chat-input.component.html. +const MIN_TEXTAREA_HEIGHT_PX = 60; +const MAX_TEXTAREA_HEIGHT_PX = 200; + interface Message { content: string; timestamp: Date; @@ -97,7 +102,6 @@ export class ChatInputComponent { // Signals for state management userInput = signal(''); - isExpanded = signal(false); isFocused = signal(false); isDraggingOver = signal(false); @@ -219,7 +223,7 @@ export class ChatInputComponent { // Clear input and pending uploads this.userInput.set(''); - this.isExpanded.set(false); + this.resetTextareaHeight(); this.fileUploadService.clearReadyUploads(); } @@ -296,10 +300,29 @@ export class ChatInputComponent { onTextareaInput(event: Event) { const textarea = event.target as HTMLTextAreaElement; this.userInput.set(textarea.value); - - // Auto-expand based on content + this.autoResize(textarea); + } + + /** + * Grow the textarea with its content up to MAX_TEXTAREA_HEIGHT_PX, past which + * it scrolls internally (the template sets overflow-y-auto). Without the clamp + * the inline height keeps growing past max-height and the scrollbar never + * becomes usable. + */ + private autoResize(textarea: HTMLTextAreaElement): void { textarea.style.height = 'auto'; - textarea.style.height = textarea.scrollHeight + 'px'; + const height = Math.min(textarea.scrollHeight, MAX_TEXTAREA_HEIGHT_PX); + textarea.style.height = `${height}px`; + } + + /** Collapse the textarea back to a single row (after submit or clear). */ + private resetTextareaHeight(): void { + const textarea = this.messageInput()?.nativeElement; + if (!textarea) { + return; + } + textarea.style.height = `${MIN_TEXTAREA_HEIGHT_PX}px`; + textarea.scrollTop = 0; } onKeyDown(event: KeyboardEvent) { diff --git a/infrastructure/lib/constructs/inference-api/inference-agentcore-construct.ts b/infrastructure/lib/constructs/inference-api/inference-agentcore-construct.ts index b17ad3c58..740ff8d23 100644 --- a/infrastructure/lib/constructs/inference-api/inference-agentcore-construct.ts +++ b/infrastructure/lib/constructs/inference-api/inference-agentcore-construct.ts @@ -310,6 +310,12 @@ export class InferenceAgentCoreConstruct extends Construct { DYNAMODB_MANAGED_MODELS_TABLE_NAME: managedModelsTableName, DYNAMODB_USER_SETTINGS_TABLE_NAME: userSettingsTableName, DYNAMODB_USER_FILES_TABLE_NAME: userFilesTableName, + // Bucket the runtime writes generated Word docs to (create/modify + // word-document tools). Without this the tool falls back to the + // literal "user-files" default and PutObject is AccessDenied. The + // runtime role's UserFilesBucketAccess statement already grants + // Get/Put/Delete/List on this bucket. + S3_USER_FILES_BUCKET_NAME: props.refs.fileUploadBucket.bucketName, DYNAMODB_SYSTEM_PROMPTS_TABLE_NAME: systemPromptsTableName, // Auth providers diff --git a/infrastructure/lib/constructs/observability/prompt-cache-observability-construct.ts b/infrastructure/lib/constructs/observability/prompt-cache-observability-construct.ts new file mode 100644 index 000000000..4f0706798 --- /dev/null +++ b/infrastructure/lib/constructs/observability/prompt-cache-observability-construct.ts @@ -0,0 +1,198 @@ +import * as cdk from 'aws-cdk-lib'; +import * as cloudwatch from 'aws-cdk-lib/aws-cloudwatch'; +import { Construct } from 'constructs'; + +import { AppConfig, getResourceName } from '../../config'; + +export interface PromptCacheObservabilityConstructProps { + config: AppConfig; +} + +/** + * PromptCacheObservabilityConstruct β€” fleet-wide dashboard + alarms for + * the prompt-cache EMF metrics emitted by + * `backend/src/apis/shared/observability/emf.py` (PR #697). + * + * The metrics arrive as raw EMF JSON on stdout from BOTH compute + * surfaces (inference-api via the AgentCore Runtime log group, app-api + * via ECS awslogs) into the shared `AgentCoreStack/PromptCache` + * namespace β€” the default of the `EMF_NAMESPACE` env var, which is + * deliberately NOT set in CDK (dev and prod are separate AWS accounts, + * so the unscoped namespace never collides). All metrics are + * dimension-less and designed for `Sum`; per-call detail (`modelId`, + * `sessionId`, `cacheStatus`) rides as queryable EMF log properties, + * hence the Logs Insights widget below rather than dimension fan-out. + * + * This is the first cross-service dashboard, so it lives in its own + * construct area instead of inside the inference-api construct. The + * per-session drill-down counterpart is the cost-anatomy admin endpoint + * (`GET /admin/costs/sessions/{id}/calls`). + * + * Alarms are console-only β€” no SNS topics exist in the stack yet (same + * posture as kb-sync and scheduled-runs). They use NOT_BREACHING for + * missing data because the `PROMPT_CACHE_OBSERVABILITY_ENABLED=false` + * kill switch (or simply zero traffic) makes the metrics absent + * entirely. + */ +export class PromptCacheObservabilityConstruct extends Construct { + constructor( + scope: Construct, + id: string, + props: PromptCacheObservabilityConstructProps, + ) { + super(scope, id); + + const { config } = props; + + // Must match the default of EMF_NAMESPACE in emf.py. + const namespace = 'AgentCoreStack/PromptCache'; + + const cacheReadTokensMetric = new cloudwatch.Metric({ + namespace, + metricName: 'CacheReadTokens', + statistic: 'Sum', + period: cdk.Duration.minutes(5), + }); + + const cacheWriteTokensMetric = new cloudwatch.Metric({ + namespace, + metricName: 'CacheWriteTokens', + statistic: 'Sum', + period: cdk.Duration.minutes(5), + }); + + const avoidableMissMetric = new cloudwatch.Metric({ + namespace, + metricName: 'AvoidableMiss', + statistic: 'Sum', + period: cdk.Duration.minutes(5), + }); + + const wastedUsdMetric = new cloudwatch.Metric({ + namespace, + metricName: 'WastedUsd', + statistic: 'Sum', + period: cdk.Duration.minutes(5), + }); + + // Cache efficiency: fraction of cache traffic served from cache. + // Healthy steady-state conversations read far more than they write, + // so this should sit near 100%; a prefix-stability regression shows + // up as a sustained drop (writes displacing reads). + const cacheEfficiencyExpression = new cloudwatch.MathExpression({ + expression: '100 * reads / (reads + writes)', + usingMetrics: { + reads: cacheReadTokensMetric, + writes: cacheWriteTokensMetric, + }, + label: 'Cache efficiency (%)', + period: cdk.Duration.minutes(5), + }); + + // ============================================================ + // Dashboard + // ============================================================ + + const dashboard = new cloudwatch.Dashboard(this, 'PromptCacheDashboard', { + dashboardName: getResourceName(config, 'prompt-cache-observability'), + defaultInterval: cdk.Duration.hours(3), + }); + + dashboard.addWidgets( + new cloudwatch.TextWidget({ + markdown: `# Prompt Cache Observability\n**Project:** ${config.projectPrefix} | **Region:** ${config.awsRegion} β€” fleet-wide EMF metrics from app-api + inference-api. Per-session drill-down: \`GET /admin/costs/sessions/{id}/calls\`.`, + width: 24, + height: 1, + }), + ); + + dashboard.addWidgets( + new cloudwatch.GraphWidget({ + title: 'Cache Tokens (Read vs Write)', + left: [cacheReadTokensMetric, cacheWriteTokensMetric], + width: 12, + height: 6, + }), + new cloudwatch.GraphWidget({ + title: 'Cache Efficiency (reads / total cache traffic)', + left: [cacheEfficiencyExpression], + leftYAxis: { min: 0, max: 100 }, + width: 12, + height: 6, + }), + ); + + dashboard.addWidgets( + new cloudwatch.GraphWidget({ + title: 'Avoidable Cache Misses', + left: [avoidableMissMetric], + width: 12, + height: 6, + }), + new cloudwatch.GraphWidget({ + title: 'Wasted USD (avoidable re-writes)', + left: [wastedUsdMetric], + width: 12, + height: 6, + }), + ); + + dashboard.addWidgets( + new cloudwatch.LogQueryWidget({ + title: 'Model Calls by cacheStatus (inference-api)', + // The runtime log group is defined by the inference-api + // construct with this deterministic name; referenced by name + // (not typed ref) since a dashboard widget creates no CFN + // dependency. app-api's ECS log group is auto-named, so its + // (much smaller) share of emissions isn't queried here. + logGroupNames: [`/aws/bedrock-agentcore/runtimes/${config.projectPrefix}`], + queryLines: [ + 'filter ispresent(cacheStatus)', + 'stats count(*) as calls by cacheStatus', + 'sort calls desc', + ], + width: 24, + height: 6, + }), + ); + + // ============================================================ + // Alarms (console-only; no SNS wiring yet) + // ============================================================ + + // AvoidableMiss is the nominated alarm target (see emf.py): a + // prefix-stability regression flips a large share of calls to + // `miss_avoidable`, showing up as a step change in this sum. + new cloudwatch.Alarm(this, 'PromptCacheAvoidableMissAlarm', { + alarmName: getResourceName(config, 'prompt-cache-avoidable-miss'), + alarmDescription: + 'Avoidable prompt-cache misses exceeded threshold β€” likely a prompt-prefix stability regression', + metric: avoidableMissMetric, + threshold: config.production ? 10 : 50, + evaluationPeriods: 3, + comparisonOperator: cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD, + treatMissingData: cloudwatch.TreatMissingData.NOT_BREACHING, + }); + + new cloudwatch.Alarm(this, 'PromptCacheWastedUsdAlarm', { + alarmName: getResourceName(config, 'prompt-cache-wasted-usd'), + alarmDescription: + 'Dollars wasted on avoidable prompt-cache re-writes exceeded threshold', + metric: wastedUsdMetric, + threshold: config.production ? 1 : 5, + evaluationPeriods: 3, + comparisonOperator: cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD, + treatMissingData: cloudwatch.TreatMissingData.NOT_BREACHING, + }); + + // ============================================================ + // Outputs + // ============================================================ + + new cdk.CfnOutput(this, 'PromptCacheDashboardName', { + value: dashboard.dashboardName, + description: 'CloudWatch Dashboard for prompt-cache observability', + exportName: `${config.projectPrefix}-PromptCacheDashboard`, + }); + } +} diff --git a/infrastructure/lib/platform-stack.ts b/infrastructure/lib/platform-stack.ts index 48bfecd49..5ed350997 100644 --- a/infrastructure/lib/platform-stack.ts +++ b/infrastructure/lib/platform-stack.ts @@ -70,6 +70,9 @@ import { AgentCoreGatewayConstruct } from './constructs/gateway/agentcore-gatewa import { McpSandboxBucketConstruct } from './constructs/mcp-sandbox/mcp-sandbox-bucket-construct'; import { McpSandboxDistributionConstruct } from './constructs/mcp-sandbox/mcp-sandbox-distribution-construct'; +// Observability (cross-service dashboards + alarms) +import { PromptCacheObservabilityConstruct } from './constructs/observability/prompt-cache-observability-construct'; + // Fine-tuning (data half lives in Platform) import { FineTuningDataConstruct } from './constructs/fine-tuning/fine-tuning-data-construct'; @@ -630,6 +633,15 @@ export class PlatformStack extends cdk.Stack { frontendUrl, documentsBucket: this.ragDocumentsBucket, }); + + // ============================================================ + // Prompt-cache observability β€” cross-service dashboard + alarms + // over the EMF metrics both APIs emit (PR #697 follow-up). + // Metric-namespace-based, so no typed refs are needed. + // ============================================================ + new PromptCacheObservabilityConstruct(this, 'PromptCacheObservability', { + config, + }); } /** diff --git a/infrastructure/package-lock.json b/infrastructure/package-lock.json index a5ac74c81..d9666af9b 100644 --- a/infrastructure/package-lock.json +++ b/infrastructure/package-lock.json @@ -1,12 +1,12 @@ { "name": "infrastructure", - "version": "1.8.0", + "version": "1.9.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "infrastructure", - "version": "1.8.0", + "version": "1.9.0", "dependencies": { "aws-cdk-lib": "2.260.0", "constructs": "10.6.0" diff --git a/infrastructure/package.json b/infrastructure/package.json index 75781bdcc..d12452285 100644 --- a/infrastructure/package.json +++ b/infrastructure/package.json @@ -1,6 +1,6 @@ { "name": "infrastructure", - "version": "1.8.0", + "version": "1.9.0", "bin": { "infrastructure": "bin/infrastructure.js" }, diff --git a/infrastructure/test/prompt-cache-observability.test.ts b/infrastructure/test/prompt-cache-observability.test.ts new file mode 100644 index 000000000..75aebfc74 --- /dev/null +++ b/infrastructure/test/prompt-cache-observability.test.ts @@ -0,0 +1,86 @@ +import * as cdk from 'aws-cdk-lib'; +import { Template } from 'aws-cdk-lib/assertions'; + +import { PromptCacheObservabilityConstruct } from '../lib/constructs/observability/prompt-cache-observability-construct'; +import { createMockConfig, MOCK_ACCOUNT, MOCK_PREFIX, MOCK_REGION } from './helpers/mock-config'; + +function synth(production: boolean): Template { + const app = new cdk.App(); + const stack = new cdk.Stack(app, 'Test', { + env: { account: MOCK_ACCOUNT, region: MOCK_REGION }, + }); + const config = createMockConfig({ production }); + new PromptCacheObservabilityConstruct(stack, 'PromptCacheObservability', { + config, + }); + return Template.fromStack(stack); +} + +describe('PromptCacheObservabilityConstruct', () => { + let t: Template; + beforeAll(() => { + t = synth(false); + }); + + it('creates the dashboard with the conventional name', () => { + t.hasResourceProperties('AWS::CloudWatch::Dashboard', { + DashboardName: `${MOCK_PREFIX}-prompt-cache-observability`, + }); + }); + + it('dashboard graphs the EMF namespace metrics and queries the runtime log group', () => { + const dashboards = t.findResources('AWS::CloudWatch::Dashboard'); + const body = JSON.stringify(Object.values(dashboards)[0].Properties.DashboardBody); + for (const metric of ['CacheReadTokens', 'CacheWriteTokens', 'AvoidableMiss', 'WastedUsd']) { + expect(body).toContain(metric); + } + expect(body).toContain('AgentCoreStack/PromptCache'); + expect(body).toContain(`/aws/bedrock-agentcore/runtimes/${MOCK_PREFIX}`); + expect(body).toContain('cacheStatus'); + }); + + it('creates exactly two alarms, both NOT_BREACHING on missing data (kill-switch tolerant)', () => { + t.resourceCountIs('AWS::CloudWatch::Alarm', 2); + const alarms = Object.values(t.findResources('AWS::CloudWatch::Alarm')); + for (const alarm of alarms) { + expect(alarm.Properties.TreatMissingData).toBe('notBreaching'); + expect(alarm.Properties.Namespace).toBe('AgentCoreStack/PromptCache'); + expect(alarm.Properties.Statistic).toBe('Sum'); + // Console-only: no SNS wiring yet anywhere in the stack. + expect(alarm.Properties.AlarmActions).toBeUndefined(); + } + }); + + it('alarms on AvoidableMiss (the nominated target) and WastedUsd', () => { + t.hasResourceProperties('AWS::CloudWatch::Alarm', { + AlarmName: `${MOCK_PREFIX}-prompt-cache-avoidable-miss`, + MetricName: 'AvoidableMiss', + Threshold: 50, + EvaluationPeriods: 3, + ComparisonOperator: 'GreaterThanThreshold', + }); + t.hasResourceProperties('AWS::CloudWatch::Alarm', { + AlarmName: `${MOCK_PREFIX}-prompt-cache-wasted-usd`, + MetricName: 'WastedUsd', + Threshold: 5, + }); + }); + + it('uses stricter thresholds in production', () => { + const prod = synth(true); + prod.hasResourceProperties('AWS::CloudWatch::Alarm', { + MetricName: 'AvoidableMiss', + Threshold: 10, + }); + prod.hasResourceProperties('AWS::CloudWatch::Alarm', { + MetricName: 'WastedUsd', + Threshold: 1, + }); + }); + + it('exports the dashboard name', () => { + t.hasOutput('*', { + Export: { Name: `${MOCK_PREFIX}-PromptCacheDashboard` }, + }); + }); +});