Release/1.9.0

CHANGELOG.md

@@ -4,6 +4,37 @@ All notable changes to this project are documented in this file. Format follows
 
 For narrative release notes written for operators and product owners, see [RELEASE_NOTES.md](RELEASE_NOTES.md).
 
+## [1.9.0] - 2026-07-20
+
+Feature release making **Bedrock prompt-cache economics stable and measurable**. Three cache-busting defects in the model-call path are fixed (per-turn history mutation, nondeterministic skill ordering, single-cachePoint fragility), and a new observability layer makes every model call's cache behavior diagnosable: prefix fingerprints and a `cacheStatus` classification on each cost row, an admin Session Cost Anatomy drill-down page, CloudWatch EMF metrics, and a dashboard with alarms. Also fixes chat-input textarea sizing and points the deployed runtime's Word-document tools at the real user-files bucket. Requires a CDK deploy (new dashboard construct + one runtime env var); no data migration.
+
+### 🚀 Added
+
+- Prompt-cache observability per model call — `PrefixFingerprintHook` hashes the three cacheable prefix components (toolConfig, effective system prompt, message history) onto each cost row; write-time `cacheStatus` classification (`first_write | hit | miss_ttl_expired | miss_avoidable | uncached`) with `wastedUsd` for avoidable misses priced from the row's own pricing snapshot; session-row cache rollups (`totalCacheReadTokens`, `totalCacheWriteTokens`, `avoidableMissCount`, `wastedUsd`); `GET /admin/costs/sessions/{sessionId}/calls` admin endpoint; and per-call CloudWatch EMF metrics. Kill switch `PROMPT_CACHE_OBSERVABILITY_ENABLED=false` (default ON) (#697)
+- Admin **Session Cost Anatomy** page at `/admin/costs/sessions/:id` — chronological per-call table with color-coded `cacheStatus` badges and prefix-fingerprint diffing that names which hash (tools/system/history) flipped versus the previous call, plus session-level cache summary and a session-id lookup form on the Cost Analytics dashboard (#700)
+
+### ⚡ Performance
+
+- Restored conversation history is now byte-stable between turns — tool-content truncation is driven by a persisted `truncation_anchor` in compaction state instead of a sliding window that re-mutated history every turn, which forced a full prompt-cache prefix re-write (~$2.50/MTok on a 35k–150k-token prefix) nearly every turn (#697)
+- Skill records reach the `<available_skills>` system-prompt block in deterministic order (sorted at the repository, RBAC-union, and injection layers), removing a per-turn prompt-cache invalidator (#697)
+- Model calls now carry 3 of Bedrock's max-4 cachePoints (toolConfig tail, system tail, last-user-message) so a message-level cache miss still reads the stable tools+system prefix from cache instead of re-writing everything; the added points are gated to models that support them (#697)
+
+### 🐛 Fixed
+
+- Chat input textarea is scrollable once content exceeds its max height, clamps its growth, and resets to its base height after sending (#696)
+- The first cache write after a run of below-threshold (uncached) calls is classified `first_write`, not `miss_avoidable` — it no longer inflates the AvoidableMiss/WastedUsd metrics or the Session Cost Anatomy page (#701)
+- Word-document tools on the deployed runtime save to the real user-files bucket — `S3_USER_FILES_BUCKET_NAME` is now set on the inference-api Runtime, where its absence made the tools fall back to a literal `user-files` bucket and fail with `AccessDenied` (#702)
+- Word-document tools fail fast with a clear "storage is not configured" message when the bucket env var is unset, instead of surfacing a confusing S3 `PermanentRedirect`/`AccessDenied` mid-run (#706)
+
+### 🏗️ Infrastructure
+
+- `PromptCacheObservabilityConstruct` (`lib/constructs/observability/`) — CloudWatch dashboard over the `AgentCoreStack/PromptCache` EMF metrics (cache read/write tokens, cache-efficiency expression, AvoidableMiss, WastedUsd) with a Logs Insights widget grouped by `cacheStatus`, plus console-only alarms on AvoidableMiss and WastedUsd (stricter in prod; `NOT_BREACHING` on missing data so the kill switch stays quiet) (#699)
+- `S3_USER_FILES_BUCKET_NAME` added to the inference-api AgentCore Runtime environment (#702)
+
+### 📚 Docs
+
+- CLAUDE.md — prompt-cache determinism/byte-stability contract, the fingerprint-based cache-miss debugging recipe, and a token-cost-effectiveness design tenet (#697, #698)
+
 ## [1.8.0] - 2026-07-19
 
 Feature release delivering **Skills v2** — skills are redesigned from tool-binding containers into pure, portable knowledge bundles that Agents load on demand, and for the first time **any signed-in user can author their own**. Skills ship enabled by default. Also completes the session-metadata static-sort-key migration (issue #175 Phases 2–3) with an operator backfill and a GSI-only read contraction, and consolidates the two artifact tool-catalog rows into a single "Artifacts" toggle. **Three backfill scripts must be run per environment** — see [RELEASE_NOTES.md](RELEASE_NOTES.md) deployment notes. No new AWS resources; a CDK deploy is needed only to set the `SKILLS_ENABLED` env var explicitly.

CLAUDE.MD

@@ -39,6 +39,8 @@ npx cdk deploy {prefix}-PlatformStack
 - **Admin endpoints** go under `/admin/<domain>/`, user-facing under `/<domain>/`
 - **Errors stream as assistant messages** via SSE (not HTTP error codes)
 - **Signal-based state** throughout frontend (`signal()`, `computed()`)
+- **Prompt-cache stability is a contract.** Bedrock prompt caching is exact-prefix-match: any list that reaches the system prompt or `toolConfig` (skills, tools, models, MCP tool listings) must be deterministically ordered at its source, and restored conversation history must be byte-stable between compaction-state changes (see the truncation anchor in `TurnBasedSessionManager`). An order flip or history mutation between turns silently re-writes a 30k–150k-token prefix at the $2.50/MTok cache-write premium. `PROMPT_CACHE_OBSERVABILITY_ENABLED=false` disables the observability layer (fingerprint hook, cacheStatus derivation, EMF metrics) — the caching itself stays on
+- **Token cost effectiveness is a design tenet — engineer against waste, not against context.** Before merging a change that touches the model call path, answer: what does this add to the prompt, on every turn, for the life of every session? (1) Anything in the cacheable prefix (system prompt, `toolConfig`, restored history) must be deterministic and append-only — see the prompt-cache contract above. (2) Per-turn payloads (tool results, MCP responses, retrieved documents) should be bounded or offloaded, never unbounded pass-through. (3) Don't guess — verify: `cacheStatus` + fingerprint hashes on the session's `C#` rows, `GET /admin/costs/sessions/{id}/calls`, and the `AgentCoreStack/PromptCache` EMF metrics exist to prove a change's cost impact. The balance: "cost effective" means eliminating waste (avoidable cache re-writes, duplicated context, oversized payloads) — it never means stripping context the model needs for a quality answer. When cost and answer quality genuinely conflict, quality wins; look for the cheaper path to the *same* quality, not a cheaper answer
 - **All dependencies use exact version pins** — no `^`, `~`, or `>=`
 - **Never install packages without explicit user approval**
 
@@ -118,6 +120,7 @@ Do not reintroduce a Bearer-only `Depends(...)` on any user-facing route. If you
 - **Tool not appearing:** Check `__init__.py` export, RBAC permissions, `enabled_tools`, ToolRegistry
 - **Session not persisting:** Check AgentCore Memory config, session_id, TurnBasedSessionManager flush
 - **SSE stream disconnecting:** Check 600s timeout, client connection, quota exceeded events
+- **Cost spike / cache miss:** Check `cacheStatus` + `toolConfigHash`/`systemPromptHash`/`historyHash` on the session's `C#` rows in sessions-metadata (or `GET /admin/costs/sessions/{id}/calls`) — the hash that changed between consecutive calls names the cache-buster
 
 ## Coding Standards
 

README.md

@@ -8,7 +8,7 @@
 **An open-source, production-ready Generative AI platform for institutions**
 *Built by Boise State University, designed for everyone.*
 
-[![Release](https://img.shields.io/badge/Release-v1.8.0-6366f1?style=flat&logo=github&logoColor=white)](RELEASE_NOTES.md)
+[![Release](https://img.shields.io/badge/Release-v1.9.0-6366f1?style=flat&logo=github&logoColor=white)](RELEASE_NOTES.md)
 [![Nightly](https://github.com/Boise-State-Development/agentcore-public-stack/actions/workflows/nightly.yml/badge.svg)](https://github.com/Boise-State-Development/agentcore-public-stack/actions/workflows/nightly.yml)
 
 ![Python](https://img.shields.io/badge/Python-3.13+-3776AB?style=flat&logo=python&logoColor=white)
@@ -296,7 +296,7 @@ agentcore-public-stack/
 
 See [RELEASE_NOTES.md](RELEASE_NOTES.md) for the full changelog, including new features, bug fixes, platform upgrades, and deployment notes for each release.
 
-**Current release:** v1.8.0
+**Current release:** v1.9.0
 
 ---
 

RELEASE_NOTES.md

@@ -1,3 +1,69 @@
+# Release Notes — v1.9.0
+
+**Release Date:** July 20, 2026
+**Previous Release:** v1.8.0 (July 19, 2026)
+
+---
+
+> 🏗️ **CDK deploy required this release** — a new CloudWatch dashboard construct and one new env var on the inference-api Runtime. No data migration, no backfills, no dependency changes. Standard order: `platform.yml` → `backend.yml` → `frontend-deploy.yml`.
+
+---
+
+## Highlights
+
+v1.9.0 makes **Bedrock prompt-cache spend stable and measurable**. A production conversation audit found 75% of one session's cost was avoidable cache re-writes; this release fixes the three defects behind that waste — restored history that mutated every turn, skills injected in nondeterministic order, and a single fragile cachePoint — and adds the observability to prove it and catch regressions: every model call now records prefix fingerprints and a `cacheStatus` verdict, admins get a per-session **Cost Anatomy** drill-down page, and operators get a CloudWatch dashboard with alarms on avoidable waste. Smaller fixes: the chat input textarea scrolls and resets correctly, and Word-document saves work on the deployed runtime (the container wasn't told which S3 bucket to use).
+
+## Prompt-cache stability — stop paying for avoidable re-writes
+
+Bedrock prompt caching is exact-prefix-match: if any byte of the cached prefix changes between turns, the whole prefix re-writes at the $2.50/MTok cache-write premium instead of reading at the ~$0.30/MTok cache-read rate. On a typical 35k–150k-token session prefix, one silent cache-buster costs more per turn than most turns' actual work. Three were found and fixed (#697):
+
+- **Restored history is now byte-stable.** Tool-content truncation ran on every session restore behind a sliding protected-turns window, so each new turn re-mutated the turn that had just aged past the window — invalidating the cache nearly every turn. Truncation is now driven by a persisted `truncation_anchor` in the session's compaction state: it advances only when the compaction checkpoint advances (where the slice already pays a single re-write) or opportunistically when the cache TTL (default 300s, `AGENTCORE_MEMORY_COMPACTION_CACHE_TTL_SECONDS`) has lapsed since the previous turn and the cache entry is dead anyway — so pending truncations apply for free.
+- **Skills inject in deterministic order.** Skill records reached the `<available_skills>` system-prompt block in whatever order DynamoDB `batch_get_item` and Python set iteration produced, changing the system prompt between turns of the same session. Ordering is now sorted at three layers: the skills repository, the RBAC grant-union resolver (which returned `list(set)` — order varies per process via hash randomization), and the injection point itself.
+- **Three cachePoints instead of one.** The auto strategy places a single message-level cachePoint; when its lookup misses (one proven mode: a wide parallel tool fan-out pushes the previous checkpoint past Anthropic's ~20-block cache lookback), *nothing* was read and the entire prefix re-wrote. Requests now carry 3 of Bedrock's max-4 cachePoints — toolConfig tail, system-prompt tail, and the existing last-user-message point — so a message-level miss still reads the stable tools+system prefix from cache. The added points are gated on `ModelConfig.bedrock_cache_points_supported()` since non-Anthropic models reject them.
+
+## Prompt-cache observability — every model call explains its cache behavior
+
+Diagnosing the waste above originally took hours of manual forensics against raw DynamoDB cost rows. That whole class of investigation is now a column diff (#697, #700, #699, #701).
+
+### Backend
+
+- `PrefixFingerprintHook` (a Strands `BeforeModelCallEvent` hook) hashes the three cacheable prefix components per model call — toolConfig (order-sensitive canonical JSON), the effective system prompt captured *after* AgentSkills injection, and message history excluding the newest message — and the stream coordinator persists them on the turn's cost rows. When a cache miss happens, the hash that changed between consecutive calls names the cache-buster.
+- Each cost row gets a write-time `cacheStatus` — `first_write`, `hit`, `miss_ttl_expired`, `miss_avoidable`, or `uncached` — derived against the session's previous cost row, plus `wastedUsd` for avoidable misses priced at the cache-write premium over cache-read from the row's own pricing snapshot. Turn rows now write sequentially so each call classifies against its true predecessor. A follow-up fix (#701) classifies the first write after a run of below-threshold calls as `first_write` rather than `miss_avoidable`, so short-prompt sessions don't inflate the waste metrics.
+- Session rows carry rollups next to `totalCost` — `totalCacheReadTokens`, `totalCacheWriteTokens`, `avoidableMissCount`, `wastedUsd` — so lists and admin views get a cache-efficiency ratio without scanning cost rows.
+- `GET /admin/costs/sessions/{sessionId}/calls` (admin-only) returns the chronological per-call rows with token splits, cost, `cacheStatus`, and fingerprints, plus a session-level cache summary.
+- Everything derived is behind `PROMPT_CACHE_OBSERVABILITY_ENABLED` (default ON, `=false` to disable the hook, the classification's extra GSI read, and EMF emission). Raw cache read/write token rollups are unaffected — they're usage passthrough, not derived.
+
+### Frontend
+
+- New **Session Cost Anatomy** page at `/admin/costs/sessions/:id`: summary tiles (total cost, cache efficiency, avoidable misses, wasted USD, cache read/write tokens) over a chronological calls table with color-coded `cacheStatus` badges. Fingerprint diffing flags which hash — tools, system, or history — flipped versus the previous fingerprinted call, which is the diagnosis on any `miss_avoidable` row. Expandable rows show full hashes and message counts; a session-id lookup form on the Cost Analytics dashboard is the entry point.
+
+### Infrastructure
+
+- `PromptCacheObservabilityConstruct` (new `lib/constructs/observability/` area, composed into `PlatformStack`) builds a CloudWatch dashboard over the `AgentCoreStack/PromptCache` EMF namespace both APIs emit into: cache read/write token trends, a cache-efficiency MathExpression, AvoidableMiss and WastedUsd, and a Logs Insights widget grouped by `cacheStatus`. Console-only alarms on AvoidableMiss and WastedUsd Sums (stricter thresholds in prod, `NOT_BREACHING` on missing data so the kill switch keeps them quiet). Deliberately no SNS — alerting infra remains out of scope, matching kb-sync and scheduled runs.
+
+### Test Coverage
+
+~1,800 lines of new tests: fingerprint/classification unit tests (including the below-threshold regression), cachePoint position and budget assertions, forced-order skill-sorting regressions, CDK construct assertions, and Vitest specs for the anatomy page, diff util, and HTTP service.
+
+## 🐛 Bug fixes
+
+- **The chat input became unusable on long prompts.** The textarea carried `overflow-hidden` with unclamped height growth, so past its 200px cap the content could neither be seen nor scrolled — and after sending, the box stayed expanded. Growth is now clamped with scrolling enabled past the cap, and the input resets to its base height on submit (#696)
+- **Word-document saves failed on the deployed runtime with `AccessDenied`.** The AgentCore Runtime env set the user-files *table* name but not `S3_USER_FILES_BUCKET_NAME`, so the Word tools fell back to a literal `user-files` bucket the role has no access to. The env var is now set on the Runtime (#702), and the tools fail fast with a clear "storage is not configured" message — before spending a Code Interpreter run — if the variable is ever missing again (#706)
+
+## 🏗️ Infrastructure
+
+- New CloudWatch dashboard + alarms construct (see spotlight above) — CloudWatch-console resources only, no SNS, no new IAM of note.
+- `S3_USER_FILES_BUCKET_NAME` on the inference-api Runtime environment; the role's existing `UserFilesBucketAccess` grant already covers the bucket (#702)
+- New env var `PROMPT_CACHE_OBSERVABILITY_ENABLED` on app-api and inference-api (default ON; set `=false` per environment to disable the observability layer — caching itself stays on).
+
+## 🚀 Deployment notes
+
+Standard order, and this release uses all three: **`platform.yml` first** (the dashboard construct and the Runtime env var are CDK changes; the runtime picks up its current image via SSM, so the infra deploy is safe on its own) → `backend.yml` → `frontend-deploy.yml`. No backfills, no data migration, no dependency changes.
+
+After deploy, the **PromptCache dashboard** appears in the CloudWatch console. Expect `first_write` rows at the start of sessions and after idle gaps — only `miss_avoidable` indicates regression. The observability layer is per-call metadata; if it ever needs to be silenced in an environment, set `PROMPT_CACHE_OBSERVABILITY_ENABLED=false` and redeploy that service — the alarms go quiet on missing data by design.
+
+---
+
 # Release Notes — v1.8.0
 
 **Release Date:** July 19, 2026

VERSION

@@ -1 +1 @@
-1.8.0
+1.9.0

backend/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "agentcore-stack"
-version = "1.8.0"
+version = "1.9.0"
 requires-python = ">=3.10"
 description = "Multi-agent conversational AI system with AWS Bedrock AgentCore"
 readme = "README.md"