From 1b0e8bb9262c1be537bc09640c7d43e4f1ec5267 Mon Sep 17 00:00:00 2001 From: "Ramanan Vellore Ramesh (from Dev Box)" Date: Thu, 17 Sep 2026 16:51:19 -0700 Subject: [PATCH 1/8] ESS Diagnostics: interactive transcript-diagnosis skill MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replace the standalone Python diagnostics tool with a single self-contained Claude skill (tools/ESS-Diagnostics-Skills/SKILL.md) that walks a Field Engineer through a Copilot Studio / PVA transcript turn-by-turn, runs a 5-check diagnostic with quoted evidence, and writes a Debug Report plus a normalized-diagnostic JSON and a faithful full-transcript JSON to the OS temp directory (never the repo — transcripts contain PII). Highlights: - Step 0 hard gate: confirm file path + problem statement before parsing. - Interactive per-turn walkthrough: structured summary, section drill-down, verdict override, continue / run all. - Real-schema parse map (Trace/p.data.kind, SynchronousIncomingActivity, AnalyticsAiMetricsSignalTraceData / KnowledgeTraceData). - 5 checks with divergence + root cause; search results show source text for every result. - scripts/transcript-to-json.js: lossless transcript-to-JSON helper. - Design + implementation docs under docs/plans/. Co-Authored-By: Claude --- tools/ESS-Diagnostics-Skills/README.md | 75 ++++ tools/ESS-Diagnostics-Skills/SKILL.md | 388 ++++++++++++++++++ ...6-09-17-ess-diagnostics-skill-v4-design.md | 149 +++++++ ...ess-diagnostics-skill-v4-implementation.md | 126 ++++++ .../scripts/transcript-to-json.js | 114 +++++ 5 files changed, 852 insertions(+) create mode 100644 tools/ESS-Diagnostics-Skills/README.md create mode 100644 tools/ESS-Diagnostics-Skills/SKILL.md create mode 100644 tools/ESS-Diagnostics-Skills/docs/plans/2026-09-17-ess-diagnostics-skill-v4-design.md create mode 100644 tools/ESS-Diagnostics-Skills/docs/plans/2026-09-17-ess-diagnostics-skill-v4-implementation.md create mode 100644 tools/ESS-Diagnostics-Skills/scripts/transcript-to-json.js diff --git a/tools/ESS-Diagnostics-Skills/README.md b/tools/ESS-Diagnostics-Skills/README.md new file mode 100644 index 000000000..40869a9d1 --- /dev/null +++ b/tools/ESS-Diagnostics-Skills/README.md @@ -0,0 +1,75 @@ +# ESS Diagnostics Skills + +A read-only Claude skill for diagnosing ESS Custom Engine Agent transcripts. The +full procedure lives in [`SKILL.md`](./SKILL.md). + +## What it is + +`SKILL.md` is a read-only Claude skill that walks a Field Engineer (FDE) +turn-by-turn through a Copilot Studio / PVA transcript, runs a 5-check +diagnostic on each turn, and produces an evidence-cited Debug Report. It +replaces the earlier standalone Python diagnostics tool. + +## How an FDE uses it + +Invoke the skill (via its slash/skill mechanism). Before it parses anything, the +skill **stops at a hard gate (Step 0)**: it asks the FDE to confirm the +transcript **file path** and to state the **specific problem** being +investigated. It will not proceed without a problem statement — even if one was +supplied in the invocation, it echoes that back for the FDE to confirm or refine. + +Only then does it read and parse the transcript. It walks each turn and **STOPS +at every turn**, presenting a structured summary (intent, search query + +rewrite, search response, plus a one-line verdict strip) and a section menu. At +each pause the FDE can: + +- **Drill into any section (1–5)** — see the full raw transcript evidence for + that section plus the reasoning behind that check's verdict. +- **Override a verdict** — the FDE's override replaces the skill's verdict for + divergence and root-cause purposes. +- **`continue`** — advance to the next turn. +- **`run all`** — finish the remaining turns without pausing. + +## The 5 checks + +- **CHECK-001 — Intent Recognition:** was the user's intent recognized (fails on + no intent event, or on `triggeredGptFallback = true`). +- **CHECK-002 — Search Query Issued:** shows the original prompt, the issued + query, and any rewrite. +- **CHECK-003 — Search Result Topical Relevance:** relevance of the returned + results to the query/keywords (can use `verifiedSearchResults[].rankScore`, + else keyword overlap). +- **CHECK-004 — Knowledge Grounding Consistency:** did the answer's grounding + match what was retrieved — judged from `isKnowledgeSearched`, + `completionState`, `citedKnowledgeSources`, and the `verifiedSearchResults` + count versus what the final answer does. +- **CHECK-005 — Final Answer vs. Retrieved Content / Guardrails:** did the final + answer align with what was retrieved (or decline appropriately). + +## Outputs + +Because transcripts contain employee PII, the outputs are written **outside the +repo**, into a per-run subfolder in the OS temporary directory, named from the +transcript's basename: + +- **Windows:** `%TEMP%\ess-diagnostics\\` +- **POSIX:** `$TMPDIR` (or `/tmp` when unset) `/ess-diagnostics//` + +Two files are written there: + +- **`-debug-report.md`** — the human-readable Debug Report with per-turn + verdicts, any FDE verdict overrides, the divergence point, and the root cause. +- **`-normalized.json`** — a clean, normalized parsed transcript artifact + containing NO verdicts. + +## Read-only guarantee + +The skill never modifies the transcript or any agent files. Its only writes are +the two output files above, in the OS temp directory — never in the repo. + +## Testing / validation + +There is no automated test suite — the skill is prose, not code. Validate it +with a manual dry run against a real (redacted) transcript. Outputs are written +outside the repo specifically to avoid committing PII; still, never commit +un-redacted real transcripts, which may contain PII or employee data. diff --git a/tools/ESS-Diagnostics-Skills/SKILL.md b/tools/ESS-Diagnostics-Skills/SKILL.md new file mode 100644 index 000000000..4268e5f96 --- /dev/null +++ b/tools/ESS-Diagnostics-Skills/SKILL.md @@ -0,0 +1,388 @@ +# Diagnose an ESS Transcript + +Read-only diagnostic skill that walks a Field Engineer (FDE) turn-by-turn +through a Copilot Studio / PVA transcript (a JSON payload exported as a `.txt` +file), running a 5-check diagnostic on each turn with evidence quoted straight +from the transcript. It produces an evidence-cited markdown Debug Report plus a +normalized-transcript JSON artifact. This skill replaces the earlier standalone +Python diagnostics tool. + +## Rules + +- This skill is **read-only**. The ONLY writes it may make are the two output + files: the markdown Debug Report and the normalized-transcript JSON. Never + modify the transcript, agent files, or anything else. +- Do NOT narrate your internal process. Say "Let me walk you through this + transcript" not "Let me read the parse map." Speak directly to the FDE. +- **Do not proceed past Step 0 until the FDE has stated the problem and you + have confirmed the transcript file path.** This is a hard stop: ask even if a + problem statement was already supplied in the invocation — echo the supplied + statement back and have the FDE confirm or refine it. Never parse the + transcript before the problem is established. +- Every check verdict MUST quote the specific transcript field value it rests + on (the actual query string, result names, and so on). Make no claim the + transcript does not support. +- Walk the transcript one turn at a time. After presenting a turn's checks, + pause and ask the FDE to confirm or add context before advancing to the next + turn. +- Track progress with todos if the transcript has many turns. + +## Step 0: Establish the Problem + +On invocation, STOP before doing anything else. Do not read or parse the +transcript yet — the problem must be established first. + +Confirm the transcript **file path** with the FDE, stating the path you were +given. Then ask explicitly: "What specific problem are you investigating in this +transcript?" + +If a problem statement was supplied in the invocation arguments, echo it back +verbatim and ask the FDE to confirm it as-is or refine it — do NOT silently +accept it. + +Only once the FDE has given or confirmed the problem statement AND the file path +is confirmed may you proceed to Step 1. Keep the confirmed problem statement — it +is cited in the final report. + +## Step 1: Read and Parse the Transcript + +Read the transcript file (a JSON payload stored in a `.txt`) and extract the +events you will need for the checks. + +**Iterate events by their array position in the transcript, NOT by `seq`.** +`seq` is unreliable — some event kinds (e.g. `TranscriptTrace`) repeat +`seq: -1` many times. The array/list order is canonical; walk it in order. + +Top-level events are objects with `t` (the event kind) and `p` (its payload). +All diagnostic signals are `t = "Trace"` events discriminated by `p.data.kind`. +Extract events using this parse map: + +| Concept | Where it lives | Fields | +| --- | --- | --- | +| User utterance | `t = SynchronousIncomingActivity` (only `p.activity.type = "message"`) | `p.activity.text` | +| Bot message | `t = OutgoingActivity` (only `p.activity.type = "message"`) | `p.activity.text` | +| Intent recognized | `t = Trace`, `p.data.kind = LlmIntentRecognized` | `intentId`, `intentMessage`, `userUtterance` | +| Search issued | `t = Trace`, `p.data.kind = PluginStart` | `p.data.input.search_query`, `p.data.input.search_keywords`, `pluginName` | +| Search results | `t = Trace`, `p.data.kind = PluginResponse` | `citableContent[]` (`source`, `chunks`), `output` | +| Grounding / answer signal | `t = Trace`, `p.data.kind = AnalyticsAiMetricsSignalTraceData` | `completionState`, `triggeredGptFallback`, `rewrittenMessage`, `rewrittenMessageKeywords`, `verifiedSearchResults[]` (`url`, `rankScore`, `snippet`), `citedKnowledgeSources`, `textCitations` | +| Knowledge search state | `t = Trace`, `p.data.kind = KnowledgeTraceData` | `isKnowledgeSearched`, `completionState`, `citedKnowledgeSources`, `failedKnowledgeSourcesTypes` | + +Only activities with `p.activity.type = "message"` are real user or bot turns; +activities with `p.activity.type = "event"` are system plumbing and must be +skipped for turn segmentation. Result shapes differ by trace: +`PluginResponse.citableContent[]` items have `source` + `chunks`, while +`AnalyticsAiMetricsSignalTraceData.verifiedSearchResults[]` items have `url`, +`rankScore`, `snippet`, `searchType` — a `rankScore` DOES exist there. + +If a mapped field or event kind is absent in a given transcript, mark the +affected check **N/A** with a note like "field not present in this transcript" +rather than failing or guessing. This map derives from observed real exports; +field names may vary by bot configuration. + +Segment the events into turns. A **turn** is a user utterance plus the bot +activity and trace events that follow it, up to (but not including) the next +user utterance. Group events into turns by walking array order. + +After parsing, tell the FDE how many turns were found before starting the +per-turn walkthrough. + +### Faithful transcript-to-JSON (helper script) + +The transcript is a large JSON-in-`.txt` payload. To give the FDE a complete, +readable JSON copy of the WHOLE transcript — every event and field preserved, +not just the diagnostic fields — run the bundled helper rather than hand-parsing +it: + +``` +node scripts/transcript-to-json.js "" +``` + +(run from the skill directory `tools/ESS-Diagnostics-Skills/`). It writes a +lossless, pretty-printed `-transcript.json` into the same OS +temp folder used for the other outputs (see Step 4), annotating each event with +a `_turn` and `_index` but removing nothing, and prints the absolute output path +(or an `Error: ...` line on failure — relay that to the FDE rather than +retrying blindly). This faithful dump is SEPARATE from the compact +normalized-diagnostic JSON written in Step 4; produce both. + +## Step 2: Walk Each Turn Through the 5 Checks + +Walk the turns in array order from Step 1. For each turn, first present the +user utterance and the bot's response, then run **CHECK-001** through +**CHECK-005** in order. Give each check a verdict of **Pass**, **Fail**, or +**N/A**, and quote the specific transcript field value that the verdict rests +on as evidence. + +If an earlier check in the same turn already failed and that failure removes a +later check's precondition (for example, no intent was recognized, so no search +could fire), mark the later check **N/A** rather than **Fail**, and note which +earlier check it depends on — this avoids counting one root cause as several +failures. + +### Interactive pause protocol + +After running the 5 checks for a turn, do NOT advance on your own. Present a +compact structured summary, then a section menu, and STOP. Wait for the FDE. +You may NOT look at or process the next turn until the FDE responds. + +First, present a **structured summary** of the turn — not just the verdicts: + +- **Intent** — the recognized intent (`intentId` / `intentMessage`), or, when + recognition did not land, the failure signal (`triggeredGptFallback`). +- **Search query** — the original user prompt AND the issued / rewritten query + and keywords. +- **Search response** — a compact one-line-per-result list of EVERY returned + result (not just the count): for each result show its identifier (e.g. the KB + number from the source text), its title / short description, and its + `rankScore`. This makes a mis-ranked but topically correct source visible at a + glance. Then note the cited sources and what the bot actually answered. +- **Verdict strip** — a one-line summary, e.g. + `001 Pass · 002 Pass · 003 Pass · 004 Pass · 005 Pass`. + +Then present this menu and STOP — wait for the FDE before touching the next turn: + +``` +Drill into a section, or advance: + 1. Intent 2. Search query 3. Search results + 4. Grounding 5. Final answer + [continue] next turn [run all] finish without pausing [override] a verdict +``` + +From this first pause onward, if the FDE types `run all`, switch to a batch +pass: auto-advance through all remaining turns without pausing, still collecting +everything the full report needs. + +If the FDE picks a **section number (1–5)**, show the FULL raw transcript +evidence for that section — the complete `search_query` + `search_keywords`, the +full `verifiedSearchResults` / `citableContent` list, the full intent object, or +the complete bot answer text, as applies — PLUS the reasoning behind that +section's check verdict (why Pass / Fail / N/A). For **section 3 (Search +results)** specifically, show the **Source Text for EVERY result returned** — for +each result, its full `citableContent[].chunks` source text along with its +`source` / `url`, `rankScore`, and `searchType`. Do not show only the top result +or a snippet; list all of them, in the order returned, so the FDE can see +whether a topically correct source was retrieved but under-ranked or not cited. +Explicitly invite the FDE to OVERRIDE the verdict if their domain knowledge says +it is wrong. After a drill-down, return to the SAME turn's menu; do not +auto-advance. + +If the FDE **overrides** a verdict (via `[override]` or during a drill-down), +record it: which check, the skill's original verdict, the FDE's override +verdict, and the FDE's stated reason. Carry the override (with its reason) into +the final Debug Report. An overridden verdict REPLACES the skill's verdict for +divergence and root-cause purposes in Step 3. + +`continue` advances to the next turn. The loop repeats — summary, menu, hard +stop — for every turn until all turns are done, unless `run all` was chosen. + +- **CHECK-001 — Intent Recognition.** **Pass** if the turn has an + `LlmIntentRecognized` event (quote its `intentId` / `intentMessage`). + **Fail** if there is no intent event at all, OR the turn's + `AnalyticsAiMetricsSignalTraceData` carries `triggeredGptFallback = true` + (quote the `triggeredGptFallback` flag) — that fallback flag is the real + signal that intent recognition did not land. (Some exports may instead + surface an explicit unknown-intent event; the real signal here is + `triggeredGptFallback`.) This is the first divergence candidate — a fail here + often explains a downstream decline. + +- **CHECK-002 — Search Query Issued.** **Pass** if a `PluginStart` carries a + non-empty `p.data.input.search_query` or `p.data.input.search_keywords`. When + reporting this check, ALWAYS show three things: + 1. **Original user prompt** — the turn's user utterance from the + `SynchronousIncomingActivity` `p.activity.text`. (Prefer this: the + `userUtterance` field on the `LlmIntentRecognized` event is often empty.) + 2. **Issued search query / keywords** — the actual + `p.data.input.search_query` and `p.data.input.search_keywords` values. + 3. **Rewrite note** — read this DIRECTLY from `rewrittenMessage` / + `rewrittenMessageKeywords` on the `AnalyticsAiMetricsSignalTraceData`; do + NOT infer it. If a rewrite is present and materially different from the + issued query / original prompt, quote it (e.g. "query was rewritten to + ''"); otherwise state "no rewrite." Dropping + filler/stopwords while keeping the key search term(s) does NOT count as a + material rewrite — mark that "no rewrite." Treat it as a rewrite only when + the key term itself changed, was dropped, or was narrowed/broadened in a + way that could change what gets retrieved. A bad rewrite (dropping the key + term) can itself be the root cause. + + **Fail** if a knowledge answer was expected but no search fired. **N/A** if + the turn is not knowledge-seeking (e.g. a greeting). + +- **CHECK-003 — Search Result Topical Relevance.** **Pass** if the returned + results are topically relevant to the query/keywords. Prefer + `verifiedSearchResults[].rankScore` together with each result's `snippet` + where present (a `rankScore` DOES exist on + `AnalyticsAiMetricsSignalTraceData.verifiedSearchResults[]`). Where that is + absent, fall back to keyword overlap between the query/keywords and the + result's `snippet` / `source` or the `PluginResponse.citableContent[].chunks` + — ignore common English stopwords and tokens shorter than 2 chars, and + require a real shared term. **Fail** if results came back but none are + relevant. **N/A** if no search was issued. When reporting this check, list + EVERY returned result with its source text (`citableContent[].chunks`) and + `rankScore`, not just the top hit — a topically correct source that was + retrieved but under-ranked or not cited is itself the finding. + +- **CHECK-004 — Knowledge Grounding Consistency.** Judge this from the mapped + fields only: `isKnowledgeSearched`, `completionState`, + `citedKnowledgeSources`, and the `verifiedSearchResults` count, versus what + the bot's final `OutgoingActivity` `p.activity.text` does with them. **Pass** + if the two are consistent — the answer presents substantive knowledge AND + knowledge was searched / results were returned, OR the answer declines / says + it lacks information AND nothing usable was retrieved. **Fail** on a + mismatch — the answer presents specific knowledge as fact but nothing + (`verifiedSearchResults` / `citedKnowledgeSources`) backed it (ungrounded), or + results were returned yet the answer ignores them and declines. Quote the + `verifiedSearchResults` count / `completionState` / `citedKnowledgeSources` + and the relevant answer text as evidence. **N/A** if the turn is not + knowledge-seeking. + +- **CHECK-005 — Final Answer vs. Retrieved Content / Guardrails.** **Pass** if + the bot's final `OutgoingActivity` text aligns with what was retrieved + (`verifiedSearchResults` / `citedKnowledgeSources` / `textCitations`), OR the + bot appropriately declined / gave a guardrail response when no relevant + content was found. **Fail** if it answered with unsupported content (not + backed by retrieved results), or it declined despite having relevant results. + +## Step 3: Identify Divergence and Root Cause + +After all turns are walked, identify the FIRST failing check across the whole +transcript — the earliest turn, then the lowest check number within that turn. +That check is the **divergence point**. + +Compose a **root cause** statement derived from that first failure, citing the +evidence that supports it (the quoted query, result names, search_results count, +or answer text behind the failing check). + +If NO check failed anywhere, state that explicitly: there is no divergence and +the transcript looks healthy. Do not invent a problem where the evidence shows +none. + +## Step 4: Write the Outputs + +After the walkthrough and diagnosis, write **three files** into a per-run +subfolder **outside the repo**, in the OS temporary directory. These outputs may +contain employee PII (utterances, retrieved policy content), so they are written +outside the repo and must NEVER be committed to source control or shared outside +approved channels. + +Derive the file basename from the transcript's file name — for a transcript +`foo.txt`, the transcript name is `foo`. Write both files into a per-run +subfolder named for the transcript: + +- On **Windows**: `%TEMP%\ess-diagnostics\\` +- On **POSIX**: `$TMPDIR` (or `/tmp` when `$TMPDIR` is unset) + `/ess-diagnostics//` + +Create the subfolder if it does not exist. + +**1. Markdown Debug Report** — `/ess-diagnostics//-debug-report.md`. +This is the human-readable diagnosis. It MUST contain: + +- The **confirmed problem statement** (the one confirmed with the FDE in Step 0). +- **Per-turn checklist results** — for each turn, the user utterance and the bot + response, followed by the 5 check verdicts (**Pass** / **Fail** / **N/A**), + each with its quoted transcript evidence. +- **FDE verdict overrides** — a section listing every verdict the FDE overrode + during the walkthrough. For each override, list: which check, the skill's + original verdict, the FDE's override verdict, and the FDE's stated reason. Note + that the overridden verdicts are the ones used for the divergence and + root-cause analysis. If there were no overrides, state that. +- The **divergence point** (from Step 3). +- The **root cause** (from Step 3). + +**2. Normalized-transcript JSON** — `/ess-diagnostics//-normalized.json`. +This is a **normalized transcript ONLY** — a clean, flattened, reusable parsed +artifact. It carries NO diagnostic verdicts, checks, divergence, or root cause; +those live only in the markdown report. It is an array of turn objects in array +order. Each turn object contains: + +- `user_utterance` — the user's prompt text, from + `SynchronousIncomingActivity.p.activity.text`. +- `intent` — an object: + `{ "kind": "LlmIntentRecognized" | null, "intentId": ..., "intentMessage": ..., "userUtterance": ..., "triggeredGptFallback": }`. + Populate `intentId` / `intentMessage` / `userUtterance` from the + `LlmIntentRecognized` event; use `null` for any field that is absent, and + `null` for `kind` when no intent event fired. Represent intent-recognition + failure via `triggeredGptFallback` (read from the + `AnalyticsAiMetricsSignalTraceData`). +- `search` — an object: + `{ "original_prompt": ..., "query": , "keywords": , "rewritten_message": , "rewritten_keywords": , "rewritten": , "rewrite_note": ... }`. + `query` / `keywords` come from `PluginStart.p.data.input.search_query` / + `search_keywords`; `rewritten_message` / `rewritten_keywords` come from + `rewrittenMessage` / `rewrittenMessageKeywords` on the + `AnalyticsAiMetricsSignalTraceData`. Set `rewritten` / `rewrite_note` by + comparing the original prompt to the issued query: dropping filler/stopwords + while the key search term(s) stay the same => `false` / `"no rewrite"`; a + change to the key term itself (changed, dropped, or narrowed/broadened so + retrieval could differ) => `true` with a note of what changed. Use `null` / + empty values when no search occurred. +- `search_results` — an array of the real result objects returned. From + `AnalyticsAiMetricsSignalTraceData.verifiedSearchResults[]` use keys `url`, + `rankScore`, `snippet`, `searchType`; and/or from + `PluginResponse.citableContent[]` use items with `source`, `chunks`. Emit them + exactly as returned; an empty array if none. +- `grounding` — an object: + `{ "isKnowledgeSearched": , "completionState": ..., "citedKnowledgeSources": [...], "triggeredGptFallback": }`, + from the `KnowledgeTraceData` / `AnalyticsAiMetricsSignalTraceData` fields. +- `bot_text` — the bot's `OutgoingActivity.p.activity.text` for the turn. + +Emit each turn object in this shape. Example of ONE populated turn object: + +```json +{ + "user_utterance": "How much parental leave do I get?", + "intent": { + "kind": "LlmIntentRecognized", + "intentId": "KnowledgeSearch", + "intentMessage": "User is asking about parental leave policy", + "userUtterance": "How much parental leave do I get?", + "triggeredGptFallback": false + }, + "search": { + "original_prompt": "How much parental leave do I get?", + "query": "parental leave entitlement", + "keywords": "parental leave", + "rewritten_message": null, + "rewritten_keywords": null, + "rewritten": false, + "rewrite_note": "no rewrite" + }, + "search_results": [ + { + "url": "https://contoso.example/policies/parental-leave-2025", + "rankScore": 0.87, + "snippet": "Eligible employees receive up to 16 weeks of paid parental leave.", + "searchType": "Semantic" + } + ], + "grounding": { + "isKnowledgeSearched": true, + "completionState": "Answered", + "citedKnowledgeSources": [ + "https://contoso.example/policies/parental-leave-2025" + ], + "triggeredGptFallback": false + }, + "bot_text": "Eligible employees receive up to 16 weeks of paid parental leave." +} +``` + +Remember: the normalized JSON contains NO verdicts, checks, divergence, or root +cause — those belong only in the markdown Debug Report. + +**3. Faithful transcript JSON** — `/ess-diagnostics//-transcript.json`. +This is the COMPLETE original transcript converted to readable JSON — every +event and every field preserved, nothing dropped (unlike the compact normalized +JSON above, which keeps only the diagnostic fields). Produce it with the bundled +helper introduced in Step 1: + +``` +node scripts/transcript-to-json.js "" +``` + +The helper writes this file into the same per-run temp folder and prints its +absolute path. Do not hand-build this file; use the helper so the dump stays +lossless. + +After writing all three files, tell the FDE the three **absolute** output file +paths in the temp directory: the markdown Debug Report, the normalized-diagnostic +JSON, and the faithful transcript JSON. diff --git a/tools/ESS-Diagnostics-Skills/docs/plans/2026-09-17-ess-diagnostics-skill-v4-design.md b/tools/ESS-Diagnostics-Skills/docs/plans/2026-09-17-ess-diagnostics-skill-v4-design.md new file mode 100644 index 000000000..49d2f51d9 --- /dev/null +++ b/tools/ESS-Diagnostics-Skills/docs/plans/2026-09-17-ess-diagnostics-skill-v4-design.md @@ -0,0 +1,149 @@ +# ESS Diagnostics Skill (v4) — Interactive Walkthrough + Real-Schema Parse Map — Design + +## Purpose + +Fix two problems found when dry-running the v3 skill against a real transcript +(`Transcript_a9b8b841-...txt`): + +1. **The skill was not interactive.** It ran the whole diagnosis in one shot: + it never stopped to ask the FDE for the specific problem, and it never paused + turn-by-turn for FDE input. The v3 rules *said* to do both, but they were soft + guidance ("ask if missing", "pause") with no hard stop, so the model flowed + straight through. +2. **The parse map did not match real data.** Real Copilot Studio / PVA exports + carry diagnostic events as `Trace` events discriminated by `p.data.kind`, use + `SynchronousIncomingActivity` for user turns, and expose grounding/rewrite via + `AnalyticsAiMetricsSignalTraceData` / `KnowledgeTraceData`. The v3 map's + `IncomingActivity` / `search_results[]`-with-`Name`/`Text` shape does not exist + in the observed transcript. + +This is a v4 revision of `tools/ESS-Diagnostics-Skills/SKILL.md`. Scope: both the +interaction model AND the parse-map reconciliation. + +## 1. Problem-statement hard gate (new Step 0) + +Add **Step 0: Establish the Problem**, run before any parsing: + +- On invocation the skill STOPS, confirms the transcript **file path**, and + explicitly asks: **"What specific problem are you investigating in this + transcript?"** +- It may NOT read/parse the transcript until the FDE answers. +- If a problem statement was supplied in the invocation args, the skill echoes it + back and asks the FDE to confirm or refine it — never silently accepts it. +- Rules bullet changes from "ask if missing" to a firm hard stop: "Do not proceed + past Step 0 until the FDE has stated the problem and you have confirmed the file + path — even if a problem was supplied in the invocation." + +## 2. Interactive per-turn walkthrough with structured pause + +Step 2 becomes a real interactive loop with a hard stop after each turn. + +At each turn's pause, the skill presents a **structured summary** (explicit, not +just verdicts): + +- **Intent** — recognized intent (id/message) or the failure signal. +- **Search query** — original user prompt + issued/rewritten query & keywords. +- **Search response** — result count / cited sources, and what the bot answered. +- A one-line **verdict strip**: `001 Pass · 002 Pass · 003 Pass · 004 Pass · 005 Pass`. + +Then a **section menu**: + +``` +Drill into a section, or advance: + 1. Intent 2. Search query 3. Search results + 4. Grounding 5. Final answer + [continue] next turn [run all] finish without pausing [override] a verdict +``` + +- **Hard stop:** the skill WAITS here; it may not look at the next turn until the + FDE responds. Default is per-turn stops. +- **Escape hatch:** at the first pause the FDE may type `run all` to switch to a + batch pass (auto-advance through remaining turns, still producing the full + report). +- **Drill-down (sections 1–5):** show the full raw transcript evidence for that + section (complete query/keywords, full result list, full intent object, full + answer text) PLUS the check's reasoning (why Pass/Fail/N/A), and explicitly + invite the FDE to **override** the verdict with their domain knowledge. +- **Override:** recorded (check, original verdict → override, FDE reason) and + carried into the final report. + +## 3. Parse-map reconciliation (real schema) + +Rewrite Step 1's parse map to the observed real export. Ordering by array +position is unchanged (that rule was correct). Structural change: most diagnostic +events are `Trace` events keyed by `p.data.kind`. + +| Concept | Where it lives | Fields | +|---|---|---| +| User utterance | `t = SynchronousIncomingActivity`, `p.activity.type = "message"` | `p.activity.text` | +| Bot message | `t = OutgoingActivity`, `p.activity.type = "message"` | `p.activity.text` | +| Intent recognized | `t = Trace`, `p.data.kind = LlmIntentRecognized` | `intentId`, `intentMessage`, `userUtterance` | +| Search issued | `t = Trace`, `p.data.kind = PluginStart` | `input.search_query`, `input.search_keywords`, `pluginName` | +| Search results | `t = Trace`, `p.data.kind = PluginResponse` | `citableContent[]` (`source`, `chunks`), `output` | +| Grounding/answer signal | `t = Trace`, `p.data.kind = AnalyticsAiMetricsSignalTraceData` | `completionState`, `triggeredGptFallback`, `rewrittenMessage`, `rewrittenMessageKeywords`, `verifiedSearchResults[]` (`url`, `rankScore`, `snippet`, `searchType`), `citedKnowledgeSources`, `textCitations` | +| Knowledge search state | `t = Trace`, `p.data.kind = KnowledgeTraceData` | `isKnowledgeSearched`, `completionState`, `citedKnowledgeSources`, `failedKnowledgeSourcesTypes` | + +Only `p.activity.type = "message"` activities are real user/bot turns; +`type = "event"` activities are system/plumbing and are skipped for turn +segmentation. + +Check reconciliation: + +- **CHECK-001 Intent Recognition** — recognized via `LlmIntentRecognized`. The + failure signal is `triggeredGptFallback = true` on + `AnalyticsAiMetricsSignalTraceData` (there is no `UnknownIntentTriggered` in + real data). Fail if intent is absent or GPT fallback was triggered. +- **CHECK-002 Search Query Issued** — original prompt from + `SynchronousIncomingActivity.p.activity.text` (the intent's `userUtterance` is + often empty). Issued query/keywords from `PluginStart.input`. **Rewrite is read + directly** from `rewrittenMessage` / `rewrittenMessageKeywords`, not inferred. +- **CHECK-003 Search Result Topical Relevance** — use + `verifiedSearchResults[].rankScore` and `snippet` where present (rankScore does + exist in this schema); fall back to keyword overlap on `snippet` / `source` / + `citableContent.chunks`. +- **CHECK-004 Knowledge Grounding Consistency** — from `isKnowledgeSearched`, + `completionState`, `citedKnowledgeSources`, and `verifiedSearchResults` count + vs. what the answer does with them. +- **CHECK-005 Final Answer vs. Retrieved Content / Guardrails** — answer text vs. + `verifiedSearchResults` / `citedKnowledgeSources` / `textCitations`. + +**Graceful missing-field handling:** if a mapped field/event is absent in a given +transcript, mark the affected check **N/A** with "field not present in this +transcript" rather than failing or guessing. Note that the map derives from +observed exports and field names may vary by bot configuration. + +## 4. Outputs, PII handling, testing + +**Outputs written OUTSIDE the repo (PII safety):** the markdown Debug Report and +normalized JSON are written to a per-run subfolder in the OS temp directory +(`%TEMP%/ess-diagnostics//` on Windows; `$TMPDIR` or `/tmp` on +POSIX), NOT under `tools/ESS-Diagnostics-Skills/`. Nothing is written into the +repo, so commits can never capture employee PII. The skill tells the FDE the +absolute temp paths when done and warns once that outputs may contain PII and +should not be shared outside approved channels. + +**Report additions:** the confirmed problem statement (from Step 0) and any FDE +verdict overrides (check, original → override, reason). Per-turn results surface +intent / search query + rewrite / search response explicitly. + +**Normalized JSON updates:** `search.rewritten` / `rewrite_note` sourced from +`rewrittenMessage` / `rewrittenMessageKeywords`; `search_results` becomes the real +`verifiedSearchResults[]` shape (`url`, `rankScore`, `snippet`, …) and/or +`citableContent[]`; add a `grounding` object (`completionState`, +`isKnowledgeSearched`, `citedKnowledgeSources`, `triggeredGptFallback`). Still NO +verdicts in the JSON. + +**Testing:** re-run the dry run against the same real transcript and confirm: +Step 0 gate fires; the structured per-turn pause + section menu + drill-down + +override work; the new parse map extracts intent/search/grounding correctly; and +outputs land in the OS temp dir, not the repo. No automated suite (prose skill). + +## Note on prior v3 decisions this revisits + +- v3's final review removed reliance on a "completion state" field because it was + unmapped. Real data DOES carry `completionState` / `isKnowledgeSearched`, so v4 + re-introduces it as a mapped field for CHECK-004. +- v3 asserted there is "no rank/score field" on results. Real + `verifiedSearchResults[]` objects DO have `rankScore`, so CHECK-003 may use it. +- These are corrections grounded in one real export; the graceful-missing-field + rule guards against over-fitting to a single transcript. diff --git a/tools/ESS-Diagnostics-Skills/docs/plans/2026-09-17-ess-diagnostics-skill-v4-implementation.md b/tools/ESS-Diagnostics-Skills/docs/plans/2026-09-17-ess-diagnostics-skill-v4-implementation.md new file mode 100644 index 000000000..ef0a55b0d --- /dev/null +++ b/tools/ESS-Diagnostics-Skills/docs/plans/2026-09-17-ess-diagnostics-skill-v4-implementation.md @@ -0,0 +1,126 @@ +# ESS Diagnostics Skill v4 Implementation Plan + +> **For Claude:** REQUIRED SUB-SKILL: Use `development/reference/executing-plans-guide.md` to implement this plan task-by-task. + +**Goal:** Revise `tools/ESS-Diagnostics-Skills/SKILL.md` to (a) make the diagnosis interactive — a hard problem-statement gate and a structured per-turn pause with section drill-down and verdict override — and (b) reconcile the parse map with the real Copilot Studio / PVA export schema, writing PII-bearing outputs to an OS temp dir outside the repo. + +**Architecture:** Prose edits to a single `SKILL.md` (read-only Claude skill, no code). Sourced from the v4 design doc `docs/plans/2026-09-17-ess-diagnostics-skill-v4-design.md` and from field paths observed in a real transcript. Validation is a manual dry run. + +**Tech Stack:** Markdown. No test framework. + +**Line endings:** repo is `core.autocrlf=true` + `core.safecrlf=true`, files are CRLF. After each edit, normalize to exactly one CRLF per line: `sed -i 's/\r$//' && sed -i 's/$/\r/' `; verify `file ` says "CRLF line terminators" and `grep -aoP '\r\r' | wc -l` prints 0. Only touch `tools/ESS-Diagnostics-Skills/SKILL.md` unless a task says otherwise. + +**Real schema reference (from the observed transcript):** +- User turn: `t = SynchronousIncomingActivity`, `p.activity.type = "message"`, text at `p.activity.text`. +- Bot turn: `t = OutgoingActivity`, `p.activity.type = "message"`, text at `p.activity.text`. (`type = "event"` activities are system plumbing — skip.) +- Diagnostic events: `t = Trace`, discriminated by `p.data.kind` ∈ {`LlmIntentRecognized`, `PluginStart`, `PluginResponse`, `KnowledgeTraceData`, `AnalyticsAiMetricsSignalTraceData`, …}. +- `PluginStart.input`: `search_query`, `search_keywords`, `enable_summarization`. +- `PluginResponse`: `citableContent[]` (`source`, `chunks`), `output`. +- `AnalyticsAiMetricsSignalTraceData`: `completionState`, `triggeredGptFallback`, `rewrittenMessage`, `rewrittenMessageKeywords`, `verifiedSearchResults[]` (`url`, `rankScore`, `snippet`, `searchType`), `citedKnowledgeSources`, `textCitations`. +- `KnowledgeTraceData`: `isKnowledgeSearched`, `completionState`, `citedKnowledgeSources`, `failedKnowledgeSourcesTypes`. + +--- + +### Task 1: Add Step 0 (problem-statement hard gate) and strengthen the Rule + +**Files:** Modify `tools/ESS-Diagnostics-Skills/SKILL.md`. + +**Step 1:** In `## Rules`, replace the bullet "Require two inputs up front… If either is missing, ask for it before proceeding." with a hard-stop bullet: the skill must not proceed past Step 0 until the FDE has stated the problem and the file path is confirmed — even if a problem was supplied in the invocation. + +**Step 2:** Insert a new `## Step 0: Establish the Problem` section BEFORE `## Step 1`. It must instruct: on invocation, STOP; confirm the transcript file path; explicitly ask "What specific problem are you investigating in this transcript?"; do not read/parse until answered; if a problem statement was supplied in the args, echo it back and ask the FDE to confirm or refine rather than silently accepting. + +**Step 3:** CRLF-normalize, verify, `git add`, commit: `feat: add Step 0 problem-statement hard gate to diagnostics skill`. + +--- + +### Task 2: Rewrite the Step 1 parse map to the real schema + +**Files:** Modify `tools/ESS-Diagnostics-Skills/SKILL.md`. + +**Step 1:** Replace the Step 1 parse-map table with the concept→location→fields table from the v4 design (SynchronousIncomingActivity; OutgoingActivity; Trace/`p.data.kind` for LlmIntentRecognized, PluginStart, PluginResponse, AnalyticsAiMetricsSignalTraceData, KnowledgeTraceData). Keep the array-position ordering rule. Add: only `p.activity.type = "message"` activities are real turns; `type = "event"` are skipped for segmentation. Replace the old "search_results[] with keys FileType/Name/Text/…, no score field" note with the real result shapes (`citableContent[]` source/chunks; `verifiedSearchResults[]` url/rankScore/snippet). + +**Step 2:** Add the graceful-missing-field rule: if a mapped field/event is absent, mark the affected check N/A with "field not present in this transcript" rather than failing/guessing; note the map derives from observed exports and may vary by bot config. + +**Step 3:** CRLF-normalize, verify, `git add`, commit: `feat: reconcile Step 1 parse map with real transcript schema`. + +--- + +### Task 3: Reconcile the 5 checks with the real fields + +**Files:** Modify `tools/ESS-Diagnostics-Skills/SKILL.md`. + +**Step 1:** Update each check bullet in Step 2: +- CHECK-001: recognized via `LlmIntentRecognized`; failure = intent absent OR `triggeredGptFallback = true`. Remove `UnknownIntentTriggered` as the failure path (note it's not present in real data; keep a mention that some exports may use it). +- CHECK-002: original prompt from `SynchronousIncomingActivity.p.activity.text`; issued query/keywords from `PluginStart.input`; rewrite read DIRECTLY from `rewrittenMessage` / `rewrittenMessageKeywords` (not inferred). +- CHECK-003: use `verifiedSearchResults[].rankScore` + `snippet` where present; fall back to keyword overlap on `snippet`/`source`/`citableContent.chunks`. (rankScore DOES exist — drop the "no score field" claim.) +- CHECK-004: from `isKnowledgeSearched`, `completionState`, `citedKnowledgeSources`, `verifiedSearchResults` count vs. the answer. +- CHECK-005: answer vs. `verifiedSearchResults` / `citedKnowledgeSources` / `textCitations`. + +**Step 2:** Keep the existing "earlier failed check removes a later precondition ⇒ N/A" rule. + +**Step 3:** CRLF-normalize, verify, `git add`, commit: `feat: reconcile the 5 checks with real transcript fields`. + +--- + +### Task 4: Make Step 2 an interactive loop with structured pause + drill-down + override + +**Files:** Modify `tools/ESS-Diagnostics-Skills/SKILL.md`. + +**Step 1:** Rewrite the Step 2 walkthrough framing so that per turn the skill presents a STRUCTURED SUMMARY: Intent (id/message or failure signal); Search query (original prompt + issued/rewritten query & keywords); Search response (result count / cited sources + bot answer); and a one-line verdict strip `001 … · 005 …`. + +**Step 2:** Add the section menu + hard stop: +- Menu options 1–5 (Intent, Search query, Search results, Grounding, Final answer) plus `[continue]`, `[run all]`, `[override]`. +- Hard stop: the skill WAITS and may not look at the next turn until the FDE responds. +- `run all` (available from the first pause) switches to a batch pass that auto-advances remaining turns but still produces the full report. + +**Step 3:** Add drill-down + override behavior: +- Drill-down (section 1–5): show full raw transcript evidence for that section PLUS the check's reasoning (why Pass/Fail/N/A), and explicitly invite the FDE to override the verdict. +- Override: record (check, original verdict → override, FDE reason); it flows into the final report. + +**Step 4:** CRLF-normalize, verify, `git add`, commit: `feat: make per-turn walkthrough interactive with drill-down and override`. + +--- + +### Task 5: Update Step 4 outputs — OS temp dir, overrides, real-schema JSON + +**Files:** Modify `tools/ESS-Diagnostics-Skills/SKILL.md`. + +**Step 1:** Change output location: write the two files to a per-run subfolder in the OS temp dir (`%TEMP%/ess-diagnostics//` on Windows; `$TMPDIR` or `/tmp` on POSIX), NOT under the repo. Remove the `reports/`-under-repo path. Add a one-time PII warning and instruct the skill to report the absolute temp paths. + +**Step 2:** Markdown report additions: include the confirmed problem statement (from Step 0) and any FDE verdict overrides (check, original → override, reason). + +**Step 3:** Normalized JSON updates: `search.rewritten`/`rewrite_note` from `rewrittenMessage`/`rewrittenMessageKeywords`; `search_results` → real `verifiedSearchResults[]` shape (`url`, `rankScore`, `snippet`, …) and/or `citableContent[]`; add a `grounding` object (`completionState`, `isKnowledgeSearched`, `citedKnowledgeSources`, `triggeredGptFallback`). Update the fenced JSON example to this shape. Keep "NO verdicts in JSON". + +**Step 4:** CRLF-normalize, verify, `git add`, commit: `feat: write outputs to OS temp dir and update JSON to real schema`. + +--- + +### Task 6: Update README for v4 behavior + +**Files:** Modify `tools/ESS-Diagnostics-Skills/README.md`. + +**Step 1:** Update the README to describe: the Step 0 problem gate; the interactive per-turn pause with section drill-down and override; outputs written to an OS temp dir (not the repo) with a PII note; and the CHECK-004 (grounding) / CHECK-003 (rankScore) reconciliations. Keep it concise. + +**Step 2:** CRLF-normalize, verify, `git add`, commit: `docs: update README for v4 interactive + real-schema behavior`. + +--- + +### Task 7: Dry-run validation against the real transcript + +**Files:** none (validation only). + +**Step 1:** Follow the revised SKILL.md against `C:\Users\rarame\Downloads\Transcript_a9b8b841-6a4f-4675-9e79-92fc6df74be9.txt`. Confirm: +- Step 0 hard gate fires (skill asks for the problem before parsing). +- Parse map extracts the real events: 3 turns; intents `LlmIntentRecognized`; PluginStart queries + `rewrittenMessage`; `verifiedSearchResults` (count 10), `completionState = Answered`. +- The per-turn structured pause shows intent/search/response + verdict strip + section menu; drill-down shows raw evidence + reasoning; an override is accepted and recorded; `run all` works. +- Outputs land in the OS temp dir (NOT the repo); `git status` under the repo stays clean. + +**Step 2:** Do NOT commit any generated report/JSON. Confirm nothing was written under `tools/ESS-Diagnostics-Skills/`. Record any gaps found; if SKILL.md needs a fix, apply it and commit `fix: address gaps found during v4 dry run`. + +--- + +## Execution Handoff + +Two options: +1. **Subagent-Driven (this session)** — fresh subagent per task, spec review between tasks. +2. **Parallel Session** — new session using `development/reference/executing-plans-guide.md`. diff --git a/tools/ESS-Diagnostics-Skills/scripts/transcript-to-json.js b/tools/ESS-Diagnostics-Skills/scripts/transcript-to-json.js new file mode 100644 index 000000000..62bdcd973 --- /dev/null +++ b/tools/ESS-Diagnostics-Skills/scripts/transcript-to-json.js @@ -0,0 +1,114 @@ +#!/usr/bin/env node +/** + * transcript-to-json.js — faithful transcript -> JSON helper for the ESS + * Diagnostics skill. + * + * Parses a Copilot Studio / PVA transcript (JSON payload stored in a .txt) and + * writes a LOSSLESS, pretty-printed JSON copy: every event and every field is + * preserved exactly. The only addition is a `_turn` number annotated on each + * event (0 for events before the first user message) so a reader can navigate + * by conversation turn. Nothing is removed, so the output round-trips back to + * the original once the `_turn` keys are dropped. + * + * This is the faithful full dump. It is NOT the compact diagnostic + * "normalized" view the skill builds for the 5 checks — it is the whole + * transcript, just as real JSON. + * + * Usage: + * node transcript-to-json.js [outFile.json] + * + * - path to the raw transcript export (required). + * - [outFile.json] where to write. If omitted, writes + * `-transcript.json` into the OS temp dir under + * `ess-diagnostics//` (outside any repo, because + * transcripts may contain employee PII). + * + * Prints the absolute output path on success. Exits non-zero with an + * `Error: ...` message on failure. + */ + +"use strict"; + +const fs = require("fs"); +const os = require("os"); +const path = require("path"); + +function fail(msg) { + process.stderr.write("Error: " + msg + "\n"); + process.exit(1); +} + +const srcArg = process.argv[2]; +if (!srcArg) fail("missing transcript path. Usage: node transcript-to-json.js [outFile.json]"); +if (!fs.existsSync(srcArg)) fail("transcript file not found: " + srcArg); + +let raw; +try { + raw = fs.readFileSync(srcArg, "utf8"); +} catch (e) { + fail("could not read transcript: " + e.message); +} + +// Strip a leading UTF-8 BOM if present, then parse. +if (raw.charCodeAt(0) === 0xfeff) raw = raw.slice(1); + +let events; +try { + events = JSON.parse(raw); +} catch (e) { + fail("transcript is not valid JSON: " + e.message); +} + +if (!Array.isArray(events)) { + fail("expected the transcript to be a JSON array of events; got " + typeof events); +} + +// Annotate each event with its conversation turn number, WITHOUT mutating any +// existing field. A turn starts at each real user message +// (SynchronousIncomingActivity with activity.type === "message"). Events before +// the first user message are turn 0. `_turn` is a first key on a shallow copy; +// every original key/value is copied through untouched. +function isUserTurnStart(e) { + return ( + e && + e.t === "SynchronousIncomingActivity" && + e.p && + e.p.activity && + e.p.activity.type === "message" + ); +} + +let turn = 0; +const annotated = events.map((e, i) => { + if (isUserTurnStart(e)) turn += 1; + // _turn and _index first, then a faithful spread of the original event. + return Object.assign({ _turn: turn, _index: i }, e); +}); + +// Resolve output path. +const base = path.basename(srcArg).replace(/\.[^.]+$/, ""); +let outFile = process.argv[3]; +if (!outFile) { + const outDir = path.join(os.tmpdir(), "ess-diagnostics", base); + try { + fs.mkdirSync(outDir, { recursive: true }); + } catch (e) { + fail("could not create output directory " + outDir + ": " + e.message); + } + outFile = path.join(outDir, base + "-transcript.json"); +} else { + const outParent = path.dirname(path.resolve(outFile)); + try { + fs.mkdirSync(outParent, { recursive: true }); + } catch (e) { + fail("could not create output directory " + outParent + ": " + e.message); + } +} + +try { + fs.writeFileSync(outFile, JSON.stringify(annotated, null, 2)); +} catch (e) { + fail("could not write output: " + e.message); +} + +process.stdout.write(path.resolve(outFile) + "\n"); From f2feb5aefa305c6da0cb0841aeea161a208ad4af Mon Sep 17 00:00:00 2001 From: "Ramanan Vellore Ramesh (from Dev Box)" Date: Thu, 17 Sep 2026 18:20:06 -0700 Subject: [PATCH 2/8] ESS Diagnostics: design for Node->Python transcript helper port Design doc for converting scripts/transcript-to-json.js to a stdlib-only Python drop-in (transcript_to_json.py) with exact behavioral parity, plus SKILL.md invocation updates and removal of the JS file. Co-Authored-By: Claude --- ...transcript-helper-node-to-python-design.md | 114 ++++++++++++++++++ 1 file changed, 114 insertions(+) create mode 100644 tools/ESS-Diagnostics-Skills/docs/plans/2026-09-17-transcript-helper-node-to-python-design.md diff --git a/tools/ESS-Diagnostics-Skills/docs/plans/2026-09-17-transcript-helper-node-to-python-design.md b/tools/ESS-Diagnostics-Skills/docs/plans/2026-09-17-transcript-helper-node-to-python-design.md new file mode 100644 index 000000000..61ad1f773 --- /dev/null +++ b/tools/ESS-Diagnostics-Skills/docs/plans/2026-09-17-transcript-helper-node-to-python-design.md @@ -0,0 +1,114 @@ +# Convert transcript-to-json helper from Node.js to Python + +**Date:** 2026-09-17 +**Status:** Approved design +**Scope:** `tools/ESS-Diagnostics-Skills/` + +## Problem + +The ESS Diagnostics skill ships a helper, `scripts/transcript-to-json.js`, that +produces a faithful (lossless) JSON dump of a Copilot Studio / PVA transcript. +The rest of the repo is Python (3.11+, Ruff-linted, pytest suite). A lone +Node.js dependency in an otherwise Python toolkit is an avoidable footgun for +FDEs and maintainers. Convert the helper to Python and remove the Node version. + +## Goals + +- Replace the JS helper with a stdlib-only Python script that is an **exact + behavioral drop-in** (same CLI contract, same output, same error/exit + behavior). +- Update `SKILL.md` to invoke the Python helper. +- Remove the JS file. No Node dependency remains. + +## Non-goals + +- No automated test (matches the JS helper's current no-test status; validate + manually). +- No CLI enhancements (`argparse`, `--help`), no packaging (`__main__.py`), no + refactor of the diagnostic flow. Faithful port only. + +## Decisions + +| Question | Decision | +| --- | --- | +| Fate of the JS file | **Delete** — Python fully replaces it. | +| Invocation | `python scripts/transcript_to_json.py ""` (plain script, not `-m`). | +| Interpreter in docs | Document both: `python` (Windows) / `python3` (POSIX), matching SKILL.md's existing Windows/POSIX split. | +| Fidelity | **Exact behavioral parity** — true drop-in. | +| Test | **None** — manual validation. | +| Internal structure | **Approach A** — faithful procedural port, stdlib only, no unused abstraction. | +| Filename | `transcript_to_json.py` (PEP 8 underscores). | + +## Files + +- **Add:** `tools/ESS-Diagnostics-Skills/scripts/transcript_to_json.py` +- **Delete:** `tools/ESS-Diagnostics-Skills/scripts/transcript-to-json.js` +- **Edit:** `tools/ESS-Diagnostics-Skills/SKILL.md` (invocation commands, ~2 spots, + platform note) +- **README.md:** no change (verified — no `node`/`.js`/`transcript-to-json` references) + +No new dependencies. Stdlib only: `sys`, `json`, `pathlib`, `tempfile`. +Target: Python 3.11+. + +## The Python script (behavioral parity) + +CLI: `python transcript_to_json.py [outFile.json]` — arg 1 +required (source), arg 2 optional (output path). + +Flow, identical to the JS original: + +1. `fail(msg)` → write `Error: \n` to stderr, `sys.exit(1)`. +2. Missing arg 1 → fail with the usage message. Source file not found → fail. +3. Read source as UTF-8. Strip a leading BOM (``) if present. +4. `json.loads`; on error → `fail("transcript is not valid JSON: ...")`. +5. Not a `list` → `fail("expected the transcript to be a JSON array of events; got ")`. +6. Annotate: walk events in array order, increment `turn` at each user-turn + start (`t == "SynchronousIncomingActivity"` and `p.activity.type == + "message"`); emit each as `{"_turn": turn, "_index": i, **event}` so + `_turn`/`_index` are the first keys and every original field is preserved + untouched. Nested access is guarded (`isinstance` / `.get()` chains) so a + malformed event never raises — mirrors the JS `e && e.p && e.p.activity` + guard. +7. Resolve output path: if arg 2 is given, use it (create its parent dir); + otherwise `tempfile.gettempdir()/ess-diagnostics//-transcript.json`, + creating directories. Basename = source filename minus its final extension. +8. Write with `json.dumps(annotated, indent=2, ensure_ascii=False)`. + **`ensure_ascii=False` is required** for byte-parity with JS + `JSON.stringify`, which emits raw (non-escaped) Unicode. +9. Print the absolute output path to stdout. + +Wrapped in a `main()` called under `if __name__ == "__main__":`. + +Parity guarantees preserved: identical `Error:` prefix (SKILL.md instructs the +FDE to relay it), exit codes (0 success / 1 failure), temp-path shape, printed +absolute path, and round-trippable output (drop `_turn`/`_index` → original). + +## SKILL.md edits + +Each `node scripts/transcript-to-json.js ""` (Step 1 "Faithful +transcript-to-JSON" and Step 4, file #3) becomes: + +``` +python scripts/transcript_to_json.py "" +``` + +Add a one-line platform note in the existing Windows/POSIX style: use `python` +on Windows, `python3` on POSIX. Surrounding prose (writes into the same temp +folder, prints the absolute path, relay `Error:` on failure, produce both dumps) +stays valid unchanged. + +## Error handling + +All failure modes route through `fail()`: missing/absent source, invalid JSON, +non-array top level, and unwritable output directory each print a single +`Error: ...` line to stderr and exit non-zero. No Python traceback leaks to the +FDE. Identical to the JS behavior. + +## Validation (manual) + +1. Run the new script against the real transcript + (`Transcript_ 4b548465-...txt`); confirm it prints the same absolute path and + writes a valid file. +2. Diff the Python output against the existing JS output for that transcript; + expect identical content (the annotation + round-trip guarantee). +3. Spot-check a failure case (non-existent path); confirm the `Error:` line and + non-zero exit. From 7ec2be0c7a0ee357c0b82089301a94fbc0ce7be2 Mon Sep 17 00:00:00 2001 From: "Ramanan Vellore Ramesh (from Dev Box)" Date: Thu, 17 Sep 2026 20:34:13 -0700 Subject: [PATCH 3/8] ESS Diagnostics: implementation plan for Node->Python helper port Co-Authored-By: Claude --- ...-09-17-transcript-helper-node-to-python.md | 278 ++++++++++++++++++ 1 file changed, 278 insertions(+) create mode 100644 tools/ESS-Diagnostics-Skills/docs/plans/2026-09-17-transcript-helper-node-to-python.md diff --git a/tools/ESS-Diagnostics-Skills/docs/plans/2026-09-17-transcript-helper-node-to-python.md b/tools/ESS-Diagnostics-Skills/docs/plans/2026-09-17-transcript-helper-node-to-python.md new file mode 100644 index 000000000..eb580f793 --- /dev/null +++ b/tools/ESS-Diagnostics-Skills/docs/plans/2026-09-17-transcript-helper-node-to-python.md @@ -0,0 +1,278 @@ +# Node→Python Transcript Helper Port — Implementation Plan + +> **For Claude:** REQUIRED SUB-SKILL: Use `development/reference/executing-plans-guide.md` to implement this plan task-by-task. + +**Goal:** Replace the Node.js helper `scripts/transcript-to-json.js` with a stdlib-only Python drop-in (`scripts/transcript_to_json.py`) of exact behavioral parity, update `SKILL.md` to invoke it, and delete the JS file. + +**Architecture:** A single faithful procedural port (design "Approach A"). The Python script mirrors the JS flow one-for-one — BOM strip, JSON-array validation, `_turn`/`_index` annotation as first keys, temp-dir output, absolute-path print, `Error:`/exit-1 failures. No dependencies, no packaging, no CLI framework. Validated manually by diffing its output against the existing JS output for a real transcript (no automated test, per the approved design). + +**Tech Stack:** Python 3.11+ stdlib only (`sys`, `json`, `pathlib`, `tempfile`). Repo lint: Ruff (E4/E7/E9/F). Git `core.autocrlf=true` — new text files must be committed with CRLF terminators. + +**Design doc:** `tools/ESS-Diagnostics-Skills/docs/plans/2026-09-17-transcript-helper-node-to-python-design.md` (committed as `f2feb5a`). + +**Working directory for all paths below:** repo root `Employee-Self-Service-Agent-Developer-Kit/`. + +**Note on git & line endings:** `core.autocrlf=true` with no `.gitattributes` caused a "LF would be replaced by CRLF" commit failure on a LF-only file. Before every commit of a NEW text file (`.py`, `.md`), convert to CRLF first: `sed -i 's/$/\r/' ` (idempotent-guard: only run on files freshly written LF-only). Existing files edited in place keep their endings. + +--- + +### Task 1: Create the Python helper + +**Files:** +- Create: `tools/ESS-Diagnostics-Skills/scripts/transcript_to_json.py` + +**Step 1: Write the script** + +Write exactly this content to `tools/ESS-Diagnostics-Skills/scripts/transcript_to_json.py`: + +```python +#!/usr/bin/env python3 +"""transcript_to_json.py — faithful transcript -> JSON helper for the ESS +Diagnostics skill. + +Parses a Copilot Studio / PVA transcript (JSON payload stored in a .txt) and +writes a LOSSLESS, pretty-printed JSON copy: every event and every field is +preserved exactly. The only additions are `_turn` and `_index`, annotated as +the first keys on each event (turn 0 for events before the first user message) +so a reader can navigate by conversation turn. Nothing is removed, so the +output round-trips back to the original once `_turn`/`_index` are dropped. + +This is the faithful full dump. It is NOT the compact diagnostic "normalized" +view the skill builds for the 5 checks — it is the whole transcript, just as +real JSON. + +Usage: + python transcript_to_json.py [outFile.json] + +- path to the raw transcript export (required). +- [outFile.json] where to write. If omitted, writes + `-transcript.json` into the OS temp dir under + `ess-diagnostics//` (outside any repo, because + transcripts may contain employee PII). + +Prints the absolute output path on success. Exits non-zero with an +`Error: ...` message on failure. +""" + +import json +import sys +import tempfile +from pathlib import Path + + +def fail(msg): + sys.stderr.write("Error: " + msg + "\n") + sys.exit(1) + + +def is_user_turn_start(e): + return ( + isinstance(e, dict) + and e.get("t") == "SynchronousIncomingActivity" + and isinstance(e.get("p"), dict) + and isinstance(e["p"].get("activity"), dict) + and e["p"]["activity"].get("type") == "message" + ) + + +def main(argv): + if len(argv) < 2: + fail( + "missing transcript path. Usage: " + "python transcript_to_json.py [outFile.json]" + ) + src_arg = argv[1] + src_path = Path(src_arg) + if not src_path.exists(): + fail("transcript file not found: " + src_arg) + + try: + raw = src_path.read_text(encoding="utf-8") + except OSError as e: + fail("could not read transcript: " + str(e)) + + # Strip a leading UTF-8 BOM if present, then parse. + if raw and raw[0] == "\ufeff": + raw = raw[1:] + + try: + events = json.loads(raw) + except ValueError as e: + fail("transcript is not valid JSON: " + str(e)) + + if not isinstance(events, list): + fail( + "expected the transcript to be a JSON array of events; got " + + type(events).__name__ + ) + + # Annotate each event with its conversation turn number, WITHOUT mutating any + # existing field. A turn starts at each real user message; events before the + # first user message are turn 0. `_turn`/`_index` are first keys on a shallow + # copy; every original key/value is copied through untouched. + annotated = [] + turn = 0 + for i, e in enumerate(events): + if is_user_turn_start(e): + turn += 1 + if isinstance(e, dict): + annotated.append({"_turn": turn, "_index": i, **e}) + else: + annotated.append({"_turn": turn, "_index": i, "_value": e}) + + base = src_path.name + dot = base.rfind(".") + if dot > 0: + base = base[:dot] + + if len(argv) >= 3 and argv[2]: + out_file = Path(argv[2]) + try: + out_file.resolve().parent.mkdir(parents=True, exist_ok=True) + except OSError as e: + fail( + "could not create output directory " + + str(out_file.resolve().parent) + + ": " + + str(e) + ) + else: + out_dir = Path(tempfile.gettempdir()) / "ess-diagnostics" / base + try: + out_dir.mkdir(parents=True, exist_ok=True) + except OSError as e: + fail("could not create output directory " + str(out_dir) + ": " + str(e)) + out_file = out_dir / (base + "-transcript.json") + + try: + out_file.write_text( + json.dumps(annotated, indent=2, ensure_ascii=False), encoding="utf-8" + ) + except OSError as e: + fail("could not write output: " + str(e)) + + sys.stdout.write(str(out_file.resolve()) + "\n") + + +if __name__ == "__main__": + main(sys.argv) +``` + +**Step 2: Lint the new file** + +Run: `cd tools/ESS-Diagnostics-Skills && python -m ruff check scripts/transcript_to_json.py` +(If Ruff is not installed, skip — it is a dev tool; note the skip.) +Expected: no errors (E4/E7/E9/F clean). + +**Step 3: Run against the real transcript** + +Run (from `tools/ESS-Diagnostics-Skills`): +`python scripts/transcript_to_json.py "C:\Users\rarame\Downloads\Transcript_ 4b548465-9be5-4b73-bc96-e3d6bbbe5229.txt"` +Expected: prints one absolute path ending in +`ess-diagnostics\Transcript_ 4b548465-9be5-4b73-bc96-e3d6bbbe5229\Transcript_ 4b548465-9be5-4b73-bc96-e3d6bbbe5229-transcript.json` +and exits 0. + +**Step 4: Diff Python output against the existing JS output (parity check)** + +The JS dump already exists from this session at the same temp path. Regenerate +it to a side path with the JS helper, then compare: +``` +node scripts/transcript-to-json.js "C:\Users\rarame\Downloads\Transcript_ 4b548465-9be5-4b73-bc96-e3d6bbbe5229.txt" /tmp/js-out.json +python scripts/transcript_to_json.py "C:\Users\rarame\Downloads\Transcript_ 4b548465-9be5-4b73-bc96-e3d6bbbe5229.txt" /tmp/py-out.json +diff /tmp/js-out.json /tmp/py-out.json && echo "IDENTICAL" +``` +Expected: `IDENTICAL` (no diff). If diffs appear, they will be Unicode escaping +(check `ensure_ascii=False`) or key ordering — fix before continuing. + +**Step 5: Spot-check a failure case** + +Run: `python scripts/transcript_to_json.py /nonexistent/file.txt; echo "exit=$?"` +Expected: stderr line `Error: transcript file not found: /nonexistent/file.txt` and `exit=1`. + +**Step 6: Commit** + +```bash +cd +sed -i 's/$/\r/' tools/ESS-Diagnostics-Skills/scripts/transcript_to_json.py +git add tools/ESS-Diagnostics-Skills/scripts/transcript_to_json.py +git commit -m "ESS Diagnostics: add Python transcript_to_json helper (drop-in for JS)" +``` +(Append the `Co-Authored-By: Claude ` line per session attribution.) + +--- + +### Task 2: Point SKILL.md at the Python helper + +**Files:** +- Modify: `tools/ESS-Diagnostics-Skills/SKILL.md` (Step 1 "Faithful transcript-to-JSON" block ~line 96-98; Step 4 file #3 block ~line 378-380) + +**Step 1: Replace both invocation code blocks** + +Both currently read: +``` +node scripts/transcript-to-json.js "" +``` +Change each to: +``` +python scripts/transcript_to_json.py "" +``` + +**Step 2: Add a platform note next to the Step 1 invocation** + +Immediately after the Step 1 code block (which says "run from the skill +directory `tools/ESS-Diagnostics-Skills/`"), add a sentence in the existing +Windows/POSIX style: +> On Windows use `python`; on POSIX use `python3`. + +**Step 3: Verify no stale JS references remain** + +Run: `grep -n "node \|transcript-to-json.js\|\.js" tools/ESS-Diagnostics-Skills/SKILL.md` +Expected: no matches. + +**Step 4: Commit** + +```bash +git add tools/ESS-Diagnostics-Skills/SKILL.md +git commit -m "ESS Diagnostics: SKILL.md invokes Python transcript helper" +``` +(SKILL.md is an existing file; no CRLF conversion needed — Edit preserves its endings. If a commit fails on CRLF, run `sed -i 's/$/\r/'` only on lines you added, or re-normalize the whole file with `unix2dos`.) + +--- + +### Task 3: Remove the Node.js helper + +**Files:** +- Delete: `tools/ESS-Diagnostics-Skills/scripts/transcript-to-json.js` + +**Step 1: Confirm nothing else references it** + +Run: `grep -rn "transcript-to-json" tools/ESS-Diagnostics-Skills/ --include=*.md --include=*.py --include=*.js` +Expected: no matches (SKILL.md already updated; README verified clean in design). +If any match remains, fix it before deleting. + +**Step 2: Delete the file** + +```bash +git rm tools/ESS-Diagnostics-Skills/scripts/transcript-to-json.js +``` + +**Step 3: Final sanity run** + +Run the Python helper once more against the real transcript (Task 1 Step 3) to +confirm the skill's dump path still works with the JS file gone. +Expected: same absolute path printed, exit 0. + +**Step 4: Commit** + +```bash +git commit -m "ESS Diagnostics: remove Node transcript helper (replaced by Python)" +``` + +--- + +### Task 4: Final verification + +**Step 1:** `git log --oneline -4` — confirm the four commits (design + 3 tasks) are present. +**Step 2:** `git status` — confirm clean working tree. +**Step 3:** Confirm `scripts/` now contains only `transcript_to_json.py` (no `.js`): +`ls tools/ESS-Diagnostics-Skills/scripts/` From f5cbe484cb3fda0ea069b9274ba66ac6c19c2078 Mon Sep 17 00:00:00 2001 From: "Ramanan Vellore Ramesh (from Dev Box)" Date: Thu, 17 Sep 2026 20:36:36 -0700 Subject: [PATCH 4/8] ESS Diagnostics: add Python transcript_to_json helper (drop-in for JS) Co-Authored-By: Claude --- .../scripts/transcript_to_json.py | 130 ++++++++++++++++++ 1 file changed, 130 insertions(+) create mode 100644 tools/ESS-Diagnostics-Skills/scripts/transcript_to_json.py diff --git a/tools/ESS-Diagnostics-Skills/scripts/transcript_to_json.py b/tools/ESS-Diagnostics-Skills/scripts/transcript_to_json.py new file mode 100644 index 000000000..c927ab860 --- /dev/null +++ b/tools/ESS-Diagnostics-Skills/scripts/transcript_to_json.py @@ -0,0 +1,130 @@ +#!/usr/bin/env python3 +"""transcript_to_json.py — faithful transcript -> JSON helper for the ESS +Diagnostics skill. + +Parses a Copilot Studio / PVA transcript (JSON payload stored in a .txt) and +writes a LOSSLESS, pretty-printed JSON copy: every event and every field is +preserved exactly. The only additions are `_turn` and `_index`, annotated as +the first keys on each event (turn 0 for events before the first user message) +so a reader can navigate by conversation turn. Nothing is removed, so the +output round-trips back to the original once `_turn`/`_index` are dropped. + +This is the faithful full dump. It is NOT the compact diagnostic "normalized" +view the skill builds for the 5 checks — it is the whole transcript, just as +real JSON. + +Usage: + python transcript_to_json.py [outFile.json] + +- path to the raw transcript export (required). +- [outFile.json] where to write. If omitted, writes + `-transcript.json` into the OS temp dir under + `ess-diagnostics//` (outside any repo, because + transcripts may contain employee PII). + +Prints the absolute output path on success. Exits non-zero with an +`Error: ...` message on failure. +""" + +import json +import sys +import tempfile +from pathlib import Path + + +def fail(msg): + sys.stderr.write("Error: " + msg + "\n") + sys.exit(1) + + +def is_user_turn_start(e): + return ( + isinstance(e, dict) + and e.get("t") == "SynchronousIncomingActivity" + and isinstance(e.get("p"), dict) + and isinstance(e["p"].get("activity"), dict) + and e["p"]["activity"].get("type") == "message" + ) + + +def main(argv): + if len(argv) < 2: + fail( + "missing transcript path. Usage: " + "python transcript_to_json.py [outFile.json]" + ) + src_arg = argv[1] + src_path = Path(src_arg) + if not src_path.exists(): + fail("transcript file not found: " + src_arg) + + try: + raw = src_path.read_text(encoding="utf-8") + except OSError as e: + fail("could not read transcript: " + str(e)) + + # Strip a leading UTF-8 BOM if present, then parse. + if raw and raw[0] == "": + raw = raw[1:] + + try: + events = json.loads(raw) + except ValueError as e: + fail("transcript is not valid JSON: " + str(e)) + + if not isinstance(events, list): + fail( + "expected the transcript to be a JSON array of events; got " + + type(events).__name__ + ) + + # Annotate each event with its conversation turn number, WITHOUT mutating any + # existing field. A turn starts at each real user message; events before the + # first user message are turn 0. `_turn`/`_index` are first keys on a shallow + # copy; every original key/value is copied through untouched. + annotated = [] + turn = 0 + for i, e in enumerate(events): + if is_user_turn_start(e): + turn += 1 + if isinstance(e, dict): + annotated.append({"_turn": turn, "_index": i, **e}) + else: + annotated.append({"_turn": turn, "_index": i, "_value": e}) + + base = src_path.name + dot = base.rfind(".") + if dot > 0: + base = base[:dot] + + if len(argv) >= 3 and argv[2]: + out_file = Path(argv[2]) + try: + out_file.resolve().parent.mkdir(parents=True, exist_ok=True) + except OSError as e: + fail( + "could not create output directory " + + str(out_file.resolve().parent) + + ": " + + str(e) + ) + else: + out_dir = Path(tempfile.gettempdir()) / "ess-diagnostics" / base + try: + out_dir.mkdir(parents=True, exist_ok=True) + except OSError as e: + fail("could not create output directory " + str(out_dir) + ": " + str(e)) + out_file = out_dir / (base + "-transcript.json") + + try: + out_file.write_text( + json.dumps(annotated, indent=2, ensure_ascii=False), encoding="utf-8" + ) + except OSError as e: + fail("could not write output: " + str(e)) + + sys.stdout.write(str(out_file.resolve()) + "\n") + + +if __name__ == "__main__": + main(sys.argv) From e36bb523aa7d5e01d18745c4c15fe35f4acad648 Mon Sep 17 00:00:00 2001 From: "Ramanan Vellore Ramesh (from Dev Box)" Date: Thu, 17 Sep 2026 21:15:40 -0700 Subject: [PATCH 5/8] ESS Diagnostics: SKILL.md invokes Python transcript helper Co-Authored-By: Claude --- tools/ESS-Diagnostics-Skills/SKILL.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/tools/ESS-Diagnostics-Skills/SKILL.md b/tools/ESS-Diagnostics-Skills/SKILL.md index 4268e5f96..5e2399bfa 100644 --- a/tools/ESS-Diagnostics-Skills/SKILL.md +++ b/tools/ESS-Diagnostics-Skills/SKILL.md @@ -94,10 +94,11 @@ not just the diagnostic fields — run the bundled helper rather than hand-parsi it: ``` -node scripts/transcript-to-json.js "" +python scripts/transcript_to_json.py "" ``` -(run from the skill directory `tools/ESS-Diagnostics-Skills/`). It writes a +(On Windows use `python`; on POSIX use `python3`.) Run from the skill directory +`tools/ESS-Diagnostics-Skills/`. It writes a lossless, pretty-printed `-transcript.json` into the same OS temp folder used for the other outputs (see Step 4), annotating each event with a `_turn` and `_index` but removing nothing, and prints the absolute output path @@ -376,7 +377,7 @@ JSON above, which keeps only the diagnostic fields). Produce it with the bundled helper introduced in Step 1: ``` -node scripts/transcript-to-json.js "" +python scripts/transcript_to_json.py "" ``` The helper writes this file into the same per-run temp folder and prints its From c309d6dd1e171354ed74b0984e688ff1fe4fae02 Mon Sep 17 00:00:00 2001 From: "Ramanan Vellore Ramesh (from Dev Box)" Date: Thu, 17 Sep 2026 21:16:08 -0700 Subject: [PATCH 6/8] ESS Diagnostics: remove Node transcript helper (replaced by Python) Co-Authored-By: Claude --- .../scripts/transcript-to-json.js | 114 ------------------ 1 file changed, 114 deletions(-) delete mode 100644 tools/ESS-Diagnostics-Skills/scripts/transcript-to-json.js diff --git a/tools/ESS-Diagnostics-Skills/scripts/transcript-to-json.js b/tools/ESS-Diagnostics-Skills/scripts/transcript-to-json.js deleted file mode 100644 index 62bdcd973..000000000 --- a/tools/ESS-Diagnostics-Skills/scripts/transcript-to-json.js +++ /dev/null @@ -1,114 +0,0 @@ -#!/usr/bin/env node -/** - * transcript-to-json.js — faithful transcript -> JSON helper for the ESS - * Diagnostics skill. - * - * Parses a Copilot Studio / PVA transcript (JSON payload stored in a .txt) and - * writes a LOSSLESS, pretty-printed JSON copy: every event and every field is - * preserved exactly. The only addition is a `_turn` number annotated on each - * event (0 for events before the first user message) so a reader can navigate - * by conversation turn. Nothing is removed, so the output round-trips back to - * the original once the `_turn` keys are dropped. - * - * This is the faithful full dump. It is NOT the compact diagnostic - * "normalized" view the skill builds for the 5 checks — it is the whole - * transcript, just as real JSON. - * - * Usage: - * node transcript-to-json.js [outFile.json] - * - * - path to the raw transcript export (required). - * - [outFile.json] where to write. If omitted, writes - * `-transcript.json` into the OS temp dir under - * `ess-diagnostics//` (outside any repo, because - * transcripts may contain employee PII). - * - * Prints the absolute output path on success. Exits non-zero with an - * `Error: ...` message on failure. - */ - -"use strict"; - -const fs = require("fs"); -const os = require("os"); -const path = require("path"); - -function fail(msg) { - process.stderr.write("Error: " + msg + "\n"); - process.exit(1); -} - -const srcArg = process.argv[2]; -if (!srcArg) fail("missing transcript path. Usage: node transcript-to-json.js [outFile.json]"); -if (!fs.existsSync(srcArg)) fail("transcript file not found: " + srcArg); - -let raw; -try { - raw = fs.readFileSync(srcArg, "utf8"); -} catch (e) { - fail("could not read transcript: " + e.message); -} - -// Strip a leading UTF-8 BOM if present, then parse. -if (raw.charCodeAt(0) === 0xfeff) raw = raw.slice(1); - -let events; -try { - events = JSON.parse(raw); -} catch (e) { - fail("transcript is not valid JSON: " + e.message); -} - -if (!Array.isArray(events)) { - fail("expected the transcript to be a JSON array of events; got " + typeof events); -} - -// Annotate each event with its conversation turn number, WITHOUT mutating any -// existing field. A turn starts at each real user message -// (SynchronousIncomingActivity with activity.type === "message"). Events before -// the first user message are turn 0. `_turn` is a first key on a shallow copy; -// every original key/value is copied through untouched. -function isUserTurnStart(e) { - return ( - e && - e.t === "SynchronousIncomingActivity" && - e.p && - e.p.activity && - e.p.activity.type === "message" - ); -} - -let turn = 0; -const annotated = events.map((e, i) => { - if (isUserTurnStart(e)) turn += 1; - // _turn and _index first, then a faithful spread of the original event. - return Object.assign({ _turn: turn, _index: i }, e); -}); - -// Resolve output path. -const base = path.basename(srcArg).replace(/\.[^.]+$/, ""); -let outFile = process.argv[3]; -if (!outFile) { - const outDir = path.join(os.tmpdir(), "ess-diagnostics", base); - try { - fs.mkdirSync(outDir, { recursive: true }); - } catch (e) { - fail("could not create output directory " + outDir + ": " + e.message); - } - outFile = path.join(outDir, base + "-transcript.json"); -} else { - const outParent = path.dirname(path.resolve(outFile)); - try { - fs.mkdirSync(outParent, { recursive: true }); - } catch (e) { - fail("could not create output directory " + outParent + ": " + e.message); - } -} - -try { - fs.writeFileSync(outFile, JSON.stringify(annotated, null, 2)); -} catch (e) { - fail("could not write output: " + e.message); -} - -process.stdout.write(path.resolve(outFile) + "\n"); From 09448f9e14adfa44850a82cbdecef756adb8b0de Mon Sep 17 00:00:00 2001 From: "Ramanan Vellore Ramesh (from Dev Box)" Date: Thu, 17 Sep 2026 23:00:18 -0700 Subject: [PATCH 7/8] ESS Diagnostics: consolidate plan docs into one design and one implementation Merge the v4-skill and Node->Python-port design docs into a single design doc, and their two implementation plans into a single implementation doc (each split into Part 1 / Part 2). No content lost. Co-Authored-By: Claude --- ...d => 2026-09-17-ess-diagnostics-design.md} | 137 +++++- ...26-09-17-ess-diagnostics-implementation.md | 415 ++++++++++++++++++ ...ess-diagnostics-skill-v4-implementation.md | 126 ------ ...transcript-helper-node-to-python-design.md | 114 ----- ...-09-17-transcript-helper-node-to-python.md | 278 ------------ 5 files changed, 551 insertions(+), 519 deletions(-) rename tools/ESS-Diagnostics-Skills/docs/plans/{2026-09-17-ess-diagnostics-skill-v4-design.md => 2026-09-17-ess-diagnostics-design.md} (58%) create mode 100644 tools/ESS-Diagnostics-Skills/docs/plans/2026-09-17-ess-diagnostics-implementation.md delete mode 100644 tools/ESS-Diagnostics-Skills/docs/plans/2026-09-17-ess-diagnostics-skill-v4-implementation.md delete mode 100644 tools/ESS-Diagnostics-Skills/docs/plans/2026-09-17-transcript-helper-node-to-python-design.md delete mode 100644 tools/ESS-Diagnostics-Skills/docs/plans/2026-09-17-transcript-helper-node-to-python.md diff --git a/tools/ESS-Diagnostics-Skills/docs/plans/2026-09-17-ess-diagnostics-skill-v4-design.md b/tools/ESS-Diagnostics-Skills/docs/plans/2026-09-17-ess-diagnostics-design.md similarity index 58% rename from tools/ESS-Diagnostics-Skills/docs/plans/2026-09-17-ess-diagnostics-skill-v4-design.md rename to tools/ESS-Diagnostics-Skills/docs/plans/2026-09-17-ess-diagnostics-design.md index 49d2f51d9..31f99daf5 100644 --- a/tools/ESS-Diagnostics-Skills/docs/plans/2026-09-17-ess-diagnostics-skill-v4-design.md +++ b/tools/ESS-Diagnostics-Skills/docs/plans/2026-09-17-ess-diagnostics-design.md @@ -1,4 +1,16 @@ -# ESS Diagnostics Skill (v4) — Interactive Walkthrough + Real-Schema Parse Map — Design +# ESS Diagnostics — Combined Design + +This document combines the design for two related efforts on the ESS Diagnostics +skill, in the order they were done: + +- **Part 1 — ESS Diagnostics Skill (v4):** the interactive-walkthrough + + real-schema parse-map revision of `SKILL.md`. +- **Part 2 — Transcript helper: Node.js → Python port:** converting the bundled + faithful-dump helper from `transcript-to-json.js` to `transcript_to_json.py`. + +--- + +# Part 1 — ESS Diagnostics Skill (v4): Interactive Walkthrough + Real-Schema Parse Map ## Purpose @@ -147,3 +159,126 @@ outputs land in the OS temp dir, not the repo. No automated suite (prose skill). `verifiedSearchResults[]` objects DO have `rankScore`, so CHECK-003 may use it. - These are corrections grounded in one real export; the graceful-missing-field rule guards against over-fitting to a single transcript. + +--- + +# Part 2 — Transcript helper: Node.js → Python port + +**Date:** 2026-09-17 +**Status:** Approved design +**Scope:** `tools/ESS-Diagnostics-Skills/` + +## Problem + +The ESS Diagnostics skill ships a helper, `scripts/transcript-to-json.js`, that +produces a faithful (lossless) JSON dump of a Copilot Studio / PVA transcript. +The rest of the repo is Python (3.11+, Ruff-linted, pytest suite). A lone +Node.js dependency in an otherwise Python toolkit is an avoidable footgun for +FDEs and maintainers. Convert the helper to Python and remove the Node version. + +## Goals + +- Replace the JS helper with a stdlib-only Python script that is an **exact + behavioral drop-in** (same CLI contract, same output, same error/exit + behavior). +- Update `SKILL.md` to invoke the Python helper. +- Remove the JS file. No Node dependency remains. + +## Non-goals + +- No automated test (matches the JS helper's current no-test status; validate + manually). +- No CLI enhancements (`argparse`, `--help`), no packaging (`__main__.py`), no + refactor of the diagnostic flow. Faithful port only. + +## Decisions + +| Question | Decision | +| --- | --- | +| Fate of the JS file | **Delete** — Python fully replaces it. | +| Invocation | `python scripts/transcript_to_json.py ""` (plain script, not `-m`). | +| Interpreter in docs | Document both: `python` (Windows) / `python3` (POSIX), matching SKILL.md's existing Windows/POSIX split. | +| Fidelity | **Exact behavioral parity** — true drop-in. | +| Test | **None** — manual validation. | +| Internal structure | **Approach A** — faithful procedural port, stdlib only, no unused abstraction. | +| Filename | `transcript_to_json.py` (PEP 8 underscores). | + +## Files + +- **Add:** `tools/ESS-Diagnostics-Skills/scripts/transcript_to_json.py` +- **Delete:** `tools/ESS-Diagnostics-Skills/scripts/transcript-to-json.js` +- **Edit:** `tools/ESS-Diagnostics-Skills/SKILL.md` (invocation commands, ~2 spots, + platform note) +- **README.md:** no change (verified — no `node`/`.js`/`transcript-to-json` references) + +No new dependencies. Stdlib only: `sys`, `json`, `pathlib`, `tempfile`. +Target: Python 3.11+. + +## The Python script (behavioral parity) + +CLI: `python transcript_to_json.py [outFile.json]` — arg 1 +required (source), arg 2 optional (output path). + +Flow, identical to the JS original: + +1. `fail(msg)` → write `Error: \n` to stderr, `sys.exit(1)`. +2. Missing arg 1 → fail with the usage message. Source file not found → fail. +3. Read source as UTF-8. Strip a leading BOM (``) if present. +4. `json.loads`; on error → `fail("transcript is not valid JSON: ...")`. +5. Not a `list` → `fail("expected the transcript to be a JSON array of events; got ")`. +6. Annotate: walk events in array order, increment `turn` at each user-turn + start (`t == "SynchronousIncomingActivity"` and `p.activity.type == + "message"`); emit each as `{"_turn": turn, "_index": i, **event}` so + `_turn`/`_index` are the first keys and every original field is preserved + untouched. Nested access is guarded (`isinstance` / `.get()` chains) so a + malformed event never raises — mirrors the JS `e && e.p && e.p.activity` + guard. +7. Resolve output path: if arg 2 is given, use it (create its parent dir); + otherwise `tempfile.gettempdir()/ess-diagnostics//-transcript.json`, + creating directories. Basename = source filename minus its final extension. +8. Write with `json.dumps(annotated, indent=2, ensure_ascii=False)`. + **`ensure_ascii=False` is required** for byte-parity with JS + `JSON.stringify`, which emits raw (non-escaped) Unicode. +9. Print the absolute output path to stdout. + +Wrapped in a `main()` called under `if __name__ == "__main__":`. + +Parity guarantees preserved: identical `Error:` prefix (SKILL.md instructs the +FDE to relay it), exit codes (0 success / 1 failure), temp-path shape, printed +absolute path, and round-trippable output (drop `_turn`/`_index` → original). + +**Accepted platform deviation:** on Windows, Python's `Path.write_text` +translates `\n` to `\r\n`, so the dump uses CRLF while the JS output used LF. +This was reviewed and accepted — the file is valid JSON consumed by re-parsing +(newline-agnostic), CRLF is native on Windows, and the content is byte-identical +after newline normalization. + +## SKILL.md edits + +Each `node scripts/transcript-to-json.js ""` (Step 1 "Faithful +transcript-to-JSON" and Step 4, file #3) becomes: + +``` +python scripts/transcript_to_json.py "" +``` + +Add a one-line platform note in the existing Windows/POSIX style: use `python` +on Windows, `python3` on POSIX. Surrounding prose (writes into the same temp +folder, prints the absolute path, relay `Error:` on failure, produce both dumps) +stays valid unchanged. + +## Error handling + +All failure modes route through `fail()`: missing/absent source, invalid JSON, +non-array top level, and unwritable output directory each print a single +`Error: ...` line to stderr and exit non-zero. No Python traceback leaks to the +FDE. Identical to the JS behavior. + +## Validation (manual) + +1. Run the new script against the real transcript + (`Transcript_ 4b548465-...txt`); confirm it prints the same absolute path and + writes a valid file. +2. Diff the Python output against the existing JS output for that transcript; + expect identical content (the annotation + round-trip guarantee). +3. Spot-check a failure case (non-existent path); confirm the `Error:` line and + non-zero exit. diff --git a/tools/ESS-Diagnostics-Skills/docs/plans/2026-09-17-ess-diagnostics-implementation.md b/tools/ESS-Diagnostics-Skills/docs/plans/2026-09-17-ess-diagnostics-implementation.md new file mode 100644 index 000000000..60ab8df74 --- /dev/null +++ b/tools/ESS-Diagnostics-Skills/docs/plans/2026-09-17-ess-diagnostics-implementation.md @@ -0,0 +1,415 @@ +# ESS Diagnostics — Combined Implementation Plan + +This document combines the implementation plans for two related efforts on the +ESS Diagnostics skill, in the order they were done: + +- **Part 1 — ESS Diagnostics Skill (v4):** prose edits to `SKILL.md` for the + interactive walkthrough + real-schema parse map. +- **Part 2 — Transcript helper: Node.js → Python port:** converting the bundled + helper to `transcript_to_json.py` and removing the Node version. + +See the combined design doc `2026-09-17-ess-diagnostics-design.md` for the +rationale behind both. + +--- + +# Part 1 — ESS Diagnostics Skill v4 Implementation Plan + +> **For Claude:** REQUIRED SUB-SKILL: Use `development/reference/executing-plans-guide.md` to implement this plan task-by-task. + +**Goal:** Revise `tools/ESS-Diagnostics-Skills/SKILL.md` to (a) make the diagnosis interactive — a hard problem-statement gate and a structured per-turn pause with section drill-down and verdict override — and (b) reconcile the parse map with the real Copilot Studio / PVA export schema, writing PII-bearing outputs to an OS temp dir outside the repo. + +**Architecture:** Prose edits to a single `SKILL.md` (read-only Claude skill, no code). Sourced from the v4 design (Part 1 of the combined design doc) and from field paths observed in a real transcript. Validation is a manual dry run. + +**Tech Stack:** Markdown. No test framework. + +**Line endings:** repo is `core.autocrlf=true` + `core.safecrlf=true`, files are CRLF. After each edit, normalize to exactly one CRLF per line: `sed -i 's/\r$//' && sed -i 's/$/\r/' `; verify `file ` says "CRLF line terminators" and `grep -aoP '\r\r' | wc -l` prints 0. Only touch `tools/ESS-Diagnostics-Skills/SKILL.md` unless a task says otherwise. + +**Real schema reference (from the observed transcript):** +- User turn: `t = SynchronousIncomingActivity`, `p.activity.type = "message"`, text at `p.activity.text`. +- Bot turn: `t = OutgoingActivity`, `p.activity.type = "message"`, text at `p.activity.text`. (`type = "event"` activities are system plumbing — skip.) +- Diagnostic events: `t = Trace`, discriminated by `p.data.kind` ∈ {`LlmIntentRecognized`, `PluginStart`, `PluginResponse`, `KnowledgeTraceData`, `AnalyticsAiMetricsSignalTraceData`, …}. +- `PluginStart.input`: `search_query`, `search_keywords`, `enable_summarization`. +- `PluginResponse`: `citableContent[]` (`source`, `chunks`), `output`. +- `AnalyticsAiMetricsSignalTraceData`: `completionState`, `triggeredGptFallback`, `rewrittenMessage`, `rewrittenMessageKeywords`, `verifiedSearchResults[]` (`url`, `rankScore`, `snippet`, `searchType`), `citedKnowledgeSources`, `textCitations`. +- `KnowledgeTraceData`: `isKnowledgeSearched`, `completionState`, `citedKnowledgeSources`, `failedKnowledgeSourcesTypes`. + +--- + +### Task 1: Add Step 0 (problem-statement hard gate) and strengthen the Rule + +**Files:** Modify `tools/ESS-Diagnostics-Skills/SKILL.md`. + +**Step 1:** In `## Rules`, replace the bullet "Require two inputs up front… If either is missing, ask for it before proceeding." with a hard-stop bullet: the skill must not proceed past Step 0 until the FDE has stated the problem and the file path is confirmed — even if a problem was supplied in the invocation. + +**Step 2:** Insert a new `## Step 0: Establish the Problem` section BEFORE `## Step 1`. It must instruct: on invocation, STOP; confirm the transcript file path; explicitly ask "What specific problem are you investigating in this transcript?"; do not read/parse until answered; if a problem statement was supplied in the args, echo it back and ask the FDE to confirm or refine rather than silently accepting. + +**Step 3:** CRLF-normalize, verify, `git add`, commit: `feat: add Step 0 problem-statement hard gate to diagnostics skill`. + +--- + +### Task 2: Rewrite the Step 1 parse map to the real schema + +**Files:** Modify `tools/ESS-Diagnostics-Skills/SKILL.md`. + +**Step 1:** Replace the Step 1 parse-map table with the concept→location→fields table from the v4 design (SynchronousIncomingActivity; OutgoingActivity; Trace/`p.data.kind` for LlmIntentRecognized, PluginStart, PluginResponse, AnalyticsAiMetricsSignalTraceData, KnowledgeTraceData). Keep the array-position ordering rule. Add: only `p.activity.type = "message"` activities are real turns; `type = "event"` are skipped for segmentation. Replace the old "search_results[] with keys FileType/Name/Text/…, no score field" note with the real result shapes (`citableContent[]` source/chunks; `verifiedSearchResults[]` url/rankScore/snippet). + +**Step 2:** Add the graceful-missing-field rule: if a mapped field/event is absent, mark the affected check N/A with "field not present in this transcript" rather than failing/guessing; note the map derives from observed exports and may vary by bot config. + +**Step 3:** CRLF-normalize, verify, `git add`, commit: `feat: reconcile Step 1 parse map with real transcript schema`. + +--- + +### Task 3: Reconcile the 5 checks with the real fields + +**Files:** Modify `tools/ESS-Diagnostics-Skills/SKILL.md`. + +**Step 1:** Update each check bullet in Step 2: +- CHECK-001: recognized via `LlmIntentRecognized`; failure = intent absent OR `triggeredGptFallback = true`. Remove `UnknownIntentTriggered` as the failure path (note it's not present in real data; keep a mention that some exports may use it). +- CHECK-002: original prompt from `SynchronousIncomingActivity.p.activity.text`; issued query/keywords from `PluginStart.input`; rewrite read DIRECTLY from `rewrittenMessage` / `rewrittenMessageKeywords` (not inferred). +- CHECK-003: use `verifiedSearchResults[].rankScore` + `snippet` where present; fall back to keyword overlap on `snippet`/`source`/`citableContent.chunks`. (rankScore DOES exist — drop the "no score field" claim.) +- CHECK-004: from `isKnowledgeSearched`, `completionState`, `citedKnowledgeSources`, `verifiedSearchResults` count vs. the answer. +- CHECK-005: answer vs. `verifiedSearchResults` / `citedKnowledgeSources` / `textCitations`. + +**Step 2:** Keep the existing "earlier failed check removes a later precondition ⇒ N/A" rule. + +**Step 3:** CRLF-normalize, verify, `git add`, commit: `feat: reconcile the 5 checks with real transcript fields`. + +--- + +### Task 4: Make Step 2 an interactive loop with structured pause + drill-down + override + +**Files:** Modify `tools/ESS-Diagnostics-Skills/SKILL.md`. + +**Step 1:** Rewrite the Step 2 walkthrough framing so that per turn the skill presents a STRUCTURED SUMMARY: Intent (id/message or failure signal); Search query (original prompt + issued/rewritten query & keywords); Search response (result count / cited sources + bot answer); and a one-line verdict strip `001 … · 005 …`. + +**Step 2:** Add the section menu + hard stop: +- Menu options 1–5 (Intent, Search query, Search results, Grounding, Final answer) plus `[continue]`, `[run all]`, `[override]`. +- Hard stop: the skill WAITS and may not look at the next turn until the FDE responds. +- `run all` (available from the first pause) switches to a batch pass that auto-advances remaining turns but still produces the full report. + +**Step 3:** Add drill-down + override behavior: +- Drill-down (section 1–5): show full raw transcript evidence for that section PLUS the check's reasoning (why Pass/Fail/N/A), and explicitly invite the FDE to override the verdict. +- Override: record (check, original verdict → override, FDE reason); it flows into the final report. + +**Step 4:** CRLF-normalize, verify, `git add`, commit: `feat: make per-turn walkthrough interactive with drill-down and override`. + +--- + +### Task 5: Update Step 4 outputs — OS temp dir, overrides, real-schema JSON + +**Files:** Modify `tools/ESS-Diagnostics-Skills/SKILL.md`. + +**Step 1:** Change output location: write the two files to a per-run subfolder in the OS temp dir (`%TEMP%/ess-diagnostics//` on Windows; `$TMPDIR` or `/tmp` on POSIX), NOT under the repo. Remove the `reports/`-under-repo path. Add a one-time PII warning and instruct the skill to report the absolute temp paths. + +**Step 2:** Markdown report additions: include the confirmed problem statement (from Step 0) and any FDE verdict overrides (check, original → override, reason). + +**Step 3:** Normalized JSON updates: `search.rewritten`/`rewrite_note` from `rewrittenMessage`/`rewrittenMessageKeywords`; `search_results` → real `verifiedSearchResults[]` shape (`url`, `rankScore`, `snippet`, …) and/or `citableContent[]`; add a `grounding` object (`completionState`, `isKnowledgeSearched`, `citedKnowledgeSources`, `triggeredGptFallback`). Update the fenced JSON example to this shape. Keep "NO verdicts in JSON". + +**Step 4:** CRLF-normalize, verify, `git add`, commit: `feat: write outputs to OS temp dir and update JSON to real schema`. + +--- + +### Task 6: Update README for v4 behavior + +**Files:** Modify `tools/ESS-Diagnostics-Skills/README.md`. + +**Step 1:** Update the README to describe: the Step 0 problem gate; the interactive per-turn pause with section drill-down and override; outputs written to an OS temp dir (not the repo) with a PII note; and the CHECK-004 (grounding) / CHECK-003 (rankScore) reconciliations. Keep it concise. + +**Step 2:** CRLF-normalize, verify, `git add`, commit: `docs: update README for v4 interactive + real-schema behavior`. + +--- + +### Task 7: Dry-run validation against the real transcript + +**Files:** none (validation only). + +**Step 1:** Follow the revised SKILL.md against `C:\Users\rarame\Downloads\Transcript_a9b8b841-6a4f-4675-9e79-92fc6df74be9.txt`. Confirm: +- Step 0 hard gate fires (skill asks for the problem before parsing). +- Parse map extracts the real events: 3 turns; intents `LlmIntentRecognized`; PluginStart queries + `rewrittenMessage`; `verifiedSearchResults` (count 10), `completionState = Answered`. +- The per-turn structured pause shows intent/search/response + verdict strip + section menu; drill-down shows raw evidence + reasoning; an override is accepted and recorded; `run all` works. +- Outputs land in the OS temp dir (NOT the repo); `git status` under the repo stays clean. + +**Step 2:** Do NOT commit any generated report/JSON. Confirm nothing was written under `tools/ESS-Diagnostics-Skills/`. Record any gaps found; if SKILL.md needs a fix, apply it and commit `fix: address gaps found during v4 dry run`. + +--- + +# Part 2 — Transcript helper: Node→Python port Implementation Plan + +> **For Claude:** REQUIRED SUB-SKILL: Use `development/reference/executing-plans-guide.md` to implement this plan task-by-task. + +**Goal:** Replace the Node.js helper `scripts/transcript-to-json.js` with a stdlib-only Python drop-in (`scripts/transcript_to_json.py`) of exact behavioral parity, update `SKILL.md` to invoke it, and delete the JS file. + +**Architecture:** A single faithful procedural port (design "Approach A"). The Python script mirrors the JS flow one-for-one — BOM strip, JSON-array validation, `_turn`/`_index` annotation as first keys, temp-dir output, absolute-path print, `Error:`/exit-1 failures. No dependencies, no packaging, no CLI framework. Validated manually by diffing its output against the existing JS output for a real transcript (no automated test, per the approved design). + +**Tech Stack:** Python 3.11+ stdlib only (`sys`, `json`, `pathlib`, `tempfile`). Repo lint: Ruff (E4/E7/E9/F). Git `core.autocrlf=true` — new text files must be committed with CRLF terminators. + +**Design:** Part 2 of the combined design doc `2026-09-17-ess-diagnostics-design.md`. + +**Working directory for all paths below:** repo root `Employee-Self-Service-Agent-Developer-Kit/`. + +**Note on git & line endings:** `core.autocrlf=true` with no `.gitattributes` caused a "LF would be replaced by CRLF" commit failure on a LF-only file. Before every commit of a NEW text file (`.py`, `.md`), convert to CRLF first: `sed -i 's/$/\r/' ` (idempotent-guard: only run on files freshly written LF-only). Existing files edited in place keep their endings. + +--- + +### Task 1: Create the Python helper + +**Files:** +- Create: `tools/ESS-Diagnostics-Skills/scripts/transcript_to_json.py` + +**Step 1: Write the script** + +Write exactly this content to `tools/ESS-Diagnostics-Skills/scripts/transcript_to_json.py`: + +```python +#!/usr/bin/env python3 +"""transcript_to_json.py — faithful transcript -> JSON helper for the ESS +Diagnostics skill. + +Parses a Copilot Studio / PVA transcript (JSON payload stored in a .txt) and +writes a LOSSLESS, pretty-printed JSON copy: every event and every field is +preserved exactly. The only additions are `_turn` and `_index`, annotated as +the first keys on each event (turn 0 for events before the first user message) +so a reader can navigate by conversation turn. Nothing is removed, so the +output round-trips back to the original once `_turn`/`_index` are dropped. + +This is the faithful full dump. It is NOT the compact diagnostic "normalized" +view the skill builds for the 5 checks — it is the whole transcript, just as +real JSON. + +Usage: + python transcript_to_json.py [outFile.json] + +- path to the raw transcript export (required). +- [outFile.json] where to write. If omitted, writes + `-transcript.json` into the OS temp dir under + `ess-diagnostics//` (outside any repo, because + transcripts may contain employee PII). + +Prints the absolute output path on success. Exits non-zero with an +`Error: ...` message on failure. +""" + +import json +import sys +import tempfile +from pathlib import Path + + +def fail(msg): + sys.stderr.write("Error: " + msg + "\n") + sys.exit(1) + + +def is_user_turn_start(e): + return ( + isinstance(e, dict) + and e.get("t") == "SynchronousIncomingActivity" + and isinstance(e.get("p"), dict) + and isinstance(e["p"].get("activity"), dict) + and e["p"]["activity"].get("type") == "message" + ) + + +def main(argv): + if len(argv) < 2: + fail( + "missing transcript path. Usage: " + "python transcript_to_json.py [outFile.json]" + ) + src_arg = argv[1] + src_path = Path(src_arg) + if not src_path.exists(): + fail("transcript file not found: " + src_arg) + + try: + raw = src_path.read_text(encoding="utf-8") + except OSError as e: + fail("could not read transcript: " + str(e)) + + # Strip a leading UTF-8 BOM if present, then parse. + if raw and raw[0] == "\ufeff": + raw = raw[1:] + + try: + events = json.loads(raw) + except ValueError as e: + fail("transcript is not valid JSON: " + str(e)) + + if not isinstance(events, list): + fail( + "expected the transcript to be a JSON array of events; got " + + type(events).__name__ + ) + + # Annotate each event with its conversation turn number, WITHOUT mutating any + # existing field. A turn starts at each real user message; events before the + # first user message are turn 0. `_turn`/`_index` are first keys on a shallow + # copy; every original key/value is copied through untouched. + annotated = [] + turn = 0 + for i, e in enumerate(events): + if is_user_turn_start(e): + turn += 1 + if isinstance(e, dict): + annotated.append({"_turn": turn, "_index": i, **e}) + else: + annotated.append({"_turn": turn, "_index": i, "_value": e}) + + base = src_path.name + dot = base.rfind(".") + if dot > 0: + base = base[:dot] + + if len(argv) >= 3 and argv[2]: + out_file = Path(argv[2]) + try: + out_file.resolve().parent.mkdir(parents=True, exist_ok=True) + except OSError as e: + fail( + "could not create output directory " + + str(out_file.resolve().parent) + + ": " + + str(e) + ) + else: + out_dir = Path(tempfile.gettempdir()) / "ess-diagnostics" / base + try: + out_dir.mkdir(parents=True, exist_ok=True) + except OSError as e: + fail("could not create output directory " + str(out_dir) + ": " + str(e)) + out_file = out_dir / (base + "-transcript.json") + + try: + out_file.write_text( + json.dumps(annotated, indent=2, ensure_ascii=False), encoding="utf-8" + ) + except OSError as e: + fail("could not write output: " + str(e)) + + sys.stdout.write(str(out_file.resolve()) + "\n") + + +if __name__ == "__main__": + main(sys.argv) +``` + +**Step 2: Lint the new file** + +Run: `cd tools/ESS-Diagnostics-Skills && python -m ruff check scripts/transcript_to_json.py` +(If Ruff is not installed, skip — it is a dev tool; note the skip.) +Expected: no errors (E4/E7/E9/F clean). + +**Step 3: Run against the real transcript** + +Run (from `tools/ESS-Diagnostics-Skills`): +`python scripts/transcript_to_json.py "C:\Users\rarame\Downloads\Transcript_ 4b548465-9be5-4b73-bc96-e3d6bbbe5229.txt"` +Expected: prints one absolute path ending in +`ess-diagnostics\Transcript_ 4b548465-9be5-4b73-bc96-e3d6bbbe5229\Transcript_ 4b548465-9be5-4b73-bc96-e3d6bbbe5229-transcript.json` +and exits 0. + +**Step 4: Diff Python output against the existing JS output (parity check)** + +The JS dump already exists from this session at the same temp path. Regenerate +it to a side path with the JS helper, then compare: +``` +node scripts/transcript-to-json.js "C:\Users\rarame\Downloads\Transcript_ 4b548465-9be5-4b73-bc96-e3d6bbbe5229.txt" /tmp/js-out.json +python scripts/transcript_to_json.py "C:\Users\rarame\Downloads\Transcript_ 4b548465-9be5-4b73-bc96-e3d6bbbe5229.txt" /tmp/py-out.json +diff /tmp/js-out.json /tmp/py-out.json && echo "IDENTICAL" +``` +Expected: no content diff. On Windows the Python output uses CRLF where JS uses +LF; that newline-only difference is the accepted deviation — normalize newlines +(`diff <(tr -d '\r' < js) <(tr -d '\r' < py)`) and expect IDENTICAL. Any other +diff (Unicode escaping, key ordering) must be fixed before continuing. + +**Step 5: Spot-check a failure case** + +Run: `python scripts/transcript_to_json.py /nonexistent/file.txt; echo "exit=$?"` +Expected: stderr line `Error: transcript file not found: /nonexistent/file.txt` and `exit=1`. + +**Step 6: Commit** + +```bash +cd +sed -i 's/$/\r/' tools/ESS-Diagnostics-Skills/scripts/transcript_to_json.py +git add tools/ESS-Diagnostics-Skills/scripts/transcript_to_json.py +git commit -m "ESS Diagnostics: add Python transcript_to_json helper (drop-in for JS)" +``` +(Append the `Co-Authored-By: Claude ` line per session attribution.) + +--- + +### Task 2: Point SKILL.md at the Python helper + +**Files:** +- Modify: `tools/ESS-Diagnostics-Skills/SKILL.md` (Step 1 "Faithful transcript-to-JSON" block; Step 4 file #3 block) + +**Step 1: Replace both invocation code blocks** + +Both currently read: +``` +node scripts/transcript-to-json.js "" +``` +Change each to: +``` +python scripts/transcript_to_json.py "" +``` + +**Step 2: Add a platform note next to the Step 1 invocation** + +Immediately after the Step 1 code block, add a sentence in the existing +Windows/POSIX style: +> On Windows use `python`; on POSIX use `python3`. + +**Step 3: Verify no stale JS references remain** + +Run: `grep -n "node \|transcript-to-json.js\|\.js" tools/ESS-Diagnostics-Skills/SKILL.md` +Expected: no matches. + +**Step 4: Commit** + +```bash +git add tools/ESS-Diagnostics-Skills/SKILL.md +git commit -m "ESS Diagnostics: SKILL.md invokes Python transcript helper" +``` +(SKILL.md is an existing file; Edit preserves its CRLF endings. If a commit fails on CRLF, re-normalize with `sed -i 's/$/\r/'`.) + +--- + +### Task 3: Remove the Node.js helper + +**Files:** +- Delete: `tools/ESS-Diagnostics-Skills/scripts/transcript-to-json.js` + +**Step 1: Confirm nothing operational references it** + +Run: `grep -rn "transcript-to-json" tools/ESS-Diagnostics-Skills/ --include=*.md --include=*.py --include=*.js` +Expected: matches only inside `docs/plans/` (historical design/plan text) — no +match in SKILL.md, README, or scripts. Fix any operational match before deleting. + +**Step 2: Delete the file** + +```bash +git rm tools/ESS-Diagnostics-Skills/scripts/transcript-to-json.js +``` + +**Step 3: Final sanity run** + +Run the Python helper once more against the real transcript (Task 1 Step 3) to +confirm the skill's dump path still works with the JS file gone. +Expected: same absolute path printed, exit 0. + +**Step 4: Commit** + +```bash +git commit -m "ESS Diagnostics: remove Node transcript helper (replaced by Python)" +``` + +--- + +### Task 4: Final verification + +**Step 1:** `git log --oneline` — confirm the port commits are present. +**Step 2:** `git status` — confirm clean working tree. +**Step 3:** Confirm `scripts/` now contains only `transcript_to_json.py` (no `.js`): +`ls tools/ESS-Diagnostics-Skills/scripts/` diff --git a/tools/ESS-Diagnostics-Skills/docs/plans/2026-09-17-ess-diagnostics-skill-v4-implementation.md b/tools/ESS-Diagnostics-Skills/docs/plans/2026-09-17-ess-diagnostics-skill-v4-implementation.md deleted file mode 100644 index ef0a55b0d..000000000 --- a/tools/ESS-Diagnostics-Skills/docs/plans/2026-09-17-ess-diagnostics-skill-v4-implementation.md +++ /dev/null @@ -1,126 +0,0 @@ -# ESS Diagnostics Skill v4 Implementation Plan - -> **For Claude:** REQUIRED SUB-SKILL: Use `development/reference/executing-plans-guide.md` to implement this plan task-by-task. - -**Goal:** Revise `tools/ESS-Diagnostics-Skills/SKILL.md` to (a) make the diagnosis interactive — a hard problem-statement gate and a structured per-turn pause with section drill-down and verdict override — and (b) reconcile the parse map with the real Copilot Studio / PVA export schema, writing PII-bearing outputs to an OS temp dir outside the repo. - -**Architecture:** Prose edits to a single `SKILL.md` (read-only Claude skill, no code). Sourced from the v4 design doc `docs/plans/2026-09-17-ess-diagnostics-skill-v4-design.md` and from field paths observed in a real transcript. Validation is a manual dry run. - -**Tech Stack:** Markdown. No test framework. - -**Line endings:** repo is `core.autocrlf=true` + `core.safecrlf=true`, files are CRLF. After each edit, normalize to exactly one CRLF per line: `sed -i 's/\r$//' && sed -i 's/$/\r/' `; verify `file ` says "CRLF line terminators" and `grep -aoP '\r\r' | wc -l` prints 0. Only touch `tools/ESS-Diagnostics-Skills/SKILL.md` unless a task says otherwise. - -**Real schema reference (from the observed transcript):** -- User turn: `t = SynchronousIncomingActivity`, `p.activity.type = "message"`, text at `p.activity.text`. -- Bot turn: `t = OutgoingActivity`, `p.activity.type = "message"`, text at `p.activity.text`. (`type = "event"` activities are system plumbing — skip.) -- Diagnostic events: `t = Trace`, discriminated by `p.data.kind` ∈ {`LlmIntentRecognized`, `PluginStart`, `PluginResponse`, `KnowledgeTraceData`, `AnalyticsAiMetricsSignalTraceData`, …}. -- `PluginStart.input`: `search_query`, `search_keywords`, `enable_summarization`. -- `PluginResponse`: `citableContent[]` (`source`, `chunks`), `output`. -- `AnalyticsAiMetricsSignalTraceData`: `completionState`, `triggeredGptFallback`, `rewrittenMessage`, `rewrittenMessageKeywords`, `verifiedSearchResults[]` (`url`, `rankScore`, `snippet`, `searchType`), `citedKnowledgeSources`, `textCitations`. -- `KnowledgeTraceData`: `isKnowledgeSearched`, `completionState`, `citedKnowledgeSources`, `failedKnowledgeSourcesTypes`. - ---- - -### Task 1: Add Step 0 (problem-statement hard gate) and strengthen the Rule - -**Files:** Modify `tools/ESS-Diagnostics-Skills/SKILL.md`. - -**Step 1:** In `## Rules`, replace the bullet "Require two inputs up front… If either is missing, ask for it before proceeding." with a hard-stop bullet: the skill must not proceed past Step 0 until the FDE has stated the problem and the file path is confirmed — even if a problem was supplied in the invocation. - -**Step 2:** Insert a new `## Step 0: Establish the Problem` section BEFORE `## Step 1`. It must instruct: on invocation, STOP; confirm the transcript file path; explicitly ask "What specific problem are you investigating in this transcript?"; do not read/parse until answered; if a problem statement was supplied in the args, echo it back and ask the FDE to confirm or refine rather than silently accepting. - -**Step 3:** CRLF-normalize, verify, `git add`, commit: `feat: add Step 0 problem-statement hard gate to diagnostics skill`. - ---- - -### Task 2: Rewrite the Step 1 parse map to the real schema - -**Files:** Modify `tools/ESS-Diagnostics-Skills/SKILL.md`. - -**Step 1:** Replace the Step 1 parse-map table with the concept→location→fields table from the v4 design (SynchronousIncomingActivity; OutgoingActivity; Trace/`p.data.kind` for LlmIntentRecognized, PluginStart, PluginResponse, AnalyticsAiMetricsSignalTraceData, KnowledgeTraceData). Keep the array-position ordering rule. Add: only `p.activity.type = "message"` activities are real turns; `type = "event"` are skipped for segmentation. Replace the old "search_results[] with keys FileType/Name/Text/…, no score field" note with the real result shapes (`citableContent[]` source/chunks; `verifiedSearchResults[]` url/rankScore/snippet). - -**Step 2:** Add the graceful-missing-field rule: if a mapped field/event is absent, mark the affected check N/A with "field not present in this transcript" rather than failing/guessing; note the map derives from observed exports and may vary by bot config. - -**Step 3:** CRLF-normalize, verify, `git add`, commit: `feat: reconcile Step 1 parse map with real transcript schema`. - ---- - -### Task 3: Reconcile the 5 checks with the real fields - -**Files:** Modify `tools/ESS-Diagnostics-Skills/SKILL.md`. - -**Step 1:** Update each check bullet in Step 2: -- CHECK-001: recognized via `LlmIntentRecognized`; failure = intent absent OR `triggeredGptFallback = true`. Remove `UnknownIntentTriggered` as the failure path (note it's not present in real data; keep a mention that some exports may use it). -- CHECK-002: original prompt from `SynchronousIncomingActivity.p.activity.text`; issued query/keywords from `PluginStart.input`; rewrite read DIRECTLY from `rewrittenMessage` / `rewrittenMessageKeywords` (not inferred). -- CHECK-003: use `verifiedSearchResults[].rankScore` + `snippet` where present; fall back to keyword overlap on `snippet`/`source`/`citableContent.chunks`. (rankScore DOES exist — drop the "no score field" claim.) -- CHECK-004: from `isKnowledgeSearched`, `completionState`, `citedKnowledgeSources`, `verifiedSearchResults` count vs. the answer. -- CHECK-005: answer vs. `verifiedSearchResults` / `citedKnowledgeSources` / `textCitations`. - -**Step 2:** Keep the existing "earlier failed check removes a later precondition ⇒ N/A" rule. - -**Step 3:** CRLF-normalize, verify, `git add`, commit: `feat: reconcile the 5 checks with real transcript fields`. - ---- - -### Task 4: Make Step 2 an interactive loop with structured pause + drill-down + override - -**Files:** Modify `tools/ESS-Diagnostics-Skills/SKILL.md`. - -**Step 1:** Rewrite the Step 2 walkthrough framing so that per turn the skill presents a STRUCTURED SUMMARY: Intent (id/message or failure signal); Search query (original prompt + issued/rewritten query & keywords); Search response (result count / cited sources + bot answer); and a one-line verdict strip `001 … · 005 …`. - -**Step 2:** Add the section menu + hard stop: -- Menu options 1–5 (Intent, Search query, Search results, Grounding, Final answer) plus `[continue]`, `[run all]`, `[override]`. -- Hard stop: the skill WAITS and may not look at the next turn until the FDE responds. -- `run all` (available from the first pause) switches to a batch pass that auto-advances remaining turns but still produces the full report. - -**Step 3:** Add drill-down + override behavior: -- Drill-down (section 1–5): show full raw transcript evidence for that section PLUS the check's reasoning (why Pass/Fail/N/A), and explicitly invite the FDE to override the verdict. -- Override: record (check, original verdict → override, FDE reason); it flows into the final report. - -**Step 4:** CRLF-normalize, verify, `git add`, commit: `feat: make per-turn walkthrough interactive with drill-down and override`. - ---- - -### Task 5: Update Step 4 outputs — OS temp dir, overrides, real-schema JSON - -**Files:** Modify `tools/ESS-Diagnostics-Skills/SKILL.md`. - -**Step 1:** Change output location: write the two files to a per-run subfolder in the OS temp dir (`%TEMP%/ess-diagnostics//` on Windows; `$TMPDIR` or `/tmp` on POSIX), NOT under the repo. Remove the `reports/`-under-repo path. Add a one-time PII warning and instruct the skill to report the absolute temp paths. - -**Step 2:** Markdown report additions: include the confirmed problem statement (from Step 0) and any FDE verdict overrides (check, original → override, reason). - -**Step 3:** Normalized JSON updates: `search.rewritten`/`rewrite_note` from `rewrittenMessage`/`rewrittenMessageKeywords`; `search_results` → real `verifiedSearchResults[]` shape (`url`, `rankScore`, `snippet`, …) and/or `citableContent[]`; add a `grounding` object (`completionState`, `isKnowledgeSearched`, `citedKnowledgeSources`, `triggeredGptFallback`). Update the fenced JSON example to this shape. Keep "NO verdicts in JSON". - -**Step 4:** CRLF-normalize, verify, `git add`, commit: `feat: write outputs to OS temp dir and update JSON to real schema`. - ---- - -### Task 6: Update README for v4 behavior - -**Files:** Modify `tools/ESS-Diagnostics-Skills/README.md`. - -**Step 1:** Update the README to describe: the Step 0 problem gate; the interactive per-turn pause with section drill-down and override; outputs written to an OS temp dir (not the repo) with a PII note; and the CHECK-004 (grounding) / CHECK-003 (rankScore) reconciliations. Keep it concise. - -**Step 2:** CRLF-normalize, verify, `git add`, commit: `docs: update README for v4 interactive + real-schema behavior`. - ---- - -### Task 7: Dry-run validation against the real transcript - -**Files:** none (validation only). - -**Step 1:** Follow the revised SKILL.md against `C:\Users\rarame\Downloads\Transcript_a9b8b841-6a4f-4675-9e79-92fc6df74be9.txt`. Confirm: -- Step 0 hard gate fires (skill asks for the problem before parsing). -- Parse map extracts the real events: 3 turns; intents `LlmIntentRecognized`; PluginStart queries + `rewrittenMessage`; `verifiedSearchResults` (count 10), `completionState = Answered`. -- The per-turn structured pause shows intent/search/response + verdict strip + section menu; drill-down shows raw evidence + reasoning; an override is accepted and recorded; `run all` works. -- Outputs land in the OS temp dir (NOT the repo); `git status` under the repo stays clean. - -**Step 2:** Do NOT commit any generated report/JSON. Confirm nothing was written under `tools/ESS-Diagnostics-Skills/`. Record any gaps found; if SKILL.md needs a fix, apply it and commit `fix: address gaps found during v4 dry run`. - ---- - -## Execution Handoff - -Two options: -1. **Subagent-Driven (this session)** — fresh subagent per task, spec review between tasks. -2. **Parallel Session** — new session using `development/reference/executing-plans-guide.md`. diff --git a/tools/ESS-Diagnostics-Skills/docs/plans/2026-09-17-transcript-helper-node-to-python-design.md b/tools/ESS-Diagnostics-Skills/docs/plans/2026-09-17-transcript-helper-node-to-python-design.md deleted file mode 100644 index 61ad1f773..000000000 --- a/tools/ESS-Diagnostics-Skills/docs/plans/2026-09-17-transcript-helper-node-to-python-design.md +++ /dev/null @@ -1,114 +0,0 @@ -# Convert transcript-to-json helper from Node.js to Python - -**Date:** 2026-09-17 -**Status:** Approved design -**Scope:** `tools/ESS-Diagnostics-Skills/` - -## Problem - -The ESS Diagnostics skill ships a helper, `scripts/transcript-to-json.js`, that -produces a faithful (lossless) JSON dump of a Copilot Studio / PVA transcript. -The rest of the repo is Python (3.11+, Ruff-linted, pytest suite). A lone -Node.js dependency in an otherwise Python toolkit is an avoidable footgun for -FDEs and maintainers. Convert the helper to Python and remove the Node version. - -## Goals - -- Replace the JS helper with a stdlib-only Python script that is an **exact - behavioral drop-in** (same CLI contract, same output, same error/exit - behavior). -- Update `SKILL.md` to invoke the Python helper. -- Remove the JS file. No Node dependency remains. - -## Non-goals - -- No automated test (matches the JS helper's current no-test status; validate - manually). -- No CLI enhancements (`argparse`, `--help`), no packaging (`__main__.py`), no - refactor of the diagnostic flow. Faithful port only. - -## Decisions - -| Question | Decision | -| --- | --- | -| Fate of the JS file | **Delete** — Python fully replaces it. | -| Invocation | `python scripts/transcript_to_json.py ""` (plain script, not `-m`). | -| Interpreter in docs | Document both: `python` (Windows) / `python3` (POSIX), matching SKILL.md's existing Windows/POSIX split. | -| Fidelity | **Exact behavioral parity** — true drop-in. | -| Test | **None** — manual validation. | -| Internal structure | **Approach A** — faithful procedural port, stdlib only, no unused abstraction. | -| Filename | `transcript_to_json.py` (PEP 8 underscores). | - -## Files - -- **Add:** `tools/ESS-Diagnostics-Skills/scripts/transcript_to_json.py` -- **Delete:** `tools/ESS-Diagnostics-Skills/scripts/transcript-to-json.js` -- **Edit:** `tools/ESS-Diagnostics-Skills/SKILL.md` (invocation commands, ~2 spots, + platform note) -- **README.md:** no change (verified — no `node`/`.js`/`transcript-to-json` references) - -No new dependencies. Stdlib only: `sys`, `json`, `pathlib`, `tempfile`. -Target: Python 3.11+. - -## The Python script (behavioral parity) - -CLI: `python transcript_to_json.py [outFile.json]` — arg 1 -required (source), arg 2 optional (output path). - -Flow, identical to the JS original: - -1. `fail(msg)` → write `Error: \n` to stderr, `sys.exit(1)`. -2. Missing arg 1 → fail with the usage message. Source file not found → fail. -3. Read source as UTF-8. Strip a leading BOM (``) if present. -4. `json.loads`; on error → `fail("transcript is not valid JSON: ...")`. -5. Not a `list` → `fail("expected the transcript to be a JSON array of events; got ")`. -6. Annotate: walk events in array order, increment `turn` at each user-turn - start (`t == "SynchronousIncomingActivity"` and `p.activity.type == - "message"`); emit each as `{"_turn": turn, "_index": i, **event}` so - `_turn`/`_index` are the first keys and every original field is preserved - untouched. Nested access is guarded (`isinstance` / `.get()` chains) so a - malformed event never raises — mirrors the JS `e && e.p && e.p.activity` - guard. -7. Resolve output path: if arg 2 is given, use it (create its parent dir); - otherwise `tempfile.gettempdir()/ess-diagnostics//-transcript.json`, - creating directories. Basename = source filename minus its final extension. -8. Write with `json.dumps(annotated, indent=2, ensure_ascii=False)`. - **`ensure_ascii=False` is required** for byte-parity with JS - `JSON.stringify`, which emits raw (non-escaped) Unicode. -9. Print the absolute output path to stdout. - -Wrapped in a `main()` called under `if __name__ == "__main__":`. - -Parity guarantees preserved: identical `Error:` prefix (SKILL.md instructs the -FDE to relay it), exit codes (0 success / 1 failure), temp-path shape, printed -absolute path, and round-trippable output (drop `_turn`/`_index` → original). - -## SKILL.md edits - -Each `node scripts/transcript-to-json.js ""` (Step 1 "Faithful -transcript-to-JSON" and Step 4, file #3) becomes: - -``` -python scripts/transcript_to_json.py "" -``` - -Add a one-line platform note in the existing Windows/POSIX style: use `python` -on Windows, `python3` on POSIX. Surrounding prose (writes into the same temp -folder, prints the absolute path, relay `Error:` on failure, produce both dumps) -stays valid unchanged. - -## Error handling - -All failure modes route through `fail()`: missing/absent source, invalid JSON, -non-array top level, and unwritable output directory each print a single -`Error: ...` line to stderr and exit non-zero. No Python traceback leaks to the -FDE. Identical to the JS behavior. - -## Validation (manual) - -1. Run the new script against the real transcript - (`Transcript_ 4b548465-...txt`); confirm it prints the same absolute path and - writes a valid file. -2. Diff the Python output against the existing JS output for that transcript; - expect identical content (the annotation + round-trip guarantee). -3. Spot-check a failure case (non-existent path); confirm the `Error:` line and - non-zero exit. diff --git a/tools/ESS-Diagnostics-Skills/docs/plans/2026-09-17-transcript-helper-node-to-python.md b/tools/ESS-Diagnostics-Skills/docs/plans/2026-09-17-transcript-helper-node-to-python.md deleted file mode 100644 index eb580f793..000000000 --- a/tools/ESS-Diagnostics-Skills/docs/plans/2026-09-17-transcript-helper-node-to-python.md +++ /dev/null @@ -1,278 +0,0 @@ -# Node→Python Transcript Helper Port — Implementation Plan - -> **For Claude:** REQUIRED SUB-SKILL: Use `development/reference/executing-plans-guide.md` to implement this plan task-by-task. - -**Goal:** Replace the Node.js helper `scripts/transcript-to-json.js` with a stdlib-only Python drop-in (`scripts/transcript_to_json.py`) of exact behavioral parity, update `SKILL.md` to invoke it, and delete the JS file. - -**Architecture:** A single faithful procedural port (design "Approach A"). The Python script mirrors the JS flow one-for-one — BOM strip, JSON-array validation, `_turn`/`_index` annotation as first keys, temp-dir output, absolute-path print, `Error:`/exit-1 failures. No dependencies, no packaging, no CLI framework. Validated manually by diffing its output against the existing JS output for a real transcript (no automated test, per the approved design). - -**Tech Stack:** Python 3.11+ stdlib only (`sys`, `json`, `pathlib`, `tempfile`). Repo lint: Ruff (E4/E7/E9/F). Git `core.autocrlf=true` — new text files must be committed with CRLF terminators. - -**Design doc:** `tools/ESS-Diagnostics-Skills/docs/plans/2026-09-17-transcript-helper-node-to-python-design.md` (committed as `f2feb5a`). - -**Working directory for all paths below:** repo root `Employee-Self-Service-Agent-Developer-Kit/`. - -**Note on git & line endings:** `core.autocrlf=true` with no `.gitattributes` caused a "LF would be replaced by CRLF" commit failure on a LF-only file. Before every commit of a NEW text file (`.py`, `.md`), convert to CRLF first: `sed -i 's/$/\r/' ` (idempotent-guard: only run on files freshly written LF-only). Existing files edited in place keep their endings. - ---- - -### Task 1: Create the Python helper - -**Files:** -- Create: `tools/ESS-Diagnostics-Skills/scripts/transcript_to_json.py` - -**Step 1: Write the script** - -Write exactly this content to `tools/ESS-Diagnostics-Skills/scripts/transcript_to_json.py`: - -```python -#!/usr/bin/env python3 -"""transcript_to_json.py — faithful transcript -> JSON helper for the ESS -Diagnostics skill. - -Parses a Copilot Studio / PVA transcript (JSON payload stored in a .txt) and -writes a LOSSLESS, pretty-printed JSON copy: every event and every field is -preserved exactly. The only additions are `_turn` and `_index`, annotated as -the first keys on each event (turn 0 for events before the first user message) -so a reader can navigate by conversation turn. Nothing is removed, so the -output round-trips back to the original once `_turn`/`_index` are dropped. - -This is the faithful full dump. It is NOT the compact diagnostic "normalized" -view the skill builds for the 5 checks — it is the whole transcript, just as -real JSON. - -Usage: - python transcript_to_json.py [outFile.json] - -- path to the raw transcript export (required). -- [outFile.json] where to write. If omitted, writes - `-transcript.json` into the OS temp dir under - `ess-diagnostics//` (outside any repo, because - transcripts may contain employee PII). - -Prints the absolute output path on success. Exits non-zero with an -`Error: ...` message on failure. -""" - -import json -import sys -import tempfile -from pathlib import Path - - -def fail(msg): - sys.stderr.write("Error: " + msg + "\n") - sys.exit(1) - - -def is_user_turn_start(e): - return ( - isinstance(e, dict) - and e.get("t") == "SynchronousIncomingActivity" - and isinstance(e.get("p"), dict) - and isinstance(e["p"].get("activity"), dict) - and e["p"]["activity"].get("type") == "message" - ) - - -def main(argv): - if len(argv) < 2: - fail( - "missing transcript path. Usage: " - "python transcript_to_json.py [outFile.json]" - ) - src_arg = argv[1] - src_path = Path(src_arg) - if not src_path.exists(): - fail("transcript file not found: " + src_arg) - - try: - raw = src_path.read_text(encoding="utf-8") - except OSError as e: - fail("could not read transcript: " + str(e)) - - # Strip a leading UTF-8 BOM if present, then parse. - if raw and raw[0] == "\ufeff": - raw = raw[1:] - - try: - events = json.loads(raw) - except ValueError as e: - fail("transcript is not valid JSON: " + str(e)) - - if not isinstance(events, list): - fail( - "expected the transcript to be a JSON array of events; got " - + type(events).__name__ - ) - - # Annotate each event with its conversation turn number, WITHOUT mutating any - # existing field. A turn starts at each real user message; events before the - # first user message are turn 0. `_turn`/`_index` are first keys on a shallow - # copy; every original key/value is copied through untouched. - annotated = [] - turn = 0 - for i, e in enumerate(events): - if is_user_turn_start(e): - turn += 1 - if isinstance(e, dict): - annotated.append({"_turn": turn, "_index": i, **e}) - else: - annotated.append({"_turn": turn, "_index": i, "_value": e}) - - base = src_path.name - dot = base.rfind(".") - if dot > 0: - base = base[:dot] - - if len(argv) >= 3 and argv[2]: - out_file = Path(argv[2]) - try: - out_file.resolve().parent.mkdir(parents=True, exist_ok=True) - except OSError as e: - fail( - "could not create output directory " - + str(out_file.resolve().parent) - + ": " - + str(e) - ) - else: - out_dir = Path(tempfile.gettempdir()) / "ess-diagnostics" / base - try: - out_dir.mkdir(parents=True, exist_ok=True) - except OSError as e: - fail("could not create output directory " + str(out_dir) + ": " + str(e)) - out_file = out_dir / (base + "-transcript.json") - - try: - out_file.write_text( - json.dumps(annotated, indent=2, ensure_ascii=False), encoding="utf-8" - ) - except OSError as e: - fail("could not write output: " + str(e)) - - sys.stdout.write(str(out_file.resolve()) + "\n") - - -if __name__ == "__main__": - main(sys.argv) -``` - -**Step 2: Lint the new file** - -Run: `cd tools/ESS-Diagnostics-Skills && python -m ruff check scripts/transcript_to_json.py` -(If Ruff is not installed, skip — it is a dev tool; note the skip.) -Expected: no errors (E4/E7/E9/F clean). - -**Step 3: Run against the real transcript** - -Run (from `tools/ESS-Diagnostics-Skills`): -`python scripts/transcript_to_json.py "C:\Users\rarame\Downloads\Transcript_ 4b548465-9be5-4b73-bc96-e3d6bbbe5229.txt"` -Expected: prints one absolute path ending in -`ess-diagnostics\Transcript_ 4b548465-9be5-4b73-bc96-e3d6bbbe5229\Transcript_ 4b548465-9be5-4b73-bc96-e3d6bbbe5229-transcript.json` -and exits 0. - -**Step 4: Diff Python output against the existing JS output (parity check)** - -The JS dump already exists from this session at the same temp path. Regenerate -it to a side path with the JS helper, then compare: -``` -node scripts/transcript-to-json.js "C:\Users\rarame\Downloads\Transcript_ 4b548465-9be5-4b73-bc96-e3d6bbbe5229.txt" /tmp/js-out.json -python scripts/transcript_to_json.py "C:\Users\rarame\Downloads\Transcript_ 4b548465-9be5-4b73-bc96-e3d6bbbe5229.txt" /tmp/py-out.json -diff /tmp/js-out.json /tmp/py-out.json && echo "IDENTICAL" -``` -Expected: `IDENTICAL` (no diff). If diffs appear, they will be Unicode escaping -(check `ensure_ascii=False`) or key ordering — fix before continuing. - -**Step 5: Spot-check a failure case** - -Run: `python scripts/transcript_to_json.py /nonexistent/file.txt; echo "exit=$?"` -Expected: stderr line `Error: transcript file not found: /nonexistent/file.txt` and `exit=1`. - -**Step 6: Commit** - -```bash -cd -sed -i 's/$/\r/' tools/ESS-Diagnostics-Skills/scripts/transcript_to_json.py -git add tools/ESS-Diagnostics-Skills/scripts/transcript_to_json.py -git commit -m "ESS Diagnostics: add Python transcript_to_json helper (drop-in for JS)" -``` -(Append the `Co-Authored-By: Claude ` line per session attribution.) - ---- - -### Task 2: Point SKILL.md at the Python helper - -**Files:** -- Modify: `tools/ESS-Diagnostics-Skills/SKILL.md` (Step 1 "Faithful transcript-to-JSON" block ~line 96-98; Step 4 file #3 block ~line 378-380) - -**Step 1: Replace both invocation code blocks** - -Both currently read: -``` -node scripts/transcript-to-json.js "" -``` -Change each to: -``` -python scripts/transcript_to_json.py "" -``` - -**Step 2: Add a platform note next to the Step 1 invocation** - -Immediately after the Step 1 code block (which says "run from the skill -directory `tools/ESS-Diagnostics-Skills/`"), add a sentence in the existing -Windows/POSIX style: -> On Windows use `python`; on POSIX use `python3`. - -**Step 3: Verify no stale JS references remain** - -Run: `grep -n "node \|transcript-to-json.js\|\.js" tools/ESS-Diagnostics-Skills/SKILL.md` -Expected: no matches. - -**Step 4: Commit** - -```bash -git add tools/ESS-Diagnostics-Skills/SKILL.md -git commit -m "ESS Diagnostics: SKILL.md invokes Python transcript helper" -``` -(SKILL.md is an existing file; no CRLF conversion needed — Edit preserves its endings. If a commit fails on CRLF, run `sed -i 's/$/\r/'` only on lines you added, or re-normalize the whole file with `unix2dos`.) - ---- - -### Task 3: Remove the Node.js helper - -**Files:** -- Delete: `tools/ESS-Diagnostics-Skills/scripts/transcript-to-json.js` - -**Step 1: Confirm nothing else references it** - -Run: `grep -rn "transcript-to-json" tools/ESS-Diagnostics-Skills/ --include=*.md --include=*.py --include=*.js` -Expected: no matches (SKILL.md already updated; README verified clean in design). -If any match remains, fix it before deleting. - -**Step 2: Delete the file** - -```bash -git rm tools/ESS-Diagnostics-Skills/scripts/transcript-to-json.js -``` - -**Step 3: Final sanity run** - -Run the Python helper once more against the real transcript (Task 1 Step 3) to -confirm the skill's dump path still works with the JS file gone. -Expected: same absolute path printed, exit 0. - -**Step 4: Commit** - -```bash -git commit -m "ESS Diagnostics: remove Node transcript helper (replaced by Python)" -``` - ---- - -### Task 4: Final verification - -**Step 1:** `git log --oneline -4` — confirm the four commits (design + 3 tasks) are present. -**Step 2:** `git status` — confirm clean working tree. -**Step 3:** Confirm `scripts/` now contains only `transcript_to_json.py` (no `.js`): -`ls tools/ESS-Diagnostics-Skills/scripts/` From 0ec9a3ad98ffa2049a0df54b86f98c34da8dfe9c Mon Sep 17 00:00:00 2001 From: "Ramanan Vellore Ramesh (from Dev Box)" Date: Fri, 18 Sep 2026 12:15:09 -0700 Subject: [PATCH 8/8] ESS Diagnostics: address PR #289 review comments - Make the output contract consistently say three files (Debug Report, normalized JSON, faithful transcript JSON) across SKILL.md Rules + Step 4 and the README; previously the Rules block and README said two, which would lead an LLM to skip the faithful dump Step 4 promises. - transcript_to_json.py: add the Microsoft copyright/license header; refuse to overwrite the source transcript when it is passed as the output path; and on POSIX restrict the PII-bearing temp dir/file to the owner (0700/0600). - README: clarify how the skill is invoked (point Claude at the skill dir; no wired slash command yet). - Replace a real user-alias path with a placeholder in the implementation plan. Co-Authored-By: Claude --- tools/ESS-Diagnostics-Skills/README.md | 14 ++++++++--- tools/ESS-Diagnostics-Skills/SKILL.md | 9 ++++--- ...26-09-17-ess-diagnostics-implementation.md | 8 +++--- .../scripts/transcript_to_json.py | 25 +++++++++++++++++++ 4 files changed, 44 insertions(+), 12 deletions(-) diff --git a/tools/ESS-Diagnostics-Skills/README.md b/tools/ESS-Diagnostics-Skills/README.md index 40869a9d1..0e2824c7b 100644 --- a/tools/ESS-Diagnostics-Skills/README.md +++ b/tools/ESS-Diagnostics-Skills/README.md @@ -12,7 +12,10 @@ replaces the earlier standalone Python diagnostics tool. ## How an FDE uses it -Invoke the skill (via its slash/skill mechanism). Before it parses anything, the +Invoke the skill by pointing Claude at this skill directory (there is no wired +slash command yet) — for example, ask Claude to "use the ESS diagnostics skill in +`tools/ESS-Diagnostics-Skills/` on ``", or open +`SKILL.md` and have Claude follow it. Before it parses anything, the skill **stops at a hard gate (Step 0)**: it asks the FDE to confirm the transcript **file path** and to state the **specific problem** being investigated. It will not proceed without a problem statement — even if one was @@ -55,17 +58,20 @@ transcript's basename: - **Windows:** `%TEMP%\ess-diagnostics\\` - **POSIX:** `$TMPDIR` (or `/tmp` when unset) `/ess-diagnostics//` -Two files are written there: +Three files are written there: - **`-debug-report.md`** — the human-readable Debug Report with per-turn verdicts, any FDE verdict overrides, the divergence point, and the root cause. - **`-normalized.json`** — a clean, normalized parsed transcript artifact - containing NO verdicts. + containing NO verdicts (only the fields the diagnostic checks use). +- **`-transcript.json`** — a faithful, lossless JSON copy of the whole + transcript (every event and field preserved, `_turn`/`_index` annotated), + produced by the bundled helper `scripts/transcript_to_json.py`. ## Read-only guarantee The skill never modifies the transcript or any agent files. Its only writes are -the two output files above, in the OS temp directory — never in the repo. +the three output files above, in the OS temp directory — never in the repo. ## Testing / validation diff --git a/tools/ESS-Diagnostics-Skills/SKILL.md b/tools/ESS-Diagnostics-Skills/SKILL.md index 5e2399bfa..96a4e1081 100644 --- a/tools/ESS-Diagnostics-Skills/SKILL.md +++ b/tools/ESS-Diagnostics-Skills/SKILL.md @@ -9,9 +9,10 @@ Python diagnostics tool. ## Rules -- This skill is **read-only**. The ONLY writes it may make are the two output - files: the markdown Debug Report and the normalized-transcript JSON. Never - modify the transcript, agent files, or anything else. +- This skill is **read-only**. The ONLY writes it may make are the three output + files written in Step 4: the markdown Debug Report, the normalized-transcript + JSON, and the faithful full-transcript JSON. Never modify the transcript, agent + files, or anything else. - Do NOT narrate your internal process. Say "Let me walk you through this transcript" not "Let me read the parse map." Speak directly to the FDE. - **Do not proceed past Step 0 until the FDE has stated the problem and you @@ -266,7 +267,7 @@ outside the repo and must NEVER be committed to source control or shared outside approved channels. Derive the file basename from the transcript's file name — for a transcript -`foo.txt`, the transcript name is `foo`. Write both files into a per-run +`foo.txt`, the transcript name is `foo`. Write all three files into a per-run subfolder named for the transcript: - On **Windows**: `%TEMP%\ess-diagnostics\\` diff --git a/tools/ESS-Diagnostics-Skills/docs/plans/2026-09-17-ess-diagnostics-implementation.md b/tools/ESS-Diagnostics-Skills/docs/plans/2026-09-17-ess-diagnostics-implementation.md index 60ab8df74..66931cf1c 100644 --- a/tools/ESS-Diagnostics-Skills/docs/plans/2026-09-17-ess-diagnostics-implementation.md +++ b/tools/ESS-Diagnostics-Skills/docs/plans/2026-09-17-ess-diagnostics-implementation.md @@ -124,7 +124,7 @@ rationale behind both. **Files:** none (validation only). -**Step 1:** Follow the revised SKILL.md against `C:\Users\rarame\Downloads\Transcript_a9b8b841-6a4f-4675-9e79-92fc6df74be9.txt`. Confirm: +**Step 1:** Follow the revised SKILL.md against `C:\path\to\Transcript_a9b8b841-6a4f-4675-9e79-92fc6df74be9.txt`. Confirm: - Step 0 hard gate fires (skill asks for the problem before parsing). - Parse map extracts the real events: 3 turns; intents `LlmIntentRecognized`; PluginStart queries + `rewrittenMessage`; `verifiedSearchResults` (count 10), `completionState = Answered`. - The per-turn structured pause shows intent/search/response + verdict strip + section menu; drill-down shows raw evidence + reasoning; an override is accepted and recorded; `run all` works. @@ -303,7 +303,7 @@ Expected: no errors (E4/E7/E9/F clean). **Step 3: Run against the real transcript** Run (from `tools/ESS-Diagnostics-Skills`): -`python scripts/transcript_to_json.py "C:\Users\rarame\Downloads\Transcript_ 4b548465-9be5-4b73-bc96-e3d6bbbe5229.txt"` +`python scripts/transcript_to_json.py "C:\path\to\Transcript_ 4b548465-9be5-4b73-bc96-e3d6bbbe5229.txt"` Expected: prints one absolute path ending in `ess-diagnostics\Transcript_ 4b548465-9be5-4b73-bc96-e3d6bbbe5229\Transcript_ 4b548465-9be5-4b73-bc96-e3d6bbbe5229-transcript.json` and exits 0. @@ -313,8 +313,8 @@ and exits 0. The JS dump already exists from this session at the same temp path. Regenerate it to a side path with the JS helper, then compare: ``` -node scripts/transcript-to-json.js "C:\Users\rarame\Downloads\Transcript_ 4b548465-9be5-4b73-bc96-e3d6bbbe5229.txt" /tmp/js-out.json -python scripts/transcript_to_json.py "C:\Users\rarame\Downloads\Transcript_ 4b548465-9be5-4b73-bc96-e3d6bbbe5229.txt" /tmp/py-out.json +node scripts/transcript-to-json.js "C:\path\to\Transcript_ 4b548465-9be5-4b73-bc96-e3d6bbbe5229.txt" /tmp/js-out.json +python scripts/transcript_to_json.py "C:\path\to\Transcript_ 4b548465-9be5-4b73-bc96-e3d6bbbe5229.txt" /tmp/py-out.json diff /tmp/js-out.json /tmp/py-out.json && echo "IDENTICAL" ``` Expected: no content diff. On Windows the Python output uses CRLF where JS uses diff --git a/tools/ESS-Diagnostics-Skills/scripts/transcript_to_json.py b/tools/ESS-Diagnostics-Skills/scripts/transcript_to_json.py index c927ab860..1d77fbe91 100644 --- a/tools/ESS-Diagnostics-Skills/scripts/transcript_to_json.py +++ b/tools/ESS-Diagnostics-Skills/scripts/transcript_to_json.py @@ -1,4 +1,6 @@ #!/usr/bin/env python3 +# Copyright (c) Microsoft Corporation. +# Licensed under the MIT License. """transcript_to_json.py — faithful transcript -> JSON helper for the ESS Diagnostics skill. @@ -27,6 +29,7 @@ """ import json +import os import sys import tempfile from pathlib import Path @@ -97,6 +100,7 @@ def main(argv): if dot > 0: base = base[:dot] + wrote_to_temp = False if len(argv) >= 3 and argv[2]: out_file = Path(argv[2]) try: @@ -115,6 +119,20 @@ def main(argv): except OSError as e: fail("could not create output directory " + str(out_dir) + ": " + str(e)) out_file = out_dir / (base + "-transcript.json") + wrote_to_temp = True + # Transcripts may contain employee PII and the default temp dir is + # world-readable on POSIX. Restrict the per-run folder to the owner + # (no-op on Windows, where os.chmod ignores these bits and %TEMP% is + # already per-user). + if os.name == "posix": + try: + os.chmod(out_dir, 0o700) + except OSError: + pass + + # Never overwrite the source transcript, even if it was passed as the output. + if out_file.resolve() == src_path.resolve(): + fail("output path is the same as the transcript; refusing to overwrite it") try: out_file.write_text( @@ -123,6 +141,13 @@ def main(argv): except OSError as e: fail("could not write output: " + str(e)) + # Owner-only on the PII-bearing temp output (POSIX; no-op on Windows). + if wrote_to_temp and os.name == "posix": + try: + os.chmod(out_file, 0o600) + except OSError: + pass + sys.stdout.write(str(out_file.resolve()) + "\n")