Instruction hardening engine: combine regex + semantic instruction review for /harden

solutions/ess-maker-skills/.github/prompts/harden.prompt.md

@@ -0,0 +1,21 @@
+---
+mode: agent
+description: "Type Enter to review and harden your agent's instructions against ungrounded or over-committing answers"
+---
+
+# Harden
+
+**Setup-state check.** Read `.local/config.json`. If it does not exist, OR `setup` is not `"complete"`, show:
+
+> Welcome to the ESS Maker Kit. Before running `/harden`, type `/setup` to set up your environment.
+
+and STOP. Otherwise proceed with the skill instructions below.
+
+You are helping a maker review their agent's **system instructions** — the standing guidance the agent
+follows on every turn — for internal contradictions and for the gaps that let an agent answer confidently
+from something other than its knowledge sources, or offer to do things it cannot do.
+
+Every change is **proposed, never applied silently**: the maker sees the exact before-and-after text and
+approves it.
+
+Read the skill instructions at `src/skills/instructions/harden/SKILL.md`, then follow the steps in order.

solutions/ess-maker-skills/.github/prompts/menu.prompt.md

@@ -19,6 +19,7 @@ Here's what I can help you with:
 | `/delete` | Type Enter to delete a topic or workflow from your agent |
 | `/scan` | Type Enter to scan your agent for compile errors and fix them |
 | `/review` | Type Enter to review a topic or evaluation test sets tagged for review |
+| `/harden` | Type Enter to review and harden your agent's instructions against ungrounded or over-committing answers |
 | `/test` | Type Enter to drive and debug a topic or workflow's runtime behaviour until it's right |
 | `/evaluate` | Type Enter to create, update, tag, review, run, view results, or delete evaluation test sets |
 | `/run` | Type Enter to run evaluation test sets or view run results |

solutions/ess-maker-skills/README.md

@@ -71,6 +71,18 @@ Catch and fix compile errors before they reach production. The `/scan` command a
 - Proposes fixes and applies them with your confirmation
 - Re-scans after each fix to verify resolution
 
+### 🛡️ Harden Agent Instructions
+
+Review your agent's **system instructions** — the standing guidance it follows on every turn — for the gaps that let it answer confidently from something other than its knowledge sources, or offer to do things it can't actually do. Run `/harden`.
+
+- **Asks what you've actually seen first** — paste an answer you didn't like and the review works backward from it to the instruction that permitted it
+- **Always checks for contradictions** — a rule contradicted elsewhere in the instructions isn't in force, however firmly it's written, and this runs even when the agent is behaving well
+- **Proposes line-level diffs** — exact before-and-after text with a reason, applied only after you approve; never a wholesale rewrite
+- **Won't tighten for the sake of it** — if nothing is wrong, it says so. Extra prohibitions make the agent decline questions it could have answered, which is a real regression traded for a hypothetical one
+- **Enforces the length ceiling** — instructions have a character limit and hardening only adds text, so the pass measures the result and proposes what comes out when it doesn't fit
+
+Instruction changes are behavioral changes, so `/harden` hands off to `/evaluate` and `/test` to check them — including the answers you reported, which are the only direct evidence of whether the change worked.
+
 ### 📊 Generate Evaluation Test Sets
 
 Create Copilot Studio-native evaluation sets from configured agent topics, or

solutions/ess-maker-skills/scripts/adk_telemetry.py

@@ -114,6 +114,7 @@
 #   restore_template_configs-> Workday template-config restore
 #   publishing              -> push / deploy to Copilot Studio
 #   flightcheck             -> pre-deployment readiness check
+#   harden                  -> agent system-instruction hardening review
 ADK_CAPABILITIES = (
     "setup",
     "connect",
@@ -130,6 +131,7 @@
     "restore_template_configs",
     "publishing",
     "flightcheck",
+    "harden",
 )
 _CAPABILITY_SET = frozenset(ADK_CAPABILITIES)
 

solutions/ess-maker-skills/scripts/check_instruction_budget.py

@@ -0,0 +1,187 @@
+#!/usr/bin/env python3
+"""
+check_instruction_budget.py — Deterministically measure an agent's system
+instructions against the character budget.
+
+Hardening usually lengthens instructions. Copilot Studio constrains how long an
+agent's instructions may be, so a hardening pass that does not measure will
+happily produce instructions that cannot be saved — or that get silently
+truncated, which is worse than not hardening at all (a truncated prompt can
+lose the very guardrail that was just added).
+
+Asking the agent to "count the characters" is model-dependent and was observed
+to drift. This script removes that variable: it reads the ``instructions`` block
+out of ``agent.mcs.yml``, measures it, compares it against the baseline copy so
+the maker can see what a pass *added*, and emits a machine-readable verdict the
+skill reads verbatim.
+
+The limit is a **working assumption, not a verified platform constant**. It
+defaults to 8000 and is overridable with ``--limit`` so a maker who knows their
+real ceiling is not blocked by ours.
+
+Usage (from solutions/ess-maker-skills/):
+    python scripts/check_instruction_budget.py --agent employee-self-service-hr
+    python scripts/check_instruction_budget.py --agent employee-self-service-hr --candidate .local/harden/candidate.txt
+    python scripts/check_instruction_budget.py --agent employee-self-service-hr --limit 6000
+
+Emits a human-readable summary plus a machine-readable block behind a sentinel:
+
+    ###INSTRUCTION_BUDGET_JSON###{"verdict": "ok", "chars": 5996, ...}
+"""
+
+import argparse
+import json
+import sys
+from pathlib import Path
+
+try:
+    import yaml
+except ImportError:  # pragma: no cover - environment guard
+    yaml = None
+
+if sys.stdout.encoding and sys.stdout.encoding.lower() != "utf-8":
+    sys.stdout.reconfigure(encoding="utf-8", errors="replace")
+
+_SENTINEL = "###INSTRUCTION_BUDGET_JSON###"
+
+DEFAULT_LIMIT = 8000
+
+# Headroom below which a maker should be warned that the next edit will not fit.
+# Not a failure — a "you are nearly out of room" signal, so the skill can tell
+# the maker to plan a removal before proposing another addition.
+TIGHT_HEADROOM = 250
+
+_AGENTS_DIR = Path(__file__).resolve().parent.parent / "workspace" / "agents"
+_SKILL_ROOT = Path(__file__).resolve().parent.parent
+
+
+def _resolve_agent_dir(value):
+    """Resolve --agent from any of the forms a caller reasonably has to hand.
+
+    ``.local/config.json`` stores ``agent.folder`` as a path relative to the
+    solution root (``workspace/agents/<slug>``) and ``activeAgent`` as a bare
+    slug. Accepting only one of them means whichever the caller reaches for
+    first is a coin flip, and the failure is an unhelpful "not found" against a
+    doubled-up path.
+    """
+    candidate = Path(value)
+    if candidate.is_absolute():
+        return candidate
+    relative_to_root = _SKILL_ROOT / value
+    if relative_to_root.is_dir():
+        return relative_to_root
+    return _AGENTS_DIR / value
+
+
+def _read_instructions(path):
+    """Return (instructions, error). ``instructions`` is None when unreadable.
+
+    A missing ``instructions:`` key and an empty one are different problems, so
+    they return different messages — an empty block usually means extraction
+    ran against an agent that was never configured, which is worth saying out
+    loud rather than reporting as "0 characters, plenty of headroom".
+    """
+    if not path.is_file():
+        return None, f"{path} not found"
+    if yaml is None:
+        return None, "PyYAML is not installed in this environment"
+    try:
+        data = yaml.safe_load(path.read_text(encoding="utf-8"))
+    except yaml.YAMLError as exc:
+        return None, f"{path.name} could not be parsed as YAML: {exc}"
+    if not isinstance(data, dict):
+        return None, f"{path.name} did not parse to a mapping"
+    if "instructions" not in data:
+        return None, f"{path.name} has no 'instructions' block"
+    value = data["instructions"]
+    if value is None:
+        return None, f"{path.name} has an empty 'instructions' block"
+    return str(value), None
+
+
+def _verdict(chars, limit):
+    if chars > limit:
+        return "over"
+    if limit - chars < TIGHT_HEADROOM:
+        return "tight"
+    return "ok"
+
+
+def main(argv=None):
+    parser = argparse.ArgumentParser(
+        description="Measure agent instructions against the character budget."
+    )
+    parser.add_argument("--agent", required=True,
+                        help="agent folder name under workspace/agents/, or the "
+                             "'agent.folder' path from .local/config.json")
+    parser.add_argument(
+        "--candidate",
+        help="path to a file holding proposed replacement instructions "
+             "(plain text, not YAML); measured instead of the live block",
+    )
+    parser.add_argument("--limit", type=int, default=DEFAULT_LIMIT,
+                        help=f"character ceiling (default {DEFAULT_LIMIT})")
+    args = parser.parse_args(argv)
+
+    if args.limit <= 0:
+        print("--limit must be a positive number of characters", file=sys.stderr)
+        print(_SENTINEL + json.dumps({"verdict": "unknown", "error": "invalid --limit"}))
+        return 2
+
+    agent_dir = _resolve_agent_dir(args.agent)
+    live_path = agent_dir / "agent.mcs.yml"
+    baseline_path = agent_dir / ".baseline" / "agent.mcs.yml"
+
+    result = {
+        "agent": args.agent,
+        "limit": args.limit,
+        "source": "candidate" if args.candidate else "working",
+    }
+
+    # Baseline is advisory context, never fatal: a freshly extracted agent has
+    # one, but an agent mid-edit may not, and that must not block the measure.
+    baseline_text, _ = _read_instructions(baseline_path)
+    result["baseline_chars"] = len(baseline_text) if baseline_text is not None else None
+
+    if args.candidate:
+        cand = Path(args.candidate)
+        if not cand.is_file():
+            print(f"Candidate file not found: {cand}", file=sys.stderr)
+            print(_SENTINEL + json.dumps({**result, "verdict": "unknown",
+                                          "error": "candidate not found"}))
+            return 2
+        text = cand.read_text(encoding="utf-8")
+        error = None
+    else:
+        text, error = _read_instructions(live_path)
+
+    if text is None:
+        print(f"Could not measure instructions: {error}", file=sys.stderr)
+        print(_SENTINEL + json.dumps({**result, "verdict": "unknown", "error": error}))
+        return 2
+
+    chars = len(text)
+    headroom = args.limit - chars
+    result.update({
+        "chars": chars,
+        "headroom": headroom,
+        "verdict": _verdict(chars, args.limit),
+        "delta": (chars - result["baseline_chars"])
+        if result["baseline_chars"] is not None else None,
+    })
+
+    print(f"Instructions: {chars} characters (limit {args.limit}, headroom {headroom})")
+    if result["delta"] is not None:
+        sign = "+" if result["delta"] >= 0 else ""
+        print(f"Change vs. last extract: {sign}{result['delta']} characters")
+    if result["verdict"] == "over":
+        print(f"OVER BUDGET by {-headroom} characters — this will not fit.")
+    elif result["verdict"] == "tight":
+        print(f"Within budget, but only {headroom} characters remain.")
+
+    print(_SENTINEL + json.dumps(result))
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

solutions/ess-maker-skills/scripts/instruction_engine/__main__.py

@@ -0,0 +1,97 @@
+"""CLI entry point for the instruction hardening engine.
+
+Usage:
+    python -m instruction_engine --instructions FILE [--problems FILE] [--json] [--fail-on error|warn|review]
+"""
+
+from __future__ import annotations
+
+import argparse
+import dataclasses
+import json
+import sys
+from pathlib import Path
+
+from .coverage_check import ReportedProblem
+from .engine import run
+
+_SEVERITY_RANK = {"review": 0, "warn": 1, "error": 2}
+
+
+def _load_problems(path: Path) -> list:
+    raw = json.loads(path.read_text(encoding="utf-8"))
+    return [
+        ReportedProblem(
+            customer=item["customer"],
+            description=item["description"],
+            example_prompt=item.get("example_prompt"),
+            example_response=item.get("example_response"),
+            prior_attempts=item.get("prior_attempts", []),
+        )
+        for item in raw
+    ]
+
+
+def _render_text(result) -> str:
+    lines = []
+    if not result.findings:
+        lines.append("No findings.")
+    for f in result.findings:
+        lines.append(f"[{f.severity.upper()}] {f.id} ({f.source}) — {f.section}: {f.message}")
+        if f.suggestion:
+            lines.append(f"    suggestion: {f.suggestion}")
+    if result.coverage_verdicts:
+        lines.append("")
+        lines.append("Coverage verdicts:")
+        for v in result.coverage_verdicts:
+            lines.append(f"  - {v.verdict}")
+            if v.evidence:
+                lines.append(f"    evidence: {v.evidence.id} — {v.evidence.message}")
+    return "\n".join(lines)
+
+
+def _render_json(result) -> str:
+    payload = {
+        "findings": [dataclasses.asdict(f) for f in result.findings],
+        "coverage_verdicts": [
+            {
+                "verdict": v.verdict,
+                "evidence": dataclasses.asdict(v.evidence) if v.evidence else None,
+                "diff": v.diff,
+            }
+            for v in result.coverage_verdicts
+        ],
+    }
+    return json.dumps(payload, indent=2)
+
+
+def main(argv=None) -> int:
+    parser = argparse.ArgumentParser(prog="instruction_engine")
+    parser.add_argument("--instructions", required=True, type=Path)
+    parser.add_argument("--problems", type=Path, default=None)
+    parser.add_argument("--json", action="store_true", dest="as_json")
+    parser.add_argument("--fail-on", choices=["error", "warn", "review"], default=None)
+    args = parser.parse_args(argv)
+
+    if not args.instructions.is_file():
+        print(f"error: instructions file not found: {args.instructions}", file=sys.stderr)
+        return 2
+
+    instructions_text = args.instructions.read_text(encoding="utf-8")
+    problems = _load_problems(args.problems) if args.problems else None
+
+    result = run(instructions_text, problems=problems)
+
+    output = _render_json(result) if args.as_json else _render_text(result)
+    print(output)
+
+    if args.fail_on:
+        threshold = _SEVERITY_RANK[args.fail_on]
+        if any(_SEVERITY_RANK.get(f.severity, 0) >= threshold for f in result.findings):
+            return 1
+
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())