diff --git a/docs/agenteye/alerts.mdx b/docs/agenteye/alerts.mdx index 2a2f1008..e06ee05b 100644 --- a/docs/agenteye/alerts.mdx +++ b/docs/agenteye/alerts.mdx @@ -4,328 +4,60 @@ description: "AgentEye Alerts documentation." --- -Alerts watch your agents for you and page you the moment something breaks, so you don't have to sit and refresh a dashboard. You set a rule once (an error-rate spike, a latency regression, an evaluator score drop, or any custom condition), and AgentEye checks it on a schedule and notifies you by email, Slack, webhook, or in the dashboard. Reach for alerts whenever there's a signal you'd want to hear about even when nobody's looking. +Find out the moment something crosses your line, on the channel your team already watches, instead of hearing about it from a customer. Set a rule once and AgentEye checks it on a schedule, then pages you by email, Slack, webhook, or right in the dashboard. -![The Alerts page: a grid of alert-rule cards, each showing its trigger type, evaluation window, channels, and an info/warning/critical severity badge](/agenteye/images/alerts.png) +![The Alerts page: a grid of alert-rule cards, each showing its trigger, evaluation window, channels, and an info, warning, or critical severity badge](/agenteye/images/alerts.png) +*Every alert rule at a glance: what it watches, how often, where it pages, and how urgent.* -## Create an alert +## Hear about problems before your users do -You author every alert in the dashboard. The form writes the underlying rule for you, so you never have to hand-write JSON. The happy path is five steps: +Stop refreshing a dashboard hoping to catch a regression. Reach for an alert whenever there is a signal you would want to hear about even when nobody is looking, and have it land where you already are: -1. Open **Alerts → New alert** and give it a name. -2. Pick a **trigger**: the condition to watch (see [Trigger types](#trigger-types) below). -3. Set the **threshold and window**: how bad, measured over how long. -4. Attach at least one **channel**, where the notification goes: email, Slack, webhook, or in-dashboard. -5. **Save**, then **Test** to send a synthetic notification and confirm each channel is wired up. +- **Email**, to whoever should know. +- **Slack**, a rich message with a button that jumps straight to the incident. +- **Webhook**, a JSON POST for PagerDuty, Opsgenie, or your own endpoint, with an optional signature so the receiver can trust it. +- **In-dashboard**, quiet by design, for when you are tuning a rule and do not want to page anyone yet. -That's the whole loop. Everything below is reference: what each trigger stores, how notifications are delivered, and the knobs you can tune. The JSON specs are what the form builds under the hood. They're handy for understanding a rule, but you rarely type them yourself. +Attach any combination to a single rule, and its severity (info, warning, or critical) rides along so the urgent ones look urgent. ---- - -## Concepts - -- **Alert**: the rule. It says *what* to check (the trigger), *how often* (the eval interval), *when to consider it broken* (compound logic), *how urgent* (severity), and *who/where to notify* (channels). -- **Dispatcher**: the background worker that evaluates each alert rule on its schedule and sends the notifications when a rule breaches. When this page says "the dispatcher fires," it means this worker; you never run it by hand. -- **Incident**: what happens when an alert fires. One alert has at most one open incident at a time. Repeated breaches update the same incident's evidence. Incidents are resolved by an operator; automatic resolution when the rule stops firing is planned but not currently enabled. -- **Channel**, where a notification goes: email, Slack, generic webhook, or in-dashboard. Each alert can attach any combination. - -Alerts and incidents belong to an organization and are shared across that org's operators (in a single-tenant deployment that is simply the built-in `default` org). Incidents can be assigned to an individual operator for triage. - ---- - -## Trigger types - -Five kinds, each with its own JSON spec. Pick the one that matches how you want to describe "broken". The dashboard's **new alert** form builds the same spec for you, switching the condition editor to match the trigger type you pick: - -![The new-alert form, with the basics (name, description, enabled) and the trigger picker showing metric threshold, custom SQL, evaluation score, eval failures, and per-event options](/agenteye/images/alert-new.png) - -### 1. `metric_threshold` - -The simplest. Choose a metric from a closed list, an operator, a threshold, and a time window. - -| Metric | What it counts | -|---|---| -| `event_count` | Total events in the window | -| `error_count` | `event_type = 'error'` OR any event with `error_type IS NOT NULL` | -| `error_rate` | `error_count / event_count` (0..1) | -| `p95_latency_ms` | `quantile(0.95)(duration_ms)` across all events with a `duration_ms` | -| `p99_latency_ms` | `quantile(0.99)(duration_ms)` | -| `token_sum` | `SUM(input_tokens + output_tokens)` | - -Operators: `>`, `>=`, `<`, `<=`, `==` (also accepted as `gt`, `gte`, `lt`, `lte`, `eq`). - -Optional filters: `environment`, `event_type`. - -Example spec: - -```json -{ - "metric": "p95_latency_ms", - "filter": { "environment": "production" }, - "window_secs": 900, - "op": ">", - "value": 5000 -} -``` - -### 2. `custom_sql` - -For anything the preset metrics don't cover. The operator-supplied SQL goes through the same `/queries/run` guard (SELECT/WITH only, single statement, 10 000-row cap) before the dispatcher runs it. Two modes: - -- **rows mode** (no `op`/`value`): the alert fires as soon as the query returns at least one row. -- **value mode**: the query must alias one column as `metric_value`; the dispatcher compares the first row's `metric_value` against `value` using `op`. - -```json -{ - "sql": "SELECT count() AS metric_value FROM agenteye.events WHERE event_type = 'error' AND ts >= now() - INTERVAL 1 HOUR", - "op": ">", - "value": 100 -} -``` - -### 3. `evaluation_score` - -Watches an AgentEye evaluation score: the numeric quality ratings (like `hallucination` or `helpfulness`) that your evaluator writes for each scored session. See the [Evaluation suite](/agenteye/evaluation-suite) for how those scores are produced and named. This trigger reads the stored evaluations and compares the average of one named score across a window. +## Build the rule in a form, not JSON -```json -{ - "score_key": "hallucination", - "op": ">", - "value": 0.6, - "window_secs": 3600, - "min_count": 5, - "environment": "production" -} -``` - -`min_count` guards against single-sample outliers; the dispatcher won't fire until at least N evaluations exist in the window. - -### 4. `eval_compound` - -Catch a quality regression that only shows up across *several* evaluator scores at once. Where `evaluation_score` watches a single named score, `eval_compound` composes multiple evaluation-score conditions into one alert and combines their results with a chosen logic; so one rule can express "fire if helpfulness drops **or** hallucination climbs", "fire only if helpfulness **and** tool-efficiency both drop", or "fire if **at least 2 of** these three checks breach". - -Each condition reads the average of one named score from `agenteye.evaluations` over the shared window and tests it with its own operator and threshold. The boolean results are then combined by the `combinator`: - -| Combinator | Logic | Fires when | -|---|---|---| -| `"any"` | OR | at least one condition breaches | -| `"all"` | AND | every condition breaches | -| `{ "at_least": N }` | M-of-N | at least N conditions breach | +You describe what "broken" means in a form, and AgentEye writes the underlying rule for you. The JSON spec is just what that form produces under the hood, so you can read it to understand a rule but you rarely type it. -```json -{ - "combinator": { "at_least": 2 }, - "conditions": [ - { "score_key": "hallucination", "op": ">", "value": 0.8 }, - { "score_key": "helpfulness", "op": "<", "value": 0.6 }, - { "score_key": "tone", "op": "<", "value": 0.5 } - ], - "window_secs": 3600, - "min_count": 5, - "environment": "production" -} -``` - -- `combinator`: `"any"`, `"all"`, or `{ "at_least": N }`. -- `conditions[]`: each `{ score_key, op, value }`, using the same operators as the other triggers (`>`, `>=`, `<`, `<=`, `==`). -- `window_secs`: the shared lookback applied to every condition (default `3600`). -- `min_count`: per-condition minimum number of evaluations before that condition can breach; a condition with too few samples in the window counts as "not breached" (default `1`). -- `environment`: optional; restricts every condition to one environment. - -The notification's evidence records each condition's observed average, the threshold it was tested against, how many evaluations were seen, and whether it breached; so you can see exactly which checks tripped. +![The new-alert form: name and description, an enabled toggle, and a trigger picker offering metric threshold, custom SQL, evaluation score, compound eval, and per-event conditions](/agenteye/images/alert-new.png) +*Pick a trigger and the form swaps in the right fields; Save writes the rule.* -### 5. `per_event` - -For "any event matching X has landed" alerts. No aggregation; the dispatcher fires as soon as it sees a match in the lookback window. +The happy path is quick: name it, pick a **trigger** (what to watch), set the **threshold and window** (how bad, over how long), attach at least one **channel**, then **Save** and hit **Test** to fire a synthetic notification and confirm every destination is wired up. Under the hood that produces a small spec like: ```json -{ - "event_type": "error", - "lookback_secs": 60, - "environment": "production", - "tool_name": "bash", - "agent_id": "caa", - "error_type": "TimeoutError", - "message_contains": "vertex-ai timed out" -} +{ "metric": "p95_latency_ms", "op": ">", "value": 5000, "window_secs": 900 } ``` -All filters are AND-combined; any field you leave out is unconstrained. +You are not limited to one kind of signal. Pick the trigger that matches how you think about the failure: -| Field | Purpose | +| Trigger | Fires when | |---|---| -| `agent_id` | Restrict to errors from a specific agent (the one shown on the `/errors` row). | -| `error_type` | Restrict to a specific error class (e.g. `TimeoutError`) rather than every error. | -| `message_contains` | Case-insensitive substring match against `payload.message`. Useful for catching one specific failure mode (e.g. `prompt is too long`) without alerting on every error from the same agent. Capped at 200 characters; matched as a literal string, not a pattern. | - -> **Tip:** Set `lookback_secs` to roughly match the alert's `eval_interval_secs` so you don't double-notify on the same event. - -**Shortcut from `/errors`:** each error group's representative row on the errors view (and the session event-detail panel for an error event) has a **+ alert** button that opens `/alerts/new` prefilled with a `per_event` trigger keyed to that row's `event_type` + environment, including a name seeded from the payload's `error_type` when present. You still pick channels and confirm the lookback, but the matcher is filled in for you. Operators need `alerts:write` for the button to appear. - ---- - -## Compound logic (M of N) - -"M of N" is a noise filter: it controls how many of the last few checks must fail before the alert actually pages you. Every alert has two integer knobs in addition to the trigger: - -- `eval_window`: how many recent evaluations to look at (default 1) -- `min_breaches`: how many of those must breach before the alert fires (default 1) - -`1 of 1` (the default) is "fire on the first breach". `3 of 5` means "fire when the rule has breached 3 of the last 5 evaluations", useful for jittery signals where a single bad measurement is noise. The dispatcher maintains a ring buffer per alert; you don't have to manage state. - ---- - -## Eval interval - -`eval_interval_secs` controls how often the dispatcher runs your rule. Clamped to `[30, 86400]`. Presets in the dashboard: 1m / 5m / 15m / 1h. Pick an interval matched to how fast the underlying signal moves: a 5-minute error-rate alert evaluated every 15 seconds wastes CPU; a per-event alert needs a short lookback or it'll silently drop events between ticks. - ---- - -## Channels +| **Metric threshold** | a preset metric (error rate, p95 or p99 latency, event or error counts, token spend) crosses your line over a window | +| **Custom SQL** | your own read-only query returns a row, or a value it computes crosses a threshold | +| **Evaluation score** | an evaluator score's average (say, hallucination) crosses a threshold | +| **Compound eval** | several score checks combine with any, all, or at-least-N logic, to catch a regression that only shows across scores | +| **Per event** | a single matching event lands: a specific agent, a specific error type, or a message substring | -Each alert can attach any combination of these four. Per-channel credentials (Slack webhook URL, generic webhook URL + signing secret, default email recipients) are configured once in **`/settings`** and referenced by key from each alert. This way one Slack channel can serve many alerts without each one storing its own copy of the webhook URL. For where SMTP and the Slack/webhook credentials live, see the operational-settings section of [Deployment](/agenteye/deployment#operational-settings). +Already staring at a failure on the [Errors page](/agenteye/error-tracking)? Every row there has a **+ alert** button that opens this same form prefilled to catch that exact failure again, so the incident you just triaged becomes the one that pages you next time. -The three external channel kinds (email, Slack, webhook) are also gated by an org-wide kill switch, `alerts.enabled_channels`. When a fired alert attaches a channel kind that is not in this set, the dispatcher skips it and records an `alert_notifications` row with status `skipped_disabled` and target `` (letting you globally pause, say, all Slack delivery without editing every rule). The in-dashboard channel is always allowed. See [Configuration](#configuration). +**Where to find it:** Alerts live at `//alerts`. Creating, editing, deleting, and testing rules needs **`alerts:write`**; `alerts:read` is enough to look. The recipient picker lists your org's members by name, so you can page a person without leaving the form. -### Email +## Page me only when it is real -Reuses the same SMTP transport that ships OTP login emails. Recipients resolve in order: +One bad measurement should not wake you. The **M of N** noise filter controls how many of the last few checks must fail before the alert actually pages you. Set it to **3 of 5** and the rule fires only after it has breached three of its last five checks, so a jittery signal stops crying wolf; leave it at the default **1 of 1** to fire on the first breach. You also choose how often the rule runs, from presets of 1m, 5m, 15m, and 1h, matched to how fast the signal really moves. -1. Per-channel `recipients[]` override (when non-empty). -2. The `alerts.email_default_recipients` setting (an array of email strings). +## What happens when an alert fires -If SMTP is unconfigured the channel is a no-op; the dispatcher still records an `alert_notifications` row with target `` so the audit trail makes the misconfiguration visible. - -### Slack - -Sends a Block Kit message to an [incoming webhook URL](https://api.slack.com/messaging/webhooks). - -- Default URL: `alerts.slack_default_webhook` (set in `/settings`). -- Per-alert override: set the channel's `webhook_setting_key` to any other URL-typed setting key, e.g. `alerts.slack_oncall`, `alerts.slack_costs`. - -The header includes a severity emoji (`:rotating_light:` / `:warning:` / `:bell:`), and the message carries a button that deep-links to the incident page. - -### Generic webhook - -A JSON-POST integration for PagerDuty, Opsgenie, or your own ingestion endpoint. Body shape: - -```json -{ - "schema_version": 1, - "event": "alert.firing", - "incident": { "id": "uuid", "url": "https://…/incidents/uuid", "opened_at": "2026-…" }, - "alert": { "id": "uuid", "name": "errors > 50/hr", "severity": "critical" }, - "breach": { "value": 73.0, "summary": "ErrorCount > 50 (observed 73.000)", "evidence": { … } } -} -``` - -When `alerts.webhook_signing_secret` is set, the request includes an `X-AgentEye-Signature: sha256=` header, the HMAC-SHA256 of the body using the secret. Verify it on the receiving side before trusting the payload. - -The header `X-AgentEye-Event` carries `alert.firing` / `alert.test`. (`alert.resolved` is reserved for the planned auto-resolve feature and is not currently emitted.) - -### In-dashboard - -No external delivery: the alert just writes an `alert_notifications` row that the dashboard's incidents page surfaces. Useful while you're tuning a rule and don't want to spam an external system, or for low-urgency alerts that operators check during normal triage. - ---- - -## Incident lifecycle - -```mermaid -stateDiagram-v2 - [*] --> firing - firing --> acknowledged: ack - firing --> resolved: operator resolve - acknowledged --> resolved: operator resolve - resolved --> [*] -``` - -- **firing**: the dispatcher just opened the incident or detected another breach. The firing notification is fanned out exactly once (gated by the `notified_firing_at` timestamp on the incident). -- **acknowledged**: an operator pressed *ack* on `/incidents/:id`. The incident is still considered open; subsequent breaches will update its evidence without re-notifying. -- **resolved**: an operator pressed *resolve*. Automatic resolution when the rule stops breaching is planned but not currently enabled, so an open incident stays open until an operator resolves it. - -A new incident can re-open on the same alert at any point after the previous one resolved. - -**Activity timeline.** Every action on an incident (opened, acknowledged, resolved) is recorded in an append-only activity log and shown on the incident's *activity* timeline, each entry attributed to the operator who performed it (by email) or to **automated** for actions the dispatcher took on its own (auto-open on breach). Acknowledgement is shared: several operators can ack the same incident and each appears as a separate, attributed entry. - -The **Incidents** inbox groups open incidents by state and lets you filter by severity and assignee: - -![The Incidents inbox showing alert-linked and ad-hoc incident cards with severity badges and assignees](/agenteye/images/incidents.png) - -Opening an incident shows the breach evidence, assignees and subscribers, the attributed activity timeline, and a comment thread: - -![An incident detail view: the parent alert, breach summary, assignees, subscribers, attributed activity log, and conversation](/agenteye/images/incident-detail.png) - ---- - -## Required permissions - -Authoring alert *rules* and triaging *incidents* are separate concerns with separate grants, so you can give an on-call rotation incident access without also handing it the ability to rewrite the rules. - -- `alerts:read`: view alert rules. -- `alerts:write`: create, edit, delete alert rules, and trigger a test notification. -- `incidents:read`: view incidents. -- `incidents:write`: open manual incidents, both standalone (ad-hoc) incidents and manual incidents attached to a parent alert. -- `incidents:ack`: acknowledge, assign, comment on, and resolve incidents. - -> **Note:** Keys and operators that were granted the older `alerts:ack` token keep working: it is honored as `incidents:ack` (and implies `incidents:read`), so existing on-callers retain their access without re-issuing credentials. Issue new grants with the `incidents:*` family. - -Grant these on API keys and operator accounts. See [API keys](/agenteye/api-keys) for how permissions map to keys. The dashboard's permission gate locks the relevant buttons when a permission is missing, so the action shows as denied. - -> **Note:** The alert editor's recipient picker lists your org's members so you can pick them by name. It loads for any operator holding `alerts:read` or `alerts:write`; viewing your team's directory for this purpose does **not** require `users:read`, and the picker returns only member email addresses, never full user records. - ---- - -## Configuration - -Environment variables consumed by the dispatcher: - -| Var | Default | Purpose | -|---|---|---| -| `ALERT_WORKERS` | `1` | Worker tasks per server instance. Most deployments only need one. | -| `ALERT_CLAIM_BATCH` | `16` | Max alerts a single worker tick processes. | -| `ALERT_POLL_IDLE_SECS` | `5` | Worker sleep when the queue is empty. | -| `ALERT_REQUEST_TIMEOUT_MS` | `15000` | Per-trigger evaluation timeout. | -| `DASHBOARD_URL` | `https://app.befailproof.ai` | Origin used to build the incident magic-link in notifications. Set to your dashboard host. | - -After a transient evaluation failure (ClickHouse unreachable, a query timeout), the dispatcher retries the rule with exponential backoff. Once a rule has accumulated **5** consecutive transient failures it is rescheduled at its normal cadence instead of continuing to back off, so a persistently failing rule still keeps being re-evaluated. This ceiling is fixed and is not operator-tunable. - -Channel settings (managed from `/settings`, not env): - -- `alerts.email_default_recipients` (`email_list`): JSON array of email strings, the default recipients for email channels. -- `alerts.slack_default_webhook` (`url`): default Slack incoming-webhook URL. -- `alerts.webhook_default_url` (`url`): default generic-webhook URL. -- `alerts.webhook_signing_secret` (`secret`): HMAC-SHA256 key. Always returned as `""` in the GET response; type a new value to rotate. -- `alerts.enabled_channels` (`channel_set`): the org-wide set of external channel kinds that are dispatched when an alert fires; defaults to all three (`email`, `slack`, `webhook`). Removing a kind here globally suppresses that channel for every alert without editing each rule. The in-dashboard channel is always delivered and is not affected by this setting. - ---- - -## Verify a new alert - -Before relying on a new alert: - -1. Save it as enabled with at least one notification channel. -2. Select **test** on the alert detail page and confirm each configured destination receives the synthetic notification. -3. After the first real breach, confirm the incident appears under **Incidents** and its measured value matches the corresponding dashboard query. - -Incidents do not resolve automatically when a condition clears. An operator must resolve them from the incident detail page. - ---- - -## Troubleshooting - -| Symptom | Likely cause | -|---|---| -| Alert never fires | `enabled = false`, or no channels attached, or the underlying ClickHouse query returns 0 rows. Use *test* to confirm channels; use the query runner (`/queries/run`) to confirm the metric. | -| Slack notification missing | `alerts.slack_default_webhook` (or the per-alert override key) is unset: check `alert_notifications.target` for `` rows; or the `slack` kind is globally disabled in `alerts.enabled_channels`: check for `alert_notifications` rows with status `skipped_disabled` and target ``. | -| Generic webhook 401 | The recipient is requiring a signature but `alerts.webhook_signing_secret` is unset. Verify on the recipient that the HMAC matches `hmac_sha256(secret, body)`. | -| Email "from all sends failed" | SMTP credentials wrong or the `from` address is rejected by your relay. Same surface that ships OTP emails: if those work, the SMTP transport is fine. | -| Incident reopens repeatedly | The compound knobs are too aggressive: try raising `min_breaches` or `eval_window` so transient spikes don't reopen incidents you resolved. | - ---- +A breach opens an **incident** and pages your channels once. From there your team acknowledges it, assigns an owner, talks it through, and resolves it, all against a clean, attributed record. That triage workflow has its own home: see [Incidents](/agenteye/incidents). -## Next steps +## Related -- [Evaluation suite](/agenteye/evaluation-suite): set up the evaluator scores that `evaluation_score` and `eval_compound` alerts watch. -- [API keys](/agenteye/api-keys): grant the `alerts:*` and `incidents:*` permissions an operator or key needs. -- [Deployment](/agenteye/deployment#operational-settings): configure SMTP and the Slack/webhook channel credentials each alert references. +- [Incidents](/agenteye/incidents): track a firing alert from open to acknowledged to resolved. +- [Error tracking](/agenteye/error-tracking): group agent failures and promote one to an alert in a click. +- [Dashboards](/agenteye/dashboards): watch the shared boards the thresholds you alert on come from. +- [CLI and agents](/agenteye/cli-and-agents): create alerts and ack incidents from your terminal, or script them into CI. diff --git a/docs/agenteye/assistant.mdx b/docs/agenteye/assistant.mdx index eb9ce08a..d16f4d8c 100644 --- a/docs/agenteye/assistant.mdx +++ b/docs/agenteye/assistant.mdx @@ -4,193 +4,60 @@ description: "AgentEye AI Assistant documentation." --- -The AI assistant is a chat panel built into the dashboard that answers plain-English questions about your agents ("how is quality trending in prod this week?", "which sessions errored today?", "summarize this session") and, with your approval, saves the SQL queries and dashboards it drafts for you. Every answer links straight to the sessions, queries, and dashboards it references, and it is **page-aware**: ask about "this session" while viewing one and it knows what you mean. It is safe by design: it reads only the data you can already see, every write waits for your explicit approval, and it can never delete anything. +Ask your agent data a question in plain English and get an answer that links straight to the evidence. No SQL to write, no dashboards to dig through: it is the fastest way for anyone on your team to get answers about your agents. -For example, ask "which sessions errored today?" and you get a short list with each session linked. Follow up with "save that as a query" and the assistant drafts the SQL and shows you an **Approve / Reject** card before anything is written. +![The AgentEye AI assistant answering a plain-English question inside the dashboard, showing a live Agent Activity table, a per-agent model-usage breakdown, and written takeaways, with the queries it ran shown inline](/agenteye/images/assistant.png) +*Ask in plain English and get an answer built from your own data. Here it breaks down which agents are busiest and which models they use, and shows the queries it ran so you can verify every number.* -> **Note:** The assistant is off until an operator turns it on. It runs as a separate agent service that only the dashboard can reach, and it needs an LLM endpoint configured (see [Enabling it](#enabling-it)). Until then, users with access see a muted "llm not configured" rail instead of a working assistant. +There is nothing to learn. Open the chat, type what you want to know, and follow the links it hands back: -### Using the dock - -The assistant lives in a slim rail on the right edge of every dashboard page. Click the rail (or press `⌘J` / `Ctrl+J`) to expand it into the full chat panel, and drag its left edge to resize it; your preferred width is remembered across reloads. Only users with the **`agent:use`** permission can use it; everyone else sees a greyed-out, disabled rail. - ---- - -## What it can and cannot do - -- **Reads the operational data the asking user can see.** Events, evaluations, sessions, the evaluation-job queue, saved queries, and saved dashboards, scoped per-request to the user's read permissions. Read tools execute immediately. -- **Writes are gated by per-action approval.** It can author saved queries (`create_saved_query`, `update_saved_query`), run draft SQL against the read-only role to validate it (`run_query`), and assemble dashboards from those queries (`create_dashboard`, `update_dashboard`, `add_query_to_dashboard`). Each write pauses for an in-chat **Approve / Reject / ask a question** prompt; the SDK does not call the tool until the operator clicks Approve. **Deletion is never available to the assistant**; destructive operations stay with operators. -- **Drafted SQL goes through the same `sql_guard` validation and read-only roles as user-written SQL** (SELECT/WITH only, no multi-statement). Execution is routed by which tables the query touches: queries that reference the analytics tables (events, evaluations, sessions) run as the organization's read-only ClickHouse user (scoped to that org by a row policy, with a 10s execution cap) while queries that touch only relational tables run on a read-only Postgres role (also 10s). Either way the result set is capped at **10,000 rows**; a larger result is truncated and flagged. The assistant cannot widen the data surface; it can only author over the query surface the operator already has. -- It uses a **dedicated assistant key** (see below) seeded with a fixed permission set; even if the model misbehaves it cannot exceed those scopes. -- Each dashboard user needs the **`agent:use`** permission to see and use the assistant. Tools are filtered per-request to match the user's own data permissions, so an `events:read` user gets event tools but no `dashboards:write` tools. - ---- - -## On the Queries page: write SQL by asking - -The assistant adapts to the page you are on. On most pages it is a chat panel: you ask, it answers, and any write waits for your approval. On the **Queries** page it becomes a SQL author instead. - -Type what you want in plain English ("show error rate by agent for the last 7 days", "parameterize this by date range") and the assistant streams SQL straight into the query editor. The editor opens a **diff view** (your current SQL on the left, the proposed SQL on the right) with **Accept** and **Reject** buttons. Accept keeps the change; Reject discards it. When you are editing an existing query, the assistant makes the smallest change that satisfies your request rather than rewriting from scratch. - -Each Queries page also offers suggestion chips: for example `add a status='error' filter` (edits the SQL) or `explain this query` (answers in prose). Free-typed questions like "why is this slow?" always get a prose answer. - -> **Note:** The assistant only ever writes read-only `SELECT` queries. If it cannot answer a request with a read-only analytics query (for example a request to change data, or to read tables outside the analytics views), it declines in the chat and inserts nothing. - -**Permissions.** To use the SQL author, a user needs the `queries:run` permission, the same one that enables the editor's **Run** button. Chat on every other page needs `agent:use`. Both require an LLM endpoint configured on the agent service; without one, the dock shows an "assistant isn't configured on this deployment" banner. - -**Model selection.** When `AGENTEYE_AGENT_MODELS` lists more than one model, a picker appears in the dock header and each user's choice is remembered. It applies to both chat and the SQL author, so an operator can offer a faster model for quick edits alongside a stronger one for chat. - ---- - -## Enabling it - -The `agent` service ships in the Docker Compose file and the Kubernetes manifests. To turn the assistant on, provide **(1)** an LLM endpoint and **(2)** the assistant's dedicated data key. - -### 1. Choose an LLM connection - -Pick one of these and set the corresponding variables on the `agent` service: - -**a) Anthropic directly** - -```bash -ANTHROPIC_API_KEY=sk-ant-... -``` - -**b) Through Portkey (recommended; model-catalog slug, key only)** - -```bash -PORTKEY_API_KEY= -AGENTEYE_AGENT_MODEL=@/claude-sonnet-4-6 -``` - -This is the simplest path: in Portkey, set up an **Anthropic integration** (Model -Catalog); it gets a **slug**. Name the model as `@/` and the slug -carries the provider + credential routing, so **no virtual key is needed**, -just your Portkey API key. The agent sends only `x-portkey-api-key` and points at -the Portkey gateway; Portkey resolves the rest. (A *plain* model name fails with -"x-portkey-config or x-portkey-provider header is required"; the `@slug/` prefix -is what makes key-only work.) For a self-hosted gateway set `PORTKEY_BASE_URL`. - -Prefer per-request routing instead of a slug? Set `PORTKEY_VIRTUAL_KEY=` (or -`PORTKEY_CONFIG=`) with a plain `AGENTEYE_AGENT_MODEL`. - -**c) Any other Anthropic-compatible gateway (LiteLLM, self-hosted, …)** - -```bash -ANTHROPIC_BASE_URL=https://your-gateway -# Newline-delimited "Name: Value" header lines (NOT JSON): -ANTHROPIC_CUSTOM_HEADERS=x-my-header: value ``` - -**d) Amazon Bedrock / Google Vertex** - -```bash -CLAUDE_CODE_USE_BEDROCK=1 # + standard AWS credentials in the environment -# or -CLAUDE_CODE_USE_VERTEX=1 # + standard GCP credentials in the environment -``` - -Optionally pin the default model with `AGENTEYE_AGENT_MODEL` (default `claude-sonnet-4-6`). To let -users **choose** among several models, set `AGENTEYE_AGENT_MODELS` to a comma-separated allowlist (e.g. -`@anthropic-prod/claude-opus-4-7,@anthropic-prod/claude-sonnet-4-6`); a model picker then appears in the -chat header, and each user's choice is remembered. The assistant only ever calls a model on this allowlist. - -### 2. Provide the assistant key - -Pick any random secret and give it to the **agent** as `AGENTEYE_API_KEY` and to the **server** as `AGENT_API_KEY` (the same value). On startup the server seeds it as a dedicated key named `dashboard-assistant` with this fixed permission set: `events:read`, `evaluations:read`, `dashboards:read`, `dashboards:write`, `queries:read`, `queries:write`, `queries:run`. The write permissions are only ever exercised through approval-gated tools (see "What it can and cannot do" above). There is **no manual key-minting step and no admin key involved**. The permission set is fixed in the server, and the seeded key is **protected**: it cannot be disabled or regenerated through the keys API; rotate it by changing the value and restarting the server. Do **not** reuse the admin/dashboard key. - -```bash -SECRET="$(openssl rand -base64 32)" -# on the agent service: -AGENTEYE_API_KEY="$SECRET" -# on the server service: -AGENT_API_KEY="$SECRET" +You: which sessions errored today? +AI: 5 sessions errored today, newest first. Each one is linked: + • checkout-agent 14:02 tool timeout + • billing-agent 11:47 unhandled error + • ...and 3 more + +You: summarize this session (asked while viewing a run) +AI: This run took 12 steps across 3 tools and failed near the end when a + payment tool returned an error. It scored low on your "resolved" eval. + Links: the session, the failing event, and that evaluation. ``` -On Kubernetes this is wired for you: put `AGENTEYE_API_KEY` in the `agenteye-agent` secret and the server Deployment already reads that same value as `AGENT_API_KEY`. - -### 3. Set the shared dashboard↔agent token - -Set the same `AGENTEYE_AGENT_TOKEN` on **both** the `dashboard` and `agent` services. The dashboard presents it when calling the internal agent service; the agent service rejects calls without it. - -### 4. Grant users access - -Give the relevant dashboard operators the `agent:use` permission (see [API keys](/agenteye/api-keys)). Users without it cannot use the assistant; they see a greyed-out, disabled rail. - -Once an LLM endpoint and the read-only key are set, restart the **server** (to seed the read-only key) and the **agent** service. The assistant dock appears on the right edge for any `agent:use` user, collapsed by default; click the rail or press `⌘J` / `Ctrl+J` to expand. +## Just ask, and jump straight to the proof ---- - -## Environment variable reference - -Set on the **`agent`** service: - -| Variable | Purpose | -|---|---| -| `PORTKEY_API_KEY` | Route through Portkey (the agent builds the gateway connection from this) | -| `PORTKEY_VIRTUAL_KEY` | Portkey virtual key for your Anthropic credentials (optional if the key has a default config) | -| `PORTKEY_CONFIG` / `PORTKEY_BASE_URL` | Named Portkey config / self-hosted Portkey gateway URL (optional) | -| `PORTKEY_PROVIDER` | Portkey provider slug, a third routing option alongside `PORTKEY_VIRTUAL_KEY` / `PORTKEY_CONFIG` (used only when neither of those is set) | -| `ANTHROPIC_API_KEY` | Direct Anthropic access (alternative to a gateway / Bedrock / Vertex) | -| `ANTHROPIC_AUTH_TOKEN` | Bearer token for a gateway that authenticates via `Authorization: Bearer` instead of `x-api-key` (optional) | -| `ANTHROPIC_BASE_URL` | Endpoint for a non-Portkey gateway | -| `ANTHROPIC_CUSTOM_HEADERS` | Extra headers for a non-Portkey gateway: newline-delimited `Name: Value` lines (not JSON) | -| `CLAUDE_CODE_USE_BEDROCK` / `CLAUDE_CODE_USE_VERTEX` | Route via Bedrock / Vertex | -| `AGENTEYE_AGENT_MODEL` | Default model id (default `claude-sonnet-4-6`) | -| `AGENTEYE_AGENT_MODELS` | Comma-separated allowlist of models the user can pick from in the chat header. Leave unset for a single fixed model. The default above must be one of these (else it's added). | -| `AGENTEYE_AGENT_MAX_CONCURRENCY` | Max concurrent chats per pod (default 4); excess requests get 429 | -| `AGENTEYE_API_KEY` | Assistant's data key. Set the **same** value as the server's `AGENT_API_KEY`, which seeds it with a fixed scoped permission set on startup (see step 2). | -| `AGENTEYE_AGENT_TOKEN` | Shared secret with the dashboard | -| `AGENTEYE_SERVER_URL` | AgentEye server URL (default `http://server:8080`) | -| `AGENTEYE_AGENT_ALLOW_NO_ORG` | **Multi-tenancy.** Off by default (fail-closed): the assistant rejects a `/chat` request that carries no organization context with `400`, because every tool it runs is scoped to one org. The dashboard always sends that context once it is org-aware, so you normally leave this unset. Set to `1` **only** during a transitional rollout where a not-yet-org-aware dashboard is talking to an org-aware agent, so the assistant falls back to the `default` org instead of refusing. Clear it once the dashboard upgrade lands. | -| `AGENTEYE_AGENT_MAX_STEPS` | Max tool-use steps per answer (default 8) | -| `AGENTEYE_AGENT_TIMEOUT_MS` | Overall `/chat` request timeout (all model turns + tool steps), in milliseconds (default 90000); the SQL tool has its own 10s cap | -| `AGENTEYE_AGENT_SELF_TELEMETRY` | `1` to record the assistant's own runs into AgentEye | -| `AGENTEYE_TELEMETRY_API_KEY` | Separate `events:add`-only key for self-instrumentation | -| `AGENTEYE_AGENT_ENV` | Environment tag applied to the assistant's own self-telemetry (default `prod`) | - -Set on the **`dashboard`** service: - -| Variable | Purpose | -|---|---| -| `AGENTEYE_AGENT_URL` | Where the dashboard reaches the agent service. The bundled Kubernetes manifests and Compose file set this to `http://agent:9100`. If it is unset, `agent:use` users see a muted "service offline" rail instead of a working assistant. | -| `AGENTEYE_AGENT_TOKEN` | Must match the agent's token | - ---- +You stop guessing and you stop writing queries. Ask "how is quality trending in prod this week?", "which sessions errored today?", or "summarize this session," and you get a straight answer in seconds instead of building a query and reading it yourself. -## Telemetry & seeing what users ask +Every answer comes with its receipts. The assistant links the exact sessions, saved queries, and dashboards it used to reach the answer, so you can click through and confirm rather than take its word for it. It is also **page-aware**: ask about "this session" while you are viewing one and it already knows which run you mean. Reopen any earlier conversation later from the history switcher and pick up where you left off. -Prompt **content stays inside your own systems** by default. Three layers: +## Turn a good answer into a saved query or dashboard -1. **Conversation store**: every prompt and answer is saved in your AgentEye database (per user, private), and reloadable from the assistant's history switcher. This is the durable record of what users ask. -2. **Product analytics**: the dashboard records **metadata only** (how often the assistant is used, tool counts, latency) to your analytics. Prompt **text** is never included on this path. -3. **Self-instrumentation (optional)**: set `AGENTEYE_AGENT_SELF_TELEMETRY=1` (plus an `events:add`-only `AGENTEYE_TELEMETRY_API_KEY`) and the assistant records its own runs into AgentEye as a `dashboard-assistant` agent. You then watch user prompts and the assistant's reasoning in the very same sessions/events views you use for everything else. Note: those events are visible to anyone with `events:read`; if that's too broad, leave this off. +When an answer is worth keeping, ask the assistant to save it. It drafts the SQL for a saved query, or assembles a dashboard from those queries, then shows you an **Approve / Reject** card. Nothing is written until you click Approve, so you get the speed of "just ask" with the last word always yours. ---- +On the **Queries** page it goes a step further and becomes a SQL author: describe the query you want ("show error rate by agent for the last 7 days") and it streams SQL straight into the editor, opening a diff view so you can **Accept** or **Reject** the change before it lands. -## Disabling it +![The AgentEye Queries page and its SQL editor](/agenteye/images/query-lab.png) +*The Queries page: this editor is where the assistant streams a draft, read-only query for you to accept or reject.* -Any of these makes the assistant non-functional: +Authoring SQL by asking here uses the `queries:run` permission, the same one behind the editor's **Run** button. Chat everywhere else needs `agent:use`. -- Unset `AGENTEYE_AGENT_URL` on the dashboard, **or** -- Leave the LLM endpoint unconfigured on the agent (no `ANTHROPIC_API_KEY` / gateway / Bedrock / Vertex), **or** -- Don't deploy the `agent` service at all. +## Safe to hand to the whole team -> **Note:** For an `agent:use` user the rail does not vanish; it switches to a muted, non-working state labelled "service offline" (no agent service reachable) or "llm not configured" (amber). To remove access for a specific user, take away their `agent:use` permission instead; they then see a greyed-out, disabled rail. +You can open the assistant up to everyone without worrying about what it might touch: ---- +- **It reads only what you can already see.** Answers are scoped to your own read permissions, so it never widens your data surface. +- **Every write waits for you.** Saved queries and dashboards are created only after your explicit Approve click, and there is no setting that turns that gate off. +- **It can never delete anything.** No delete tool is exposed and the assistant holds no delete permission. Deletions stay in your hands, in the dashboard. +- **It stays inside your org.** The assistant only ever sees the organization you are currently viewing. +- **Your questions stay yours.** Prompts and answers live in your own AgentEye database; product analytics records usage metadata only, never your prompt text. -## Security summary +## Where to find it -- **No silent writes**: the assistant's write tools (`create_saved_query`, `update_saved_query`, `create_dashboard`, `update_dashboard`, `add_query_to_dashboard`) cannot execute without an explicit operator click on the in-chat Approve button; the SDK's pre-call gate blocks the tool until an approval reaches the agent over a back-channel. There is no setting that disables this gate. -- **Fixed, narrow data scope**: the assistant authenticates to the server with a dedicated key whose permission set is fixed in the server (`events:read`, `evaluations:read`, `dashboards:read`, `dashboards:write`, `queries:read`, `queries:write`, `queries:run`). The only writes it can author are saved queries and dashboards; the server rejects anything outside that scope regardless of what the model attempts. -- **No deletion surface**: the key carries no delete permission and no delete tool is exposed. Operators delete through the dashboard UI, never the assistant. -- **Internal-only**: the agent has no public route; only the dashboard can call it, and only with the shared token. (In Kubernetes, a NetworkPolicy restricts the agent to reaching just the AgentEye server and the LLM endpoint.) -- **Per-user scoping**: only `agent:use` users can use the assistant (others see a disabled rail), and it is given only the tools matching each user's read permissions. -- **No raw HTML, safe links**: answers render as sanitized markdown; unsafe-scheme links (`javascript:`, `data:`) are stripped to plain text, and any external `http(s)` link opens in a new tab with `rel="noopener noreferrer"`. +The assistant rides along on the right edge of every page under your org (`//...`). Click the rail, or press `⌘J` / `Ctrl+J`, to expand it into the full chat panel, and drag its edge to resize; your width is remembered across reloads. You need the **`agent:use`** permission to use it, otherwise the rail is greyed out. If it has not been switched on for your deployment yet (it needs an LLM connection), you will see a muted rail in place of a working chat. -## Next steps +## Related -- [API keys](/agenteye/api-keys): grant users the `agent:use` permission so they can use the assistant. -- [Deployment](/agenteye/deployment): wire the agent service into your stack. -- [Troubleshooting](/agenteye/troubleshooting): common assistant issues and fixes. +- [CLI and agents](/agenteye/cli-and-agents) +- [Queries](/agenteye/queries) +- [Dashboards](/agenteye/dashboards) +- [Evaluation suite](/agenteye/evaluation-suite) diff --git a/docs/agenteye/audits.mdx b/docs/agenteye/audits.mdx index edc06e00..ad334691 100644 --- a/docs/agenteye/audits.mdx +++ b/docs/agenteye/audits.mdx @@ -1,116 +1,52 @@ --- -title: "Audits: agentic improvement detection" -description: "AgentEye Audits: agentic improvement detection documentation." +title: "Audits: your automatic reliability analyst" +description: "AgentEye Audits: your automatic reliability analyst documentation." --- -Audits tell you what to fix in your AI agents. An audit is a scheduled job that reads across all your agent sessions, hunts for patterns worth improving (a leaked API key, a silently failing tool, a prompt that is quietly degrading) and writes up concrete recommendations with the evidence behind each one. Use audits to answer "what should I improve?"; use alerts to be paged the moment a threshold you already know about is crossed. +AgentEye goes looking for the failures you never wrote a rule for and hands you a ranked, evidence-backed to-do list of exactly what to fix. It is like having an analyst comb your logs every night, then leaving the short list on your desk by morning. -Every recommendation links to the exact sessions and queries behind it, and a one-click shortcut drafts a recurrence alert so you catch the issue if it comes back. You will find audits in the dashboard at **`//audits`** (sidebar → *analyze* → *audits*), gated by the `audits:read` / `audits:write` permissions. +
+ +
-[![Watch: Automatically find where your agent is failing (2 min)](/agenteye/images/video-audit.jpg)](https://youtu.be/C5nHKGQIqZQ) +*A two-minute tour: from a scheduled run to a fix you can act on.* -*A two-minute tour of automatic failure audits: from a scheduled run to a fix you can act on.* +![The Audits page: recurring jobs that scan your sessions for failure patterns, each with a schedule and sensitivity](/agenteye/images/audits.png) +*Each audit is a recurring job that mines your sessions and writes up ranked, evidence-backed recommendations.* -![The Audits page: recurring jobs that mine your logs for failure patterns, each with a schedule and analysis mode](/agenteye/images/audits.png) +## Stop guessing what to fix next ---- - -## How a run works - -Each run has two layers: a deterministic floor and an agentic investigation. - -### 1. The policy pass (deterministic) - -Before any model runs, the audit executes a small catalog of **SQL policy checks** over the window: bounded aggregate queries that flag known-bad patterns and report *how many* events / *which* sessions matched, never the matched text itself. The catalog includes: - -- **Secret / credential leakage** in event payloads: AWS access keys, `sk-…` API keys, PEM private keys, JWT / bearer tokens, and `KEY=…` credential assignments. -- **Prompt-injection markers**: "ignore previous instructions", "reveal your system prompt", and similar. -- **PII**: SSN-shaped numbers (heuristic). -- **Tool-permission denials** and **runaway tool-call loops**. - -Policy hits are persisted as findings (kind `policy`) that **always surface** (they are never trimmed by the per-run cap), and they are handed to the AI agent as starting leads. Because this layer needs no model, an audit still produces its most important security signals even if the AI agent is unavailable. - -### 2. The agentic investigation (AI) - -The audit then runs an **autonomous reliability agent**: the same agent service that powers the AI assistant, running an audit-specific prompt. Given the audit's **scope** (selected agents × environments) and **time window**, the agent: - -- runs read-only SQL queries against your analytics tables, -- reads a handful of representative session transcripts, -- optionally writes and runs short **Python scripts in a locked-down in-pod sandbox** (no network, no filesystem access, secrets scrubbed) for analysis SQL can't express: clustering errors, computing distributions, sweeping payloads it already fetched, -- and records each well-evidenced **improvement** it finds. - -It works through several investigative lines (error clustering, drift vs. a baseline, goal failure in transcripts, tool misuse, quality/cost trade-offs, and coverage gaps) at the audit's **sensitivity** (low / medium / high). Every improvement **must cite evidence**: the session ids the agent actually inspected and/or the SQL it ran. The server validates that cited sessions exist and **discards any improvement with no surviving evidence**, so the agent investigates but never invents. - -Each improvement carries: - -- a **recommendation** (the concrete change to make: a prompt tweak, a tool-schema fix, a retry policy, a guardrail, more eval coverage), -- an **expected impact** and an **effort** estimate (low / medium / high), -- a **magnitude** (how loudly it should surface): `big` (an operator should be paged), `medium` (belongs in the run report), or `small` (dashboard context), -- a stable **fingerprint**: an identity derived from the issue's category and scope, *not* this run's sessions, so the same issue is tracked run-over-run even as the evidence changes, -- and, where a simple watcher could catch the issue coming back, a **suggested alert**: a read-only recommendation shown on the run page as a "graduation candidate" for what to watch. (Separately, every finding page has a one-click shortcut that drafts a recurrence alert prefilled with a sensible starting trigger, which you then tune.) - -> **Note:** The AI layer is optional but recommended. If no AI agent is configured for the audit pipeline, runs still execute, persist the policy findings, and honestly report "analysis unavailable" for the agentic layer rather than silently passing. - -### Failure modes - -Improvements classify into your org's durable **failure-mode catalog** (the running list of distinct ways your agents fail) or propose a new mode. Modes give patterns a stable identity across runs and long-horizon recurrence tracking. - -## Triage lifecycle - -On a finding page (`/audits//findings/`): - -| Action | Effect | -|---|---| -| **acknowledge** | Keeps the finding visible but halves its priority. | -| **resolve** | Marks it fixed. If the pattern genuinely returns later, it re-opens as **new**, so a regression is loud, not silently folded into history. | -| **mute** / **dismiss** | Durable suppression: the pattern's fingerprint is remembered and never surfaces again, even across runs. Use mute for "known, accepted"; dismiss for "not useful". | -| **reopen** | Clears suppression / resolution and ranks the pattern again. | - -Low-signal noise is controlled per audit with a per-run findings cap (`top_k`) on the agentic improvements. Policy findings bypass the cap (they are security-relevant and always shown). Anything cut by the cap is counted in the run's stats; nothing is silently dropped. - -## Scheduling - -- **Cadence** (`schedule_interval_secs`): hourly to weekly; **daily is the default**. Audits are deliberately coarser than alerts: an agentic investigation scans whole windows and runs for minutes. -- **Window**: either a fixed rolling lookback (e.g. "each run scans the last 7 days") or **since-last-run** (the default): each run picks up where the previous successful one ended, with a small overlap so boundary events are never missed. -- The next run is scheduled a full interval after the previous one **completes**, so a slow run never stacks a second concurrent run of the same audit. -- **Run now** on the audit page makes it due immediately. - -## Model selection - -When creating an audit you can pick which model the investigation uses, from the **list of models your operator has configured** for the agent service. With a single model configured, the picker shows it as a caption; with several, you choose. Leaving it unset uses the configured default. +Alerts catch the problems you already know to watch for. Audits catch the ones you don't. On a schedule you set, an audit reads across all of your agent sessions and hunts for the patterns worth fixing, so you spend your time acting on findings instead of scrolling logs hoping to spot them yourself. -## Notifications +A single run goes after the failure modes that actually break agents in production: -When a run surfaces **new** findings, the audit notifies your organization's configured channels, the same `alerts.enabled_channels` gate and settings the alerts pipeline uses: +- **Error clusters**: the same failure repeating under a shared root cause. +- **Drift versus a baseline**: behaviour quietly sliding away from a known-good window. +- **Goal failure in transcripts**: runs that technically finished but never did the job. +- **Tool misuse**: the wrong tool, bad arguments, or loops that burn calls. +- **Quality and cost trade-offs**: where you are overpaying for output you could get cheaper. +- **Coverage gaps**: behaviour that no eval or alert is watching. -- **Slack**: a summary of the significant (`big`) new items with a deep link. -- **Email**: a designed **audit report** listing the new improvements (top severity, per-item recommendations, deep link), sent when the audit has an **email** channel attached and there is at least one new finding. +You decide how hard it looks with a single **sensitivity** setting (low, medium, or high), so a noisy staging agent and a locked-down production one can each be tuned to the signal you want. -Recurring-but-known findings do not re-notify. +## Every recommendation comes with receipts -## Configuration reference +You never have to take a finding on faith. Each recommendation cites the exact sessions it came from and the SQL that surfaced it, so you can open the evidence and confirm the problem in a click instead of reverse-engineering a claim. -Audit definitions are managed in the dashboard (`/audits/new`) or via the API. Per-audit settings include the schedule cadence and window, the scope (`{"environments": [...], "agent_ids": [...]}`), the sensitivity (`low` / `medium` / `high`), the notification channels, the per-run findings cap (`top_k`), and the model (via `llm_budget.model`). Operator-level server settings (timeouts, sandbox, the agent-service URL) are documented in [Deployment](/agenteye/deployment). +That is also what keeps audits honest. The server checks that every cited session actually exists and **discards any recommendation whose evidence does not hold up**, so the audit investigates but never invents. What lands on your list is real, reproducible, and ranked by how much it matters, with the biggest wins at the top. -## API +## Turn a fix into a guardrail -All endpoints are org-scoped and follow the standard bearer-key auth (see [API keys](/agenteye/api-keys)). +Fixing an issue is only half the win. The other half is making sure it cannot quietly come back. Every finding carries a **one-click shortcut that drafts a recurrence alert**, prefilled with a sensible starting trigger you can tune. Close the finding, arm the alert, and the next time that pattern reappears you get paged instead of rediscovering it in a future audit. -| Endpoint | Permission | Purpose | -|---|---|---| -| `GET /audits` · `POST /audits` | `audits:read` / `audits:write` | List / create audit definitions. | -| `GET` / `PUT` / `DELETE /audits/:id` | read / write / write | Inspect, edit, delete an audit. | -| `POST /audits/:id/run` | `audits:write` | Make the audit due immediately. | -| `GET /audits/:id/runs` | `audits:read` | Run history (window, status, stats, finding counts). | -| `GET /audits/findings` | `audits:read` | Org-wide findings, filterable by `audit_id`, `status`; sorted by priority. | -| `GET /audits/findings/:fid` | `audits:read` | Full finding detail (recommendation, evidence, priority). | -| `POST /audits/findings/:fid/status` | `audits:write` | Triage: `{"action": "ack" \| "mute" \| "dismiss" \| "resolve" \| "reopen" \| "assign"}`. | +## Where to find it -For "audit ran but found nothing", "the code sandbox is disabled", and "audit email not delivered", see [Troubleshooting](/agenteye/troubleshooting#audits). +Audits live in the dashboard at **`//audits`** (sidebar to *analyze* to *audits*). Viewing runs and findings needs **`audits:read`**; creating, editing, and triaging audits needs **`audits:write`**. Set an audit's scope and cadence, then hit **Run now** whenever you want results immediately instead of waiting for the next scheduled pass. -## Next steps +## Related -- [Alerts](/agenteye/alerts): get paged the moment a specific threshold you already know about is crossed. -- [Evaluation suite](/agenteye/evaluation-suite): score every completed run so quality regressions surface on their own. -- [AI assistant](/agenteye/assistant): ask questions about your agents in natural language and dig into any finding. +- [Alerts](/agenteye/alerts): get paged the moment a threshold you already know about is crossed. +- [Evaluations](/agenteye/evaluations): score every run so quality regressions surface on their own. +- [Error tracking](/agenteye/error-tracking): group and follow the errors your agents throw. +- [Incidents](/agenteye/incidents): track an issue an audit turns up through to its fix. diff --git a/docs/agenteye/cli-and-agents.mdx b/docs/agenteye/cli-and-agents.mdx new file mode 100644 index 00000000..430a778f --- /dev/null +++ b/docs/agenteye/cli-and-agents.mdx @@ -0,0 +1,80 @@ +--- +title: "CLI" +description: "AgentEye CLI documentation." +--- + + +Your entire AgentEye deployment, one command away. Check production, cut an API key, or ack an incident without leaving your terminal, then script any of it into CI, or let a coding agent do it for you in plain English. + +```bash +pipx install agenteye +agenteye login --email you@example.com # a 6-digit code lands in your inbox +agenteye --json sessions --since 24h # every agent run from the last day, newest first +``` + +*The `agenteye` CLI talks to your dashboard. It is a different tool from the collector, which ships events to the server.* + +## Your whole deployment, one command away + +Stop tab-hopping to answer a quick question. The `agenteye` CLI reads your data and administers your org from a single binary, so a check that used to mean clicking through the dashboard becomes one line you can rerun, alias, or paste into a runbook. You get four surfaces: + +- **Read your data:** `sessions`, `events`, `evals`, and `errors`, filtered by time, agent, and environment. +- **Manage your org:** `keys`, `users`, `settings`, `alerts`, and `incidents`. +- **Run analytics:** saved SQL plus an ad-hoc `query` runner over your event data. +- **Ask the assistant:** `agent ask` reaches the same read-only analyst you chat with in the dashboard. + +Install it once with `pipx`, sign in with an emailed 6-digit code, and you are ready. The session lasts about a day; rerun `agenteye login` when it expires. Reach for it to spot-check production, provision a key, or triage a firing incident, all without opening a browser: + +```bash +agenteye errors --since 24h --aggregate # what is breaking, grouped by error type +agenteye incidents list --state firing # what is on fire right now +agenteye keys create ci --add events:add # a key that can only push events, secret shown once +``` + +One habit to know: global options like `--json` go before the command. `agenteye --json sessions` is right; `agenteye sessions --json` is not. + +## Script it, wire it into CI + +Every command takes `--json`, and that changes everything. Clean JSON goes to stdout while human status and warnings go to stderr, so a `--json` capture pipes straight into `jq` with no stray line to strip. That is what makes the CLI equally good for you at a prompt and for a coding agent parsing output: + +```bash +agenteye --json sessions --since 24h --status error | jq -r '.sessions[].session_id' +``` + +It is built to run unattended. Confirmation prompts auto-skip when no terminal is attached, so nothing hangs in a pipeline, and every command returns a meaningful exit code: `0` success, `4` not logged in, `5` missing a permission (the message names it, for example `alerts:write`), `3` dashboard unreachable. A script can branch on a `4` to reauthenticate or a `5` to tell you exactly what to ask an admin for, instead of failing blind. + +## Let a coding agent drive it in plain English + +Better yet, you should not have to remember any of these flags at all. The **CLI skill** is a small Agent Skill folder named `agenteye-cli` that teaches a coding agent such as Claude Code or Codex to drive the CLI from plain-English requests. Ask "is anything broken today?" and the agent picks the command, runs it as you, and answers in prose. + +For Claude Code, drop the `agenteye-cli` folder into `~/.claude/skills/` and it is auto-discovered. AgentEye provides the folder; there is nothing to deploy and no image to pull, because it only drives the CLI you already installed. Log in yourself first: the skill cannot complete the emailed-code login for you. + +Because the agent runs the CLI as you, it can do everything your login permits, reads and writes alike: create keys, change settings, resolve incidents. The CLI's "are you sure?" prompt does not fire for an agent, so the skill is written to state the exact command and wait for your OK before any change. You are the confirmation step. + +```text +you Why did session run-001 fail? + +agent Running: agenteye --json events --session-id run-001 --all + The checkout-agent hit a TimeoutError on its third tool call. +``` + +Reads stay instant, and every write pauses for you: + +```text +you Give CI a key that can only push events. + +agent This creates an API key. I'll run: + agenteye keys create ci --add events:add + OK to proceed? + +you yes + +agent Done. Key "ci" created with events:add only. The secret is shown once, so store it now. +``` + +## Related + +- [CLI reference](/agenteye/cli): every command, flag, and JSON shape. +- [CLI recipes for agents](/agenteye/cli-recipes): copy-paste `jq` patterns and exit-code handling. +- [CLI agent skill](/agenteye/cli-skill): install and run the `agenteye-cli` skill. +- [AI assistant](/agenteye/assistant): the in-dashboard analyst that `agent ask` talks to. diff --git a/docs/agenteye/cli-recipes.mdx b/docs/agenteye/cli-recipes.mdx index 8b71ec16..57a00d71 100644 --- a/docs/agenteye/cli-recipes.mdx +++ b/docs/agenteye/cli-recipes.mdx @@ -111,7 +111,7 @@ A multi-org login without `--org` exits non-zero and prints the orgs to choose f ## Provision an API key for the SDK/collector ```bash -# the secret is printed ONCE — with --json it's the .key field +# the secret is printed ONCE, with --json it's the .key field key=$(agenteye --json keys create ci-bot --add events:read.add | jq -r '.key') agenteye keys regenerate ci-bot --yes # rotate; agenteye keys disable ci-bot --yes to revoke ``` diff --git a/docs/agenteye/cli-skill.mdx b/docs/agenteye/cli-skill.mdx index 54be1217..71aaa670 100644 --- a/docs/agenteye/cli-skill.mdx +++ b/docs/agenteye/cli-skill.mdx @@ -102,7 +102,7 @@ agent ▸ This creates an API key. I'll run: you ▸ yes agent ▸ Done. Key "ci" created with events:add only. - The secret is shown only once — store it now, I can't reprint it. + The secret is shown only once, so store it now. I can't reprint it. ``` The skill maps each plain-English intent to the right `agenteye` command, discovering valid values first (`list `, `whoami`) so it doesn't guess, and stating the exact command before any change. More examples: diff --git a/docs/agenteye/cli.mdx b/docs/agenteye/cli.mdx index 6cbd8bf1..392656e3 100644 --- a/docs/agenteye/cli.mdx +++ b/docs/agenteye/cli.mdx @@ -212,23 +212,23 @@ agenteye orgs perms # your permissions in the active org, grouped by resou None of these need a confirmation. Shared filters: `--session-id`, `--agent-id`, `--env` (**not** `--environment`), and the time range (`--since` / `--from` / `--to`). ```bash -# events (alias: the raw per-step trail) — newest first +# events (alias: the raw per-step trail), newest first agenteye --json events --session-id run-001 --event-type tool_use,tool_result --all --limit 1000 agenteye --json events --since 1h --search timeout --all | jq '.events[].payload' -# sessions — one row per agent run (time/env/agent/session/status; no score filtering) +# sessions: one row per agent run (time/env/agent/session/status; no score filtering) agenteye --json sessions --since 24h --status error agenteye --json sessions --agent-id checkout-bot --env prod --all --limit 1000 -# evals — evaluation results + scores; --score filters by metric, --aggregate rolls up +# evals: evaluation results + scores; --score filters by metric, --aggregate rolls up agenteye --json evals --score helpfulness:0.5..0.8 --score tool_efficiency:..0.3 agenteye --json evals --aggregate --since 7d --env prod # status mix + per-key score stats -# errors — errored events; --aggregate for counts/sessions/agents/last-seen +# errors: errored events; --aggregate for counts/sessions/agents/last-seen agenteye --json errors --since 24h --aggregate agenteye --json errors --since 24h --error-type timeout --all --limit 1000 -# list — discover valid filter values before you filter +# list: discover valid filter values before you filter agenteye list envs # also: agents event_types score_filters models hooks tools error_types ``` diff --git a/docs/agenteye/dashboards.mdx b/docs/agenteye/dashboards.mdx new file mode 100644 index 00000000..c07e6dbb --- /dev/null +++ b/docs/agenteye/dashboards.mdx @@ -0,0 +1,46 @@ +--- +title: "Dashboards" +description: "AgentEye Dashboards documentation." +--- + + +Turn your live agent data into one shared picture your whole team watches. Pin the queries that matter as charts, and everyone opens the same numbers at a glance, without re-running a single query. + +![A dashboard built from saved queries: an events-per-hour line, an errors-by-type bar, a latency area chart, and tokens-by-model](/agenteye/images/dashboard-fleet.png) + +*One board, four saved queries: events per hour, errors by type, latency, and tokens by model.* + +## Everyone sees the same truth + +Stop pasting screenshots into chat and stop re-running the same query five times a day. A dashboard is a shared, org-wide board anyone on your team can open to the exact same view. When the underlying data moves, the charts move with it, so the board is always current and nobody is arguing over stale numbers. + +The fleet dashboard above is a good starting shape for day-to-day operations: + +- an **events-per-hour** line, so you can watch throughput and catch a sudden drop +- an **errors-by-type** bar, so your biggest failure categories jump out +- a **latency** area chart, so slow-downs show up before users complain +- a **tokens-by-model** breakdown, so cost stays in view + +You'll find your boards at `//dashboards`. + +## Pin the queries you already saved + +Every tile starts as a saved query. Build and save the query you care about in the [Queries](/agenteye/queries) library (built-in presets plus your own, over your events and evaluations), then pin it to a dashboard as the chart that fits the data: a **line** for trends over time, a **bar** for comparing categories, an **area** for volume, or a **pie** for a share breakdown. + +Because a tile is just your saved query rendered as a chart, there's nothing to keep in sync by hand. Update the query once and every dashboard that uses it updates too. + +## Watch quality, not just volume + +Volume tells you the agents are busy. Quality tells you they're actually doing the job. Point a dashboard at your [evaluation scores](/agenteye/evaluations) and you get a board that tracks how well runs are going over time, so a quality regression shows up as a dip on a chart instead of a surprise from a customer. + +![A quality-focused dashboard built from saved evaluation queries](/agenteye/images/dashboard-quality.png) + +*A quality board keeps your evaluation scores front and center, right beside the operational numbers.* + +Keep an operations board and a quality board side by side and your team has one place to answer both "is it working?" and "is it good?", without anyone re-running a query. + +## Related + +- [Queries](/agenteye/queries): build and save the queries that become your tiles. +- [Evaluations](/agenteye/evaluations): score your runs so you can chart quality over time. +- [Alerts](/agenteye/alerts): turn a threshold on any of these metrics into a page. diff --git a/docs/agenteye/error-tracking.mdx b/docs/agenteye/error-tracking.mdx new file mode 100644 index 00000000..e2d51cfd --- /dev/null +++ b/docs/agenteye/error-tracking.mdx @@ -0,0 +1,41 @@ +--- +title: "Error Tracking" +description: "AgentEye Error Tracking documentation." +--- + + +See every failure your agents produce in one place, grouped so a noisy burst reads as a single problem. You get a one-click path from "something is red" to the exact run that broke, without scrolling a live feed to find it. + +![The Errors page: a histogram of failures over time above grouped red error rows, each with a one-click "+ alert" button](/agenteye/images/errors.png) +*The Errors page: a histogram of failures over time, with repeat failures collapsed into one row per incident.* + +## Every failure, already collected for you + +When an agent breaks, you should not have to scroll a live event stream hoping to catch the red rows before they scroll away. The **Errors** page does the collecting for you. It pulls together everything the dashboard would paint red into one triage surface, so the first thing you see is what is failing, not where to go looking for it. + +And it catches more than the obvious ones. Alongside explicit `error` events, AgentEye surfaces the quiet failures too: any `tool_result`, `hook_completed`, or `agent_end` whose payload carries a failure shows up here. A tool that returned an error, or a hook that exited badly, no longer slips past you just because nothing threw a loud exception. + +Across the top, a histogram plots errors over time. One look tells you whether this is a steady background trickle or a spike that started a few minutes ago, so you know right away whether to drop what you are doing. + +Like every observe surface, the Errors page is scoped to your organization and filters by date range, environment, agent, and session. That means you can take a fleet-wide list and narrow it to the one agent or one environment you actually care about. + +## One incident, not a hundred identical rows + +A single broken dependency can fire the same error hundreds of times a minute. Left raw, that is a wall of near-identical lines that buries the one thing you actually need to see. + +AgentEye collapses repeat failures that share the same session and error type into a single row. A burst reads as one incident. You end up counting problems, not log lines, and the signal that matters stays on top instead of being drowned out by its own volume. + +## From "something is red" to the exact event + +Click any row to land straight inside that run's session, positioned on the exact event that failed. No copying session IDs, no scrolling to hunt for the moment it went wrong: you arrive right on it, with the full execution graph one glance away so you can see what the agent did in the moments before it broke. + +If you have `alerts:write`, every row also carries a **+ alert** button. Click it and AgentEye opens a new alert rule already filled in to catch that same failure again. The incident you just triaged becomes the one that pages you next time, instead of surprising you twice. + +**Where to find it:** the **Errors** page lives in the observe section of the dashboard, at `//errors`. + +## Related + +- [Alerts](/agenteye/alerts): turn any failure into a paging rule. +- [Incidents](/agenteye/incidents): track a firing alert from open to resolved. +- [Sessions](/agenteye/sessions): open the full run behind any error. +- [Audits](/agenteye/audits): let AgentEye find failure patterns across your runs for you. diff --git a/docs/agenteye/evaluations.mdx b/docs/agenteye/evaluations.mdx new file mode 100644 index 00000000..5589adc9 --- /dev/null +++ b/docs/agenteye/evaluations.mdx @@ -0,0 +1,48 @@ +--- +title: "Evaluations" +description: "AgentEye Evaluations documentation." +--- + + +Quality problems find you now, instead of you hearing about them in a user complaint. Connect your own scoring service once and AgentEye grades every finished run automatically, so a drop in helpfulness or a spike in hallucinations shows up on its own, before a customer feels it. + +![The Sessions grid with a score column: each run carries an evaluation status pill and colour-coded helpfulness, factuality, and tool-efficiency badges](/agenteye/images/sessions-list.png) + +*Every run on the sessions grid carries its scores; red, amber, and green badges make the weak runs jump out without you opening a single transcript.* + +## Stop sampling runs by hand + +You used to spot-check a handful of runs and hope the rest were fine. Now every completed session is scored the moment it finishes, on the dimensions you care about: helpfulness, tool efficiency, factuality, safety, whatever your quality bar is. You define the score keys; AgentEye stores, trends, and displays whatever your evaluator sends back. No run slips through unscored, and you stop learning about a regression from a support ticket. + +The scores ride along on the sessions grid at **`//sessions`** (sidebar → *observe* → *sessions*), one badge cluster per row. Want just the runs that fell short? Filter the grid by score range, say helpfulness below 0.5, and pull up exactly the runs worth reading. Viewing scores needs the `evaluations:read` permission. + +## See why a run scored low + +A number tells you a run was weak; the session page tells you why. Open any run and the right rail leads with the headline summary, then shows a bar per dimension with your evaluator's own reasoning under each one, so you go from "this scored 0.4 on factuality" to the exact claim it got wrong in seconds. + +![A session's right rail: the evaluation summary on top, then per-dimension score bars each with a line of reasoning, beside the full event timeline](/agenteye/images/session-detail.png) + +*The session detail view: summary, per-dimension score bars, and the reasoning behind each score, right next to the run's event timeline.* + +Shipped a sharper evaluator, or looking at a run that crashed before it could be scored? A **re-evaluate** button (gated by `evaluations:trigger`) re-scores the session in place and appends the fresh result to its timeline, so earlier scores stay visible as history. You will find it at **`//sessions/`**. + +## Watch quality trend across the fleet + +One run scoring low is noise; a whole cohort sliding is a signal. Saved dashboards turn your scores into a trend you can watch at a glance: average helpfulness this week against last, per agent, per environment. + +![A quality dashboard: average-score bars per evaluator dimension alongside a trend over time](/agenteye/images/dashboard-quality.png) + +*A saved quality dashboard trends the score keys you feature, so a slow drift is obvious long before it becomes an incident.* + +Dashboards live at **`//dashboards`** (sidebar → *analyze* → *dashboards*), are shared across your whole organization, and each card rolls up the matching sessions: how many, the average of each featured score, and a trend sparkline. "Open in sessions" drops you straight into the pre-filtered runs behind any number. Viewing needs `dashboards:read` plus `evaluations:read`. + +## Connect an evaluator once + +Scoring is opt-in and stays completely off until you point AgentEye at a scorer. You stand up one small HTTP service (AgentEye ships a working reference you can copy), set two values on your server, and every run from then on is scored for you. The full walkthrough, the scoring contract, and the SDK live in the deep guide. + +## Related + +- [Evaluation suite](/agenteye/evaluation-suite): connect your evaluator, the scoring contract, and the SDK. +- [Sessions](/agenteye/sessions): the run-by-run grid where scores appear. +- [Dashboards](/agenteye/dashboards): save and share quality trends across your org. +- [Audits](/agenteye/audits): AgentEye's other automatic quality feature, for cross-session investigations. diff --git a/docs/agenteye/event-stream.mdx b/docs/agenteye/event-stream.mdx new file mode 100644 index 00000000..72965d8d --- /dev/null +++ b/docs/agenteye/event-stream.mdx @@ -0,0 +1,50 @@ +--- +title: "Event Stream" +description: "AgentEye Event Stream documentation." +--- + + +The moment your agent does something, you see it. The Event Stream is your live pulse on every agent in production: no waiting, no grepping logs, no guessing what just happened. + +![The live Event Stream: colour-coded event rows tailing in real time, filterable by environment, agent, session, event type, and free text](/agenteye/images/events-stream.png) + +*Every event from every agent in your org, newest first, updating as it happens.* + +## Your live pulse on every agent + +When an agent starts a run, calls a model, fires a tool, runs a hook, or hits an error, the row appears at the top of the stream the moment it happens. It tails every event across every agent in your organization, newest first, so you always have a current picture instead of a stale one. + +That means no tailing log files on a box somewhere, no grepping across machines, no stitching timestamps together by hand. You open one page and you are already watching production. + +Rows are colour-coded by type, so you can read the stream at a glance instead of parsing every line. At a glance, each row shows you: + +- **Its type**, colour-coded: `agent_start`, `model_response`, `tool_use`, `hook_completed`, `error`, and more. +- **A one-line summary** of what happened, so you rarely need to open anything just to get the gist. +- **Token counts** for the step. +- **A context-window fill badge** where it applies, so prompt growth and an approaching compaction are visible before they bite. + +Watching it live means you catch a bad deploy, a runaway loop, or a burst of errors as it happens, not in tomorrow's log review. + +## Find the one run that matters + +When something looks off, you don't want the firehose. You want the single run that broke. The stream filters down fast: by environment, by agent, by session, by event type, or by free text. + +Filter by session id or agent id to follow one run from its first event to its last. Filter by event type to isolate a single kind of activity, for example every `error` across the org in one view. Stack filters to narrow from "everything, everywhere" to "this agent, in prod, erroring" in a couple of clicks, then act on what you find. + +Free-text search cuts straight to a message, a tool name, or an id you already have in hand, so a customer report turns into the exact run in seconds. + +## Where to find it + +The Event Stream is your org home. Sign in and it is the first surface you land on, at `//`, so triage starts the second you arrive. + +Behind it, your agents emit events through the SDK, the collector ships them to your AgentEye server, and the stream tails them as they arrive in infrastructure you control. When you want the rolled-up view instead of the raw trail, each run's events collapse into a single row on Sessions, one click away. + +This is the raw source of truth that every other observe surface builds on, so when a number looks wrong elsewhere, the stream is where you confirm what actually happened. + +## Related + +- [Sessions](/agenteye/sessions): the same events rolled up into one row per run, with a git-style execution graph. +- [Telemetry](/agenteye/telemetry): what your agents send and how events reach the stream. +- [Error tracking](/agenteye/error-tracking): one triage surface for everything that went wrong. +- [Alerts](/agenteye/alerts): turn any threshold into a paging rule. +- [CLI and agents](/agenteye/cli-and-agents): the same live trail from your terminal. diff --git a/docs/agenteye/getting-started.mdx b/docs/agenteye/getting-started.mdx index e433ba18..61a7a989 100644 --- a/docs/agenteye/getting-started.mdx +++ b/docs/agenteye/getting-started.mdx @@ -36,7 +36,9 @@ What you get: - **Alerts & incidents**: threshold rules that page you (email, Slack, webhook, in-dashboard) plus an incident workflow to triage them. - **CLI & AI assistant**: a terminal client (`agenteye`) and the in-dashboard **AI assistant** for asking questions in plain English. -[![Watch: Agent Tracing by Failproof AI (2 min)](/agenteye/images/video-tracing.jpg)](https://youtu.be/VWxukZc5k7s) +
+ +
*See a live agent run trace itself in the AgentEye dashboard: every tool call, model request, and hook on one timeline.* @@ -235,7 +237,32 @@ Click any session to open its **execution graph**, a git-style view of how agent --- -## Step 8: Explore, chart, and alert +## Step 8: Connect your coding agents + +The events you are now browsing are one plain-English question away from your terminal. The **`agenteye` CLI** reads and administers your deployment (sessions, events, evals, keys, alerts, incidents), and the **[CLI skill](/agenteye/cli-skill)** teaches a coding agent like Claude Code or Codex to drive it for you, so you can just ask "is anything broken today?" instead of memorizing commands. + +Install the CLI and sign in. This is a separate tool from the collector: the collector ships events to the server, while the CLI talks to your dashboard. + +```bash +pipx install agenteye +agenteye --base-url http://your-dashboard-host:3000 login --email you@example.com # emailed 6-digit code +agenteye whoami # confirm user and active org +agenteye --json sessions --since 24h # the runs from Step 7, as JSON +``` + +To let an agent answer for you, place the `agenteye-cli/` skill folder (AgentEye provides it) in `~/.claude/skills/`; Claude Code discovers it automatically. Then ask in plain English: + +```text +you ▸ Why did session run-001 fail? +agent ▸ Running: agenteye --json events --session-id run-001 --all + checkout-agent hit TimeoutError on its third tool call +``` + +The agent runs the CLI as you, so it can do only what your login permits, and it can change things (rotate a key, resolve an incident) as well as read. It states each command and waits for your OK before any write. See [CLI and agents](/agenteye/cli-and-agents) for the full picture, the [CLI reference](/agenteye/cli) for every command, and [CLI recipes](/agenteye/cli-recipes) for automation patterns. + +--- + +## Step 9: Explore, chart, and alert With events flowing, the **analyze** pages turn raw activity into answers, so you can measure agent behavior, share findings across the team, and get paged the moment something regresses. Dashboard pages are organization-scoped, so the URLs you see in the address bar are prefixed with your org slug (`//…`). diff --git a/docs/agenteye/health-monitoring.mdx b/docs/agenteye/health-monitoring.mdx index edbb967f..d6bc1927 100644 --- a/docs/agenteye/health-monitoring.mdx +++ b/docs/agenteye/health-monitoring.mdx @@ -65,11 +65,11 @@ pulling a replica out of rotation. (swap in your server's address and port, `8080` in the bundled manifests): ```bash -# Liveness — always 200 while the process is up +# Liveness: always 200 while the process is up curl -s http://localhost:8080/health # {"status":"ok"} -# Readiness — 200 when Postgres + ClickHouse are reachable, 503 otherwise +# Readiness: 200 when Postgres + ClickHouse are reachable, 503 otherwise curl -s -o /dev/null -w '%{http_code}\n' http://localhost:8080/ready # 200 body: {"status":"ready", "checks": { ... }} # 503 body: {"status":"not_ready", "checks": { ... }} if a hard dependency is down diff --git a/docs/agenteye/images/assistant.png b/docs/agenteye/images/assistant.png new file mode 100644 index 00000000..e6a5e5a3 Binary files /dev/null and b/docs/agenteye/images/assistant.png differ diff --git a/docs/agenteye/incidents.mdx b/docs/agenteye/incidents.mdx new file mode 100644 index 00000000..26749a8e --- /dev/null +++ b/docs/agenteye/incidents.mdx @@ -0,0 +1,50 @@ +--- +title: "Incidents" +description: "AgentEye Incidents documentation." +--- + + +When an alert fires, the first question is always "who's on it?" Incidents answer it: the moment something breaches, everyone can see the incident is open, who owns it, and exactly what has happened so far, with a clean, attributed record you can hand straight to a post-mortem. + +![The Incidents inbox: alert-linked and manually opened incident cards, grouped by state, each with a severity badge and an assignee](/agenteye/images/incidents.png) +*The inbox groups open incidents by state and filters by severity and assignee, so you see what needs a human now.* + +## Know who has it, at a glance + +No more "is anyone looking at this?" in a chat thread. A breach opens an incident automatically and drops it into a shared inbox, grouped by state. Acknowledge it and your name is on it, so the rest of the team knows it is handled. Acknowledgement is shared: several operators can ack the same incident and each is recorded on its own, so a full war room shows up by name instead of stepping on each other. Assign one owner for triage, and filter the inbox by severity or assignee to cut it down to what is yours. + +## The whole story, in one timeline + +When the incident is over, you already have the write-up. Open any incident and you get the breach evidence, its assignees and subscribers, a comment thread for coordinating in place, and an append-only activity timeline. + +![An incident detail view: the parent alert and breach summary, assignees and subscribers, an attributed activity timeline, and a comment thread](/agenteye/images/incident-detail.png) +*Everything that happened, in order, each line signed by whoever did it.* + +Every action (opened, acknowledged, resolved, and so on) is written to that timeline and never edited away. Each entry is attributed: to the operator who took it, by email, or to **automated** for anything AgentEye did on its own, like opening the incident on the breach. Nothing is anonymous and nothing is lost, so the post-mortem more or less writes itself. + +## How an incident moves + +```mermaid +stateDiagram-v2 + [*] --> firing + firing --> acknowledged: an operator acks + firing --> resolved: an operator resolves + acknowledged --> resolved: an operator resolves + resolved --> [*] +``` + +- **Open (firing):** the breach opens the incident and pages your channels once. Repeated breaches fold into the same incident and refresh its evidence instead of paging you again and again. +- **Acknowledged:** an operator picks it up. It stays open, and later breaches update the evidence quietly. +- **Resolved:** an operator closes it out. Automatic resolution when the condition clears is planned but not yet enabled, so an incident stays open until a human resolves it, which keeps everyone honest about what has actually cleared. A fresh incident can open on the same alert later. + +One alert holds at most one open incident at a time, so a flapping rule cannot bury you in duplicates. You can also open an incident by hand: a standalone one for something no alert caught, or one attached to an existing alert, if you have `incidents:write`. + +## Where to find it + +Incidents live at `//incidents`. Viewing needs **`incidents:read`**; opening a manual incident needs **`incidents:write`**; acknowledging, assigning, commenting, and resolving need **`incidents:ack`**. Older keys granted the retired `alerts:ack` keep working, since it is honored as `incidents:ack`, so your on-call rotation does not need re-issuing. + +## Related + +- [Alerts](/agenteye/alerts): the rules that open these incidents when a threshold breaches. +- [Error tracking](/agenteye/error-tracking): see every failure in one place and promote one to an alert. +- [Audits](/agenteye/audits): the scheduled analyst that finds the failures no rule was watching. diff --git a/docs/agenteye/observability.mdx b/docs/agenteye/observability.mdx index 7249c931..97107597 100644 --- a/docs/agenteye/observability.mdx +++ b/docs/agenteye/observability.mdx @@ -1,64 +1,23 @@ --- -title: "Observability" -description: "AgentEye Observability documentation." +title: "Observe" +description: "AgentEye Observe documentation." --- -The **observe** section of the dashboard is where you watch what your agents are doing right now and drill into any single run. This page tours its six surfaces (Events, Sessions, Models, Tools, Hooks, and Errors) so you know where to look when something happens. Every surface is scoped to your organization and filterable by date range, environment, agent, and session. +The observe surfaces are where you watch what your agents are doing right now and drill into any single run. Everything here is live, scoped to your organization, and filterable by date range, environment, agent, and session, so you go from "something feels off" to the exact run in seconds. ---- - -## Events: the live trail - -The **Events** page (your org home, `//`) tails every event across all agents in real time, newest first. Each row is colour-coded by type (`agent_start`, `model_response`, `tool_use`, `hook_completed`, `error`, …) and carries a one-line summary, token counts, and a context-window fill badge where it applies. - -![The live Events stream: colour-coded event rows filterable by environment, agent, session, and free text](/agenteye/images/events-stream.png) - -Filter by `session_id` or `agent_id` to follow one run, or by event type to isolate, say, every error. - ---- - -## Sessions: one row per run - -The **Sessions** page rolls events up into one row per run. Once you connect an [evaluator](/agenteye/evaluation-suite), each row also shows the run's quality scores at a glance, and you can filter by any score range. - -![The Sessions list: one row per run, across environments and agents, with status pills and evaluation score badges](/agenteye/images/sessions-list.png) - -Click any session to open its **execution graph**: a git-style view of how agents, tools, hooks, and model calls unfolded over time, with parallel sub-agents on their own lanes and a per-run breakdown in the right rail. - -![A session's git-style execution graph beside its event timeline, with the tool/model/hook breakdown panel](/agenteye/images/session-detail.png) - ---- +![The live Event Stream, colour-coded by type and filterable by environment, agent, and session](/agenteye/images/events-stream.png) -## Models, Tools, Hooks: per-surface latency +Four surfaces, each with its own page: -Each of **Models**, **Tools**, and **Hooks** is a dedicated page with the same shape: a 24-bin sparkline, a vitals strip with p50/p95/p99 latency, a latency **heat-map** (24 time bins × latency buckets), and a percentile band (p50 line with p25-p75 and p10-p90 shaded ribbons and p99 dots). The heat-map and band share a hover crosshair, so a tail spike stands out instead of being hidden by a single mean line. - -![The Models page: a latency heat-map, a percentile band, and per-model token consumption over time](/agenteye/images/models.png) - -*Models* adds token consumption and estimated cost per model. *Tools* adds a success/failure breakdown and a tool-distribution bar. *Hooks* breaks activity down by hook name and trigger event. - -![The Tools page: latency heat-map, percentile band, and a tool-distribution bar](/agenteye/images/tools.png) - -![The Hooks page: latency heat-map, percentile band, and a hook-distribution bar](/agenteye/images/hooks.png) - -> **Tip:** Model rows show context-window fill, making prompt growth and impending compaction visible. AgentEye recognizes common model IDs automatically; you can correct a model's window or add a private one under **Settings → model context windows**. - ---- - -## Errors: one triage surface - -The **Errors** page collects everything the dashboard would paint red (explicit `error` events plus any `tool_result`, `hook_completed`, or `agent_end` whose payload carries a failure) from the most recent matching events. Repeat failures with the same `(session, error_type)` collapse into a single row so a burst reads as one incident. - -![The Errors page: a histogram of errors over time and grouped red error rows, each with a one-click "+ alert" button](/agenteye/images/errors.png) - -Click a row to jump into its session at that exact event. If you have `alerts:write`, each row has a **+ alert** button that opens a new alert rule prefilled to catch that failure again. - ---- +- **[Event stream](/agenteye/event-stream)**: the live, per-step trail of every run across every agent, newest first. Your org home and first stop for triage. +- **[Sessions and execution graph](/agenteye/sessions)**: those events rolled up into one row per run, plus a git-style picture of how each run unfolded. +- **[Performance metrics](/agenteye/telemetry)**: latency heat-maps and p50/p95/p99 vitals for your models, tools, and hooks, so a tail spike stands out from the median. +- **[Error tracking](/agenteye/error-tracking)**: one triage surface for everything that went wrong, one click from a firing alert to the run that broke. -## Next steps +## Related +- [Evaluations](/agenteye/evaluations): score every run for quality. - [Alerts](/agenteye/alerts): turn any threshold into a paging rule. - [Audits](/agenteye/audits): let AgentEye find failure patterns across sessions for you. -- [Evaluation suite](/agenteye/evaluation-suite): score every run for quality. -- [CLI](/agenteye/cli): the same observability from your terminal. +- [CLI and agents](/agenteye/cli-and-agents): the same observability from your terminal. diff --git a/docs/agenteye/overview.mdx b/docs/agenteye/overview.mdx index 58836b27..82930aec 100644 --- a/docs/agenteye/overview.mdx +++ b/docs/agenteye/overview.mdx @@ -20,11 +20,15 @@ If you ship AI agents and you're tired of guessing why a run went wrong, this is Two short videos show the two things teams reach for first: tracing a run, and finding failures automatically. -[![Watch: Agent Tracing by Failproof AI (2 min)](/agenteye/images/video-tracing.jpg)](https://youtu.be/VWxukZc5k7s) +
+ +
*Agent tracing: follow a single run step by step, from goal to tools to final answer.* -[![Watch: Automatically find where your agent is failing (2 min)](/agenteye/images/video-audit.jpg)](https://youtu.be/C5nHKGQIqZQ) +
+ +
*Failproof Audit: let AgentEye mine your logs across sessions and tell you what to fix.* @@ -47,10 +51,10 @@ AgentEye is organized around three ideas (**observe**, **analyze**, and **admin* **Observe** (the raw truth of what happened): -- **Events**: the live, per-step trail of every run (tool calls, model calls, hooks, errors). -- **Sessions**: those events rolled up into one row per run, each ready to be scored. -- **Models, Tools, Hooks**: per-surface latency heat-maps and p50/p95/p99 vitals, so a tail spike stands out from the median. -- **Errors**: one triage surface for everything that went wrong, one click from a firing alert. +- **[Event stream](/agenteye/event-stream)**: the live, per-step trail of every run (tool calls, model calls, hooks, errors). +- **[Sessions](/agenteye/sessions)**: those events rolled up into one row per run, each ready to be scored, with a git-style execution graph. +- **[Performance metrics](/agenteye/telemetry)**: per-surface latency heat-maps and p50/p95/p99 vitals for models, tools, and hooks, so a tail spike stands out from the median. +- **[Error tracking](/agenteye/error-tracking)**: one triage surface for everything that went wrong, one click from a firing alert. ![The Tools observe page: a latency heat-map, a percentile band, and a tool-distribution bar over 24 time bins](/agenteye/images/tools.png) @@ -58,15 +62,20 @@ AgentEye is organized around three ideas (**observe**, **analyze**, and **admin* **Analyze** (turn activity into answers): -- **Queries & dashboards**: saved SQL over your events and evaluations, charted into shared, org-scoped dashboards. -- **Evaluations**: quality scores produced by your own evaluator service, with per-score reasoning. -- **Audits**: recurring investigations that surface failure patterns across sessions. -- **Alerts & incidents**: threshold rules that page you, plus an incident workflow to triage them. +- **[Queries](/agenteye/queries)** and **[dashboards](/agenteye/dashboards)**: saved SQL over your events and evaluations, charted into shared, org-scoped dashboards. +- **[Evaluations](/agenteye/evaluations)**: quality scores produced by your own evaluator service, with per-score reasoning. +- **[Audits](/agenteye/audits)**: recurring investigations that surface failure patterns across sessions. +- **[Alerts](/agenteye/alerts)** and **[incidents](/agenteye/incidents)**: threshold rules that page you, plus an incident workflow to triage them. + +**Interfaces** (reach your data your way): + +- **[CLI](/agenteye/cli-and-agents)**: drive your whole deployment from the terminal or a script, and let a coding agent do it for you in plain English. +- **[AI assistant](/agenteye/assistant)**: ask questions about your agents in plain English, right inside the dashboard. **Admin** (run it for your team): -- **API keys**: scoped tokens for the collector, the dashboard, and the assistant. -- **Users**: passwordless, email-based sign-in with an allowlist. +- **[API keys](/agenteye/api-keys)**: scoped tokens for the collector, the dashboard, and the assistant. +- **[Users](/agenteye/tenant-management)**: passwordless, email-based sign-in with an allowlist. - **Settings**: per-org configuration, including model context-window overrides. --- diff --git a/docs/agenteye/python-sdk.mdx b/docs/agenteye/python-sdk.mdx index 69efbea5..e2b9a858 100644 --- a/docs/agenteye/python-sdk.mdx +++ b/docs/agenteye/python-sdk.mdx @@ -10,7 +10,9 @@ Under the hood, the SDK writes structured events to local JSONL files, and the c > **Tip:** New to AgentEye? Start with [Getting started](/agenteye/getting-started) for the full picture. This page is the complete SDK event reference. -[![Watch: Agent Tracing by Failproof AI (2 min)](/agenteye/images/video-tracing.jpg)](https://youtu.be/VWxukZc5k7s) +
+ +
--- diff --git a/docs/agenteye/queries.mdx b/docs/agenteye/queries.mdx new file mode 100644 index 00000000..8652cb09 --- /dev/null +++ b/docs/agenteye/queries.mdx @@ -0,0 +1,56 @@ +--- +title: "Queries" +description: "AgentEye Queries documentation." +--- + + +Ask any question of your agent data and get an answer in seconds. AgentEye gives you a library of saved, ready-to-run queries over your events and evaluations, so you start from a working example instead of a blank SQL editor. + +![The saved-queries library: a grid of reusable queries, both built-in presets and custom ones](/agenteye/images/queries.png) + +*Your saved-queries library at `//queries`: built-in presets sitting alongside the queries your team has saved.* + +## Start from a preset, not a blank page + +You do not have to remember table names or write SQL from scratch. The library opens with built-in presets for the questions teams ask most, sitting right next to the queries your own team has saved and named. Pick one that is close to what you want and you are most of the way to an answer. + +Every saved query is org-scoped and shared, so the useful ones your teammates write become yours too. Name a query and give it a description once, and anyone in your org can find it, run it, or pin its results onto a dashboard later. + +Find it at `//queries`. + +## Tweak it and run it in the SQL composer + +Open any query and it lands in the SQL composer, where you can adjust it and see the answer immediately: no export, no round-trip, no waiting on someone else. + +![The SQL query composer running a saved query, with a schema sidebar and a live result grid](/agenteye/images/query-lab.png) + +*The SQL composer: your query on the left, a schema sidebar so you never guess a column name, and a live result grid below.* + +- **A schema sidebar** lays out the analytics tables and their columns, so you can shape a query without hunting for field names. +- **A live result grid** returns rows the moment you run, so you iterate in seconds rather than guessing and re-guessing. +- **Read-only by design.** Queries run against your analytics store (ClickHouse) and are validated on the server: only `SELECT` and `WITH` statements are allowed, with a statement timeout and a row cap. An exploratory query can never modify your data, and a runaway one gets stopped for you. + +Happy with the result? Save it back to the library so the whole team inherits it, or pin its output onto a dashboard as a line, bar, area, or pie tile. + +## Run them from the terminal, or let the assistant write them + +The same saved queries follow you wherever you work: + +- **From the terminal.** The `agenteye` CLI lists, runs, and saves the very same queries, so you can drop a result into a script, wire it into CI, or hand it to a coding agent. + +```bash +agenteye query list # the same saved queries, from your terminal +agenteye query run errs --arg prod # run one and print the rows (add --json to pipe it) +``` + + See [CLI and agents](/agenteye/cli-and-agents) for the full command set. + +- **From the AI assistant.** Not sure how to phrase the SQL? Ask the in-dashboard [AI assistant](/agenteye/assistant) in plain English and it will draft the query and save it to your library for you. + +Running a saved query is gated by the `queries:run` permission, kept separate from the permissions to create or delete queries, so you can grant read access without letting everyone rewrite the library. + +## Related + +- [Dashboards](/agenteye/dashboards): pin query results into shared, org-wide charts. +- [AI assistant](/agenteye/assistant): ask questions in plain English and get a query back. +- [CLI and agents](/agenteye/cli-and-agents): run and save the same queries from your terminal. diff --git a/docs/agenteye/sessions.mdx b/docs/agenteye/sessions.mdx new file mode 100644 index 00000000..d8c44632 --- /dev/null +++ b/docs/agenteye/sessions.mdx @@ -0,0 +1,55 @@ +--- +title: "Sessions & Execution Graph" +description: "AgentEye Sessions & Execution Graph documentation." +--- + + +Stop guessing why a run failed. AgentEye rolls every event from a run into one readable row, then draws the whole run as a git-style picture you can read in seconds, so you see exactly what your agent did, step by step. + +![The Sessions list: one row per run, across environments and agents, with status pills and evaluation score badges](/agenteye/images/sessions-list.png) + +*One row per run: the status pill tells you how the run ended at a glance, and a score badge rides along once an evaluator is connected.* + +
+ +
+ +*Agent tracing: follow a single run step by step, from goal to tools to final answer.* + +--- + +## See every run at a glance + +The raw event trail is the truth of every step, but when you have thousands of steps across dozens of runs, you need the run, not the step. The Sessions page rolls all of a run's events up into one row, so a day of activity becomes a scannable list instead of a firehose. + +Each row carries a status pill, so a failed run stands out from a healthy one before you click anything. Filter by date range, environment, agent, or session to go from "everything" to "the run I care about" in a couple of clicks. + +Once you connect an evaluator, every completed run is scored automatically and its latest score shows up on the row as a badge. You can filter by any score range, so "show me every low-scoring prod run this week" is a filter, not a manual review. Until you set one up, sessions still capture the full run; they just don't carry a score yet. + +--- + +## Read the whole run as a picture + +![A session's git-style execution graph beside its event timeline, with the tool, model, and hook breakdown panel](/agenteye/images/session-detail.png) + +*The execution graph (left) sits beside the event timeline; the right rail breaks down the tools, models, hooks, and token spend for the run.* + +Click any session to open its execution graph: a git-style view of how agents, tools, hooks, and model calls unfolded over time. Parallel sub-agents each branch onto their own lane, so you can see which work ran side by side, which sub-agent stalled, and where the run went off course, without replaying it in your head from a wall of logs. + +The right rail gives you the per-run breakdown: which tools and models ran, which hooks fired, and what the run spent in tokens. That is the answer to "why did this run cost so much?" or "which tool is the slow one?" sitting right next to the graph that caused it. + +--- + +## Where to find it + +Every dashboard page is scoped to your org (`//…`). Sessions lives under **Observe** in the left sidebar, next to Events, with the date range, environment, agent, and session filters across the top of the list. Every row is one click from its full execution graph. + +To turn on the score badges and score-range filtering, connect an evaluator: see [Evaluations](/agenteye/evaluations). + +--- + +## Related + +- [Event stream](/agenteye/event-stream): the raw, per-step trail every session is rolled up from. +- [Evaluations](/agenteye/evaluations): connect an evaluator so each run gets a score badge you can filter by. +- [Telemetry](/agenteye/telemetry): how runs get from your agent into these sessions. diff --git a/docs/agenteye/telemetry.mdx b/docs/agenteye/telemetry.mdx new file mode 100644 index 00000000..a993790d --- /dev/null +++ b/docs/agenteye/telemetry.mdx @@ -0,0 +1,52 @@ +--- +title: "Performance Metrics" +description: "AgentEye Performance Metrics documentation." +--- + + +See the instant your models, tools, or hooks slow down or run up a bill, and catch a tail-latency spike before your users ever feel it. Three dedicated pages turn raw timings into p50, p95, and p99 you can read at a glance. + +![The Models page showing a latency heat-map, a percentile band, and per-model token, cost, and context-window figures](/agenteye/images/models.png) +*The Models page: a latency heat-map, a percentile band, and per-model tokens, estimated cost, and context-window fill.* + +## Stop letting averages hide your worst runs + +An average latency number is comforting and useless: it smooths over the one call in fifty that stalls and pages your on-call at 2am. The Models, Tools, and Hooks pages refuse to do that. Each shares the same shape, so you learn it once: + +- A **24-bin sparkline** for the trend at a glance: is this getting worse? +- A **vitals strip** with p50, p95, and p99 latency, so the typical run and the tail sit side by side. +- A **latency heat-map**, 24 time bins by latency buckets, that shows *when* the slow calls clustered. +- A **percentile band**: a p50 line with p25 to p75 and p10 to p90 shaded ribbons and p99 dots, so the spread stays visible instead of averaged away. + +A shared hover crosshair links the heat-map and the band, so a tail spike lines up in time across both instead of hiding behind a single mean line. Find all three pages in the **observe** section of your dashboard, each scoped to your organization and filterable by date range, environment, agent, and session. + +## Models: see exactly what each model costs you + +The Models page (shown up top) answers the two questions a bill always raises: which model, and how much. On top of the shared latency view, it adds **per-model token consumption**, **estimated cost**, and **context-window fill**, so runaway prompt growth and an impending compaction are visible before they surprise you. + +AgentEye recognizes common model IDs automatically. If a window looks wrong, or you run a private model of your own, correct it or add one under **Settings**, in **model context windows**, and the fill readouts follow. + +## Tools: tell the slow apart from the broken + +A tool call can be slow, or it can be quietly failing, and you want to know which one in seconds, not after digging through logs. + +![The Tools page showing the shared latency heat-map and percentile band beside a success and failure breakdown and a tool-distribution bar](/agenteye/images/tools.png) +*The Tools page: the same heat-map and percentile band, plus a success and failure breakdown and a tool-distribution bar.* + +Alongside the shared latency view, the Tools page adds a **success and failure breakdown** and a **tool-distribution bar**, so you see at a glance which tools you lean on most and which are eating your error budget. + +## Hooks: pinpoint the exact hook and trigger + +When a lifecycle hook drags a run, "hooks are slow" is not something you can act on. The Hooks page gets you to the one that matters. + +![The Hooks page showing latency broken down by hook name and trigger event over the shared heat-map and percentile band](/agenteye/images/hooks.png) +*The Hooks page: latency broken down by hook name and trigger event.* + +Over the same latency heat-map and percentile band, the Hooks page breaks activity down by **hook name** and **trigger event**, so you land on the single hook and the single event that need attention. + +## Related + +- [Event stream](/agenteye/event-stream): the live, colour-coded trail of every event. +- [Sessions](/agenteye/sessions): roll events up into one row per run and open its execution graph. +- [Error tracking](/agenteye/error-tracking): one triage surface for everything the dashboard paints red. +- [Dashboards](/agenteye/dashboards): roll-up views across your fleet.