From 79e4bf1bc76d8a5dc3bb18e125102484693563ab Mon Sep 17 00:00:00 2001 From: Andrew Harris Date: Tue, 21 Jul 2026 07:38:14 +0100 Subject: [PATCH 1/3] Add skill guidance on handling dependent import steps --- .claude/skills/build-plugin/SKILL.md | 22 +++++++++++---- .../build-plugin/references/index-defs.md | 28 +++++++++++++++++++ 2 files changed, 45 insertions(+), 5 deletions(-) diff --git a/.claude/skills/build-plugin/SKILL.md b/.claude/skills/build-plugin/SKILL.md index d508a634..05f0898e 100644 --- a/.claude/skills/build-plugin/SKILL.md +++ b/.claude/skills/build-plugin/SKILL.md @@ -47,7 +47,7 @@ Create a TaskCreate task for each phase. The flow deploys early and tests as it - [ ] **Phase 3** — Scaffold files (icon, file structure, `docs/README.md`) - [ ] **Phase 4** — Write `metadata.json`, `ui.json`, `configValidation.json` + its backing stream — the deployable **shell** → [metadata.md](references/metadata.md), [ui.md](references/ui.md) - [ ] **Checkpoint A** — Deploy the shell and authenticate (invoke `deploy-plugin`, probe auth) → [checkpoints.md](references/checkpoints.md) -- [ ] **Phase 5** — Write import definitions and import streams; test each in parallel sub-agents → [index-defs.md](references/index-defs.md), [test-agent.md](references/test-agent.md) +- [ ] **Phase 5** — Write import definitions and import streams; test each in parallel sub-agents. Repeat per dependency level if any step `dependsOn` another → [index-defs.md](references/index-defs.md), [test-agent.md](references/test-agent.md) - [ ] **Checkpoint B** — Redeploy, then trigger + await the import via the CLI so objects exist → [checkpoints.md](references/checkpoints.md) - [ ] **Phase 6** — Build + test data streams in parallel sub-agents → [test-agent.md](references/test-agent.md), [data-streams.md](references/data-streams.md) - [ ] **Phase 7** — Build OOB default content in a sub-agent (it reads [oob-content.md](references/oob-content.md)) @@ -79,7 +79,7 @@ This phase produces a written plan and a user-approval gate before any files are ### The plan must cover 1. **Object types** — Every type that should appear in the SquaredUp graph. These go in `objectTypes` in `metadata.json` and as `sourceType` throughout. -2. **Import steps** — Let the API shape dictate: one step returning many types, or separate steps per type. +2. **Import steps** — Let the API shape dictate: one step returning many types, or separate steps per type. If an object type is only listable in the context of an already-imported parent (the API has "list X for parent Y" but no "list all X"), plan it as a **dependent step** instead — `dependsOn` the parent step and `scope` to its objects; see [index-defs.md](references/index-defs.md#scoped-dependent-steps). 3. **Data streams** — For each object type, plan: - A **summary/current state** stream (`"timeframes": false`, returns current values) - A **history/metrics** stream (supports timeframes, returns time-series rows) @@ -244,9 +244,21 @@ Do not proceed to Phase 5 until auth is confirmed. See [checkpoints.md](referenc ## Phase 5: Import definitions & import streams -Write `indexDefinitions/default.json` and the unscoped list/import streams it calls — these are coupled (the index steps reference the stream columns), so author them here in the main agent. Read [index-defs.md](references/index-defs.md) and [data-streams.md](references/data-streams.md). +Write `indexDefinitions/default.json` and its import streams — these are coupled (the index steps reference the stream columns), so author them here in the main agent. Read [index-defs.md](references/index-defs.md) and [data-streams.md](references/data-streams.md). -Then **test the import streams in parallel sub-agents** rather than inline — the raw paged response bodies are large and the streams are independent. Spawn **one test-mode sub-agent per import stream, all in a single message**, with `model: "sonnet"` (this is run-and-report testing, not deep authoring), passing the `--plugin-id --datasource-id ` captured at Checkpoint A. Each sub-agent tests its (already-written) unscoped stream, confirms it returns one flat row per object, and returns a compact report (per [test-agent.md](references/test-agent.md)). Fix any stream a sub-agent flags before Checkpoint B. +**Root steps first** — the ones with no `dependsOn`. These call a global/unscoped list endpoint and don't wait on anything. Build them, then **test them in parallel sub-agents** rather than inline — the raw paged response bodies are large and these streams are independent of each other. Spawn **one test-mode sub-agent per stream, all in a single message**, with `model: "sonnet"` (this is run-and-report testing, not deep authoring), passing the `--plugin-id --datasource-id ` captured at Checkpoint A. Each sub-agent tests its (already-written) unscoped stream, confirms it returns one flat row per object, and returns a compact report (per [test-agent.md](references/test-agent.md)). Fix any stream a sub-agent flags before Checkpoint B. + +### Dependent steps (`dependsOn` + `scope`) + +Skip this if the Phase 2 plan has no dependent steps — go straight to Checkpoint B. + +A dependent step's stream is scoped per-object (like a Phase 6 data stream), so it can't be tested until the objects its `scope.query` needs actually exist in the graph. Build one **dependency depth level at a time**: + +1. Write that level's step(s) and stream(s) in `indexDefinitions/default.json`. +2. Test each using the same **scoped** procedure [test-agent.md](references/test-agent.md) already uses in Phase 6 (`squaredup objects` to find a real object of the depended-on type, then `squaredup test --object `) — those objects come from the level below, already landed by its own Checkpoint B run. +3. Run [Checkpoint B](#checkpoint-b-redeploy--run-the-first-import) again to land this level's objects before building the next. + +Repeat per level until every dependent step is built, tested, and imported — one dependent step needs a single extra pass; a three-level chain needs two. ### The reconciliation pass @@ -256,7 +268,7 @@ Sub-agents run blind to each other, so several can independently rediscover — 2. **Propagate every API-level discovery to all sibling streams** that share the endpoint family or scoping — timeframe/granularity limits that 404, payload caps that 500, object property/id names, auth quirks. A constraint one sub-agent hit and fixed almost always applies to its siblings too; apply the same edit to each affected stream and **re-test every stream you changed** (re-spawn its sub-agent — a propagated edit is unproven until tested). 3. **Resolve conflicting assumptions** before continuing — if two reports name the same object property or id differently (e.g. one filters on `projectId`, another on `rawId`), determine the correct one against the real response and fix every stream that used the wrong one, so none ships with a scope filter comparing `undefined === undefined`. Do not proceed with an unresolved contradiction. -Run this pass at the end of both Phase 5 and Phase 6. If reconciliation edits an `indexDefinitions/*.json` mapping or an import stream, that edit only shapes a **future** import. Here in Phase 5 the import hasn't run yet, so the Checkpoint B import picks it up naturally — nothing is stale, no extra re-index needed. +Run this pass at the end of each Phase 5 build (the root pass, and each dependent-step level) and at the end of Phase 6. If reconciliation edits an `indexDefinitions/*.json` mapping or an import stream, that edit only shapes a **future** import. Here in Phase 5, reconciliation happens before that pass's own Checkpoint B run, so the next import picks it up naturally — nothing is stale, no extra re-index needed. --- diff --git a/.claude/skills/build-plugin/references/index-defs.md b/.claude/skills/build-plugin/references/index-defs.md index 5229699e..375a1e65 100644 --- a/.claude/skills/build-plugin/references/index-defs.md +++ b/.claude/skills/build-plugin/references/index-defs.md @@ -30,6 +30,22 @@ Defines what gets imported into the SquaredUp graph. { "deviceType": "dbusServiceType" } ] } + }, + { + "name": "deviceAlerts", + "dataStream": { "name": "deviceAlerts" }, + "scope": { + "query": "g.V().has(\"sourceType\", \"Router\")" + }, + "timeframe": "none", + "objectMapping": { + "id": "uid", + "name": "message", + "type": { "value": "Device Alert" }, + "properties": ["severity", "raisedAt", "deviceId"] + }, + "optional": true, + "dependsOn": ["devices"] } ] } @@ -51,6 +67,18 @@ The stream called by an import step must return one flat row per object with at --- +## Scoped, dependent steps + +A step can be scoped to objects a *previous* step already imported, instead of calling a global list endpoint. Use this when an object type is only listable in the context of a parent (the API has "list alerts for device X", not "list all alerts") — the `deviceAlerts` step above is one: + +- **`scope.query`** — a gremlin query run against the tenant's graph. It selects which already-imported objects this step's `dataStream` runs against, once per matched object (like a scoped data stream in Phase 6, not one call for everything). The value you filter on must be a `sourceType` the dependency actually produces — a fixed `{ "value": ... }` type, or one of the values a dynamic type column resolves to (here, one of the `deviceType` values the `devices` step's `dbusServiceType` column can return). +- **`dependsOn`** — names of steps that must finish (succeed or warn) before this one runs, so the objects `scope.query` needs already exist. Omit it (or leave it `[]`) for a root step, which runs immediately. It can list more than one step, and chains can be more than one level deep (`c` depends on `b`, which depends on `a`) — ordering across the whole graph is resolved automatically, so just name whichever step(s) must land first. Names must match real steps and can't form a cycle — both are rejected at deploy time. +- **`optional`** — when `true`, a failed step is recorded as a `warning` instead of a hard failure, which still counts as a green light for anything depending on it. Set it on a dependent step whose data isn't essential to a successful import, so one failing sub-resource doesn't cancel every step after it. + +Building and testing a dependent step needs objects from its dependency to already be in the graph — see SKILL.md Phase 5 for the build/test order. + +--- + ## Stream-level requirements Import streams are ordinary data stream files in `dataStreams/` — they need `name`, `displayName`, `description`, and `tags` like any other stream. Mark import-only streams as hidden so they don't clutter the tile editor: From d8055415953db13782a27d09be0dcdeb6f17812a Mon Sep 17 00:00:00 2001 From: Andrew Harris Date: Tue, 21 Jul 2026 07:52:34 +0100 Subject: [PATCH 2/3] Dependant step is not always per-object --- .claude/skills/build-plugin/SKILL.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.claude/skills/build-plugin/SKILL.md b/.claude/skills/build-plugin/SKILL.md index 05f0898e..a096a112 100644 --- a/.claude/skills/build-plugin/SKILL.md +++ b/.claude/skills/build-plugin/SKILL.md @@ -252,7 +252,7 @@ Write `indexDefinitions/default.json` and its import streams — these are coupl Skip this if the Phase 2 plan has no dependent steps — go straight to Checkpoint B. -A dependent step's stream is scoped per-object (like a Phase 6 data stream), so it can't be tested until the objects its `scope.query` needs actually exist in the graph. Build one **dependency depth level at a time**: +A dependent step's stream is scoped (like a Phase 6 data stream), so it can't be tested until the objects its `scope.query` needs actually exist in the graph. Build one **dependency depth level at a time**: 1. Write that level's step(s) and stream(s) in `indexDefinitions/default.json`. 2. Test each using the same **scoped** procedure [test-agent.md](references/test-agent.md) already uses in Phase 6 (`squaredup objects` to find a real object of the depended-on type, then `squaredup test --object `) — those objects come from the level below, already landed by its own Checkpoint B run. From c7c24550cc8f5733008f8f4ac5a9e7b2f90c3675 Mon Sep 17 00:00:00 2001 From: Andrew Harris Date: Tue, 21 Jul 2026 07:55:04 +0100 Subject: [PATCH 3/3] Add note on single vs multi object stream --- .claude/skills/build-plugin/references/index-defs.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.claude/skills/build-plugin/references/index-defs.md b/.claude/skills/build-plugin/references/index-defs.md index 375a1e65..cc7bc710 100644 --- a/.claude/skills/build-plugin/references/index-defs.md +++ b/.claude/skills/build-plugin/references/index-defs.md @@ -71,7 +71,7 @@ The stream called by an import step must return one flat row per object with at A step can be scoped to objects a *previous* step already imported, instead of calling a global list endpoint. Use this when an object type is only listable in the context of a parent (the API has "list alerts for device X", not "list all alerts") — the `deviceAlerts` step above is one: -- **`scope.query`** — a gremlin query run against the tenant's graph. It selects which already-imported objects this step's `dataStream` runs against, once per matched object (like a scoped data stream in Phase 6, not one call for everything). The value you filter on must be a `sourceType` the dependency actually produces — a fixed `{ "value": ... }` type, or one of the values a dynamic type column resolves to (here, one of the `deviceType` values the `devices` step's `dbusServiceType` column can return). +- **`scope.query`** — a gremlin query run against the tenant's graph. It selects which already-imported objects this step's `dataStream` runs against. The stream must be a scoped stream (like a Phase 6 data stream, not the unscoped kind root steps use) whose `matches` covers the same `sourceType` the query selects — use `httpRequestScopedSingle` when the API takes one parent at a time (one request per matched object), or `httpRequestScoped` to receive all matched objects in one call via `{{objects}}`; see [data-streams.md](data-streams.md). The value you filter on must be a `sourceType` the dependency actually produces — a fixed `{ "value": ... }` type, or one of the values a dynamic type column resolves to (here, one of the values the `devices` step's `deviceType` type column can return). - **`dependsOn`** — names of steps that must finish (succeed or warn) before this one runs, so the objects `scope.query` needs already exist. Omit it (or leave it `[]`) for a root step, which runs immediately. It can list more than one step, and chains can be more than one level deep (`c` depends on `b`, which depends on `a`) — ordering across the whole graph is resolved automatically, so just name whichever step(s) must land first. Names must match real steps and can't form a cycle — both are rejected at deploy time. - **`optional`** — when `true`, a failed step is recorded as a `warning` instead of a hard failure, which still counts as a green light for anything depending on it. Set it on a dependent step whose data isn't essential to a successful import, so one failing sub-resource doesn't cancel every step after it.