Record why a keyword failed, instead of only logging it - #215
Merged
Conversation
…n article The anchored gate shipped and the hourly cron wrote its first 162 keywords across 11 sites. The vendor and junk classes are gone. Reading the actual output found two things the fixtures could not have. **Partial matches leaked.** Eight keywords, all one shape: one generic word of a multi-word subject matched, and a word from the generic commercial vocabulary rescued it. "open standards" matched on "open", "software" did the rest, and logicsrc was queued to write about OBS Studio. "supply chain security" matched on "supply" and admitted "industrial automation supply". "online shopping" matched on "online" and admitted "quickbooks online pricing". So a partial match on a multi-word subject may now only be rescued by an anchor the site itself supplied — never by the defaults, which are generic by construction. A complete match is unaffected, including a single-word subject, where "iptv alternatives" is a real query. **And the cross-product floor was publishing constructions.** That one is mine and it was new. Used as per-subject filler it produced "saving money pricing", "deals platform", "coordination d0rz" and "ai content loop" — on-niche, gate-passing, and not topics anybody searches. bl0ggers' niche is "human-in-the-loop AI publishing", so its derived modifiers are literally "human" and "loop", and crossing a subject with those yields nonsense. The crosses' real value was always upstream: as DataForSEO seeds they are what turns "peptide" into "peptide merchant account", and they still seed every expansion. They are simply no longer published on the strength of being grammatically adjacent to the niche. They remain the floor, but site-level rather than per-subject — only a run that would otherwise insert nothing falls back to them, which keeps "a blog with every upstream down still publishes" without letting constructions pad a healthy run. The result reports usedCrossFloor so a site sitting on that for weeks is visible. All eight leaked keywords are now regression cases, with the on-niche keywords from the same run that must still pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013cbcXdpLFUgJZuVfCB4Gzv
Companion to purge-offniche-keywords.ts, for rows written before the gate was tightened. Two verdicts, both computed with the real planner rather than approximated in SQL. The second one needed care. A construction cannot be found by checking for a null search volume, which was the obvious approach: the buyer-journey model also returns keywords with no volume, and those are the best output in the run — "zeek vs suricata for network security monitoring" and "langchain alternatives for agents" both sit in the same null bucket as "deals platform". So a construction is identified by fingerprint against the cross set the planner would build for that site, which is exact rather than heuristic. Ran against the 162 rows from the first production cron: kept 53, removed 95 constructions and 14 partial-match leaks. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013cbcXdpLFUgJZuVfCB4Gzv
Publishing stopped on 2026-08-19 and nobody could tell why. All 308 failed
lx_keyword rows carry a null status_reason, because failKeyword() wrote the
reason to console.warn and nothing else — so it lived in the Railway log
buffer and expired there. From the database and the dashboard the outage was
invisible: a number going up, with no cause attached.
Every caller already computes a precise reason ("embedding failed: …",
"quality gate failed after N attempts (slop=…)", "insert failed: …"). It was
simply not being kept. Now it lands on the row, truncated, with a fallback
write so a keyword that cannot record its reason is still marked failed rather
than retried forever.
The quota path gets the same treatment even though it requeues rather than
fails, because it is the one outcome that leaves no trace anywhere: the
keyword looks untouched, no article row is created, nothing reaches the model,
and the dashboard shows a healthy queue while the site quietly stops
publishing. That is the exact shape of this outage.
Diagnosis so far, recorded here because the evidence is easy to lose: article
generation is dying BEFORE the writer model. ai_usage shows zero lx_article
calls since 2026-08-19 while buyer_journey_keyword_research ran today on the
same OpenAI key; credits are not the blocker (3,583 on the account, 20 per
article); and text-embedding-3-small answers normally when called directly.
The next 09:00 run will now say which step it is.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013cbcXdpLFUgJZuVfCB4Gzv
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| console.log("\n-- SQL:"); | ||
| for (let i = 0; i < allIds.length; i += 200) { | ||
| console.log( | ||
| `delete from lx_keyword where status='queued' and id in (${allIds |
ThreatCrush Security Scan40 finding(s) HIGH/CRITICAL: 3 | MEDIUM: 28 | LOW: 9
Snippets are redacted; ThreatCrush never prints matched credential material. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Publishing stopped on 2026-08-19 and nobody could tell why. All 308 failed
lx_keywordrows carry a nullstatus_reason, becausefailKeyword()wrote the reason toconsole.warnand nothing else — so it lived in the Railway log buffer and expired there.Every caller already computes a precise reason. It was simply not being kept. Now it lands on the row.
The quota path gets the same treatment even though it requeues rather than fails, because it is the one outcome that leaves no trace anywhere: the keyword looks untouched, no article row is created, nothing reaches the model, and the dashboard shows a healthy queue while the site quietly stops publishing.
Diagnosis so far
Article generation is dying before the writer model:
ai_usageshows zerolx_articlecalls since 2026-08-19buyer_journey_keyword_researchran today on the same OpenAI keytext-embedding-3-smallanswers normally when called directlyThe next 09:00 run will say which step it is.
1931 tests pass, clean typecheck.
🤖 Generated with Claude Code
https://claude.ai/code/session_013cbcXdpLFUgJZuVfCB4Gzv