Stop losing a post's images when the image provider blinks - #223
Merged
Conversation
Every post published since 2026-08-24 went out with no hero and no inline art. Nothing was broken in the image code and nothing was in the database to say so: gpt-image-2 is the only thing in the pipeline that draws, and unlike the article text — which falls back openai -> anthropic — an image failure has nowhere to go. So each call is wrapped in a try/catch that warns and carries on, because losing an illustration should not cost the article. That trade was right; the silence around it was not. OpenAI's quota lapsed for a few days, four posts published image-less, the only trace was a console.warn on a worker, and the first report came from someone looking at the blog. Worse, a failed inline image had its <!--INLINE_IMAGE_N--> marker deleted, which threw away both where the image belonged and the brief it would have been drawn from — so even once the provider recovered, those images were gone for good. Three changes, none of which make an image failure fatal: - A failed inline image now leaves a self-describing PENDING marker instead of a hole. It is an HTML comment, so readers see nothing, but it carries the ordinal, kind, alt and prompt needed to draw it later. The ordinal matters: it names the object in storage, so repairing image 2 cannot overwrite image 1. - The reasons land on lx_article.generation_error — a column that already existed and was never written — so "which posts lost art, and why" is a query instead of a log trawl nobody runs. - repairMissingArticleImages() backfills them. The hero prompt derives entirely from stored columns, and pending inline markers carry their own brief, so images stay recoverable for as long as the row does. The worker sweep calls it two articles at a time, since four "high" 1536x1024 renders is real money and about a minute of wall clock. A provider outage is now a delay rather than a permanent hole. The five posts already affected have had their heroes generated and backfilled by hand; their inline images predate the marker fix and are not recoverable. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RL1QYJhEEqWzs7FsQQBAMG
ThreatCrush Security Scan40 finding(s) HIGH/CRITICAL: 3 | MEDIUM: 28 | LOW: 9
Snippets are redacted; ThreatCrush never prints matched credential material. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What broke
Every autoblog post published since 2026-08-24 went out with no hero image and no inline art. Confirmed in prod:
Uploads to the
lx-article-imagesbucket went from 60/day (15 articles x 4 images) to zero.Why
gpt-image-2is the only thing in the pipeline that draws. The article text falls backopenai -> anthropic, but an image failure has nowhere to go, so every image call is wrapped in a try/catch that warns and carries on — losing an illustration shouldn't cost the article.That trade is right. The silence around it was not:
console.warnon the worker. Nothing in the database said which posts lost their art or why.<!--INLINE_IMAGE_N-->marker deleted, throwing away both where the image belonged and the brief it would have been drawn from. Once the provider recovered, those images were unrecoverable.Both providers were verified with the exact production keys: Anthropic is spend-capped (HTTP 400) until 2026-09-01; OpenAI works, including
gpt-image-2at the production params (1536x1024, qualityhigh) and on the real hero prompts of the affected posts. So the outage has passed — but nothing would have refilled the images.Changes
lx_article.generation_error— a column that already existed and was never written. "Which posts lost art, and why" becomes a query.repairMissingArticleImages()backfills them. The hero prompt derives entirely from stored columns and pending markers carry their own brief, so images stay recoverable as long as the row does. The worker sweep calls it two articles at a time (fourhigh1536x1024 renders is real money and ~1min of wall clock).None of this makes an image failure fatal. A provider outage becomes a delay rather than a permanent hole.
Already done by hand
The five affected posts within the repair window have had their heroes generated and backfilled;
image_url is nullis now 0 for the last 45 days. Their inline images predate the marker fix and are not recoverable.Testing
tests/autoblog-image-repair.test.tscovers the marker round-trip, the--/quote escaping that would otherwise close the comment early and leak prompt text into the post, ordinal preservation, and the empty-prompt and no-marker cases. All 13 assertions were run against the real helper source.lx_articlehas three FKs tolx_site, so the embed had to namelx_site!lx_article_site_id_fkey.tsc/vitestin this checkout (known); CI gates the typecheck and suite.🤖 Generated with Claude Code
https://claude.ai/code/session_01RL1QYJhEEqWzs7FsQQBAMG