investigate: storage-target descriptors and misconfiguration diagnostics for .lnk / assets.json

[zackees]

Summary

A .lnk today describes where an asset comes from, never where it should end up. There is no notion of a storage target (LittleFS, SPIFFS, SD card, embedded-in-firmware, WASM VFS), so a build cannot tell a correctly-configured project from a broken one — and says nothing either way.

This issue is to investigate the design, not to lock one in. The shape below is a starting point; the corner cases are the real content.

Verified current behavior

fbuild lnk add writes:

{ "sha256": "164804d9…", "size": 6051840, "url": "https://…", "v": 1 }

No destination field. And fbuild lnk pull resolves into the global cache only — nothing is materialized beside the .lnk:

lnk pull: 1 ok, 0 failed (of 1)
  → ~/.fbuild/prod/cache/archives/lnk-blobs/raw-video1.rgb/…/<sha256>/video1.rgb

$ ls examples/Fx/FxLedmapper32x32/data/
.gitignore  readme.txt  screenmap.json  video1.rgb.lnk     # <- no video1.rgb

So the bytes are fetched and verified, then stop. Nothing consumes them.

Related context: FastLED has no LittleFS backend at all — FsImpl is implemented only for Arduino SD (SPI), the WASM browser VFS, and a host stub. And fbuild has no buildfs/uploadfs. See #1354.

The gap

data/*.lnk → [lnk pull ✅] → global cache → ??? → device filesystem
                                            ▲
                                   no destination concept

Direction to investigate

1. A destination on the descriptor

Something like a dest block naming the storage target:

{
  "sha256": "164804d9…",
  "size": 6051840,
  "url": "https://…",
  "dest": { "target": "littlefs", "path": "/video1.rgb" },
  "v": 2
}

Candidate targets: littlefs, spiffs, sdcard, firmware (baked as a byte array), vfs (WASM preview), none (host/test only).

assets.json would carry the same vocabulary, plus a file-level default so many assets can share one target without repeating it:

{
  "defaults": { "target": "littlefs" },
  "assets": { "video1.rgb": { "url": "…", "sha256": "…" } }
}

Version bump required. Adding dest to a v1 .lnk changes its meaning, so this is v: 2. A v1 .lnk must keep working — which forces the "no destination declared" question below.

2. Diagnostics — the actual ask

Every declaration should produce a clear verdict. A proposed severity split:

Error (fails the build)

• sha256 mismatch — already the rule; never silently accept
• asset exceeds the filesystem partition
• target unsupported on the selected board (e.g. littlefs on AVR)
• littlefs target with no filesystem partition in the partition table
• two declarations naming the same file with conflicting targets

Warning (yellow)

• .lnk with no destination descriptor — the case that prompted this issue. Fetches and caches, but goes nowhere. Should say so and name the default it fell back to.
• sdcard target — fbuild cannot write to a card it does not have; the copy is manual and the warning must say that plainly
• a large file in data/ with no .lnk (committed binary that arguably should be remote)
• .lnk missing sha256 — fetch works, integrity does not

Info

• cache hit vs fetched, and the resolved target per asset

Warnings must name the offending file and the concrete fix. A yellow line reading "no destination" is only useful if it also says what to add.

Corner cases to work through

These are the reason this needs investigation rather than a patch:

1. No destination declared — hard error, or warn-and-default? A default of littlefs breaks SD-based sketches like FxLedmapper32x32; a default of sdcard silently does nothing at build time. Neither is obviously right, and v1 .lnk compatibility forces an answer.
2. .lnk sidecars must be excluded from any generated image — shipping video1.rgb.lnk to the device is always wrong.
3. Plain files in data/ with no .lnk — do they join the image implicitly? Mixed remote and committed assets in one directory is the common case.
4. Partition sizing — total data/ versus partition capacity must fail at build time, never at runtime mount.
5. Platform capability matrix — LittleFS support differs across ESP32 / ESP8266 / RP2040 / Teensy / AVR. Needs a real table, not assumptions.
6. SD card is not deployable by a build tool. Is it a target at all, or just documentation?
7. Conflicting declarations — .lnk currently overrides assets.json for the same filename (FastLED's ci/asset_resolver.py rule). Should a target conflict silently follow that precedence, or error?
8. WASM preview ignores targets entirely — everything is browser VFS. Should a littlefs target warn under a WASM build, or be silently ignored?
9. Filename constraints — LittleFS filename length and path-depth limits can reject names that were fine on disk.
10. Offline builds — cache miss plus no network needs a distinct, actionable message, not a generic fetch failure.
11. Shared assets across sketches with different targets — is the target a property of the asset or of the sketch that uses it?
12. Firmware-embedded target — bakes into the binary and interacts with bloat/size budgets rather than a filesystem.

Open questions

• Does target belong on the asset, on the sketch, or on the board environment? An asset reused by two sketches may need different answers.
• Should lnk pull gain a --materialize mode that writes beside the .lnk, or does only buildfs consume the cache?
• Is assets.json the right home for defaults, or should that live in platformio.ini / the environment?

Related

• feat: resolve remote example assets at build time (sha256-verified, globally cached) #1354 — build-time resolution and the buildfs/uploadfs gap
• tracking: move FxLedmapper32x32 binary assets to FastLED/assets with build-time resolution FastLED#3974 — asset migration that surfaced this
• feat: structured manifest.json index + CDN serving for FastLED assets assets#1 — manifest and CDN hosting
• ci/asset_resolver.py in FastLED/FastLED — reference parser for both formats, with 19 tests

[zackees]

Audit of the open questions

Independent audit against the actual fbuild source (crates/fbuild-toolchain/src/lnk/* at 9e2efae1 / 2.5.17), FastLED's reference parser, PlatformIO's documented filesystem model, and four pointer-file precedents (git-LFS, DVC, Bazel, Nix). I spot-verified every load-bearing claim below against the source before posting; corrections to my own issue body are called out explicitly.

Two corrections to the issue body above

1. board_partitions is NOT dormant — I was wrong. The comment at ci/boards.py:65 says "Reserved for future use", but two boards actually set it (ci/boards.py:702 and :763, both huge_app.csv). Only board_build_filesystem_size is genuinely unused. The stale comment misled me.

2. There are TWO incompatible .lnk formats in the FastLED ecosystem right now. This is the most important finding here and it changes the compatibility story completely.

examples/AudioUrl/data/track.mp3.lnk is committed on origin/master today, in a plain-text format from FastLED#2284:

# .lnk asset link file — see FastLED issue #2284.
# The first non-comment line is the URL this asset resolves to.
# Lines starting with '#' and blank lines are ignored.
# Additional lines (metadata like sha256=, fallback=) are reserved for
# future use and are ignored by the v1 parser.
https://www.soundhelix.com/examples/mp3/SoundHelix-Song-1.mp3

fbuild's parser is JSON-only and rejects this outright (format.rs::from_raw). So the one .lnk actually in the wild cannot be read by the tool named lnk. Note also that the text format's own comments reserve sha256= and fallback= — i.e. FastLED's convention was heading toward key=value while fbuild shipped JSON.

Resolving this format split should probably precede the destination design, since it determines what "backward compatible" even means. Options: teach fbuild's parser to accept the text form, migrate AudioUrl to JSON, or formally deprecate one. Worth its own issue.

---

The finding that reframes this issue

fbuild already has a materialization pipeline and a working firmware-embed consumer. crates/fbuild-toolchain/src/lnk/mod.rs:1-22 documents it:

  .lnk file (in source tree)
        |
    [scanner]  walks tree, parses every *.lnk
        |
    [resolver] cache lookup by sha256; miss -> download; verify
        |
  [materializer] hardlink/copy cached blob into build/resources/<rel>
        |
  downstream build steps (objcopy, embed_files, ...) consume the
  materialized file as if it had been in the source tree all along.

And crates/fbuild-build-esp/src/esp32/orchestrator/embed_stage.rs:38-46 already resolves .lnk entries named in board_build.embed_files through materialize_lnk_entry and embeds them. Verified in source.

So the cache-only behavior I reported for fbuild lnk pull is the CLI's behavior, not the architecture's. The missing consumer is buildfs — not a dest field on the descriptor.

Recommendations at a glance

Question	Recommendation	Rationale
1. Where does target live?	Board environment (platformio.ini), never the asset	All four pointer-file precedents keep destination out of the pointer; PlatformIO declares FS type per-environment
2. lnk pull --materialize?	Yes, into the build tree, with --in-place escape hatch	Build stages are the real consumers; in-place serves interop/debug
3. Defaults in assets.json?	No — it stays a source manifest; targets live in platformio.ini	One source of truth eliminates the conflict corner cases outright
Hard case: .lnk with no destination	Neither error nor default. Reframe as end-of-build "fetched but unconsumed" warning	Destination isn't the descriptor's job, so its absence isn't the descriptor's defect

Q1 — target belongs on the board environment

Keep .lnk v1 exactly as-is: a source descriptor. Do not add dest. Do not bump to v2.

1. Every precedent agrees. git-LFS pointers carry only version/oid/size. DVC uses destination-by-position with project-wide remotes. Bazel's http_file declares source only; placement is the consuming rule's job (pkg_tar package_dir). Nix fetchurl yields a store path; consumers wire destination. A .lnk embedding "deploy me to littlefs" would be novel against all four, in the wrong direction.
2. fbuild's own conventions agree. The .lnk's path already encodes logical destination — data/video1.rgb.lnk materializes as data/video1.rgb. What storage data/ maps to is exactly PlatformIO's per-environment board_build.filesystem / filesystem_size / embed_files.
3. It dissolves corner case 11 for free. An asset shared by two sketches gets a different target per environment, because target was never the asset's property.

Tradeoff accepted: diagnostics can no longer be computed from the descriptor alone — the checker needs environment config too. That is correct: "is this project configured right" was always a whole-project question. Pretending a sidecar could answer it is how you get descriptors that lie (asset marked littlefs, board has no LittleFS).

Q2 — both, with a hierarchy

Primary consumers are build stages reading the cache via materialize_all() — the embed stage already does; buildfs should too (project data/ into the build tree: committed files plus materialized blobs, minus *.lnk, then run mklittlefs/mkspiffs).

Additionally expose fbuild lnk pull --materialize for humans and non-fbuild consumers (fastled-wasm's VFS packer, FastLED CI), defaulting to the build-tree location, with --in-place to write beside the .lnk.

format.rs:6-8 states the intent: materialize "in the build tree, not the source tree... keeps the source tree clean (no .gitignore wildcards)". That intent is right and already implemented — the CLI just never exposed it. --in-place is still needed because SD-card users and fastled-wasm need a real file to copy.

Tradeoff accepted: --in-place reintroduces the stale-file/accidental-commit risk the build-tree design avoids. Mitigate by having it ensure a .gitignore entry per materialized name; sha verification makes staleness self-healing on next pull.

Q3 — no defaults in assets.json

assets.json stays a multi-asset source manifest. Target config goes in platformio.ini per environment. A defaults.target block would create a third home for deployment intent (after .lnk and the environment), multiplying exactly the conflict matrix corner case 7 worries about. The .lnk-overrides-assets.json precedence then only ever adjudicates source fields, where last-writer-wins is benign.

Tradeoff accepted: an author cannot express "this is FS-image material" in one self-contained file beside the data. In practice they never could — the environment decides whether an FS image exists at all, and a per-asset claim the environment cannot honor is worse than no claim.

Corner cases

#	Case	Resolution	Status
1	No destination declared	Reframe as end-of-build "unconsumed asset" warning; no error, no behavioral fallback (detail below)	resolved
2	.lnk sidecars in images	Non-issue under projection: the image builds from the build-tree projection containing foo.ext, never foo.ext.lnk. Add a guard erroring if a literal *.lnk reaches the image input set	resolved
3	Plain files in data/ with no .lnk	Join the image implicitly — that is PlatformIO's data_dir contract. Projection = committed plus materialized. "Large committed binary, consider .lnk" is a good info lint	resolved
4	Partition sizing	Build-time error. Pre-check total sizes plus FS overhead against partition size from the CSV before invoking mklittlefs; report per-file sizes on failure	resolved
5	Platform capability matrix	Built — see below	resolved
6	SD card as a target	Not a target. fbuild cannot flash removable media. SD is a runtime concern (FileSystem::beginSd()). Its only build footprint is suppressing the case-1 warning	resolved
7	Conflicting declarations	Keep .lnk-wins, but warn when sha256 differs between the two — silent shadowing of a different hash is a footgun. Target conflicts cannot occur (targets are not in descriptors)	resolved
8	WASM ignores targets	Correct behavior, info at most. Nothing to "ignore" since assets carry no annotation	resolved
9	Filename constraints	Validate at image-build time per FS. SPIFFS's 32-char full-path limit is the one that bites; LittleFS's 255-char name limit rarely does. Error with offending name and limit	resolved
10	Offline plus cache miss	Distinct error; already half-structured (cli/lnk.rs:100-113 prints MISSING ... run 'fbuild lnk pull'). Build-time version names file, sha, cache root, and the fix	resolved
11	Shared assets, different targets	Dissolved by Q1	resolved
12	Firmware-embedded	Already shipped via board_build.embed_files plus objcopy. Keep it there; do not add a firmware target to any asset vocabulary. Count embedded bytes against the app partition budget	resolved

Case 1 in full — the hard one

The issue frames this as "error, or warn-and-default?" — reject the framing. Both horns assume the descriptor owes us a destination. Under Q1 it does not, so a v1 .lnk with no dest is not defective and no version bump is needed.

The compatibility argument runs opposite to my assumption in the issue body. format.rs:110-116 hard-rejects v != 1, so bumping to v2 would break every older fbuild on sight, while adding fields to v1 is safe (both LnkFileRaw and asset_resolver.py ignore unknown keys). Staying on v1 is the compatible move; v2 is the breaking one.

The real defect observed is: bytes fetched, verified, consumed by nothing. Diagnose exactly that, at end of build, when fbuild knows what was consumed:

• Consumed by embed_files -> info: embedded: video1.rgb (5.8 MiB)
• Consumed by buildfs -> info: littlefs image: video1.rgb (5.8 MiB of 8 MiB partition)
• Consumed by nothing -> warning:

warning: data/video1.rgb.lnk fetched (5.8 MiB, sha256 ok) but no build step consumes it.
  - to embed in firmware:       board_build.embed_files = data/video1.rgb.lnk
  - to pack a filesystem image: board_build.filesystem = littlefs  (+ run buildfs)
  - if it belongs on an SD card: fbuild cannot deploy removable media; copy it manually
    (`fbuild lnk pull --materialize --in-place`), silence with `lnk.warn_unconsumed = false`

This handles FxLedmapper32x32 (SD-based: a littlefs default breaks it, an sdcard default silently no-ops) by doing nothing behaviorally and saying everything diagnostically.

The one genuine product decision left: whether the suppression knob is a per-env ini flag (recommended above) or something richer. That sliver, and only that sliver, is needs-human-decision.

Capability matrix — on-chip filesystem support

Platform	Flash FS	Declared via	PIO buildfs/uploadfs
ESP32 (Arduino-ESP32)	LittleFS, SPIFFS (legacy), FFat	board_build.filesystem plus partition CSV; data partition subtype spiffs	Yes
ESP8266	LittleFS (recommended); SPIFFS formally deprecated in core docs	board_build.filesystem (SPIFFS default for back-compat)	Yes (plus uploadfsota)
RP2040/RP2350 (arduino-pico)	LittleFS at end of flash	board_build.filesystem_size (LittleFS only, no filesystem option)	Yes
Teensy 4.x	LittleFS_Program (runtime lib in program flash; re-upload wipes files)	runtime myfs.begin(bytes)	No — platform-teensy's builder defines only nobuild/buildprog/size/upload
AVR	None — 1 KB EEPROM only	n/a	No
All	SD/SdFat over SPI	runtime library; removable media — no build-tool deployment path exists	n/a

Consequences: filesystem = littlefs on AVR/Teensy environments -> error ("platform has no build-time filesystem image support"; for Teensy, point at LittleFS_Program and its wipe-on-upload caveat).

FastLED-side blocker: a LittleFS FsImpl does not exist (only SD-SPI, WASM VFS, host stub), so a littlefs image is unreachable from FastLED's FileSystem API until that drop-in lands. buildfs should not ship pointing at an API gap — needs a paired FastLED tracking issue.

What could not be determined

• ESP32 SPIFFS "deprecated" is ecosystem direction, not a formal notice — current ESP-IDF SPIFFS docs carry no deprecation banner (unlike ESP8266's, which is explicit). The matrix says "legacy" accordingly.
• Local fbuild checkout is 2.5.17 (9e2efae1); installed CLI is 2.5.19. The two lnk modules were not diffed; all crates/ citations are against 2.5.17. Observed CLI behavior matches 2.5.17's cli/lnk.rs, so drift risk is low.
• fastled-wasm's actual asset path (feat: resolve remote example assets into the WASM VFS zackees/fastled-wasm#215) was not inspected. The claim that WASM bundles data/ wholesale comes from the FxLedmapper readme, not fastled-wasm source.
• No committed assets.json exists anywhere yet — it lives only as a format ci/asset_resolver.py is prepared to parse. Its schema is therefore fully steerable, another reason not to grow it a defaults.target.

[zackees]

Tracked in the meta issue: FastLED/FastLED#3986 — remote assets: hosting, build-time resolution, and on-device filesystem deployment.