Overlay state but unique to each deployment

[JasonN3]

Is is possible to create something similar to an overlay-state where /opt is writable, but doesn't carry any of the changes to that directory forward to a new deployment/upgrade?

Use case:
The situation I'm coming across is that an application (mainly security scanning tools) needs to be installed in /opt but needs to be able to upgrade automatically and separately from the bootc image. However, when changing the major OS version, those changes cannot be carried forward, but should still be available if a rollback is required. The idea is to put a version (current at build time, but may not be current at deployment time) of the application inside of the bootc image so there's something installed but then let it automatically upgrade itself on boot. That way bootc doesn't need to carry anything forward as a version of the application will be available in every image, but if the directory is writable, it can maintain itself. This would also allow a bootc rollback to utilize the previous version.

overlay-state wouldn't work because from my understanding, it would function like /etc and try to carry forward any changes.
/var wouldn't work because it's a persistent directory, which wouldn't allow for a rollback and would likely cause issues during an OS upgrade anyway.

[jmarrero]

One option that can work today is the approach from #1038: make /opt a symlink (or bind mount) to a directory under /var, e.g. /var/lib/scanner, and add a systemd service that re-seeds it from the image when needed.

For this to work you would need to keep a pristine copy of the tool under /usr in the image (once /opt points at /var, the image's /opt content only lands there on the initial install, not on upgrades), and trigger the re-seed on a stamp mismatch... for example compare the booted image digest from bootc status --json (or VERSION_ID in /usr/lib/os-release) with a stamp stored next to the tool not only when the tool fails to start. The trade-off is that /var is shared by all deployments, so a rollback boots with whatever version is currently in /var; the same service running at boot on the rollback deployment restores the version that image carries.

Another thing we are considering is a mount API for container image content, tracked in #2433: bootc would expose a selected path from the booted or staged image read-only, so a systemd unit can check whether the OS version matches what the tool expects and, if not, rsync the content from the image into /var/opt (or wherever /opt points). That is still being discussed/designed.