Update settings handling; add provenance and allow archiving unused settings

[dale-wahl]

What this does

4CAT's settings table has no record of where a setting came from, so settings left behind by renames and removed features accumulate forever and show up in the settings panel as junk. They could not be cleaned up safely, because an undeclared setting is indistinguishable from one whose extension is merely uninstalled, disabled, or failing to import.

This records which module declares each setting, uses that to tell those cases apart, and gives us a way to remove the ones that really are gone — reversibly.

Why not the extension. prefix

The issue proposed prefixing extension settings so they could be identified. I went a different way, for two reasons.

Renaming a setting like selenium.browser means changing every config.get() call, every required_settings entry and every config block at the same time. Any extension not updated would back to defaults and we would lose the value, not just the row.

Additionally, a prefix says a setting is extension-owned. It cannot say whether the owner is absent because it was uninstalled, switched off, or crashed on import — and that last case applies to core processors too, so anything keyed on "is this an extension?" would miss that.

For collision-proofing, new settings should go under the extension's own id — my_extension.api_key and so on. Not extensions..: that is one of the namespaces reserved for core (see below), so anything declared under it is refused. The extensions README.md says so now.

How all this works

ModuleCollector now records which worker declared each setting instead of flattening them all together. That goes into a new config/module_config_provenance.bin alongside the existing module_config.bin, which keeps its shape unchanged because every reader merges it straight into the config definition — including a front-end container that may still be running older code. I kept the two files separate on purpose because of that (old code will never touch the new file).

After the modules load, the back-end writes this into settings_declarations (common/config_manager.py, backend/bootstrap.py). Every stored setting can then be sorted into one of five states, only one of which is ever offered for removal:

• dormant — extension installed but switched off; kept, so switching it on restores its configuration
• absent_extension — extension uninstalled; kept indefinitely for the same reason
• vanished — last declared by core, and gone for longer than the grace period; the only removable class
• recently_absent — gone, but not for long enough to be sure
• unknown — nothing ever recorded declaring it; no evidence either way, so never removed automatically (all our accumulated old settings are here)

Age is measured against the last start-up on which every module imported, not against the clock (4cat.declarations_last_clean_scan). While an import is broken that marker stops advancing, so nothing can age into looking removed just because 4CAT currently cannot see it (mostly in case we ship something broken and do not notice it immediately).
Age is measured from the first complete start-up that found a setting missing, recorded per setting in settings_declarations.absent_since. There is a last seen, but the gap between two start-ups is however long the server happened to be up, so measuring from that would make anything an upgrade removes look long gone the moment it went. absent_since is only ever written on a start-up where every module imported 4cat.declarations_last_scan_complete), so a broken import still cannot age anything out.

A daily worker reports what it finds to admins (backend/workers/audit_settings.py), behind 4cat.report_orphan_settings, off by default — on a server nobody is developing against there is nothing to act on. It remembers which set it last reported in 4cat.declarations_reported, so dismissing the notification does not bring it back tomorrow. I discovered canonical_id means "came from the phone-home server" so could not be used and I had to match the notification text itself (this might be worth an improvement on the notifications table).

Undeclared settings are no longer rendered in the settings panel at all. A line at the foot of the page links to a new Unused settings page (webtool/templates/controlpanel/settings-unused.html) listing them with why each is kept, and an Archive button for the ones that qualify.

Nothing is destroyed

Archiving moves a setting's values — the global one and every tag — into settings_archive, and they can be restored from the same page. The guard lives in ConfigManager.archive_setting(), not in the view, so a request naming a setting directly cannot get past it. We could allow archiving other settings too if we want.

Two related fixes fell out of this:

A module could previously override a core setting's definition, because module config was merged over core with a plain dict.update(). That is now refused rather than merged, and namespaces belonging to core (privileges., flask., 4cat., path., datasources., extensions., logging.) are refused outright so an extension cannot claim a name a later 4CAT version might use. This matters because a definition controls a setting's global flag — a module setting that on a core privilege would have made every per-tag restriction silently stop applying. Probably not a risk so much as a mistake (installing an extension by nature has lots of risks anyway).

module_config.bin is now written atomically. It was truncated and rewritten in place, which a reader in the other container could catch mid-write; the retry that was meant to survive that but probably would have failed, because it re-read the same file handle after a failed unpickle. This should all but avoid that happening (I think it could still try to read right as a file was being replaced so I left the retry).

Needs an upgrade for the database

VERSION goes to 1.57. migrate-1.56-1.57.py creates the two tables and archives 18 core settings for features 4CAT no longer has — renames, the Reddit datasource, the removed scheduler, the old per-processor proxy settings, and so on. he list is written out in the migration with a note on what became of each one, since none of it can be figured out automatically. Now that won't be a problem!

Extension settings are not touched, including ones an extension has renamed itself. That is the extension author's call, not ours, and an older version of it may still be in use somewhere. I am not sure if or how we could encourage extensions to remove old settings, but we can track them now so perhaps with versioning of extensions.

On the my DevCAT I tested against: 182 settings attributed, 27 undeclared, 18 archived, 9 deliberately left (extensions and future feature settings).

Two things found but not fixed here

api.youtube.key was declared in both config_definition.py and youtube_metadata.py. Since module config was merged after core, the processor's copy had been silently winning. Its help text is now the one in the core declaration so
nothing changes in the interface, and the duplicate is gone.

api.openai.api_key is read in processors/machine_learning/audio_to_text.py but declared nowhere, so it cannot be set
from the interface and the OpenAI Whisper fallback never fires. Left alone here — it either wants re-declaring or retiring in favor of the LLM settings. Probably LLM, but need to sort out what that looks like. Out of this PR anyway.