From 6a759df91216071a555c68e3461e341c69367b32 Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Fri, 18 Sep 2026 06:07:52 +0200 Subject: [PATCH 1/2] [HWORKS-2895] Global Hopsworks Dashboard using Superset https://hopsworks.atlassian.net/browse/HWORKS-2895 Correct the tag-history guide against what ships. Four things in it were either wrong or had gone stale since #637 merged. The interval query ordered CLOSED before OPENED at an equal event_time. That is backwards for an attach and a detach that share a millisecond: it puts the CLOSED ahead of the OPENED it followed, LEAD leaves that OPENED with no removed_at, and a removed tag reads as current from then on. The dashboards in okr-dashboards order by (event_time, id) instead, because id is the insertion order and the writer emits a change as CLOSED then OPENED in one transaction. The guide now matches them, and states the residual: RonDB allocates id per SQL node, so two events written a millisecond apart through different nodes can still order arbitrarily. The baseline paragraph said an existing attachment is timed from when it was attached without saying what that means for a reader. That start predates recording, so the first interval of a pre-existing attachment covers time nobody observed and raises the average time-in-state of any report including it. It is a lower bound, and now says so. The note titled "Records intent, no behaviour yet" said nothing reads the flag. Something does now, so it pointed users at the opposite of what will ship. The creation section also named a checkbox, Archive deleted tags, that the UI does not have; the label is Archive tag history. Finally, a new subsection documents the upgrade step. Releases before 5.2 accepted archive at schema creation and stored it with no reader, so a schema created with it on has the flag set and no history, and an upgrade starts one only when the flag is set. Re-issuing PUT /tags/{name}/archive seeds the baseline, and is safe to repeat: the backfill covers only attachments that have no history yet. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5 (1M context) --- docs/user_guides/fs/tags/tags.md | 52 ++++++++++++++++++++++++-------- 1 file changed, 40 insertions(+), 12 deletions(-) diff --git a/docs/user_guides/fs/tags/tags.md b/docs/user_guides/fs/tags/tags.md index ab5371e16a..f393106f6a 100644 --- a/docs/user_guides/fs/tags/tags.md +++ b/docs/user_guides/fs/tags/tags.md @@ -77,7 +77,7 @@ Most tags are only ever read as they are now: who owns this feature group, wheth Some are interesting over time, and for those the current value is the least useful part. Marking a schema as archived says that attachments of this tag are worth keeping once they stop being current, so the tag's history can be analysed and not just its present state. -Tick `Archive deleted tags` when defining the schema, or pass `archive=True` through the API: +Tick `Archive tag history` when defining the schema, or pass `archive=True` through the API: === "Python" @@ -118,10 +118,12 @@ This history is not the same thing as the [attachment time][when-a-tag-was-attac That timestamp deliberately stays at the first attachment when a value is corrected, so it records when an artifact was first classified and not when it entered its current state. The per-value history is what the archive is for. -!!! note "Records intent, no behaviour yet" - Setting `archive` today only records the decision on the schema. - Nothing reads it: copying the retained attachments into an offline feature group, where they can be queried as above, is a later change. - Set it now on the schemas whose history you expect to want, because the flag cannot recover attachments that were already discarded while it was off. +!!! note "Set it before you need it" + Setting `archive` makes Hopsworks record every change to the tag's values, which is what makes + the analysis above possible. See [Archive tag history][archive-tag-history] for what gets + recorded, how to read it, and how to turn it on for a schema that already exists. + Set it on the schemas whose history you expect to want. The flag cannot reconstruct changes that + happened while it was off, because the live tag keeps only its current value. ## Step 2: Attach a tag to an artifact @@ -267,7 +269,11 @@ Two things are worth knowing before you turn it on: - **History starts when you turn it on.** Changes made before that are not recoverable, because the live tag keeps only its current value. Attachments that already exist are backfilled with the - state they are in, timed from when they were attached. + state they are in, timed from when they were attached. That start is the attachment time and not + the moment you turned archiving on, so the first interval of an attachment that already existed + covers time that was never observed. Read it as a lower bound on how long that state has held, + rather than as a measurement, and expect it to raise the average time-in-state of any report that + includes it. - **Turning it off stops recording but keeps what was recorded.** The rows already written are still true, and the tag is still attached, so nothing is deleted. @@ -275,6 +281,23 @@ History is recorded per key of the schema, not per tag. Changing one key of a mu a change to that key alone and leaves the others untouched, so a correction to one field does not make every other field look like it changed at the same moment. +### Turning it on for a schema archived before the history existed + +Releases before 5.2 accepted `archive` when a schema was created, and stored it, but recorded +nothing: the flag had no reader. A schema created with it on before upgrading therefore has the flag +set and no history, and the upgrade does not start one. The baseline is written when the flag is +set, and an upgrade sets nothing, so such a schema stays silent until each artifact's tag next +changes, and the state it held before that change is gone. + +After upgrading, a cluster administrator turns it on once more for each schema that already had it, +with the same call used to turn it on for any existing schema, +`PUT /hopsworks-api/api/tags/{name}/archive?value=true`. `GET /hopsworks-api/api/tags` lists the +schemas with their `archive` flag, which is how to find the ones to repeat it for. + +Repeating the call costs nothing on a schema that is already recording. The backfill covers only +attachments that have no history yet, so a schema part-way through is completed rather than +duplicated, and one that is fully recorded gets no new rows. + ### Reading the history The history is stored in the `tag_history` table of the Hopsworks metadata database, one row per @@ -299,9 +322,7 @@ FROM ( event_time AS added_on, LEAD(event_time) OVER ( PARTITION BY artifact_type, artifact_id, tag_name, tag_key - ORDER BY event_time, - CASE WHEN event_type = 'CLOSED' THEN 0 ELSE 1 END, - id + ORDER BY event_time, id ) AS removed_at FROM hopsworks.tag_history ) e @@ -314,9 +335,16 @@ Two details in that query are easy to get wrong and produce numbers that look re filtering inside would hide every `CLOSED` row from `LEAD`, and anything that ended without a successor, a detached tag or a deleted artifact, would report as still current with its duration growing forever. -- The ordering has to put `CLOSED` before `OPENED` at the same timestamp. Both halves of a value - change share one `event_time` by design, so the ordering needs a tie-break, and `id` is not one: - rows are not written in the order the two halves were built. +- The tie-break has to be `id`, and not the event type. Both halves of a value change share one + `event_time` by design, so the ordering needs one. `id` is the insertion order and the writer emits + a change as `CLOSED` then `OPENED` inside one transaction, so `id` already puts the two halves in + the order they happened. Forcing `CLOSED` first instead breaks an attach and a detach that share a + millisecond: it orders that `CLOSED` ahead of the `OPENED` it followed, `LEAD` leaves the `OPENED` + with no `removed_at`, and a tag that was removed reads as current from then on. + + One case is still open. RonDB allocates `id` per SQL node, so two events written a millisecond + apart through different nodes can order arbitrarily with respect to each other. Ordering those + exactly needs a per-key sequence rather than a tie-break. A `removed_at` of `NULL` means the artifact is still in that state. An `added_on` of `NULL` means the tag was attached before Hopsworks began recording attachment times, so the start is unknown; it is From fbe728caacc097fd8ea3285539b40ae013994cbb Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Fri, 18 Sep 2026 15:12:22 +0200 Subject: [PATCH 2/2] [HWORKS-2895] Global Hopsworks Dashboard using Superset https://hopsworks.atlassian.net/browse/HWORKS-2895 The baseline figure is an upper bound, not a lower one. Baselining the current value at the original attachment time credits that value with every stretch nobody observed: attached as dev in January, changed to prod in February with nothing recording, archived in March, and the dashboard reports prod as current since January. The reported dwell therefore exceeds the truth, which is what the following clause about the average rising already said, so the paragraph contradicted itself and told a reader to treat an inflated number as a floor. Describe the backfill in the units the code uses. It has worked per key rather than per attachment since the facade fix: it opens the keys with no open interval, and reopens a key whose last event closed one. The old wording, attachments with no history yet, is the pre-fix unit and is the sentence the whole upgrade procedure rests on. Say that the documented step can be refused. setArchive counts the events the backfill would write and refuses above tag_history_archive_max_events, 20000 by default, so an administrator following the procedure on a widely attached schema needs the limit raised alongside NDB's MaxNoOfConcurrentOperations rather than finding out when the call throws. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5 (1M context) --- docs/user_guides/fs/tags/tags.md | 22 ++++++++++++++++------ 1 file changed, 16 insertions(+), 6 deletions(-) diff --git a/docs/user_guides/fs/tags/tags.md b/docs/user_guides/fs/tags/tags.md index f393106f6a..fac940836c 100644 --- a/docs/user_guides/fs/tags/tags.md +++ b/docs/user_guides/fs/tags/tags.md @@ -271,9 +271,11 @@ Two things are worth knowing before you turn it on: live tag keeps only its current value. Attachments that already exist are backfilled with the state they are in, timed from when they were attached. That start is the attachment time and not the moment you turned archiving on, so the first interval of an attachment that already existed - covers time that was never observed. Read it as a lower bound on how long that state has held, - rather than as a measurement, and expect it to raise the average time-in-state of any report that - includes it. + covers time that was never observed, and it credits the current value with all of it. A tag + attached as `dev` in January, changed to `prod` in February with nothing recording, and archived in + March reports `prod` as current since January. The figure is therefore an upper bound on how long + that state has really held, never a lower one, and it raises the average time-in-state of any + report that includes it. - **Turning it off stops recording but keeps what was recorded.** The rows already written are still true, and the tag is still attached, so nothing is deleted. @@ -294,9 +296,17 @@ with the same call used to turn it on for any existing schema, `PUT /hopsworks-api/api/tags/{name}/archive?value=true`. `GET /hopsworks-api/api/tags` lists the schemas with their `archive` flag, which is how to find the ones to repeat it for. -Repeating the call costs nothing on a schema that is already recording. The backfill covers only -attachments that have no history yet, so a schema part-way through is completed rather than -duplicated, and one that is fully recorded gets no new rows. +Repeating the call costs nothing on a schema that is already recording. The backfill works per key, +not per attachment: it opens only the keys that do not already have an open interval, so a schema +part-way through is completed rather than duplicated, and one that is fully recorded gets no new +rows. A key whose last recorded event closed an interval is opened again at this point, since +nothing is known about the stretch when recording was off. + +On a widely attached schema the call can be refused rather than run. It counts the events the +backfill would write first, one per tag key of every attachment, and refuses above +`tag_history_archive_max_events`, which defaults to 20000. The error names the count and the limit. +Raising it is an administrator decision that belongs with NDB's `MaxNoOfConcurrentOperations`, +because the backfill is one transaction and is bounded by both. ### Reading the history