[HWORKS-3240] Feature view inference logging: real-time online logs from the Arrow inference logger

[jimdowling]

Summary

User guide for feature logging v2: the two transports and how a view switches, the job transport's flush and shutdown settings, the termination grace a stopping pod gets, and the Feature logging card for both transports.

The API protocol guide and the deployment schema guide say what choosing a protocol changes: it is picked per deployment and defaults to REST, which is what curl and the published OpenAPI document use; a deployment served by the default predictor can serve gRPC instead, where the rows travel as KServe v2 tensors, predict() returns the same dictionary either way and the predictor validates the rows it decodes. A gRPC deployment answers no HTTP, so the deployment page shows it no curl example and no OpenAPI reference. The python blocks of the feature logging guide are normalised, which the snakeoil gate requires.

Tests

mkdocs build -s passes with the branch hopsworks-api client installed (the hsml.deployment.* cross-references resolve against it); merge after hopsworks-api.

Cluster verification

The behaviour the guide describes was verified on jim-feature-logger through the other PRs of this ticket: both transports, the switch between them, the stop path and the deployment page card.

Update, 2026-09-17

• New user guide docs/user_guides/fs/feature_group/delta_maintenance.md: the four Delta operations (delta_optimize, also reachable as delta_compact, delta_checkpoint, delta_cleanup_metadata, delta_vacuum), their order, the retention that protects a running reader, and the two delta-rs defaults that make a vacuum a silent no-op.
• feature-vectors.md gains "Retrieving feature vectors without blocking" for get_feature_vector_async and get_feature_vectors_async; predictor.md notes that the default predictor's predict is a coroutine, with predict_blocking for callers without a loop and HOPSWORKS_PREDICTOR_ASYNC_LOOKUP=false to opt out.
• Benchmark figures in the feature view docstrings were brought in line with R13 and R14 (23.6 percent more throughput, p99 down 72 percent, 218 to 270 requests per second).

Cluster latency and throughput, 2026-09-15 and 2026-09-16 (jim-feature-logger)

Open-loop runs at a fixed arrival rate, one deployment measured against itself, 1 core request, 2 core limit, 2 GiB. Full tables in section 4 of the Feature logging v2 page.

Run	Question	Result
R11 loadtest-1789510032	The job transport under a sustained 150 req/s, 45,000 requests per phase	p50 9.20 to 9.96 ms (+0.76), p99 39.8 to 41.6 ms (+1.8), p99.9 220 to 555 ms, saturation 282.9 to 225.4 req/s; writer subprocess a flat 90.1 MiB; the commit job sustains 1,500 to 1,630 rows/s with flat memory as the table doubles
R12 loadtest-1789542401	Delta maintenance beside live traffic at 120 req/s	p50 +0.12 ms; p99 15.8 to 31.2 ms during the 4 s compaction, unchanged during checkpoint, log cleanup and vacuum
R13 loadtest-1789549788	The online lookup's share of a request	2.48 ms of wall time for 0.18 ms of CPU; 79.7 percent of p99 (89.6 vs 18.2 ms); saturation 218.6 vs 310.2 req/s
R14 loadtest-1789558725	Awaiting the lookup instead of blocking on it	saturation 218.6 to 270.1 req/s (+23.6 percent), p99 89.6 to 25.1 ms (-72 percent), mean -17 percent; the deployment with nothing to look up also gains, p99 -24 percent

These runs predate the 2026-09-17 review fixes below; nothing in those fixes touches the request path they measure except the per-loop pool opening, which R13 and R14 exercised warm.

Merge order

hopsworks-api first (the loadtests, the docs build and the loadtest runner install the client from it), then hopsworks-ee, model-serving-webhook and docker-images (inference logger, Grafana dashboard, runtime wrapper), then hopsworks-helm, hopsworks-front, loadtest and logicalclocks.github.io.

Related PRs

One PR per repository on branch HWORKS-3240-feature-logging: hopsworks-api, hopsworks-ee, model-serving-webhook, hopsworks-front, docker-images, loadtest, logicalclocks.github.io, hopsworks-helm.

🤖 Generated with Claude Code

[Copilot]

🟡 Changes recommended

Unresolved documentation accuracy, import, link, and formatting issues remain.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

This documentation-only PR updates HSML deployment references and documents feature logging v2 and REST/gRPC serving protocols.

Changes:

• Documents feature-logging transports, lifecycle, materialization, and monitoring.
• Adds REST/gRPC protocol and deployment configuration guidance.
• Updates HSML imports and API references.

File summaries

File	Summary
docs/user_guides/projects/python-deployment/troubleshooting.md	Updates Deployment API references.
docs/user_guides/mlops/serving/troubleshooting.md	Updates serving API references.
docs/user_guides/mlops/serving/transformer.md	Updates Transformer imports and references.
docs/user_guides/mlops/serving/resources.md	Updates resource imports and references.
docs/user_guides/mlops/serving/predictor.md	Updates predictor imports and references.
docs/user_guides/mlops/serving/inference-logger.md	Updates inference logger documentation.
docs/user_guides/mlops/serving/inference-batcher.md	Updates inference batcher documentation.
docs/user_guides/mlops/serving/deployment.md	Updates ModelServing references.
docs/user_guides/mlops/serving/deployment-state.md	Updates deployment state references.
docs/user_guides/mlops/serving/deployment-schema.md	Adds protocol and feature-logging configuration guidance.
docs/user_guides/mlops/serving/autoscaling.md	Updates scaling configuration references.
docs/user_guides/mlops/serving/api-protocol.md	Documents REST/gRPC protocol behavior.
docs/user_guides/mlops/model_monitoring/model_monitoring.md	Updates serving API references.
docs/user_guides/fs/feature_view/feature_logging.md	Documents logging transports, materialization, and monitoring.
docs/user_guides/fs/feature_view/deployment.md	Updates predictor and deployment references.

Review details

Suppressed comments (7)

docs/user_guides/fs/feature_view/feature_logging.md:50

• The switch example hard-codes transport="job", so following it when the current view already uses job cannot switch the view to realtime. Document the destination transport for both directions.

To move a view from one transport to the other, drop its log and recreate the logging group for the new transport with `feature_view.delete_log(transport="job")`.

docs/user_guides/mlops/serving/api-protocol.md:18

• This newly added sentence is wrapped across two lines instead of keeping one sentence per line, contrary to the documentation rule used by this repository. Keep the complete sentence on one line.

A deployment that runs your own predictor script has to stay on REST unless the script is written for gRPC:
under gRPC the model server hands `predict()` KServe v2 tensors rather than rows, which a script written for REST cannot read.

docs/user_guides/mlops/serving/api-protocol.md:66

• This newly added sentence is wrapped across two lines instead of keeping one sentence per line, contrary to the repository's Markdown rule. Keep the complete admonition sentence on one line.

    A gRPC deployment answers no HTTP requests, so `curl` cannot test it and the deployment page shows no curl example
    and no OpenAPI reference for it.

docs/user_guides/mlops/serving/api-protocol.md:9

• This says every non-Python client uses REST, but a non-Python client can use the gRPC endpoint when it implements KServe v2. Only curl and the published OpenAPI document are inherently REST-specific; qualify this sentence so it does not rule out other gRPC clients.

REST is what `curl`, the published OpenAPI document and any client that is not the Python library use.

docs/user_guides/mlops/serving/api-protocol.md:13

• This new sentence implies that a default-predictor deployment exposes both protocols simultaneously, which contradicts the one-protocol restriction below. Reword it to say that either protocol can be selected, and use the heading-reference form for the cross-page link so it remains valid after mike versioning.

is the only client. A deployment served by the [default predictor](deployment-schema.md) supports both protocols,

docs/user_guides/mlops/serving/deployment-schema.md:321

• This new cross-reference uses a relative .md path, which can break when the site is versioned with mike. Use the target heading-ID reference form instead.

How often the rows reach the offline store is a property of the feature view, not the deployment; see [Choosing the Materialization Interval](../../fs/feature_view/feature_logging.md#choosing-the-materialization-interval).

docs/user_guides/mlops/serving/resources.md:111

• This standalone Python block uses Resources on lines 113–114 but imports only TransformerResources, so copying Step 3 raises NameError. Import Resources in this block as well.

  from hsml.deployment.resources import TransformerResources

• Files reviewed: 15/15 changed files
• Comments generated: 4
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[javierdlrm]

Cross-repository review of the feature logging v2 change against the Hopsworks code review checklist and the Confluence spec (revision 5). Pinned to 7d48cfaac. Numbering is shared across the eight PRs of this ticket so cross-repo findings can be followed; the full report with break attempts, spec coverage, cluster-verification list and merge order is with the human reviewer. This review comments only; approval is the human reviewer's call.

Findings in this PR

• 1 (Must Fix): general
• 6 (Must Fix): general
• 27 (Should Fix): general
• 38 (Should Fix (process)): general
• 39 (Should Fix (release plan)): general
• 46 (Nitpick): general

---

Review finding 1 (Must Fix). The Arrow path is advertised only when inferenceLogger.digest is set and the full image reference is listed in inferenceLogger.capabilities (inference_logger_injector.go:220-236). Both default empty in charts/kserve/values.yaml:581,586, and the default tag 5.1.0-SNAPSHOT is mutable so it can never qualify. The shipped default is therefore the v1 legacy row path: the predictor logs sidecar capability absent and runs _log_legacy (feature_logging_arrow.py:331-335), with the 7 ms/request cost and the row loss the spec set out to remove. The spec, ticket and user docs present realtime Arrow as the default and none mentions the digest/capability prerequisite. Either ship the capability for the released tag (or let the webhook trust the chart's own image reference), or make the opt-in explicit in the spec, the docs and the chart README.

Review finding 6 (Must Fix). The branch is conflicting with upstream/main, and upstream added 12 mkdocstrings references (9 distinct) to the flat hsml.* paths in files this PR rewrites to hsml.deployment.*: hsml.scaling_config.PredictorScalingConfig and TransformerScalingConfig, hsml.predictor_state.PredictorState, hsml.predictor.Predictor, hsml.resources.Resources, hsml.transformer.Transformer (autoscaling.md 5, predictor.md 3, one each in deployment-state.md, resources.md, transformer.md, troubleshooting.md). After a plain conflict resolution the strict mkdocs build -s fails on those unresolved references. Re-sweep for hsml\.(scaling_config|resources|predictor_state|predictor|transformer)\. after the rebase.

Review finding 27 (Should Fix). Every serving sample was rewritten to hsml.deployment.*, but no sentence anywhere says the flat hsml.* import paths still work as deprecated shims. A reader on an older client copies from hsml.deployment.resources import ... and gets ModuleNotFoundError; a reader with old code sees deprecation warnings with no pointer to the new paths. One admonition on deployment.md or predictor.md would do.

Review finding 38 (Should Fix (process)). Every PR of this ticket is out of draft with iteration commits still on the branch (two each; hopsworks-ee has four and hopsworks-api three, each including a merge commit), against the team's squash-before-ready rule. PR bodies are stale in places: the helm body says four variables where the diff seeds thirteen; the ee body describes a Retirer class that no longer exists; the JIRA description still says the job transport is not implemented. Five of the eight PRs report no CI checks at all and the webhook PR has no Copilot review. Please squash, refresh the bodies, request Copilot on model-serving-webhook #88, and confirm why CI does not report on the five.

Review finding 39 (Should Fix (release plan)). JIRA fixVersion is 5.1.0, the 5.1 release branches are at 5.1.0-RC1, and every PR targets the SNAPSHOT default branch; the docs repo has no branch-5.1 at all, and packages.md says "pre-5.2 flat paths", which implies main is treated as 5.2. Either the fixVersion is wrong or eight backports are needed, one of them a package move into an RC line. Please decide before merging and state it in the PR bodies.

Review finding 46 (Nitpick). Style: feature_logging.md:84 has two sentences on one line (pre-existing, adjacent to rewritten prose); api-protocol.md:6 and :14-18 explain the gRPC restriction twice.

[jimdowling]

Replies to the general findings, which have no line to anchor to. Every inline finding on this PR has its own reply; this covers 8, 23, 27, 38, 39, 42, 43, 45, 46 and 50 in one place, and the same text is posted on the PRs each applies to.

8 (scope, the package move). Not split. You are right about the size and right that it caused the conflicting state the merge commit repaired by hand, and finding 5 (two @public modules removed with no shim) was a direct consequence. That one is fixed, and the deployment package now exports the names the architecture doc claims for it.

Splitting it now means unpicking it from an eleven-commit upstream merge that re-applied upstream's edits at the moved paths, and the docs PR is rewritten around the new paths. I do not think I should make that call unilaterally, so it is for the human reviewer: either it goes in as is with the move acknowledged in the release notes, or this ticket waits on a separate move PR and everything here rebases onto it.

23 (conflicting with base). Fixed for hopsworks-api (merged, 8106757e6) and logicalclocks.github.io (merged, 026aa0fd4, with the reference sweep finding 6 asked for). hopsworks-helm and docker-images are not rebased yet; your warning about the teleport rows and the 120 s grace template is recorded so whoever does it does not resolve them away.

27 (no note that the flat paths still work). Fixed. The deployment guide carries an admonition saying the serving classes live under hsml.deployment, that the flat paths still work as deprecated aliases and will be removed later, and that a client older than the move has only the flat paths.

38 (process). Partly. The bodies are refreshed and the swapped schedule flags are named in the api body, which they were not. Copilot has been requested on model-serving-webhook #88.

Not squashed. There are now more iteration commits, not fewer, because this review round added one per repo on top of the upstream merges, and squashing would destroy the merge commits that carry the conflict resolutions. Whether to squash before merge is the human reviewer's call, and if the answer is yes it should happen at merge time rather than now.

The missing CI on five repos I cannot explain from here: those are fork PRs and the workflows do not report on them. Someone with repository settings access needs to say whether that is intended.

39 (release plan). Unresolved and I have not touched the fixVersion. It is a release decision: either the fixVersion is wrong or this is eight backports, one of them a package move into an RC line. Flagging rather than guessing.

42, 43, 45, 46, 50 (nitpicks). Fixed: the pytest.raises(Exception) now names ModelServingException (46); the one-line copyright headers are the repo block in hopsworks-api and model-serving-webhook, and the two new docker-images test files agree with each other and their neighbours (50, 48, 49); the Prometheus TYPE ordering (44). Not fixed and recorded as follow-ups: the unbounded failed/<claim>/ directory, the malformed-frame restart counting rows as sent, the non-atomic create_feature_view(logging_transport=), the bare ValueError from zip(strict=True), numpy scalars taking the JSON path, extract_fields_from_json popping the caller's dict, the FeatureLogging.tsx empty state, hideLogo on every embedded dashboard, and the remaining docstring slips.

[jimdowling]

Findings 8 and 39 are both settled now.

8 (scope, the package move): reverted. The hsml.deployment subpackage is gone. The seventeen modules are back at their flat paths, the fifteen shims are deleted, and the only new module is the one the ticket actually introduces, hsml/deployment_logging_config.py, named after its neighbour deployment_tracing_config.py. Imports rewritten across the client, the utils, the loadtests and the documentation.

The PR goes from 11,352 added lines across about 130 files to 6,718 across 60. Both consequences you named go with it: the two @public modules that had been left without a shim are simply themselves again, and the shim mechanism that never surfaced a deprecation anywhere no longer exists, so finding 19 has nothing left to fix either. The architecture note describing the subpackage is withdrawn.

logicalclocks.github.io now differs from upstream/main only in the two pages this ticket needs, the deployment schema and API protocol guides.

Commits: hopsworks-api 15988f386, loadtest 0205fa2, docs 456679c3e. 4,151 unit tests pass, with the same 38 pre-existing Spark and pyspark environment failures as before; ruff, ruff format and check_pep8_public.py clean.

39 (release plan): fixVersion is now 5.2.0. Your reading of the branches was right. hopsworks-api branch-5.1 is at 5.1.0rc1 and hopsworks-ee branch-5.1-ee at 5.1.0-RC1, while main and master are still 5.1.0.dev1 and 5.1.0-SNAPSHOT, so the default branches are the 5.2 line and a change landing there is not in 5.1.0.

Rather than eight backports: this carries a Flyway migration, makes the logging feature group online-enabled, and changes the transport contract between the backend, the webhook and the predictor. None of that belongs in a release candidate. So it is not backported, the fixVersion is 5.2.0, and the reasoning is recorded in a Release section on the ticket. The stale sentence saying the job transport was not yet implemented is corrected there too, and the fail-open decision from finding 15 is stated.

[javierdlrm]

Follow-up review (2026-09-16), after the replies and the new commits. Numbering continues across the eight PRs: E = backend, W = webhook, F = front/docker/docs, L = loadtest, A = api, N = commits pushed after the follow-up pass. Each reply claim from the previous round was re-checked against the code at this head; the per-thread outcome is in a reply on that thread. Every finding below was independently re-verified by a second reviewer (Codex) before posting.

All findings are inline.

[javierdlrm]

Follow-up finding F1 (Must Fix). test-docs-build is red at 456679c3e because of this PR: the added heading id { #api-protocol } collides with two existing anchors (user_guides/mlops/serving/#api-protocol and user_guides/mlops/serving/predictor/#api-protocol), and mkdocs strict mode aborts on the autorefs warning "Multiple primary URLs found for 'api-protocol'". Rename the id (for example { #api-protocol-guide }) and update the [api-protocol] reference in deployment-schema.md:76. Everything else in the docs delta checks out: the custom predictor section survives (deployment-schema.md:272), feature_view.feature_logging.transport exists on the client, and the flat-path admonition is gone with the revert, so findings 6 and 27 are moot.

[jimdowling]

Fixed in e014d3a18, and there was a second abort behind the first.

The anchor collision. The guide's heading id is now { #api-protocol-guide } and the one cross-reference to it, deployment-schema.md:75, follows. The collision was with predictor.md:573, whose ## API protocol heading generates #api-protocol implicitly; the third URL you saw, user_guides/mlops/serving/#api-protocol, is the section index picking up the same heading. Only one { #api-protocol id exists in docs/ now.

What that uncovered. mkdocs strict aborts on the first warning, so with the collision gone the build failed again on three cross-references in delta_maintenance.md: hsfs.feature_group.FeatureGroup.delta_optimize, .delta_checkpoint and .delta_cleanup_metadata. They cannot resolve, and not only locally: mkdocs-test.yml:19 checks the client out at ${{ github.base_ref }}, so the build resolves against main whatever this branch adds to the client. Those three are rendered as plain code until a release ships them, which is what docs#648 did for the same reason. delta_vacuum is in the released client and keeps its link. The guide now says why, so it does not read as an oversight.

mkdocs build -s is clean at e014d3a18 apart from the javadoc nav warning, which is local only: mkdocs-test.yml:98 generates the Javadoc into docs/javadoc before building. hopsworks-docs snakeoil and markdownlint both pass.

[jimdowling]

Review round (2026-09-17). Two commits from this round of the cross-repo review.

7c76fd9c0 renames the async lookup opt-out to SERVING_PREDICTOR_ASYNC_LOOKUP. The guide told the reader to set a HOPSWORKS_-prefixed name, which the backend reserves, so the opt-out was reachable by nobody.

69ccbcc9c says how many worker processes a deployment that logs features serves on and why, and corrects three rows of the environment variable table: the three unprefixed logging limits are reserved now, so they are set through DeploymentLoggingConfig; the async lookup opt-out and KSERVE_WORKERS were missing; and the sentence reserving SERVING_* wholesale stopped being true when the feature logging marker was left editable.

Verified: markdownlint-cli2 clean across all 244 docs.