Repository navigation
Read HDFS natively in get balance, drop the namenode pod exec - #8
Merged
fjammes merged 3 commits intoSep 30, 2026
Conversation
get balance read HDFS by exec-ing hdfs dfs into the Stackable namenode pod. With an HDFS outside the cluster (fink-broker#1241) there is no such pod. The HDFS reads (list, count, cat) now go through an hdfsReader interface: - podReader keeps the exec into the namenode pod, still the default, used by CI and when finkctl runs outside the cluster; - nativeReader, selected by --namenode host:port[,host:port], talks to the NameNode RPC port and the DataNodes with github.com/colinmarc/hdfs, a pure Go client: the image stays a static binary. DataNodes are reached by IP, as their host names often do not resolve outside the HDFS cluster, and every dial is bounded so an unreachable HDFS fails instead of hanging. --hdfs-user defaults to $HADOOP_USER_NAME, else 185 like the Spark containers. The reporter logic is unchanged and now unit tested against an in-memory reader.
On a fresh deployment, or any HDFS no run has written to yet, <prefix>/raw does not exist: hdfs dfs -ls failed and get balance exited in error. The daily report CronJob of fink-broker then failed every time, and kept restarting until its backoffLimit. A missing path now reads as os.ErrNotExist in both readers (the pod reader recognises the "No such file or directory" message of hdfs dfs, the native client already returns it), and a missing raw dataset lists no night: the command says so and succeeds. Any other failure to list it, such as a denied exec or an unreachable NameNode, still fails the report.
get balance read HDFS either by exec-ing hdfs dfs into the Stackable namenode pod or, with --namenode, through the native client. The exec path needed pods/exec rights in the hdfs namespace, tied the command to the Stackable pod names and could not work with an HDFS outside the cluster. --namenode is now required and the native client is the only reader. The exec reader, the parsing of the hdfs dfs text output and the HDFS pod constants are removed. The command must run where the NameNode and the DataNodes are reachable, typically in a pod of the cluster.
fjammes
added a commit
to astrolabsoftware/fink-broker
that referenced
this pull request
Sep 30, 2026
v3.1.3-rc6 reads HDFS natively (--namenode) and reports an empty balance before the first run instead of failing. It replaces the development image the report was pinned on while astrolabsoftware/finkctl#8 was under review.
fjammes
added a commit
to astrolabsoftware/fink-broker
that referenced
this pull request
Oct 5, 2026
v3.1.3-rc6 reads HDFS natively (--namenode) and reports an empty balance before the first run instead of failing. It replaces the development image the report was pinned on while astrolabsoftware/finkctl#8 was under review.
fjammes
added a commit
to astrolabsoftware/fink-broker
that referenced
this pull request
Oct 6, 2026
v3.1.3-rc6 reads HDFS natively (--namenode) and reports an empty balance before the first run instead of failing. It replaces the development image the report was pinned on while astrolabsoftware/finkctl#8 was under review.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
finkctl get balancenow reads HDFS with a native Go client (github.com/colinmarc/hdfs/v2) instead of exec-inghdfs dfsinto the Stackable namenode pod. Part of astrolabsoftware/fink-broker#1241, where the broker moves to an HDFS outside Kubernetes and there is no namenode pod to exec into.--namenode host:port[,host:port]is required. List every namenode of an HA pair: the client follows the active one.--hdfs-userdefaults to$HADOOP_USER_NAME, else185.<prefix>/rawdoes not exist: the command now printsNo observing night under <prefix>/raw yetand succeeds. Before, the fink-broker report CronJob failed on every fresh deployment.hdfs dfstext output and the HDFS pod constants are removed. The image stays a static binary on distroless.Kafka is still read by exec into the broker pod; see #7.
Validation
go vet,go test ./...fink-reportServiceAccount:diffempty), 281 s vs 512 s;No observing night under /user/185/raw yet, rc=0 (the exec version failed there);Follow-up in fink-broker
Pin the released image in
chart/values.yaml(report.image.tag) and in.ciux. The chart now passes--namenodeand no longer grants the report any right in thehdfsnamespace.🤖 Generated with Claude Code