docs(agent): document trusting a private CA for the collector and relay - #850
mayankpande88 wants to merge 2 commits into
Conversation
Self-hosted installs behind an internal CA fail with 'x509: certificate signed by unknown authority' on every discovery POST while the runner pod stays Running and healthy, so the only visible symptom is a cluster that never reports data. Nothing in the docs said how to fix it. Document the CA mount plus SSL_CERT_DIR in helm_values, add the error to the installation troubleshooting reference, and point the on-prem page at it. Notes cover why SSL_CERT_FILE is the wrong knob (it replaces the trust store instead of extending it) and that additional_env_vars replaces the chart's list, so the ClickHouse variables must be restated.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request adds documentation for configuring the nudgebee-agent runner when the collector or relay is behind a private CA, including troubleshooting steps and Helm values configuration. The review feedback suggests improving the clarity of the documentation by rewording an awkward sentence about disabling certificate verification and providing a more detailed explanation of why SSL_CERT_DIR is required in Go environments to preserve the default system trust store.
| ``` | ||
|
|
||
| **Resolution:** | ||
| Mount your CA into the runner and set `SSL_CERT_DIR`. The full recipe is in [Collector or relay behind a private CA](../operate/helm_values.md#collector-or-relay-behind-a-private-ca). Verification cannot be disabled instead. |
There was a problem hiding this comment.
The phrase 'Verification cannot be disabled instead' is slightly awkward. It would be clearer to state that disabling certificate verification is not supported. For example:
Mount your CA into the runner and set SSL_CERT_DIR. The full recipe is in [Collector or relay behind a private CA](../operate/helm_values.md#collector-or-relay-behind-a-private-ca). Note that disabling certificate verification is not supported.
There was a problem hiding this comment.
Taken — reworded to "Disabling certificate verification is not supported." in 5f6554a.
|
|
||
| Notes: | ||
|
|
||
| - Use `SSL_CERT_DIR` and keep `/etc/ssl/certs` in the list. `SSL_CERT_FILE` **replaces** the trust store rather than adding to it, which drops the public roots the agent still needs for every other HTTPS target. |
There was a problem hiding this comment.
In Go, setting either SSL_CERT_FILE or SSL_CERT_DIR bypasses the loading of default system trust stores entirely. Since SSL_CERT_FILE only supports a single file path, using SSL_CERT_DIR with a colon-separated list is the only way to load both the default system certificates (from /etc/ssl/certs) and your custom CA certificates. Clarifying this helps users understand why SSL_CERT_DIR is required here. Consider updating the note to:
- Use SSL_CERT_DIR and keep /etc/ssl/certs in the list. In Go, setting either SSL_CERT_FILE or SSL_CERT_DIR bypasses the loading of default system trust stores. Since SSL_CERT_FILE only supports a single file path, using SSL_CERT_DIR with a colon-separated list is the only way to load both the default system certificates (from /etc/ssl/certs) and your custom CA certificates.
There was a problem hiding this comment.
Half right, so I took the direction but not the wording.
Setting SSL_CERT_DIR does not bypass the default trust store "entirely" — it overrides only the directory list. The default file list (/etc/ssl/certs/ca-certificates.crt, /etc/ssl/cert.pem, …) is still read as long as SSL_CERT_FILE is unset, per crypto/x509/root_unix.go, where the file loop and the directory loop are independent.
Rewritten in 5f6554a to state the mechanism that is actually load-bearing: Go splits SSL_CERT_DIR on : and reads every directory listed, so keeping /etc/ssl/certs in the list loads the system roots alongside the private CA. Also softened the SSL_CERT_FILE claim — it replaces the default bundle with a single path, rather than definitively dropping the public roots.
…ding State the mechanism rather than a blanket claim: Go splits SSL_CERT_DIR on ':' and reads every listed directory, so keeping /etc/ssl/certs in the list loads the system roots alongside the private CA. SSL_CERT_FILE takes one path and replaces the default bundle; whether public roots survive that then depends on what the base image leaves loose in /etc/ssl/certs, which is not something to rely on.
1 participant
Problem
Self-hosted installs where the collector/relay Ingress uses an internally-issued certificate fail on every outbound call:
The runner pod stays
Runningand its probes pass, so the only visible symptom is a cluster that never reports data. Nothing in the docs covered this.Changes
operate/helm_values.md— new "Collector or relay behind a private CA" section: create the ConfigMap, mount it viarunner.extraVolumes/extraVolumeMounts, setSSL_CERT_DIR. One setting covers discovery POSTs, the relay WebSocket, task polling and config refresh, since every one of those clients uses the Go default root pool.installation/index.md— thex509error in the diagnostic quick reference, plus an in-depth scenario with thegrep x509check.onprem-setup.md— short TLS pointer, since self-hosted is where this happens.Notes captured in the doc
SSL_CERT_DIR, notSSL_CERT_FILE: the latter replaces the trust store rather than extending it, dropping the public roots the agent needs elsewhere.additional_env_varsreplaces the chart's list rather than merging, so the threeCLICKHOUSE_*entries have to be restated alongsideSSL_CERT_DIR— omitting them breaks the ClickHouse connection.runner.es.sslVerifylooks related but only applies to Elasticsearch.