Shaping the Future Together: Share Your Wishlist for Addon-Controller Features

[gianlucam76]

Hi Sveltos community!

We're constantly striving to improve Sveltos and deliver the best possible experience for managing your Kubernetes add-ons across multiple clusters. Your feedback is invaluable in guiding our development efforts, and we're eager to hear directly from you.

What we're asking:

• What features are most important to you at this stage?
• Are there any specific pain points you'd like to see addressed?
• Do you have any innovative ideas for enhancing Sveltos's capabilities?

Feel free to share any thoughts, suggestions, or feature requests you may have. We encourage detailed descriptions, use cases, and even mockups if you have them!

[oliverbaehler]

I would wish for a strong integration with FluxCD. In my environments ii have everything with fluxCD iand i would like to leverage the capabilities provided by the kustomize-controller and helm-controller.

Btw. in vcluster scenarios, can you deploy within the cluster and in the cluster namespace? That's a common one when staging vclusters.

[gianlucam76]

Thank you @oliverbaehler
Can you give me a concrete example of integration with the Kustomize controller for instance?

So far Sveltos integrates with Flux by using as solution for gitops approach. Meaning Flux sync from git repository, then Sveltos takes it from there and deploy using Kustomize sdk (and so replacing the Flux Kustomize controller). But I would love to hear more uses cases on how sveltos can integrate with Flux.

Regarding the vcluster, I will try it out tomorrow.

Thank you again!

[oliverbaehler]

for me, the dependency management is difficult. Meaning the order of which helm charts are deployed matters. (helmRelease has a depends_on property). I need a property to ignore the changes on sync, because they are manipulated (helmreleases have that property as well).

For the vcluster scenario, how would you build that? It's mainly relevant for the CNI which is going to be running on the cluster. Because otherwise if you eg. deploy helmreleases to the vcluster, they can't be executed. But thinking about it, it's anyway different with sveltos 🤔 nvm

btw can you register clusters in a kubernetes native way?

[gianlucam76]

Thank you @oliverbaehler.

With respect to order, I agree that it matters. And Sveltos honours that in different ways:

When you create a ClusterProfile/Profile and you list multiple helm charts those are deployed in the same exact order. For instance creating this ClusterProfile, in every matching cluster, grafana will be deployed only after prometheus is correctly deployed.

apiVersion: config.projectsveltos.io/v1alpha1
kind: ClusterProfile
metadata:
  name: prometheus-grafana
spec:
  clusterSelector: env=fv
  helmCharts:
  - repositoryURL:    https://prometheus-community.github.io/helm-charts
    repositoryName:   prometheus-community
    chartName:        prometheus-community/prometheus
    chartVersion:     23.4.0
    releaseName:      prometheus
    releaseNamespace: prometheus
    helmChartAction:  Install
  - repositoryURL:    https://grafana.github.io/helm-charts
    repositoryName:   grafana
    chartName:        grafana/grafana
    chartVersion:     6.58.9
    releaseName:      grafana
    releaseNamespace: grafana
    helmChartAction:  Install

ClusterProfiles support dependency management. A ClusterProfile can specify another ClusterProfile (e.g., "kyverno") as a dependency. The deployment of the dependent ClusterProfile will only begin after the successful deployment of all add-ons and applications listed in the dependency.

Here is an example

  apiVersion: config.projectsveltos.io/v1alpha1
  kind: ClusterProfile
  metadata:
    name: kyverno-policies
  spec:
    clusterSelector: env=fv
    dependsOn:
    - kyverno
    policyRefs:
    - deploymentType: Remote
      kind: ConfigMap
      name: disallow-latest-tag
      namespace: default
      kind: ConfigMap
      name: restrict-wildcard-verbs
      namespace: default

With respect to ignore changes on sync, that is a good idea. And I guess you want this per field (so for instance you deploy a deployment but the replicas is managed by Autoscaler and you don't want the replicas to be re-written on sync right)?

[oliverbaehler]

@gianlucam76 Thanks for the response. You have convinced me enough so i will be implementing sveltos for our Solution. I will surely have follow-up proposals. 🤔

[gianlucam76]

Looking forward to your proposals! Those will help making sure Sveltos tackles real problems with a correct approach.

[oliverbaehler]

Just to clarify why I am "Misusing" this issue for my onboarding process. We are automating against a non-CAPI environment and there might be some interesting situations which arise from that, which were not yet considered. Let me know if I should move this to slack, but might be interesting for other users.

First Question/Proposal: How do you solve the case, where you have a new cluster registered, let's say this eg.

apiVersion: v1
items:
- apiVersion: lib.projectsveltos.io/v1alpha1
  kind: SveltosCluster
  metadata:
    creationTimestamp: "2024-06-12T09:13:55Z"
    generation: 1
    labels:
      capsule.clastix.io/tenant: xmodels
      env: production
      is-credential-preset: "true"
      overwrite: this
      project-id: xmodels
    name: pdl6npm7qp
    namespace: cluster-pdl6npm7qp
    spec: {}

Now I have seen here (https://projectsveltos.github.io/sveltos/template/template/ - Variables) that you have access to a lot of CAPI attributes. In our case we don't have access but I still need to configure my charts with certain cluster properties.

So there needs to be a way to add more data to sveltosclusters, which are non CAPI provided. A simple approach, adding a extra context field to sveltosclusters, where I can pass my additional information (essentially an interface of values):

apiVersion: lib.projectsveltos.io/v1alpha1
kind: SveltosCluster
metadata:
  creationTimestamp: "2024-06-12T09:13:55Z"
  generation: 1
  labels:
    capsule.clastix.io/tenant: xmodels
    env: production
    is-credential-preset: "true"
    overwrite: this
    project-id: xmodels
  name: pdl6npm7qp
  namespace: cluster-pdl6npm7qp
  uid: 02f9f1d7-b93a-4944-a449-d77c3bb3672d
spec:
  context:
    address:
      ip: 10.123.11.5
      port: 6443
      url: https://pdl6npm7qp.some.cloud:6443
    versions:
      apiserver: 1.28.9
      controlPlane: 1.28.9
      controllerManager: 1.28.9
      oldestNodeVersion: 1.28.9
      scheduler: 1.28.9

Maybe something as generic as might also be interesting:

---
apiVersion: config.projectsveltos.io/v1alpha1
kind: ClusterProfile
metadata:
  name: deploy-resources
spec:
  clusterSelector: env=fv
  templateResourceRefs:
  - resource:
      apiVersion: kubermatic.k8c.io/v1
      kind: Cluster
      name: "{{.Cluster.metadata.name}}"
    identifier: "KubermaticCluster"
  policyRefs:
  - deploymentType: Remote
    kind: ConfigMap
    name: sveltos
    namespace: default

Altough that might be different features. This would aim towards having more flexible template sources on cluster basis.

Or am I just missing a trick? :D

[gianlucam76]

Thanks @oliverbaehler. This space is good for such design discussion, so please keep using it.

One comment regarding the templateResourceRefs section

• namespace can be set or left empty (as in your example). If empty the cluster namespace will be used.
• namespace and name can be defined as template and currently use ClusterNamespace and ClusterName (so not .Cluster.metadata.name though I feel it is better to change that and make it consistent with rest of templating). This is needed in case where for instance I need to fetch a cluster wide resource and there is one resource for managed cluster. (please let me know your opinion on this).

  templateResourceRefs:
  - resource:
      apiVersion: kubermatic.k8c.io/v1
      kind: Cluster
      name: "{{.ClusterName}}"

With respect to allow adding extra information to SveltosCluster I am OK with that. If information is static I think it is useful (though it would be good to extend sveltosctl register cluster command so there is no need of multiple steps when registering a cluster.

If the information is instead dynamic (like anything that depends on Kubernetes version and so can change if cluster is upgraded), if it is possible to get this information in the managed cluster we could use sveltos event framework. So Sveltos will collect from managed cluster and report back to the management cluster. And if that changes, Sveltos automatically updates it (and templating automatic react to such change already). So we have no stale information nor we require user to manually change it (which user might forget to do).

In other words, I am fine adding a generic field in SveltosCluster Spec (maybe a map[string]string) but also present in a way that let user know to add static information there and use sveltos event framework whenever possible for dynamic information.

What do you think?

[vgromanov]

Hi there!
Been playing with sveltos for couple of days, this seems to be a product with great idea behind it and great potential ahead. But (always some "but") it lacks some features that are considered as "must have" by those who's been in gitops world for quite a while, particulary with FluxCD.

1. Support for kustomize components is absolutely necessary, for those who's infrastructure manifests is heavily multilayered and some parts are re-used often
2. Also FluxCD has a postBuild.substitute feature and that would come in handy for a smooth transition.
3. In clusterReports I personally would like to see a complete BOM, not just GVKs. Once again I refer to FluxCD and it's inventory in kustomization status.
4. My personal hell with sveltos happens when it comes to troubleshooting - i constantly get absolutely meaningless error message like 'incorrect key at line XYZ' which are extremely hard to work with. So at least I should be able to use CLI to render complete set of manifests and then pass them through some linters to find the root cause, I'd be quite happy with that. But preferably a more humane messages with resource names and faulty lines would be extremely nice and useful.
5. I wasn't able to use my local chart repo, my best guess due to lack of passCredentials option in repo configuration. Not a big fan of helm though, I'm more like kustomize guy, but sometimes there's joust no other way. As a solution you might just re-use FluxCDs HelmRepository/HelmChart.
6. Instead of validateHealth the CEL rules might be an alternative.

UPD.
7. Also inline manifests for policies would be nice - eliminates the need to create additional resources for some small changes.
8. A history limit for helm releases, because helm tends to pollute clusters with it's heavy secrets.

[gianlucam76]

Hi @vgromanov first of all thank you for the feedbacks. Really appreciate those (and where to improve is what I personally love the most).

I have few questions for me to understand better.

Point 1, Sveltos does support kustomize. Sveltos though does rely on FluxCD to get the content from a Git repo (so it needs the Flux Source controller, but just that). That is mainly because we did not want to re-invent the wheel (FluxCD solves nicely GitOps, Sveltos aims to be extend for multi clusters). And for each release there is an extra addon-controller image (named addon-controller-git) which contains git and so supports kustomization that points to a oci/git repo that utilizes remote kustomization references (this is not documented but has been available for quite some time).

Regarding point 2, Sveltos has post render patches which allows you to substitute any value or add/remove some.
Also for Kustomize Sveltos offers this also covered here in the documentation.
Wouldn't this solve your point 2?

Regarding point 3, to see complete list of deployed resources you can check the ClusterConfiguration. There is one ClusterConfiguration instance per cluster and it contains full list of all deployed resources and helm charts. This is what is used by sveltosctl show addon command and by the dashboard to display what s deployed by Sveltos on each managed cluster.

Point 5, isn't this enough? Likely I am missing something.

Regarding point 8, this is likely what you are looking for.

Point 4 unfortunately those errors are what go templating returns. Not sure how to improve there. If you have any idea, happy to explore it.

can you share an example for your point 7? That is something I have been wanting to add for some time but so far nobody else requested it so had no chance to discuss with anyone yet.

Thank you so much again.

[vgromanov]

Hi @gianlucam76!

I'm sorry, I should have thrown in some backstory, without it my comment lacks the important context.

We have a couple of hundred clusters, about 400-ish last time I checked, all of them are CAPI based, the clusters are split between different environments, datacenters, networks and etc., e.g. we need to maintain different component profiles. Currently we manage these profiles via an intricate combination of git branching/tagging/submodules and FluxCD as a gitops instrument. And at some point not too far in the past the whole system became too complicated - too many variations, too many moving parts, too many places where something can go wrong and it often did. So the Sveltos seems to be a perfect solution for us, mostly because it has a builtin cluster profiles based on selectors.

So, done with preamble, let's get to the points. )

Considering p.1, speaking of kustomize components I was referring to this particular feature. In our case it allows to have an out-of-tree sets of patches/resources that we can combine and apply as we need, using something like this:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
# ...
spec:
  path: /profiles/capi/baseline
  components:
    - /profiles/flavors/<purpose-specific>
    - /profiles/topologies/dc-spread
    - /profiles/envs/devtest

Using just kustomize alone we would have to cover all possible variations with different overlays, but kustomize components add great flexibility. And it would be simply great to be able to do the same with Sveltos like:

spec:
  kustomizationRefs:
    - name: profiles-bundle
      path: /profiles/capi/baseline
      components:
      - /profiles/flavors/<purpose-specific>
      - /profiles/topologies/dc-spread
      - /profiles/envs/devtest

As for p.2 - yes, all of the solutions you've mentioned do the job just fine, but as I said, we currently using FluxCD and It's postBuild.substitute mechanics, which invokes envsubst as a final rendering step. Which means that we have several hundred manifests with ${variableName} references in it and to use Sveltos in full we need to convert them go templates. Apparently this is in no way a show-stopper, but having this feature could help in transition to Sveltos not only for us but also for those who have similar struggles.

P.3 - yes! Thank you! That's exactly what I've been looking for. However the ClusterConfiguration doesn't seem to include resources that are deployed locally in the management cluster which sometimes might be far more important.

As for the most painful p.4 I think it might be somewhat mitigated via CLI with a new command, let's say, render profile with an option to render a profile, that already exists in mgmt cluster, or accept a file as an input. The flow can look like this:

• helmCharts section
  • Render values
  • Render valuesFrom
  • Pull chart using repo url and credentials or use
  • Invoke helm template with previously rendered values and valuesFrom
  • Interrupt on error with verbose output that would help to locate it’s cause
  • If everything is ok then store rendered manifest for later use
• kustomizeRefs section:
  • Render values
  • Render valuesFrom
  • Pull kustomization sources
  • Invoke kustomize build
  • Render resulting manifests one resource at a time to catch any templating errors
  • Same as with helm - interrupt on error
  • Otherwise store the resulting manifests
• policyRefs - pretty much the same
• Final steps with patches
• If no errors encountered then output all stored manifests for external linting via kubeconform or any other tool

The main disadvantage I see that rendering process by design cannot be done purely offline just because it relies on existing kubernetes resources. Also it should be tightly coupled with the addon-controller or even use it to offload some of the tasks. All of that might turn out quite tricky.

P.5 - I've tried all possible combinations I could think of. In my case the root cause of the problem, supposedly, that repo server returns chart URLs with hostname part that differs from the initial one. As I understand passCredentials in FluxCD (--pass-credentials in helm CLI) where specifically designed to address such situation.

P.7 - the most common example can be made with helm - since it supports namespace creation, but does not allow to assign any metadata to it (unless it's not incorporated into chart, which is quite rare). So the common scenario is to use separate kustomization that creates a namespace and populates its metadata, then the helm deploys stuff into said namespace.
Yes, it can be done with patches but assigning a bunch of labels and annotations looks a bit messy. With inline policyRefs it might look like this:

spec:
  policyRefs:
    - inline: |
      ---
      apiVersion: v1
      kind: Namespace:
      metadata:
        name: some-package
        labels:
          prometheus: some-instance
          alerting: some-target

Another scenario is when you need some minor additions to base package and you don't want to keep them separately in configmaps/secrets, but rather have them right in the profile for better visibility. For example when you want to throw in in some extra Cluster/RoleBindings or anything of this sort.

And the last p.8 - thanks! Somehow I just missed it.

Best regards!

[gianlucam76]

Thank you so much @vgromanov. I would love to help.

point 5

if I understood it correctly it translates into "--pass-credentials pass credentials to all domains" helm option. If so, I added to Sveltos Helm chart options
Here is the PR

point 2

already works. Just tested it. I created this repo that uses components

then posted this ClusterProfile

apiVersion: config.projectsveltos.io/v1beta1
kind: ClusterProfile
metadata:
  name: component-demo
spec:
  clusterSelector:
    matchLabels:
      env: fv
  syncMode: Continuous
  kustomizationRefs:
  - namespace: flux-system
    name: flux-system
    kind: GitRepository
    path: ./components/overlays/community/
    targetNamespace: eng

note path is pointing to components/overlays/community/

the outcome is

sveltosctl show addons
+-----------------------------+-----------------+-----------+----------------------------------+---------+--------------------------------+-------------------------------+
|           CLUSTER           |  RESOURCE TYPE  | NAMESPACE |               NAME               | VERSION |              TIME              |           PROFILES            |
+-----------------------------+-----------------+-----------+----------------------------------+---------+--------------------------------+-------------------------------+
| default/clusterapi-workload | apps:Deployment | eng       | my-app                           | N/A     | 2025-09-01 21:24:25 +0200 CEST | ClusterProfile/component-demo |
| default/clusterapi-workload | :ConfigMap      | eng       | db-settings                      | N/A     | 2025-09-01 21:24:25 +0200 CEST | ClusterProfile/component-demo |
| default/clusterapi-workload | :ConfigMap      | eng       | db-config-h4gkbf4m6k             | N/A     | 2025-09-01 21:24:25 +0200 CEST | ClusterProfile/component-demo |
| default/clusterapi-workload | :Secret         | eng       | recaptcha-config                 | N/A     | 2025-09-01 21:24:25 +0200 CEST | ClusterProfile/component-demo |
| default/clusterapi-workload | :Secret         | eng       | recaptcha-credentials-6kdtt8h4h4 | N/A     | 2025-09-01 21:24:25 +0200 CEST | ClusterProfile/component-demo |
+-----------------------------+-----------------+-----------+----------------------------------+---------+--------------------------------+-------------------------------+

The Flux GitRepository

  apiVersion: source.toolkit.fluxcd.io/v1
  kind: GitRepository
  metadata:
    creationTimestamp: "2025-09-01T19:16:34Z"
    finalizers:
    - finalizers.fluxcd.io
    generation: 1
    name: flux-system
    namespace: flux-system
    resourceVersion: "4745"
    uid: c0b77c25-88f0-482f-9a74-2c5c6205eb4b
  spec:
    interval: 1m0s
    ref:
      branch: main
    secretRef:
      name: flux-system
    timeout: 60s
    url: ssh://git@github.com/gianlucam76/kustomize

point 7

we can add PolicyRefs inline option. Originally I did not add it because I thought having it would result in very long ClusterProfiles. But I understand in certain scenarios like the one you describe, it is easier to keep it as part of the ClusterProfile (easier to read and easier to maintain). That can be definitely done. I will add it as part of next Sveltos release.

Regarding point 4, I believe some other Sveltos user implemented something similar. Will figure out and share here.

PS: if possible, please consider adding to adopters list (simple PR here is all that is needed)