fix(argocd-pr-env-deploy): reach ArgoCD over the in-cluster Service (ENG-2079)

[lucas-koontz]

User story

As an engineer with a deploy-labelled PR, I want the environment rollout to reach ArgoCD without depending on ingress or DNS state, so that an ops-network cutover cannot take every repo's PR environments down with it.

What is broken today

rpc error: code = Unknown desc = POST https://argo.dev.mindsdb.com/application.ApplicationService/Get failed with status code 404

That is an HTTP 404 wrapped as gRPC Unknown, not an ArgoCD NotFound, so it matches none of the tolerated states in the app set retry loop and fails the job outright.

The internal-ops cutover moved argo.dev.mindsdb.com to the private ingress class before its private DNS record existed. The name still resolves through the *.dev.mindsdb.com wildcard to the public ALB, whose controller no longer reconciles that Ingress and returns its default 404. Confirmed:

$ dig +short argo.dev.mindsdb.com
dualstack.k8s-infrastr-nginxing-8c3d4cc805-641080428.us-east-1.elb.amazonaws.com.

That is the public ALB.

The change

flowchart LR
    R["deploy-pr-env<br/>on mdb-dev<br/>(pod in newdev)"]
    R -->|"before"| DNS["argo.dev.mindsdb.com"]
    DNS --> PUB["public ALB"] --> NG["public nginx"] --> X["404"]
    R -->|"after"| SVC["argo-cd-argocd-server<br/>.argocd.svc.cluster.local:80"]
    SVC --> A["argocd-server"]
    style X fill:#c0392b,color:#fff
    style A fill:#1e8449,color:#fff

Loading

Every caller of this action runs on mdb-dev, a runner pod in the same cluster as ArgoCD, so the public hostname was never the shortest path. The Service route depends on no ingress, no ALB and no DNS record, so it is correct both now and after the cutover finishes.

argocd-plaintext is new and defaults to auto. argocd-server runs with server.insecure: true, serving cleartext on Service port 80 and expecting TLS to terminate at the ingress, so --plaintext is required for the Service and wrong for the hostname. Deriving it from argocd-server means the pair cannot drift; true/false still override.

Acceptance criteria

• Default endpoint is the in-cluster Service
• Transport derives from the endpoint, with an explicit override
• An invalid argocd-plaintext fails loudly rather than silently
• All five argocd call sites share one resolved flag array
• The public hostname still works when passed explicitly
• No caller repo needs a change

Verified locally

Check	Result
pytest tests/ -q	148 passed
actionlint (as CI runs it)	clean
scripts/workflow_graph.py --allow-external-reusables	12 workflows, all compose
helm template Service name and port	argo-cd-argocd-server, ns argocd, port 80 → 8080
Transport resolution across 9 endpoint shapes	table below
Invalid value bogus	rejected with ::error::

argo-cd-argocd-server.argocd.svc.cluster.local     auto   -> plaintext=true
argo-cd-argocd-server.argocd.svc.cluster.local:80  auto   -> plaintext=true
argo-cd-argocd-server.argocd.svc                   auto   -> plaintext=true
argo.dev.mindsdb.com                               auto   -> plaintext=false
localhost:8080                                     auto   -> plaintext=true
127.0.0.1:8080                                     auto   -> plaintext=true
argo.dev.mindsdb.com                               true   -> plaintext=true
argo-cd-argocd-server.argocd.svc.cluster.local     false  -> plaintext=false
argo.dev.mindsdb.com                               bogus  -> rejected

The Service name is render-verified, not guessed: the subchart sets nameOverride: argocd, so the fullname is argo-cd-argocd, and argo-cd-server... would not resolve.

Reviewer notes

• This is a workaround for a missing DNS record, and it should be read as one. The end state is still the internal-ops cutover: populate internal_ops_dns_cutover.newdev_alb and add argo.dev.mindsdb.com to hosts in the terraform repo. That is blocked on the internal ALB existing (terraform#188 is still open) and on AWS metadata I cannot read. Once it lands, this change stays correct but stops being load-bearing.
• Honest tradeoff: the ci-pr-envs JWT now crosses the pod network in cleartext instead of inside TLS, and NetworkPolicy enforcement is off in this cluster (--enable-network-policy=false). The internal ALB's allowlist already admits every pod in the cluster, so no control that was constraining this caller is removed, but the token is no longer wrapped. Say so if you want TLS kept and I will instead route through the ingress and gate this PR on the DNS apply.
• Second tradeoff: CI and browsers no longer share a route. The only reason this outage surfaced quickly is that they did. After this, an ingress or DNS regression can leave CI green while the UI 404s. KFS#133 adds the class assertion that covers the ingress half of that.
• Prod callers are unaffected: publish.yml and prod-build-deploy.yml set build-runner: mdb-prod but leave pr-environment false, so deploy-pr-env never runs there.
• bash -n on the extracted script fails identically on origin/main and on this branch. That is macOS bash 3.2, which cannot parse a pre-existing case in the Deploys: parser; runners use bash 5.
• This repo is public. The added default is an in-cluster Service name, unreachable from outside, and the file already named the public hostname and the ECR naming scheme.

Ships with

• mindsdb/Kubernetes-Foundational-Services#133 — binds the argo Ingress to the internal class by spec and adds the assertion that would have caught this