Add pod-level TLS termination for Keystone and Barbican - 007 WP-4 by bbobrov · Pull Request #12641 · sapcc/helm-charts

bbobrov · 2026-08-24T14:20:17Z

Terminates TLS inside the API pod with Apache (mod_ssl) instead of at the
ingress. The API is exposed on port 443 through a LoadBalancer Service that
pins a configured public IP, with a matching DNS A record. All behavior is
gated behind tls.enabled and is off by default, so existing deployments
render unchanged.

Keystone chart: 0.14.0
Barbican chart: 0.10.0

Changes

Terminate TLS in Apache via read-only conf (mod_ssl, no runtime a2enmod),
with TLS 1.2/1.3 cipher hardening included outside the VirtualHost and the
packaged default site dropped.
Add HTTP security response headers (HSTS, CSP, X-Frame-Options,
X-Content-Type-Options, Referrer-Policy) on the TLS endpoint, and stop
trusting client-supplied Forwarded/X-Forwarded-* and trust-key headers
when the pod terminates TLS directly.
Keystone only: reconstruct the x509 client-certificate headers (cc_x509)
from the verified TLS session and mount the client-cert trust anchor, so
client-certificate auth keeps working in the pod.
Keystone only: store the OIDC federation session in the client cookie so it
works across multiple replicas.
Expose the API through a type: LoadBalancer Service on port 443 that pins a
configured public IP (via spec.externalIPs by default, or the Calico
loadBalancerIPs annotation when calicoLoadBalancer: true), plus a DNS A
record. The public ingress renders only when TLS is disabled.
Manage certificates with a bootstrap Job and a weekly rotation CronJob backed
by cert-manager (issuer kind configurable, defaults to ClusterIssuer), with
TLS-manager RBAC as a pre-install hook and fsGroup so the non-root tls-init
container can read the Secret.
Require explicit key generation/wrapping/storage choices when TLS is enabled,
validated in-chart, with an allowInsecureStorage escape hatch for
non-production.
Add an optional blackbox endpoint Probe (off by default) and Prometheus alerts
for certificate expiry, stale rotation, and rotation failures.

Upgrade note

Keystone now stores the OIDC federation session in the client cookie so it works
across multiple replicas. On upgrade, existing federated users re-authenticate
once. Operators with unusually large OIDC claim sets should confirm the ingress
large_client_header_buffers is adequate.

rajivmucheli · 2026-08-24T18:17:24Z

just realised, should we also split the PR's per service ?

bbobrov · 2026-08-24T19:23:01Z

just realised, should we also split the PR's per service ?

I don't see why this would be needed. I would rather leave it as is, but if you want i can do it.

tz3

A few things worth checking before this lands.

rajivmucheli

Did another review, found the below :

When Apache terminates TLS in the pod, the cc_x509 auth method must still receive the client certificate the way it did when NGINX terminated TLS. Reconstruct the SSL-Client-Cert and SSL-Client-Verify request headers from the verified TLS session, percent-encoding the certificate to the format the plugin already parses. Strip every dash/underscore spelling of these headers before setting them so a client cannot inject a forged certificate; only the values derived from the verified handshake reach the WSGI app. Scope this to the external :443 virtual host.

The tls-init init container runs as the distroless nonroot user (uid/gid 65532), but the tls-input Secret is mounted 0440 and, without an fsGroup, is owned root:root. The unwrap step then fails with "permission denied" on /tls-input/tls.key and the pod never starts. Add a pod-level securityContext.fsGroup: 65532 (gated on tls.enabled) to the Keystone and Barbican API deployments. The Secret files become group readable while staying non-root and non-world-readable, matching the BSI least-privilege requirement. tls-input keeps defaultMode 0440.

The public TLS service only set type: LoadBalancer with the Calico projectcalico.org/loadBalancerIPs annotation. On metal clusters (e.g. qa-de-3) there is no Calico LoadBalancer controller, so the IP is never assigned or advertised and the /32 is unreachable (No route to host). Add a calicoLoadBalancer toggle (matching designate/bind, default true to preserve existing Calico regions). When false, render spec.externalIPs so kube-parrot/externalip-operator announces the address via BGP instead.

Add a blackbox Probe for the public HTTPS endpoint of Keystone and Barbican that observes the certificate actually served on the wire, and alert when it expires within 7 days. Also alert when the rotation CronJob has not completed successfully in over 8 days, covering the case where it never runs at all. The Probe is disabled by default and only renders where TLS and the external endpoint are configured, so it stays inert in regions that have not switched over. The expiry and rotation alerts carry no_alert_on_absence and match on the specific endpoint and CronJob, so they never fire where the probe or rotation is absent.

The Keystone start script mutated Apache state at boot: a2enmod ssl/headers, a2dissite 000-default, and a2dismod status. These runtime changes write into /etc/apache2 and prevent a read-only root filesystem. Replace them with static config: - Load mod_ssl, mod_socache_shmcb and mod_headers via a conf-enabled snippet (ssl-load.conf). a2enmod ssl previously pulled in socache_shmcb implicitly; static loading has no dependency resolution, so it is listed explicitly. - Because conf-enabled is parsed after ports.conf, the ssl_module-gated Listen 443 there no longer fires, so the TLS vhost declares Listen 443. - Suppress the default :80 vhost with an emptyDir over sites-enabled instead of a2dissite. - mod_status is enabled by default in the image and ships a status.conf that exposes /server-status with ExtendedStatus On. Neutralize it with a conf-enabled override (status-off.conf) rather than a2dismod, since the module cannot be unloaded statically. mod_wsgi, mod_auth_openidc and mod_shib are already enabled in the image and need no action. The root filesystem stays writable for now; this change removes the config mutations that would block making it read-only later. Verified against the deployed API image: Apache starts, ssl/socache_shmcb/ headers load, port 443 binds, TLS 1.3 negotiates, the default :80 vhost is gone, and /server-status returns 403.

…ican Mirror the Keystone exposure model for pod-level TLS termination: a LoadBalancer Service on port 443 that announces the public /32 via Calico loadBalancerIPs (or spec.externalIPs on metal clusters), plus a disco DNS Record for the public host. The external IP is sourced from global.barbican_external_ip with services.externalip kept as an alias. Co-authored-by: Rajiv Mucheli <rajiv.mucheli@sap.com>

Most clusters do not perform automatic IP assignment, so announce the public TLS address via Service.spec.externalIPs by default. Clusters that assign IPs via the Calico annotation set calicoLoadBalancer: true.

- Log external HTTPS requests on the Barbican TLS VirtualHost and cap the request body size, mirroring the internal VirtualHost. - Add a PodDisruptionBudget (minAvailable: 1) for the Barbican API so a single pod eviction cannot drop all external HTTPS traffic. - Pin the tls-lifecycle-manager image tag in the Keystone and Barbican chart defaults instead of tracking latest.

…ning - Refine the TLS 1.3 cipher suites to AES-GCM and AES-CCM, and disable TLS compression on both Keystone and Barbican. - Set HSTS and standard HTTP security response headers on both TLS vhosts, loading mod_headers for Barbican. - Add a TLS-gated Apache hardening snippet (suppress the version banner, disable TRACE, and turn off directory listing) for both services. - Bump Keystone to 0.14.2 and Barbican to 0.9.3.

The TLS rotation failure alerts for Keystone and Barbican lacked no_alert_on_absence, unlike the other TLS alerts. When TLS is disabled the rotation jobs do not exist and the metric is absent, so add the label to avoid spurious absence alerts.

…one external IP Rebuild and sanitize the client-certificate issuer and subject DN request headers on the Keystone TLS vhost so tokenless x509 auth reads verified values and clients cannot inject them. Collapse the Barbican TLS vhost onto the single barbican-api WSGI daemon group to keep one PKCS#11 session per pod and honor api.threads. Restore the services.externalip entry on the plain Keystone API Service.

The TLS vhost CSP default-src 'self' blocked the inline auto-submit script in the WebSSO callback, leaving browser federation logins stuck. Allow just that script via its sha256 hash, and guard the pinned hash at render time so a change to the callback template fails loudly instead of silently breaking WebSSO.

mod_auth_openidc's default server-cache session lives in per-pod shared memory, so the post-callback request had to return to the same replica. Without the ingress cookie affinity that pod-level TLS removes, that request can hit any of the keystone replicas and lose the session. Store the session in the client cookie (encrypted with the already-shared crypto passphrase) so any replica can complete the WebSSO login.

Spell out the ServiceAccount subject namespace on the tls-manager RoleBinding so it matches the convention used by the other charts and is explicit about which ServiceAccount it binds. Co-authored-by: Rajiv Mucheli <rajiv.mucheli@sap.com>

The empty apache-sites volume that blanks out sites-enabled to suppress the default Apache site is only needed when Apache terminates TLS in the pod. Gate the mount and volume on tls.enabled so deployments without pod-level TLS keep the site config shipped in the image unchanged. Co-authored-by: Rajiv Mucheli <rajiv.mucheli@sap.com>

The tls-hardening snippet is mounted into conf-enabled and is already loaded automatically at server scope before the virtual host is parsed. Including it again re-declares the SSL session cache, which is redundant and can emit warnings, so rely on the automatic load only. Co-authored-by: Rajiv Mucheli <rajiv.mucheli@sap.com>

The TLS config validation previously only ran from the bootstrap hook, so rendering with hooks disabled skipped it and a later change to the rotation CronJob was not checked. Call the validation from the main Deployment and the rotation CronJob so an invalid TLS configuration is caught on every install and upgrade. Co-authored-by: Rajiv Mucheli <rajiv.mucheli@sap.com>

The rotation CronJob schedule was hard-coded to the same time for both services, so barbican and keystone rotated together and could hit issuer rate limits at once. Add a tls.rotationSchedule value and stagger the defaults (barbican Monday 03:00, keystone Monday 04:00) so operators can further adjust the timing per service. Co-authored-by: Rajiv Mucheli <rajiv.mucheli@sap.com>

When the pod terminates TLS itself there is no trusted proxy in front, so disable oslo proxy-header parsing in that mode to stop clients spoofing Forwarded/X-Forwarded-* (client address, host, scheme). Also strip the external-auth trust-key header at the TLS vhost, matching what the ingress did when it fronted the endpoint.

The external TLS (:443) vhost previously defined its own 8-process keystone-tls daemon group, so enabling pod-level TLS doubled the worker count to 16 under the same memory limit and left the TLS workers outside the metrics sampler. Define the keystone-public group once at server scope and have both vhosts use it, keeping the worker count and memory footprint fixed and letting the sampler cover every worker.

The alert matched any failed rotation Job still retained by kube-state-metrics. With failedJobsHistoryLimit set, a single old failed Job kept the alert firing for weeks, even after a later weekly run succeeded. Restrict the expression to the most recently created rotation Job so it clears as soon as a newer Job appears and still fires immediately on a first-run failure.

This alert is out of scope for the TLS work and is broken: it queries apache_error_log_saml_failures_total, a metric no component in the repo produces (there is no log-to-metric exporter, and the statsd sidecar only maps the WSGI worker gauge), so it can never fire and gives false confidence in SAML monitoring. Remove it.

bbobrov · 2026-09-02T16:18:34Z

I did this:

k create job keystone-tls-forcea \
  --from=cronjob/keystone-tls-rotation --dry-run=client -o yaml \
  | sed 's/--renewal-threshold-days=30/--renewal-threshold-days=9999/' \
  | k apply -f -

To trigger forced certificate rotation. And it worked, the certificate got reissued by the job.

tz3 · 2026-09-04T09:28:26Z

+# ports.conf, so its ssl_module-gated Listen 443 does not apply here.
+Listen 0.0.0.0:443
+
+<VirtualHost *:443>


Why in Barbican you didn't set the server name similar to what has been done in keystone?

tz3 · 2026-09-04T09:46:26Z

+                {{- if .Values.tls.allowInsecureStorage }}
+                - --allow-insecure-storage
+                {{- end }}
+      backoffLimit: 3


No activeDeadlineSeconds on the job spec — if the lifecycle manager hangs for whatever reason, the pod runs forever and concurrencyPolicy: Forbid blocks all subsequent weekly runs until someone manually kills it. is this intentional?

tz3 · 2026-09-04T09:48:37Z

+                {{- if .Values.tls.allowInsecureStorage }}
+                - --allow-insecure-storage
+                {{- end }}
+      backoffLimit: 3


Same comment: https://github.com/sapcc/helm-charts/pull/12641/changes#r3932809451

tz3 · 2026-09-04T09:52:30Z


+  - alert: OpenstackKeystoneTLSRotationFailed
+    expr: >
+      kube_job_status_failed{job_name=~"keystone-tls-rotation.*"} > 0


kube_job_status_failed is a kube-state-metrics metric — TLSRotationStale sits in the kubernetes/ alerts file for the same reason. Should this one move there too for consistency?

The same goes for barbican..

tz3 · 2026-09-04T13:52:10Z

    support_group: identity

+# Pod-level TLS termination in Apache
+tls:


When tls.enabled: true is flipped, global.keystone_external_ip becomes the public endpoint IP and the Designate record for identity-3.<region>.cloud.sap needs to be updated to match. The reverse is also true — if you roll back or deploy a revision without the flag, keystone-public-tls is gone but DNS still points to its IP until someone fixes it manually.

Hit this in qa-de-3 today: deployed a chart revision without the flag on top of a TLS-enabled state, keystone-public-tls disappeared, endpoint was unreachable until rollback.

Worth adding a note in the tls: block in values.yaml next to the other operator requirements — something like "when enabling, update the Designate record for the identity endpoint to global.keystone_external_ip; revert on rollback."

bbobrov requested review from databus23, drochow, giobbe and rajivmucheli August 24, 2026 14:20

bbobrov requested review from BerndKue, JoJoPuppe, dorneanu, galkindmitrii, stanislav-zaprudskiy and tz3 as code owners August 24, 2026 14:20

rajivmucheli previously approved these changes Aug 24, 2026

View reviewed changes

This was referenced Aug 24, 2026

barbican: TLS termination at Apache (BSI ADR-406 Step 2) #12555

Closed

[Barbican] TLS Implementation #11217

Closed