feat: selective-molecule-ci

[Tadas Sutkaitis (fitbeard)]

No description provided.

[Tadas Sutkaitis (fitbeard)]

recheck

[Tadas Sutkaitis (fitbeard)]

recheck

[Tadas Sutkaitis (fitbeard)]

recheck

[Tadas Sutkaitis (fitbeard)]

recheck

[Tadas Sutkaitis (fitbeard)]

recheck

[Tadas Sutkaitis (fitbeard)]

Selective Molecule CI timing results

This branch directly builds on the parallel deployment work: its merge-base with origin/feat/parallel-deploy-role-splits is exactly that branch's tip, 5cde0db6.

Full CI comparison

Implementation	Wall time	Executor time	Improvement vs. original
Original sequential CI	1h59m04s	4h22m36s	baseline
Parallel orchestrator	59m54s	2h37m10s	49.7% faster
Current combined full fallback	1h12m30s	3h34m16s	39.1% faster

The original Open vSwitch job took 1h35m56s and OVN took 1h58m49s. Full-fallback acceleration comes primarily from the inherited parallel orchestrator and role splits. The selective planner's larger gain comes from avoiding a full deployment for scoped changes.

These historical runs used different repository states and infrastructure dates, so they are strong operational measurements but not a controlled A/B benchmark. In particular, the current combined full fallback is 12m36s slower than the historical parallel-only run.

Selective results

Compared with the successful 1h59m04s original full cycle:

Changed component	Whole CI cycle	Improvement	Result
cert-manager	8m35s	92.8% faster	passed
local-path CSI	9m28s	92.0% faster	passed
RBD CSI	25m47s	78.3% faster	passed
Keystone	30m02s	74.8% faster	passed
Placement	32m19s	72.9% faster	passed
Glance	32m43s	72.5% faster	passed
Barbican	33m25s	71.9% faster	passed
Cinder	36m57s	69.0% faster	passed
Horizon	47m52s	59.8% faster	passed
Heat	50m01s	58.0% faster	passed
Nova	1h04m43s	45.6% faster	target Open vSwitch job passed; buildset had an unrelated zero-duration node failure

The first Horizon attempt exposed an incomplete focused environment: Tempest lands on Horizon's project Overview panel, which requires Compute. The planner now includes Nova's image, network, placement, and libvirt requirements for Horizon while still omitting Cinder, Heat, Octavia, Manila, Magnum, and other unrelated services. The rerun passed deployment, idempotence, and Tempest dashboard verification.

The first Heat attempt exposed that Tempest did not trust the self-signed public ClusterIssuer. The next attempt proved the issuer CA was copied correctly but exposed an unrelated TLS injection into the internal HTTP Keystone bootstrap hook. The final run kept that hook on its internal endpoint, mounted the original issuer CA only into the Tempest test job, completed deployment and idempotence, and ran 119 smoke tests: 105 passed, 14 skipped, and 0 failed. Public Heat endpoint discovery completed with certificate verification enabled.

[Tadas Sutkaitis (fitbeard)]

recheck

[Tadas Sutkaitis (fitbeard)]

Pure-main selective CI validation

The current implementation is based directly on main and does not contain code from the Parallel orchestrator branch.

The authoritative full-fallback run for commit f9d4ed66 passed all five Zuul jobs: buildset f5b613c9.

Job	Result	Duration
AIO Open vSwitch	passed	2h04m16s
AIO OVN	passed	1h59m41s
local-path CSI	passed	7m37s
RBD CSI	passed	18m36s
Keycloak	passed	23m54s

Full-cycle comparison

Implementation	Wall time	Executor time	Change vs. original
Original sequential CI	1h59m04s	4h22m36s	baseline
Parallel orchestrator	59m54s	2h37m10s	49.7% faster
Pure-main selective CI, full fallback	2h04m16s	4h54m04s	4.4% slower

The full fallback is intentionally the standard sequential main deployment, so it is not expected to be faster. This PR's own CI-policy and runner changes correctly trigger that conservative fallback. The expected savings come from scoped service changes that omit unrelated roles and tests.

This run validates that:

• the implementation is independent from the Parallel orchestrator;
• all standard AIO, CSI, and Keycloak lifecycles still pass;
• full fallback remains safe;
• both AIO jobs now finish beyond the previous 7,000-second command limit;
• the RBD job retains CEPH_CONTAINER_IMAGE and CEPH_CONTAINER_BINARY.

Remaining work

1. Fix the GitHub Actions pre-commit failure. All hooks pass except Black, which reformats one dictionary comprehension in tests/unit/ci/test_molecule_plan.py.
2. Measure pure-main selective service runs using dependent test PRs based on this branch. CI-runner changes in this PR always select full fallback, so a separate PR is required to exercise real Glance, Cinder, Nova, Horizon, Heat, and other scoped plans.
3. Use those runs to publish authoritative per-service speed improvements. The older selective timing table came from the earlier Parallel orchestrator-based prototype and should not be treated as performance evidence for this pure-main implementation.

[Tadas Sutkaitis (fitbeard)]

Selective CI validation matrix — final

Validated on the pure-main implementation at 2ef127a7 (no Parallel orchestrator code).

Coverage summary

Coverage	Result
Selectable components tested in isolation	48/48 passed
Mixed component groups	6/6 passed
Local isolated-target invariants	48/48 passed
Local pairwise invariants	1,128/1,128 passed
Planner unit tests	69 passed
Final fallback jobs	5/5 passed
Final AIO network scenarios	Open vSwitch passed; OVN passed

Full isolated-component matrix

Component	Evidence	Wall time
Ceph	#4210 ✅	20m14s
Kubernetes	#4230 ✅	23m41s
CSI	#4213 ✅ (local + RBD)	19m31s
cert-manager	#4215 ✅	22m18s
Cluster issuer	#4221 ✅	23m15s
Ingress NGINX	#4209 ✅	21m02s
RabbitMQ operator	#4219 ✅	24m45s
PXC operator	#4227 ✅	20m26s
PXC	#4214 ✅	20m59s
Valkey	#4216 ✅	20m01s
Keycloak	#4226 ✅	28m08s
Keepalived	#4220 ✅	8m00s
Node Feature Discovery	#4228 ✅	8m28s
kube-prometheus-stack	#4211 ✅	35m18s
Loki	#4208 ✅	20m24s
Vector	#4224 ✅	20m08s
Goldpinger	#4222 ✅	8m29s
IPMI exporter	#4207 ✅	8m06s
smartctl exporter	#4229 ✅	35m40s
Prometheus pushgateway	#4218 ✅	34m10s
LPFC	#4212 ✅	4m10s
Multipathd	#4231 ✅	5m06s
iSCSI	#4225 ✅	3m55s
Udev	#4223 ✅	4m18s
Memcached	#4217 ✅	8m17s
Keystone	#4175 ✅	39m31s
Barbican	#4178 ✅	40m19s
Rook Ceph	#4195 ✅	8m07s
Rook Ceph cluster	#4190 ✅	39m38s
Ceph provisioners	#4192 ✅	18m30s
Glance	#4164 ✅	44m50s
Staffeln	#4189 ✅	37m40s
Cinder	#4165 ✅	53m10s
Placement	#4176 ✅	38m01s
Open vSwitch	#4199 ✅	1h04m38s
FRR Kubernetes	#4198 ✅	1h05m55s
OVN	#4197 ✅	1h09m27s
Libvirt	#4196 ✅	1h06m09s
CoreDNS	#4191 ✅	1h07m25s
Nova	#4166 ✅	1h08m50s
Neutron	#4181 ✅ (OVS + OVN)	OVS 1h06m04s; OVN 1h07m46s
Heat	#4177 ✅	1h05m59s
Octavia	#4193 ✅	1h29m38s
Magnum	#4194 ✅	1h33m52s
Manila	#4169 ✅	1h15m11s
Horizon	#4180 ✅	1h03m15s
OpenStack exporter	#4200 ✅	1h10m12s
OpenStack CLI	#4185 ✅	37m26s

Designate and Ironic are intentionally not counted among the 48 selectable targets: the current AIO scenario does not enable them, so their paths conservatively select the full fallback.

Mixed-component evidence

Group	Components / purpose	Result	Wall time
orchestration-ui	Heat + Horizon	#4201 ✅	1h09m58s
platform-apis	broad API union including Octavia/Magnum	#4202 ✅	1h47m04s
network-storage	Neutron + storage, both network backends	#4203 ✅	OVS 1h18m46s; OVN 1h10m11s
cross-job-foundation	union across all five job families	#4204 ✅	21m11s
core-api	focused identity/API combination	#4205 ✅	41m36s
storage-compute	Glance + Cinder + Nova/Placement closure	#4206 ✅	1h12m28s

An additional Glance + Keystone pair passed in 43m59s.

Final fallback on 2ef127a7

Job	Result	Duration
AIO Open vSwitch	✅	2h08m37s
AIO OVN	✅	2h02m41s
CSI RBD	✅	19m15s
CSI local-path	✅	8m16s
Keycloak	✅	21m01s

Final fallback wall time was 2h08m37s because the jobs run concurrently.

Speed compared with the existing ~90-minute baseline

Change scope	Observed time	Approximate saving
Small host roles	3m55s–8m29s	91–96%
Foundation services	18m30s–35m40s	60–79%
Keystone / Placement / Barbican	38m01s–40m19s	55–58%
Glance	44m50s	50%
Cinder	53m10s	41%
Horizon	1h03m15s	30%
Heat / Neutron / Nova	1h05m59s–1h08m50s	24–27%
Manila	1h15m11s	16%
Octavia	1h29m38s	effectively neutral
Magnum	1h33m52s	slightly slower
Conservative full fallback	2h08m37s	deliberately slower; only for unbounded/shared changes

The useful result is not that every target is fast: dependency-heavy Octavia and Magnum genuinely need almost the whole cloud. The gain is that most component changes avoid unrelated services, while ambiguous/shared changes remain conservative.

One non-Zuul GitHub Actions image/build check is red because Ansible Galaxy returned a downloaded artifact whose hash did not match. That is an external, retryable download failure; all other GitHub code checks and the complete Zuul runtime matrix passed.

[Mohammed Naser (mnaser)]

Could we split the non-CI runtime changes out before continuing with the selective CI implementation? These are useful fixes, but they change production role behavior independently from test selection and should be reviewed, released, and backported on their own.

In particular, I think these should become several independently mergeable PRs:

• RabbitMQ transient deployment retries (356f2a98).
• Octavia quota-request retries (8868a471).
• Namespace and service-scope self-containment fixes for Keepalived, Ceph provisioners, IPMI exporter, Valkey, and Rook Ceph cluster (5a0de044, 3082f3b9, c5970093, and 7f2d3b82). These can be split further by role where appropriate.
• The Rook Ceph cluster identity-management SDK rewrite (effb1f51).
• The Keycloak, Neutron, Nova, Octavia, and Manila timeout/default changes. Please separate these by independently backportable concern rather than keeping them in the CI feature PR.

For each extracted fix PR, please check which stable branches contain the affected code and add the applicable backport stable/<branch> labels. The repository currently has labels from backport stable/zed through backport stable/2025.2; the exact cutoff can differ per fix.

Ideally those fixes can land first and this PR can then rebase on them, leaving #4152 focused on the selector, policy, scheduler filters, and selective verification behavior.