test(ci): exercise Keystone selective plan - #4161
Closed
Tadas Sutkaitis (fitbeard) wants to merge 73 commits into
Closed
test(ci): exercise Keystone selective plan#4161Tadas Sutkaitis (fitbeard) wants to merge 73 commits into
Tadas Sutkaitis (fitbeard) wants to merge 73 commits into
Conversation
Add a Go binary (cmd/atmosphere) that deploys Atmosphere components in parallel waves using a DAG-based dependency graph, reducing deployment time from ~60 minutes to ~22 minutes. Key components: - pkg/dag: Generic Graph[T] library with topological sort, subgraph extraction, and parallel wave execution via errgroup - internal/deploy: Component registry (42 components), Deployer interface with AnsibleDeployer, and 3-mode Orchestrator - cmd/atmosphere: CLI with deploy subcommand (--inventory, --tags, --playbook-dir, --concurrency flags) Three operating modes: - No tags: full DAG parallel deployment (11 waves) - Single tag: pass-through to ansible-playbook (backwards compatible) - Multiple tags: DAG-aware subgraph with parallel waves The orchestrator spawns concurrent ansible-playbook processes with generated per-component playbooks piped via /dev/stdin, avoiding multi-play parsing overhead. Output is streamed with [component] prefixes for clear CI log interleaving. Backwards compatibility: existing ansible-playbook usage, tags, and variables are completely unchanged. The orchestrator is additive. Assisted-By: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Mohammed Naser <mnaser@vexxhost.com>
Update molecule converge playbooks to build and use the atmosphere binary for deployment: - default: full DAG deploy (no tags) - csi: multi-tag with ceph,kubernetes,csi (or kubernetes,csi) - keycloak: multi-tag with all keycloak dependencies - pxc: single-tag pass-through for percona-xtradb-cluster The multi-tag mode resolves DAG ordering automatically, running independent components in parallel where possible. Assisted-By: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Mohammed Naser <mnaser@vexxhost.com>
Adjust DAG dependencies based on actual role analysis: - ingress-nginx: drop cluster-issuer dep (only needs kubernetes) - pxc, valkey, kube-prometheus-stack, loki: add csi dep (all use PVCs) - lpfc, multipathd, iscsi, udev: remove kubernetes dep (pure host config) - rook-ceph: depend on kubernetes only (operator, not storage consumer) - rook-ceph-cluster: add ceph dep (needs ceph monitors) - nova: add neutron dep, drop ovn/coredns (transitive via neutron) - neutron: add coredns dep (dnsmasq_dns_servers uses coredns) - magnum: depend on octavia, barbican, heat (configures all three clients) - openstack-exporter: depend on cinder, neutron (only hard runtime deps) Assisted-By: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Mohammed Naser <mnaser@vexxhost.com>
- Add ensure-go role (v1.24.4) to molecule pre-run playbook - Set CGO_ENABLED=0 and explicit Go PATH in all converge build tasks - Add kubernetes, csi, valkey to keycloak scenario tags (transitive deps) Assisted-By: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Mohammed Naser <mnaser@vexxhost.com>
When rendering playbooks piped via /dev/stdin, ansible-playbook has no collection context. Prefix bare role names with vexxhost.atmosphere. so Ansible can resolve them from the installed collection. Assisted-By: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Mohammed Naser <mnaser@vexxhost.com>
Use vexxhost.atmosphere.* fully-qualified collection names for both playbooks (PlaybookType) and roles (RoleType). This removes the need for --playbook-dir since Ansible resolves collection references directly. Also removes the openstacksdk prerequisite step since dependent roles already call it and Ansible does atomic writes. Assisted-By: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Mohammed Naser <mnaser@vexxhost.com>
Add a ResourceCoordinator that serializes components sharing a named resource (e.g., 'apt'). Components ceph and kubernetes declare the apt resource since they come from external collections where we cannot add retries. For all roles within vexxhost.atmosphere that use package management, add retries (5 attempts, 10s delay) to gracefully handle dpkg lock contention during parallel deployment. Assisted-By: Copilot <223556219+Copilot@users.noreply.github.com> Change-Id: Ic2974d32c7865995d82be5cbc3b1c12003fe7e5d Signed-off-by: Mohammed Naser <mnaser@vexxhost.com>
Mark multipathd and iscsi with the 'apt' resource since they install packages on the same hosts as ceph/kubernetes (external collections without retries). Also set changed_when: false on all molecule converge build/deploy tasks to pass idempotence checks. Assisted-By: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Mohammed Naser <mnaser@vexxhost.com>
Environment values containing Jinja expressions with single quotes (e.g., ceph container image) broke YAML parsing when wrapped in single-quoted YAML strings. Switch to Go's %q format which uses double quotes, safely containing single quotes in the values. Assisted-By: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Mohammed Naser <mnaser@vexxhost.com>
ipmi-exporter deploys directly into the monitoring namespace using kubernetes.core.k8s (not Helm with create_namespace: true), so it needs the namespace to exist first. kube-prometheus-stack creates it. Assisted-By: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Mohammed Naser <mnaser@vexxhost.com>
keepalived and percona-xtradb-cluster deploy raw k8s resources into the openstack namespace without creating it. memcached (via Helm with create_namespace: true) creates the namespace. Add memcached as a dependency so the namespace exists before these components run. Assisted-By: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Mohammed Naser <mnaser@vexxhost.com>
prometheus-pushgateway enables serviceMonitor which requires the ServiceMonitor CRD from kube-prometheus-stack. Without this dep, the Helm install fails with 'no matches for kind ServiceMonitor'. Assisted-By: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Mohammed Naser <mnaser@vexxhost.com>
The vexxhost.kubernetes collection uses kubernetes.core.k8s modules in early plays before the Python kubernetes package is installed by later plays. When running in parallel mode, this race becomes more visible. Install the package in pre-run to ensure it's available system-wide before any playbooks execute. Assisted-By: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Mohammed Naser <mnaser@vexxhost.com>
libvirt, kube-prometheus-stack, and valkey all create Certificate resources using cert-manager.io/v1 CRDs directly via kubernetes.core.k8s. They also reference a ClusterIssuer named 'self-signed' created by the cluster-issuer role. Add cluster-issuer as a dependency so the CRDs and issuer exist before these components deploy. Assisted-By: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Mohammed Naser <mnaser@vexxhost.com>
The kube_prometheus_stack role starts by waiting for the Keycloak StatefulSet to be ready and then creates realms/clients. Without keycloak in its dependency list, it can start before keycloak is deployed, causing 'list object has no element 0' errors when checking the StatefulSet status. Assisted-By: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Mohammed Naser <mnaser@vexxhost.com>
The rook_ceph_cluster role creates Keystone users, services, and endpoints for Swift/RGW integration using openstack.cloud modules. Without keystone being deployed first, these calls fail with SSL connection errors to the identity endpoint. Assisted-By: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Mohammed Naser <mnaser@vexxhost.com>
Manila creates compute flavors (needs Nova endpoint), uploads images (needs Glance via Nova chain), and its Helm values reference endpoints for nova, neutron, and cinder. Without these services deployed first, manila fails with EndpointNotFound for the compute service. Assisted-By: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Mohammed Naser <mnaser@vexxhost.com>
rook-ceph-cluster creates an OpenStack user in the 'service' domain using openstack.cloud.identity_user. The 'service' domain is created by OpenStack-Helm's ks-user bootstrap jobs (via helm-toolkit). By depending on barbican (the first core service deployed), we ensure the service domain exists before rook-ceph-cluster tries to use it. Assisted-By: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Copilot <copilot@github.com>
Move Go binary build to pre-run and add a custom Zuul run playbook that runs molecule prepare, atmosphere deploy, br-ex networking (AIO), and molecule verify as separate plays. This replaces the parent job's molecule test invocation so deploy output streams directly to Zuul logs instead of being buffered through molecule. Also adds atmosphere_deploy_tags to CSI and keycloak job definitions so each scenario deploys only its required components. Assisted-By: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Copilot <copilot@github.com>
All converge playbooks now use the atmosphere_deploy_tags variable instead of hardcoded tags. The Zuul run.yml imports the molecule converge playbook directly, so the same converge logic runs both locally (molecule converge) and in CI (Zuul run playbook). Assisted-By: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Copilot <copilot@github.com>
Cinder's Helm chart creates PVCs that need the Ceph CSI provisioner to be running. Add ceph-provisioners as a dependency so the storage class and provisioner are ready before cinder deploys. Assisted-By: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Copilot <copilot@github.com>
…build ceph-provisioners only needs ceph monitors and CSI driver, not rook-ceph-cluster. Also removes duplicate Go binary build from pre.yml since converge.yml already handles it. Assisted-By: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Copilot <copilot@github.com>
Each Zuul job now sets run: to the scenario's converge.yml followed by a verify playbook, so Zuul streams deploy output directly. Molecule prepare and inventory setup move to pre.yml. Converge playbooks use hosts: all with delegate_to/run_once so they work in both molecule (localhost) and Zuul (remote node) contexts. Also fixes ceph-provisioners to depend only on ceph (not rook-ceph-cluster) since it only needs ceph monitors. Assisted-By: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Copilot <copilot@github.com>
In Zuul, Go is installed on the remote instance (via ensure-go) not on the executor (localhost). Remove delegate_to: localhost so the go build and atmosphere deploy commands run where Go and the collection are available. Assisted-By: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Copilot <copilot@github.com>
The `zuul.project.src_dir` variable is a relative path (e.g. `src/github.com/vexxhost/atmosphere`). When the deploy task uses it as a prefix in `cmd` while also having `chdir` set to the same relative path, Ansible resolves the binary path as if it were relative to the new cwd, doubling the path and causing a FileNotFoundError. Fix by using `./bin/atmosphere` and `./inventory.yaml` in the cmd field since `chdir` already navigates to the correct directory. Also fix pre-commit end-of-file issue in orchestrator.go and add a release note for the parallel deployment orchestrator feature. Agent-Logs-Url: https://github.com/vexxhost/atmosphere/sessions/60b11e18-b92e-476f-86db-2a6c2ac4db06 Co-authored-by: mnaser <435815+mnaser@users.noreply.github.com>
The atmosphere deploy binary calls ansible-playbook internally. Add .venv/bin to PATH in the deploy task so the binary can find ansible-playbook installed in the uv virtual environment. Agent-Logs-Url: https://github.com/vexxhost/atmosphere/sessions/60b11e18-b92e-476f-86db-2a6c2ac4db06 Co-authored-by: mnaser <435815+mnaser@users.noreply.github.com>
…ok path Go 1.19+ refuses to execute binaries found via relative PATH entries (CVE-2022-30580). Using `PATH=.venv/bin:...` fails because `.venv/bin` is a relative entry. Switch to `ansible.builtin.shell` with `. .venv/bin/activate` so that the shell activation script adds the ABSOLUTE path of `.venv/bin` to PATH before invoking `./bin/atmosphere deploy`. The atmosphere binary then finds ansible-playbook via an absolute path, satisfying Go 1.19+ security requirements. Agent-Logs-Url: https://github.com/vexxhost/atmosphere/sessions/60b11e18-b92e-476f-86db-2a6c2ac4db06 Co-authored-by: mnaser <435815+mnaser@users.noreply.github.com>
In the original serialized playbook, nova was deployed before neutron. Neutron's post-install network creation requires the nova availability zone to exist. Swap the dependency so nova deploys first, then neutron. Assisted-By: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Copilot <copilot@github.com>
The parallel orchestrator generates minimal single-role playbooks for RoleType components, which bypasses pre_tasks defined in the original sequential playbooks (e.g., playbooks/openstack.yml). This means the atmosphere_ceph_enabled deprecation guard was silently skipped. Add a runPreflightChecks() method that runs the same validation checks before any component deployment begins, called from both deployFullDAG and deployMultipleTags. The deploySingleTag path is unaffected since it passes through to the full site.yml which already includes pre_tasks. Change-Id: If068daa27a3f4475e570f08ab6d2cd52effb2914 Signed-off-by: Dong Ma <dong.ma@vexxhost.com>
The Acquire method now respects context cancellation instead of blocking indefinitely. If the context is cancelled while waiting for a resource semaphore, already-acquired resources are released and an error is returned. This prevents goroutine leaks when Ansible processes hang or when the orchestrator is interrupted. Assisted-By: Copilot <223556219+Copilot@users.noreply.github.com> Change-Id: I342f30e9b822226e2dc2632e41ec4431623189a3 Signed-off-by: ricolin <rlin@vexxhost.com>
Signed-off-by: Tadas Sutkaitis <tadas.sutkaitis@vexxhost.com>
Signed-off-by: Tadas Sutkaitis <tadas.sutkaitis@vexxhost.com>
Signed-off-by: Tadas Sutkaitis <tadas.sutkaitis@vexxhost.com>
Signed-off-by: Tadas Sutkaitis <tadas.sutkaitis@vexxhost.com>
Signed-off-by: Tadas Sutkaitis <tadas.sutkaitis@vexxhost.com>
Signed-off-by: Tadas Sutkaitis <tadas.sutkaitis@vexxhost.com>
Signed-off-by: Tadas Sutkaitis <tadas.sutkaitis@vexxhost.com>
Signed-off-by: Tadas Sutkaitis <tadas.sutkaitis@vexxhost.com>
Signed-off-by: Tadas Sutkaitis <tadas.sutkaitis@vexxhost.com>
Signed-off-by: Tadas Sutkaitis <tadas.sutkaitis@vexxhost.com>
Signed-off-by: Tadas Sutkaitis <tadas.sutkaitis@vexxhost.com>
Signed-off-by: Tadas Sutkaitis <tadas.sutkaitis@vexxhost.com>
Signed-off-by: Tadas Sutkaitis <tadas.sutkaitis@vexxhost.com>
Tadas Sutkaitis (fitbeard)
changed the title
Tadas Sutkaitis (fitbeard)
marked this pull request as ready for review
July 25, 2026 08:03
Contributor
Author
|
recheck |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: af7fe27f23
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "Codex (@codex) review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".
| @@ -0,0 +1 @@ | |||
| This temporary marker exercises Keystone selective CI planning. | |||
There was a problem hiding this comment.
The reviewed commit message has no Signed-off-by: trailer, so it does not satisfy the repository's requirement that every commit carry a DCO sign-off. Recreate this commit with git commit -s before submitting it.
AGENTS.md reference: AGENTS.md:L3-L9
Useful? React with 👍 / 👎.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Temporary validation PR stacked on #4152. Expected selective result: run AIO Open vSwitch and Keycloak; skip AIO OVN, CSI local-path, and CSI RBD. This PR contains only an inert Keystone-path marker and will be closed after CI validation.