Conversation
Deployment side of moving DE group data out of Grouper. The groups role follows the usual service shape and is wired into all four playbooks. Its config points at the permissions schema of the DE database rather than a database of its own, because that is where group data now lives, and at Keycloak for user attributes only. The importer ships in the same image as the service and deploys alongside it as a CronJob, created suspended. Suspended because there is nothing to keep in sync until the permissions service reads group data from this database -- but it must be resumed then, and left running for as long as Grouper stays authoritative: terrain and apps still write group changes there, and those changes are invisible to the permissions service until the next import. grouper_import.yml runs it once, for the initial import and for dry runs, and prints the report rather than just the exit status. The permissions config gains the username suffix it now shares with the groups service and the expansion cap. Its grouperdb section stays, annotated: builds from permissions#31 onwards ignore it, but the previous image will not start without it, so it is the rollback path until that window closes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The groups service reads user attributes from Keycloak over HTTPS, so on a deployment with a private CA it needs the same de_ca_* wiring every other service template carries. The importer deliberately gets none: it reaches Grouper and the DE database over postgresql with sslmode disabled and the permissions service over plain HTTP, so it makes no TLS connection. It also needs a confidential client with a service account. Unlike terrain's admin client, which lives in master because terrain builds its token URL against realms/master by name, groups logs in against the DE realm and calls GetUsers on that same realm -- so its client belongs there, and the view-users it needs is a client role of the built-in realm-management client rather than a realm role. keycloak_config could express neither: one list grants realm roles in the DE realm, the other client roles in master. Granting view-users by name alone would find no such realm role and leave the service account able to authenticate and then 403 on every user lookup. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Records which of keycloak_config's three service-account role lists applies, since choosing wrong yields a client that authenticates and then 403s on every call. Also notes why the groups deployment carries the de_ca_* partials and the importer deliberately does not. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The CronJob is applied with kubernetes.core.k8s rather than through skaffold, so nothing substituted the image the way it does for the Deployment. The bare name resolved to harbor.cyverse.org/de/groups:latest, which is never pushed, and every scheduled import sat in ImagePullBackOff -- a failure that looks like a registry problem rather than a manifest one. Read the tag from the same build descriptor deploy-service uses. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Left unset, k0s re-detects spec.api.address at every start from the default route's source address and bakes it into the component kubeconfigs and the kubernetes Service endpoint. On a workstation that is a DHCP lease. When the lease moves the cluster does not go down, it goes half-down, which is why this costs an afternoon each time. kube-proxy and CoreDNS lose their watches and keep serving what they already programmed, so everything that existed beforehand keeps working while anything created or redeployed afterwards is silently never programmed: a new Service does not resolve, a redeployed pod is healthy but its ClusterIP does not route, and nothing anywhere reports an outage. The failures present as bugs in whatever you were working on at the time. bootstrap-local-k0s.sh now writes /etc/k0s/k0s.yaml before installing, pinning the address to the host's Tailscale address when it has one -- stable across reboots and lease changes, and already in the API server's certificate SANs. This is the server-side half of a problem the script already handled for clients by rewriting the admin kubeconfig to 127.0.0.1. repair-local-k0s-api-address.sh covers clusters that already drifted or predate the pinning. It reads by default so it is safe to run on a hunch, and repairs with --fix. It edits the address in place when the config exists rather than regenerating it, because that file's network block is load-bearing and writing defaults over customised CIDRs would renumber the cluster. It also refuses to act unless the kubeconfig names a loopback server: ~/.kube/config here points at QA, and both contexts are named k0s-cluster, so the context name cannot distinguish them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The previous version pinned the advertised address to the host's Tailscale
address, falling back to the DHCP lease with a warning. That only works for
machines on the tailnet: a coworker without Tailscale got the fallback, which
is the drift this was meant to prevent.
Use an address that belongs to no physical network instead -- 10.255.255.1 on a
dummy interface, brought up by a systemd unit ordered before k0scontroller.
Nothing outside the host can move it, there is nothing to detect, and it is the
same everywhere, so the runbook and the skill can name it.
Rejected, with reasons worth keeping:
the LAN address the DHCP lease this exists to avoid
a VPN address stable, but only on machines running that VPN
10.244.0.1 the kube-bridge; reachable and stable once the cluster is
up, but the CNI creates it and cannot start until the node
has registered, so advertising it deadlocks a fresh install
Verified on this host that pods reach arbitrary host-local addresses (172.17.0.1
and 172.18.0.1 both answer), which is what makes a dummy interface viable. TLS
is not a constraint either: k0s regenerates the API server certificate on start
and puts every local interface address in its SANs.
The diagnostic now separates broken from merely non-standard. A cluster on a
reachable non-dummy address works, and reporting that as a fault would cry wolf
on a healthy cluster, so it exits 0 with advice; only a real fault exits 1.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Migrating a healthy cluster from a non-standard address falls through to the drift reporting, so --fix printed "DRIFT DETECTED." immediately after saying there was none. Guard the block on the fault actually having been found. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A freshly deployed local DE has one DE group and two LDAP subjects, which is enough to prove the importer runs and nothing about whether it is correct: no nesting, one group type, no privileges worth mapping. Every interesting failure mode was untestable. seed-local-grouper.py creates the shapes that exercise it -- every group type with owners, a two-level nesting chain, a member reachable by two paths, and the privilege vocabulary including GrouperAll. It goes through iplant-groups rather than writing to the Grouper database, so the groups are formed exactly the way the DE forms them, and it is idempotent so it can be re-run after a teardown or a partial failure. The two structures are the point. Field Team -> Genomics Lab -> msmith's default puts a member in Field Team at depth 2, so an expansion that follows one hop drops them and still looks plausible; and one member reaching a community both directly and through a nested group makes grouper_memberships_v return that pair twice, which is the case an importer assuming the view is a set gets wrong. LDAP_ROOT_PW comes from the environment. This repo is public and the credential lives in the private inventory. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both services read groups over HTTP, so switching them is a config change plus the client rewrites in their own repos. Their iplant_groups.* sections become groups.*, and baseurls_groups joins the common defaults alongside the other service base URLs. group-propagator's folder_name_prefix and public_group are gone. The groups service holds exactly one deployment's group data, so there is no folder to scope a crawl by, and the de-users group is now named plainly and resolved by type rather than spelled as a colon-delimited path. grouper_folder_name_prefix stays: the permissions config and the Grouper importer still use it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The "no longer in Grouper" list could not distinguish a group Grouper lost from one created natively, and apps creates workshop-users lazily, so during the soak that report names groups nothing deleted. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
johnworth
changed the title
community_tags.yml runs the community-tag rewrite as a one-off Job: it changes an app's community tag from the community's name to its ID, so renaming a community no longer drops every app tagged with it out of its own listing. Credentials go in through a Secret rather than a command line. apps_communities_attr is now rendered into both apps' config and the migration Job. They have to agree: pointed at the wrong attribute the migration finds nothing and reports a clean run, which looks exactly like success. grouper_import.yml carried the same missing-digest bug already fixed for the importer CronJob -- it named the image without a tag, which resolves to :latest and is never pushed, so the playbook the runbook tells operators to run would have sat in ImagePullBackOff. It now reads the build descriptor too. Verified end to end against the local cluster: dry run, then a real run rewriting 2 rows and reporting 1 orphan, then a re-run reporting zeros. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The terrain template rendered no terrain.groups.* at all, so the only way to move terrain onto the groups service was to rely on its compiled defaults. That left the toggle which is terrain's rollback path inexpressible from an inventory. terrain_groups_backend defaults to iplant-groups so no environment flips by inheriting it. Setting it to groups before the deployed apps image tags communities by ID orphans every community tag on the first rename, silently, so the default is the safe one and the comment says why. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Built from groups d81806f, which stops returning Postgres constraint names and answers 400 rather than 409 for values the schema rejects. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Built from groups 39450ed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Measuring the group-expansion paths at scale previously meant restoring a copy of production, which puts real usernames and the real membership graph on a workstation. These build an equivalent dataset instead. Row counts are the easy part. Three shape properties decide whether the result means anything: de-users must hold essentially every user, since one enormous group is what makes an expansion expensive; ownership must be skewed rather than uniform, or the planner sees statistics it never sees in production; and nesting must be disjoint rather than chained, or each group absorbs the tail below it and the closure comes out at 102,082 against production's 54,467. The generator also creates permissions_resource_id_idx, without which its own cleanup does not terminate: the schema indexes (subject_id, resource_id) and nothing on resource_id, so the FK check behind ON DELETE CASCADE from resources scans the whole table per deleted row. That gap predates this work and belongs in a migration. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A group is also a resource, and that resource's name is the group's 32-hex external id -- the value the groups service passes to the permissions service when it authorizes a request. Naming them perf-grp-<n> meant no grant was ever findable, so every synthetic group was unadministrable and its own owner got a 403 on it. Also adds the GrouperAll read grants that mark public teams and communities. They confer no read access -- GrouperAll has no members for a user to expand to -- but terrain reads them to decide whether to show a group as public and joinable, and without them nothing in the dataset looks public at all. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Records that a public team or community is marked by a read grant held by GrouperAll, that it is checked without expansion because no user is ever a member of it, and that a group's permissions resource is named by the group's external id. Corrects an earlier claim that the grant conferred no access. That came from checking grouper_memberships_v, which is membership; the marker is a privilege field in grouper_memberships_all_v, and Grouper resolved it as everyone. Pins the groups image carrying the fix. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Records the viewers/readers distinction, that it rides on groups.members_public from migration 000055, and that a public group withholding its membership answers with an empty list rather than a 403 because that is what Grouper did and the DE's team page renders from it. Pins the groups image carrying the change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Records that GET /groups returns only what the caller may read, that the filter mirrors the read check exactly so a group lists if and only if it opens, and why it is one SQL predicate rather than a check per group. Also records migration 000056, which lets a set of nested groups be deleted in one statement. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
groups_admin_users defaulted to [], so terrain, apps, and group-propagator saw only what an ordinary user sees. Everything worked locally because the local inventory happened to set it; QA and production would not have. Consequences: apps 500s resolving a community, because the groups service answers 403 rather than 404 for a group a non-admin cannot see and the client only catches 404; group-propagator receives withheld member lists and truncated crawls. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Group management requires `admin`, not `write`/`own` -- the DE's precedence puts `admin` below write, so asking for write locked out every co-admin. groups_admin_users must contain de_grouper, and now defaults to it. A withheld member list carries "redacted": true so a machine caller can tell it from an empty group. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Grouper's optins, recorded in groups.joinable by migration 000058. Gating a join on the public marker alone let any user add themselves to any public team, bypassing the join-request approval Grouper enforced. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Grouper returned them tagged g:gsa and the DE's sharing dialog keys on that tag, so returning only users made sharing with a team unreachable. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both reproduced deterministically. 000057 serializes recomputations sharing an ancestor; 000059 covers the second, which it cannot fix because the writers lock disjoint sets. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- community_tags.yml built its two database URIs without url-encoding the password, unlike every other URI on this branch; the shipped default contains an @, which splits the authority at the wrong place. - seed-local-grouper.py embedded the LDAP Manager password in kubectl's --overrides argv, visible in ps on the workstation and stored in the pod spec; it now flows through a Secret and secretKeyRef piped over stdin, deleted with the rest of the seed artifacts. - The groups_* defaults were spliced into the middle of the formation_* block with a leftover draft of the groups_admin_users comment run into the final one; moved to prefix order and deduplicated. - The importer-image comment claimed the CronJob reads "the same build descriptor deploy-service uses"; it reads the role's own copy, which coincides only at build_json_dir's default. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
GET / on the groups service now gates readiness on the database (503 on a failed ping), and the importer exits nonzero when its own verification fails rather than printing the mismatch and reporting success. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
johnworth
marked this pull request as ready for review
August 5, 2026 23:26
Resolves four conflicts. ansible/roles/common/defaults/main.yml: the baseurls block took main's info-typer entry and its repointing of baseurls_iplant_email to notifications/mail (de-mailer is retired on main) alongside this branch's baseurls_groups. wiki/playbooks/index.md and wiki/services/index.md: matched each line to the page frontmatter that survived the merge -- main's retitled ci-to-qa entry, this branch's community-tags, groups, and repointed group-propagator entries, and no get-analysis-id line, that page having been deleted on main. wiki/log.md: both sides prepended entries, so the file is rebuilt in reverse-chronological order with the two 2026-08-04 sections merged into one. okf check reports 0 errors and 0 warnings, and deploy_it.yml, build_it.yml, kubernetes.yml, community_tags.yml and grouper_import.yml all pass a syntax check. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LKP3w4fkB6itwPawxKu1ye
The cutover procedure was a mix of playbooks and hand-run kubectl, psql, and inventory edits. The parts that were not automated were the ones that matter: the group_data_source flip is the moment Grouper stops being authoritative, and it was a copy-paste UPDATE with the operator's own username and date pasted in. grouper_cutover.yml, modeled on notifications_db_merge.yml, closes that gap: - preflight is read-only and runs any time before the window. It requires an operator name for the audit row, the group tables, a suspended grouper-import CronJob, available groups and permissions replicas, and evidence the import actually landed rows. - flip updates the marker WHERE source = 'grouper' and asserts one row moved. changed_at is left to its trigger. - verify re-reads the marker from the database and re-checks the CronJob. - rollback carries the never tag, so a default run cannot reach it. The reconciliation gate is in preflight rather than verify because RequireGrouperAuthoritative runs ahead of the importer's dry-run check: past the flip, -e dry_run=true refuses too. For the same reason verify asserts the row instead of starting a Job to watch the importer refuse -- the refusal is driven by that row alone, so a Job would test the same value twice. Preflight deliberately does not re-check the community tag rewrite. community_tags.yml reports on its own run, and deciding which stored values are still rewritable means the legacy-name and short-name precedence rules that live in the community-tags command; a second copy here would be a copy that drifts. docs/production-release.md gains the cutover under section 7, alongside the other one-time migrations, and the wiki gains a page for the playbook. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wjd21NTp4Ead7JYhx5sQUT
The notifications database merge, the job-status consolidation, and the subscriptions/qms merge are all running in production. Section 7 is transient by design -- each cutover goes once it has shipped everywhere -- so all three come out, leaving the Grouper cutover as the only outstanding one. The playbooks and their wiki pages stay; other environments have not necessarily run them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wjd21NTp4Ead7JYhx5sQUT
The preparation section deploys groups early, so the neighbouring instruction to deploy groups and permissions together invites doing both. The new permissions image expands groups from the database, so deploying it before the import finishes reads every group-granted permission as absent. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wjd21NTp4Ead7JYhx5sQUT
Section 7 is a command and order reference, so the parts that need to be exact are the gates and the way back. The import gate was one sentence naming three report lines. It now separates what the playbook enforces -- rejected members, a closure disagreeing with Grouper, and an abort when two colliding groups both have content -- from what an operator still has to read and judge, including the resolved-collision list that names production's one known empty pair. Rollback said to redeploy the previous images without saying where to find them. Those descriptors live at the commit before the release merge, which is worth noting before the window rather than hunting for during it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wjd21NTp4Ead7JYhx5sQUT
psycopg2 and kubernetes were mentioned only inside the macOS note, so on Linux nothing said they were needed at all. Both the notifications merge and the Grouper cutover drive community.postgresql and kubernetes.core modules, and a missing library shows up as a module failure partway through a run rather than at startup. Found by running grouper_cutover.yml against the local cluster on a workstation that had neither. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wjd21NTp4Ead7JYhx5sQUT
Exercising the round trip on the local cluster recorded "wregglej" from -e "grouper_cutover_changed_by=wregglej (via claude)": Ansible's key=value form splits on the first space and keeps the head, so an audit field silently loses part of its value. Also note that $USER is the workstation account rather than the CyVerse username, which is what the local run actually wrote. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wjd21NTp4Ead7JYhx5sQUT
The importer's suspend guidance contradicted the cutover. The role defaults, the CronJob template, and the wiki all told the operator to resume the importer once permissions was deployed and to suspend it only after the services had cut over -- but the whole migration ships in one window, the release doc says the CronJob stays suspended throughout, and the cutover preflight hard-fails the flip while it is unsuspended. Following the old text left the playbook refusing to run on the day. All three now say it stays suspended, and name the rollback as the one case that resumes it. The groups service and the importer hardcoded http://permissions where every other role reads baseurls_permissions, so an environment that overrode the permissions base URL had the override silently ignored. A Job that outlived its timeout aborted grouper_import.yml and community_tags.yml on the retry loop, so the report those playbooks exist to print -- the orphan list, the closure comparison -- was replaced by "retries exceeded" during a maintenance window. The wait is no longer fatal on its own; the explicit failure after the report now covers the timed-out case as well as the failed one. The cutover's marker read moves into read_marker.yml, imported by both preflight and flip: --tags=flip alone died on an undefined grouper_cutover_source, and indexing the query result turned a missing seed row into a list-index error. In the scripts: synthesize-perf-dataset.sql tested only whether confirm was defined, so -v confirm=no wrote 1.5M rows anyway, and it derived group owners from a hardcoded 40000 rather than n_users, leaving every group unowned at any scale below 1. benchmark-permissions-lookup.sql turned an unknown subject into "cannot determine type of empty array". seed-local-grouper.py ignored the ldapload exit status and seeded groups whose members were never loaded. bootstrap-local-k0s.sh rewrote an existing k0s.yaml on a re-run, against its own docstring. repair-local-k0s-api-address.sh called drift on a single log line, so a cluster inspected shortly after boot reported a fault and --fix restarted k0scontroller for nothing; it also left the old address in spec.api.sans and never checked that its sed matched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wjd21NTp4Ead7JYhx5sQUT
The deployed manifest pointed both probes at /, which answers 503 while the database is unreachable. With periodSeconds 5 and the default failureThreshold, about fifteen seconds of database trouble killed every replica and they crash-looped until it came back -- a restart cannot reach a database. Liveness now hits /healthz, added to the service for this, and readiness keeps /. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wjd21NTp4Ead7JYhx5sQUT
The groups service resolves a display name, email, and institution for every member it lists, and through Keycloak that costs one request per member -- Keycloak has no bulk lookup by username. It can now read the same people through portal-conductor instead, which queries the directory Keycloak federates and answers a whole listing in one request. That backend also restores the institution. The service reads it from the `o` attribute, and keycloak_config creates no LDAP attribute mapper for `o`, so under the Keycloak backend it is empty for everyone -- where Grouper reported it because iplant-groups read the directory directly. portal-conductor does the same, so nothing about the Keycloak realm has to change. groups_userinfo_backend defaults to keycloak, so this changes nothing until a deployment asks for it, and the portal-conductor settings are rendered only when it does. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wjd21NTp4Ead7JYhx5sQUT
Decided: the DE will not add an LDAP attribute mapper for `o` to its Keycloak realm, and will read user attributes through portal-conductor instead. Keycloak cannot report an institution without that mapper, so under it every subject's institution is empty -- a field Grouper reported, because iplant-groups read the directory directly, and one the sharing dialog displays. portal-conductor reads that same directory and resolves a whole member listing in one request, where Keycloak has no bulk lookup by username and costs one request per member. It ships with every DE deployment already. The template now renders only the selected backend's settings, so a config no longer carries credentials for a service it does not consult. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wjd21NTp4Ead7JYhx5sQUT
The service no longer authenticates to Keycloak: it reads user attributes from portal-conductor, and it read the realm for nothing else. Its client registration, its view-users grant, and the groups_keycloak_* variables go with it, so the rendered config no longer carries credentials for a service the process never contacts. keycloak_config_client_service_account_roles is empty rather than removed. It is the list a DE-realm client role has to be granted from, and the note about view-users belonging to realm-management rather than to the realm is worth keeping for whatever needs it next. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wjd21NTp4Ead7JYhx5sQUT
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Deployment for the new
groupsservice, the Grouper importer, and the config that pointspermissions,apps, andgroup-propagatorat group data in the DE database.What's here
roles/services/groups— the service role, with the four wiring edits (source_repos,build_it.yml,deploy_it.yml,kubernetes.yml). Carries thede_ca_*partials, since its Keycloak call is HTTPS.ansible/grouper_import.yml— runs the importer as a one-off Job, with dry-run support. It waits, prints the report, and fails on a non-success exit so a bad import cannot look like a good one.grouper_import_suspended: trueby default. Deliberately carries no CA partials: it reaches Grouper and the DE database over postgresql withsslmodedisabled and permissions over plain HTTP, so it makes no TLS connection at all.keycloak_configgains a client for the groups service and a newkeycloak_config_client_service_account_roleslist.roles/services/permissions— config gainsusers.suffixandpermissions.max_expanded_subjects. Thegrouperdbsection is annotated as rollback-only rather than removed: the current build ignores it, but the previous image needs it to start.appsandgroup-propagator—iplant_groups.*config becomesgroups.*, andbaseurls_groupsjoins the common defaults.ansible/scripts/seed-local-grouper.py— an idempotent, representative dataset for a local Grouper, so the importer can be tested against something with nesting and privileges rather than one group.ansible/community_tags.yml— runs the one-off community-tag rewrite as a Job, with dry-run support, credentials in a Secret rather than on a command line. It changes an app's community tag from the community's name to its ID, so renaming a community stops dropping every app tagged with it out of its own listing. Run it once, after the group import.apps_communities_attr— rendered into both apps' config and the migration Job. They have to agree: pointed at the wrong attribute the migration finds nothing and reports a clean run, which looks exactly like success.okf index+okf checkreport 0 errors and 0 warnings across 90 files.Draft, and why
The original blocker — a seed descriptor with no digest — is resolved;
files/groups.jsonnow carries a real pushed image.What remains is ordering. The groups service needs migration
000054(cyverse-de/de-database#43) applied first; the permissions config pairs with cyverse-de/permissions#31; the apps and group-propagator config pairs with cyverse-de/apps#303 and cyverse-de/group-propagator#8. None are merged, and the apps/group-propagator config fails fast against their current images, which is intended but means these land together or not at all.The four build descriptors must be bumped in the same merge as this branch. The committed diff rewrites the apps, group-propagator, and permissions config templates for the new images while
files/apps.json,files/group-propagator.json,files/permissions.json, andfiles/terrain.jsonstill pin pre-migration builds — deployed as committed, apps and group-propagator fail at startup on missing config keys. A reviewer reading only this diff sees a self-consistent change that is not; the descriptor bumps ride uncommitted in the working tree until the paired service PRs merge and release images exist.appsandterrainbuild against the releasedcommon-swagger-api3.4.22(cyverse-de/common-swagger-api#101), pinned on both branches, so the former release gate on this set is cleared.Two product bugs this deployment found
The importer CronJob had no image digest. It is applied with
kubernetes.core.k8srather than through skaffold, so nothing substituted the tag the way it does for a Deployment. The bare name resolved togroups:latest, which is never pushed — every scheduled import in every environment would have sat inImagePullBackOff, reading as a registry problem rather than a manifest one.grouper_import.ymlhad the same missing-digest bug. It named the groups image without a tag, which resolves to:latestand is never pushed — so the playbook the runbook tells operators to use would have sat inImagePullBackOffexactly as the CronJob would have. Two instances of one defect, both found by running the thing rather than reading it.The groups service had no Keycloak client, and needed a grant
keycloak_configcould not express. Unlike terrain's admin client, which lives in master because terrain builds its token URL againstrealms/masterby name, groups logs in against the DE realm and queries it — so its client belongs there, andview-usersis a client role of the built-inrealm-managementclient, not a realm role. Granting it through the wrong list would have produced a service account that authenticates and then 403s on every lookup.The operational thing worth knowing
The importer has to run on a schedule through the soak. Once the permissions service reads group data from this database while terrain and apps still write to Grouper, any group change made in between is invisible until the next import — a user added to a collaborator list simply does not get access. The CronJob exists for exactly that window and should be suspended again once terrain, apps, group-propagator, and Sonora have cut over.
Note also that once
appscuts over it createsworkshop-usersnatively, so the import report will list groups Grouper never had. The importer distinguishes those from groups Grouper actually lost; both lists are printed separately.Verified
The whole stack runs end to end on a single-node local cluster alongside a full Grouper deployment: migration applied through ansible, all five services on their migration images, the importer converging to zeros against a seeded 9-group dataset with two-level nesting, and nested-group iRODS propagation carrying a member reachable only at depth 2. The community-tag rewrite was exercised through this playbook against the deployed image: a dry run, a real run rewriting 2 rows and reporting 1 orphan, then a re-run reporting zeros.