diff --git a/developer_docs/pulp_upgrade.md b/developer_docs/pulp_upgrade.md index a65d9f3191..28343f4338 100644 --- a/developer_docs/pulp_upgrade.md +++ b/developer_docs/pulp_upgrade.md @@ -4,6 +4,8 @@ Pulp is Katello's content management backend, responsible for storing and syncin Katello maintainers upgrade the y-version of Pulp and all Pulp plugins every two Katello releases (currently odd-numbered Katello y-versions). Pulp y-versions may only be updated with thorough testing, while Pulp z-versions may be updated with checks to the changelog. We do not typically re-record VCRs with Pulp z-version updates since the API should not change. +**This guide covers the containerized Pulp workflow** using foremanctl development environments where Pulp runs in Podman containers. + The following guide demonstrates a typical Pulp upgrade procedure: ### Phase 1 @@ -11,52 +13,156 @@ Begin this phase no later than 1 month after Katello branching. 1. **Coordinate with Pulp team**: Alert the Pulp team to upgrade plans and request version recommendations. Ensure Pulpcore version has full plugin support. 2. **Check for breaking changes**: Review deprecations and functionality changes in the new Pulp version that may require Katello code changes. -3. **Backup your environment**: Create a VM snapshot or use a fresh katello-devel Vagrant box. +3. **Backup your environment**: Create a VM snapshot or use a fresh Vagrant development box (`./forge vms start`). 4. **Update client bindings only**: - - Update all `pulp-*-client` dependencies in `katello.gemspec` using versions from [PyPI](https://pypi.org/project/pulpcore/) + - Update all `pulp-*-client` dependencies in `~/katello/katello.gemspec` using versions from [RubyGems](https://rubygems.org/) - In `~/foreman`, run `bundle update && bundle pristine` - Run Pulp tests: `bundle exec rake test TEST=../katello/test/services/pulp3/` - Check for failures (early warning for N-1 smart proxy sync issues) -5. **Install target Pulpcore and plugins** -Using values from gemspec (verify pulp-\*-client was released alongside pulp-\* package), run the following: +5. **Build custom Pulp container with target versions** + + Follow the [Building Custom Pulp Containers](https://github.com/theforeman/foremanctl/blob/master/docs/developer/development-environment.md#building-custom-pulp-containers) guide in foremanctl documentation to build a container with your target Pulp versions. Make sure to verify that pulp-\*-client bindings are available on [PyPI](https://pypi.org/) for all versions you pin in `requirements.txt`. +6. **Deploy the custom Pulp container** + + Deploy the development environment with your custom Pulp image (see step 5). Database migrations run automatically during deployment via the `pulpcore-manager-migrate.service`. + +7. **Verify Pulp upgrade** + + SSH to the VM and verify the new Pulp version: ```bash - sudo python3.12 -m pip install --upgrade --force-reinstall \ - pulpcore==X.Y.Z \ - pulp-ansible==X.Y.Z \ - pulp-container==X.Y.Z \ - pulp-deb==X.Y.Z \ - pulp-rpm==X.Y.Z \ - pulp-python==X.Y.Z \ - pulp-ostree==X.Y.Z + vagrant ssh quadlet + curl -k https://localhost/pulp/api/v3/status/ | jq '.versions[] | select(.component == "core")' ``` - **Notes:** - - `pulp-file` and `pulp-certguard` plugins have merged into pulpcore. Only client bindings are required. - - This step may require installing/updating many dependencies. Be sure to record the upgrade dependency requirements and have patience. Ensure these new dependencies are understood by all parties. -6. **Update systemd service files** to use pip-installed binaries (do not switch configuration away from the 'pulp' user): - - `/etc/systemd/system/pulpcore-api.service`: `ExecStart=/usr/local/bin/pulpcore-api` - - `/etc/systemd/system/pulpcore-content.service`: `ExecStart=/usr/local/bin/pulpcore-content` - - `/etc/systemd/system/pulpcore-worker@.service`: `ExecStart=/usr/local/bin/pulpcore-worker` -7. **Restart Pulp services**: + + Verify all Pulp services are running: ```bash - sudo systemctl daemon-reload - sudo systemctl restart pulpcore* --all + sudo systemctl is-active pulp-api pulp-content pulp-worker@{1..4} ``` -8. **Run Pulp migrations**: - ```bash - sudo -u pulp PULP_SETTINGS='/etc/pulp/settings.py' \ - DJANGO_SETTINGS_MODULE='pulpcore.app.settings' \ - /usr/local/bin/pulpcore-manager migrate - ``` -9. **Verify versions for installed plugins**: `sudo pulp status` -10. **Run a quick smoke test**: Restart Katello and try syncing content of all content types (RPM, container, deb, etc.) -11. **Request RPM builds**: Post to Foreman community requesting RPM builds for new Pulpcore & plugins. Anticipate 1 month for RPM builds. [Example](https://community.theforeman.org/t/request-for-pulpcore-3-85-builds/44413) + +8. **Run a quick smoke test**: Try syncing content of all content types (RPM, container, deb, etc.) +9. **Request RPM builds**: Post to Foreman community requesting RPM builds for new Pulpcore & plugins. Anticipate 1 month for RPM builds. [Example](https://community.theforeman.org/t/request-for-pulpcore-3-85-builds/44413) ### Phase 2 Begin this phase once Pulp RPMs are ready. 1. **Run unit tests with new bindings**: Run unit tests with new Pulp client bindings but old VCR recordings 2. **Remove old monkey patches**: Check for N-1/N-2 patches that can be removed. N-1/N-2 testing will prove removal safety. -3. **Re-record VCRs**: Follow instructions in [Testing & Code Quality - VCR Testing](./testing_and_code_quality.md#vcr-video-cassette-recorder-testing) +3. **Re-record VCRs**: After deploying the custom Pulp container (step 6), prepare the environment for VCR recording. + + **Important**: Do not redeploy after this point, as `./forge deploy-dev` will reset git repositories and undo your gemspec changes. + + SSH to the VM and configure Pulp for VCR recording: + ```bash + vagrant ssh quadlet + + # Set PULP_ORPHAN_PROTECTION_TIME=0 for VCR recording + sudo mkdir -p /etc/containers/systemd/pulp-api.container.d + sudo mkdir -p /etc/containers/systemd/pulp-content.container.d + sudo mkdir -p /etc/containers/systemd/pulp-worker@.container.d + + sudo tee /etc/containers/systemd/pulp-api.container.d/vcr.conf <<'EOF' + [Container] + Environment=PULP_ORPHAN_PROTECTION_TIME=0 + EOF + + sudo tee /etc/containers/systemd/pulp-content.container.d/vcr.conf <<'EOF' + [Container] + Environment=PULP_ORPHAN_PROTECTION_TIME=0 + EOF + + sudo tee /etc/containers/systemd/pulp-worker@.container.d/vcr.conf <<'EOF' + [Container] + Environment=PULP_ORPHAN_PROTECTION_TIME=0 + EOF + + sudo systemctl daemon-reload + sudo systemctl restart pulp-api pulp-content pulp-worker@{1..4} + ``` + + Update Katello code for the new Pulp version: + ```bash + cd ~/katello + # Update all pulp-*-client dependencies in katello.gemspec to match target versions + # Fix any breaking API changes (e.g., renamed classes in client bindings) + ``` + + Update bundle and restart Rails: + ```bash + cd ~/foreman + rm Gemfile.lock + bundle install --path .vendor --jobs 3 --without journald + pkill -f "rails.*server" + bundle exec foreman start & + ``` + + Follow the remaining VCR recording steps from [Testing & Code Quality - VCR Testing](./testing_and_code_quality.md#vcr-video-cassette-recorder-testing). + + **Note for containerized environments**: The VCR testing guide uses `bundle exec rake katello:reset` which is designed for pip-based Pulp installations. For containerized environments, manually reset backend databases and the Foreman database: + + Reset Candlepin database: + ```bash + sudo systemctl stop candlepin + sudo podman exec postgresql bash -c "dropdb -U postgres candlepin" + sudo podman exec postgresql bash -c "createdb -U postgres -O candlepin candlepin" + sudo systemctl start candlepin + sleep 5 + sudo podman exec candlepin /usr/share/candlepin/cpdb --update + ``` + + Reset Pulp database: + ```bash + sudo systemctl stop pulp-api pulp-content pulp-worker@*.service + sudo podman exec postgresql bash -c "dropdb -U postgres pulp" + sudo podman exec postgresql bash -c "createdb -U postgres -O pulp pulp" + sudo podman exec postgresql bash -c "psql -U postgres -d pulp -c 'create extension hstore'" + sudo systemctl restart pulpcore-manager-migrate.service + sudo systemctl restart pulpcore-manager-admin-password.service + sudo systemctl start pulp-api pulp-content pulp-worker@{1..4} + ``` + + Reset Foreman databases (step 6-7 in VCR guide): + ```bash + # Stop Rails server + pkill -f "rails.*server" + pkill -9 puma + bundle exec spring stop + + # Reset development database + cd ~/foreman + bundle exec rake db:drop + bundle exec rake db:create + bundle exec rake db:migrate + bundle exec rake db:seed + + # Reset test database + RAILS_ENV=test bundle exec rake db:drop + RAILS_ENV=test bundle exec rake db:create + RAILS_ENV=test bundle exec rake db:migrate + RAILS_ENV=test bundle exec rake db:seed + ``` + + Restart Rails server: + ```bash + cd ~/foreman + bundle exec foreman start & + ``` + + After database resets, re-register the Pulp smart proxy: + ```bash + cd ~/foreman + bundle exec rails console + ``` + + In the Rails console: + ```ruby + hostname = `hostname -f`.strip + proxy = SmartProxy.find_or_create_by!( + url: "https://#{hostname}/pulp/api/v3/smart_proxy", + name: "#{hostname}-pulp" + ) + proxy.refresh + exit + ``` 4. **File Pulp bugs**: Investigate errors and file any upstream issues. 5. **Test N-1 and N-2 compatibility**: Create smart proxies with last Pulp version (N-1) and previous (N-2). Test syncing with/without alternate content sources and updating content counts. 6. **Handle binding compatibility issues**: If new Pulp bindings don't work with older Pulp versions, create monkey patches as workarounds.