Fixes #39727 - Update Pulp upgrade guide for containerized workflow

developer_docs/pulp_upgrade.md

@@ -4,59 +4,165 @@ Pulp is Katello's content management backend, responsible for storing and syncin
 
 Katello maintainers upgrade the y-version of Pulp and all Pulp plugins every two Katello releases (currently odd-numbered Katello y-versions). Pulp y-versions may only be updated with thorough testing, while Pulp z-versions may be updated with checks to the changelog. We do not typically re-record VCRs with Pulp z-version updates since the API should not change.
 
+**This guide covers the containerized Pulp workflow** using foremanctl development environments where Pulp runs in Podman containers.
+
 The following guide demonstrates a typical Pulp upgrade procedure:
 
 ### Phase 1
 Begin this phase no later than 1 month after Katello branching.
 
 1. **Coordinate with Pulp team**: Alert the Pulp team to upgrade plans and request version recommendations. Ensure Pulpcore version has full plugin support.
 2. **Check for breaking changes**: Review deprecations and functionality changes in the new Pulp version that may require Katello code changes.
-3. **Backup your environment**: Create a VM snapshot or use a fresh katello-devel Vagrant box.
+3. **Backup your environment**: Create a VM snapshot or use a fresh Vagrant development box (`./forge vms start`).
 4. **Update client bindings only**:
-   - Update all `pulp-*-client` dependencies in `katello.gemspec` using versions from [PyPI](https://pypi.org/project/pulpcore/)
+   - Update all `pulp-*-client` dependencies in `~/katello/katello.gemspec` using versions from [RubyGems](https://rubygems.org/)
    - In `~/foreman`, run `bundle update && bundle pristine`
    - Run Pulp tests: `bundle exec rake test TEST=../katello/test/services/pulp3/`
    - Check for failures (early warning for N-1 smart proxy sync issues)
-5. **Install target Pulpcore and plugins**
-Using values from gemspec (verify pulp-\*-client was released alongside pulp-\* package), run the following:
+5. **Build custom Pulp container with target versions**
+
+   Follow the [Building Custom Pulp Containers](https://github.com/theforeman/foremanctl/blob/master/docs/developer/development-environment.md#building-custom-pulp-containers) guide in foremanctl documentation to build a container with your target Pulp versions. Make sure to verify that pulp-\*-client bindings are available on [PyPI](https://pypi.org/) for all versions you pin in `requirements.txt`.
+6. **Deploy the custom Pulp container**
+
+   Deploy the development environment with your custom Pulp image (see step 5). Database migrations run automatically during deployment via the `pulpcore-manager-migrate.service`.
+
+7. **Verify Pulp upgrade**
+
+   SSH to the VM and verify the new Pulp version:
    ```bash
-   sudo python3.12 -m pip install --upgrade --force-reinstall \
-       pulpcore==X.Y.Z \
-       pulp-ansible==X.Y.Z \
-       pulp-container==X.Y.Z \
-       pulp-deb==X.Y.Z \
-       pulp-rpm==X.Y.Z \
-       pulp-python==X.Y.Z \
-       pulp-ostree==X.Y.Z
+   vagrant ssh quadlet
+   curl -k https://localhost/pulp/api/v3/status/ | jq '.versions[] | select(.component == "core")'
    ```
-   **Notes:**
-   - `pulp-file` and `pulp-certguard` plugins have merged into pulpcore. Only client bindings are required.
-   - This step may require installing/updating many dependencies. Be sure to record the upgrade dependency requirements and have patience. Ensure these new dependencies are understood by all parties.
-6. **Update systemd service files** to use pip-installed binaries (do not switch configuration away from the 'pulp' user):
-   - `/etc/systemd/system/pulpcore-api.service`: `ExecStart=/usr/local/bin/pulpcore-api`
-   - `/etc/systemd/system/pulpcore-content.service`: `ExecStart=/usr/local/bin/pulpcore-content`
-   - `/etc/systemd/system/pulpcore-worker@.service`: `ExecStart=/usr/local/bin/pulpcore-worker`
-7. **Restart Pulp services**:
+
+   Verify all Pulp services are running:
    ```bash
-   sudo systemctl daemon-reload
-   sudo systemctl restart pulpcore* --all
+   sudo systemctl is-active pulp-api pulp-content pulp-worker@{1..4}
    ```
-8. **Run Pulp migrations**:
-    ```bash
-    sudo -u pulp PULP_SETTINGS='/etc/pulp/settings.py' \
-      DJANGO_SETTINGS_MODULE='pulpcore.app.settings' \
-      /usr/local/bin/pulpcore-manager migrate
-    ```
-9. **Verify versions for installed plugins**: `sudo pulp status`
-10. **Run a quick smoke test**: Restart Katello and try syncing content of all content types (RPM, container, deb, etc.)
-11. **Request RPM builds**: Post to Foreman community requesting RPM builds for new Pulpcore & plugins. Anticipate 1 month for RPM builds. [Example](https://community.theforeman.org/t/request-for-pulpcore-3-85-builds/44413)
+
+8. **Run a quick smoke test**: Try syncing content of all content types (RPM, container, deb, etc.)
+9. **Request RPM builds**: Post to Foreman community requesting RPM builds for new Pulpcore & plugins. Anticipate 1 month for RPM builds. [Example](https://community.theforeman.org/t/request-for-pulpcore-3-85-builds/44413)
 
 ### Phase 2
 Begin this phase once Pulp RPMs are ready.
 
 1. **Run unit tests with new bindings**: Run unit tests with new Pulp client bindings but old VCR recordings
 2. **Remove old monkey patches**: Check for N-1/N-2 patches that can be removed. N-1/N-2 testing will prove removal safety.
-3. **Re-record VCRs**: Follow instructions in [Testing & Code Quality - VCR Testing](./testing_and_code_quality.md#vcr-video-cassette-recorder-testing)
+3. **Re-record VCRs**: After deploying the custom Pulp container (step 6), prepare the environment for VCR recording.
+
+   **Important**: Do not redeploy after this point, as `./forge deploy-dev` will reset git repositories and undo your gemspec changes.
+
+   SSH to the VM and configure Pulp for VCR recording:
+   ```bash
+   vagrant ssh quadlet
+
+   # Set PULP_ORPHAN_PROTECTION_TIME=0 for VCR recording
+   sudo mkdir -p /etc/containers/systemd/pulp-api.container.d
+   sudo mkdir -p /etc/containers/systemd/pulp-content.container.d
+   sudo mkdir -p /etc/containers/systemd/pulp-worker@.container.d
+
+   sudo tee /etc/containers/systemd/pulp-api.container.d/vcr.conf <<'EOF'
+   [Container]
+   Environment=PULP_ORPHAN_PROTECTION_TIME=0
+   EOF
+
+   sudo tee /etc/containers/systemd/pulp-content.container.d/vcr.conf <<'EOF'
+   [Container]
+   Environment=PULP_ORPHAN_PROTECTION_TIME=0
+   EOF
+
+   sudo tee /etc/containers/systemd/pulp-worker@.container.d/vcr.conf <<'EOF'
+   [Container]
+   Environment=PULP_ORPHAN_PROTECTION_TIME=0
+   EOF
+
+   sudo systemctl daemon-reload
+   sudo systemctl restart pulp-api pulp-content pulp-worker@{1..4}
+   ```
+
+   Update Katello code for the new Pulp version:
+   ```bash
+   cd ~/katello
+   # Update all pulp-*-client dependencies in katello.gemspec to match target versions
+   # Fix any breaking API changes (e.g., renamed classes in client bindings)
+   ```
+
+   Update bundle and restart Rails:
+   ```bash
+   cd ~/foreman
+   rm Gemfile.lock
+   bundle install --path .vendor --jobs 3 --without journald
+   pkill -f "rails.*server"
+   bundle exec foreman start &
+   ```
+
+   Follow the remaining VCR recording steps from [Testing & Code Quality - VCR Testing](./testing_and_code_quality.md#vcr-video-cassette-recorder-testing). 
+
+   **Note for containerized environments**: The VCR testing guide uses `bundle exec rake katello:reset` which is designed for pip-based Pulp installations. For containerized environments, manually reset backend databases and the Foreman database:
+
+   Reset Candlepin database:
+   ```bash
+   sudo systemctl stop candlepin
+   sudo podman exec postgresql bash -c "dropdb -U postgres candlepin"
+   sudo podman exec postgresql bash -c "createdb -U postgres -O candlepin candlepin"
+   sudo systemctl start candlepin
+   sleep 5
+   sudo podman exec candlepin /usr/share/candlepin/cpdb --update
+   ```
+
+   Reset Pulp database:
+   ```bash
+   sudo systemctl stop pulp-api pulp-content pulp-worker@*.service
+   sudo podman exec postgresql bash -c "dropdb -U postgres pulp"
+   sudo podman exec postgresql bash -c "createdb -U postgres -O pulp pulp"
+   sudo podman exec postgresql bash -c "psql -U postgres -d pulp -c 'create extension hstore'"
+   sudo systemctl restart pulpcore-manager-migrate.service
+   sudo systemctl restart pulpcore-manager-admin-password.service
+   sudo systemctl start pulp-api pulp-content pulp-worker@{1..4}
+   ```
+
+   Reset Foreman databases (step 6-7 in VCR guide):
+   ```bash
+   # Stop Rails server
+   pkill -f "rails.*server"
+   pkill -9 puma
+   bundle exec spring stop
+
+   # Reset development database
+   cd ~/foreman
+   bundle exec rake db:drop
+   bundle exec rake db:create
+   bundle exec rake db:migrate
+   bundle exec rake db:seed
+
+   # Reset test database
+   RAILS_ENV=test bundle exec rake db:drop
+   RAILS_ENV=test bundle exec rake db:create
+   RAILS_ENV=test bundle exec rake db:migrate
+   RAILS_ENV=test bundle exec rake db:seed
+   ```
+
+   Restart Rails server:
+   ```bash
+   cd ~/foreman
+   bundle exec foreman start &
+   ```
+
+   After database resets, re-register the Pulp smart proxy:
+   ```bash
+   cd ~/foreman
+   bundle exec rails console
+   ```
+
+   In the Rails console:
+   ```ruby
+   hostname = `hostname -f`.strip
+   proxy = SmartProxy.find_or_create_by!(
+     url: "https://#{hostname}/pulp/api/v3/smart_proxy",
+     name: "#{hostname}-pulp"
+   )
+   proxy.refresh
+   exit
+   ```
 4. **File Pulp bugs**: Investigate errors and file any upstream issues.
 5. **Test N-1 and N-2 compatibility**: Create smart proxies with last Pulp version (N-1) and previous (N-2). Test syncing with/without alternate content sources and updating content counts.
 6. **Handle binding compatibility issues**: If new Pulp bindings don't work with older Pulp versions, create monkey patches as workarounds.