[sf-brahm-gaur]

Why

The CumulusCI connected app (the built-in app used by cci org connect) is adopting Salesforce connected app security controls:

• PKCE is required on the authorization code flow
• Refresh token rotation: every refresh returns a new refresh token and invalidates the old one
• 30-day idle limit: a refresh token that isn't used for 30 days expires

Current cci releases don't send PKCE, so cci org connect fails once the app requires it. Under rotation, current cci also loses tokens: several commands refresh without saving the new refresh token, so the next command uses a rotated-out token, and Salesforce revokes the whole token family.

What changed

PKCE (cumulusci/oauth/client.py, cumulusci/cli/org.py)

• Adds generate_pkce_pair() (RFC 7636, S256, 128-character verifier).
• Adds an opt-in pkce flag to OAuth2ClientConfig. When it's set, the authorize URL carries code_challenge and code_challenge_method=S256, and the token request carries code_verifier.
• cci org connect turns it on. Sending PKCE to an app that doesn't require it is harmless, so this works before and after the app changes. Marketing Cloud and other OAuth clients are unchanged.

Refresh token rotation (cumulusci/core/config/org_config.py, cumulusci/core/keychain/)

• OrgConfig.refresh_oauth_token() holds a per-org lock (<org>.org.lock, via filelock; the holding process can take it again). Under the lock it re-reads the org and adopts the stored tokens if another cci process changed them since this process loaded or last saved the org. It doesn't adopt them if the org was connected again as another org or user.
• When a refresh returns a new refresh token, the org is saved immediately and marked with refresh_token_rotation. This covers every caller, including the ones that didn't save the rotated token: SalesforceCommand tasks, the Metadata API INVALID_SESSION_ID retry, and org shell when its script raises.
• With rotation, each refresh also invalidates the access token issued by the previous refresh, which would end any other cci process's session. So for orgs marked with refresh_token_rotation, cci checks the stored access token against /services/oauth2/userinfo and refreshes only when it gets 401/403. Other errors are raised rather than triggering a refresh.
• Saving any org with a refresh token takes the same lock and keeps tokens another process rotated, so a process that hasn't seen a rotation yet can't write back a dead token. An org from a new cci org connect still replaces the stored one.
• Org files are written to a temporary file, synced, and moved into place, so other processes never read a half-written file. If that isn't possible (directory not writable, file open on Windows, file mounted on its own), cci writes in place as before.
• Timeouts: the userinfo check gives up after 20s. The token refresh has a 10s connect timeout and no read timeout, because it isn't safe to retry: if Salesforce rotated the token but the response was lost, sending the old token again would revoke the token family. A process that waits more than 60s for the lock gets "Timed out after 60 seconds waiting for another cci process to finish updating org X. Try again."
• Orgs on connected apps that don't rotate never get the flag and keep refreshing as before.
• New keychain methods: org_lock() and reload_org(). The base keychain and SubprocessKeychain implement them as no-ops; EncryptedFileProjectKeychain uses the org file, except with CUMULUSCI_KEYCHAIN_CLASS=EnvironmentProjectKeychain, which never writes org files.
• Requires filelock>=3.15.2.

Error message

• An invalid_grant refresh failure now raises: "Salesforce rejected the refresh token for org X: <Salesforce's error_description>. If the token expired from inactivity or was revoked, run cci org connect X to connect the org again." Other OAuth errors are unchanged. OAuth2Error now carries the HTTP response.

Not affected: JWT auth (SFDX_CLIENT_ID + SFDX_HUB_KEY), sf CLI–backed orgs (scratch orgs and cci org import), and the GitHub device flow.

Notes for upgraders

• CI: orgs passed in CUMULUSCI_ORG_* environment variables with a refresh token don't work under rotation across CI jobs: the secret holds a fixed refresh token that's dead after the first job's refresh. Unused secrets also expire after 30 days. Use JWT auth instead, as docs/headless.md already describes.
• Older cci versions sharing ~/.cumulusci with this one don't take the lock or save rotated tokens. Upgrade every install that uses the same orgs.
• Custom OrgConfig.OAuth2Client: refresh_token() is now called with a timeout keyword argument.
• Known limitation (follow-up PR, called out in the release notes): Snowfakery parallel workers don't save a refresh token they rotate. Workers reuse the access token the parent just checked, but if it expires during a load, a worker refreshes without saving, and the next refresh with the old token (by the parent or another worker) makes Salesforce revoke the whole token family. Run cci org connect <org> to recover. This already happens on main, where every worker refreshes when it starts.

Testing

• New unit tests for PKCE, saving rotated tokens, adopting tokens rotated by another process, access-token reuse, the lock (timeout, re-entry, unwritable directory, Windows lock file), the atomic write (fallbacks, full disk, symlinks), reconnecting as another user (an identity URL on another host still counts as the same user), SubprocessKeychain, and the invalid_grant message. Each fix was checked by reverting it and confirming a test fails.
• Full suite with network calls blocked: 3269 passed. The 15 failures (test_cci ×8, test_org_list ×2, test_config_expensive ×3, test_cross_project_tasks, UninstallPackagedIncremental) also fail on main.
• Manual, against a test external client app with PKCE, rotation, and the idle limit turned on:
  • org connect (PKCE login accepted), then repeated org info, org browser, org browser -r, org shell, a SalesforceCommand task, and a flow
  • 3 parallel flows on the same org
  • 3 parallel flows with an expired session (exactly one refresh happened, the others used the new token)
  • a second process waits for the lock, and gives up with the timeout message after 60 seconds
  • a revoked or rotated-out refresh token gives the reconnect message

[sumeet-parmar]

please review the feedback comments and make necessary changes

[sumeet-parmar]

[Fix before merge] EnvironmentProjectKeychain mode: reloading an out-of-date file makes the next refresh reuse a rotated-out token

When CUMULUSCI_KEYCHAIN_CLASS=EnvironmentProjectKeychain, _set_org updates self.orgs in memory but doesn't write the .org file. But reload_org still reads that file when it exists. _load_latest_tokens then sees a refresh token that's different from the one in memory and takes it, even though it's older.

Reproduced (org file on disk, then two refreshes in one process with _access_token_works → False):

refresh 1: sends TOKEN_1 → gets TOKEN_2 (kept in memory only)
refresh 2: reload_org reads the file → TOKEN_1 again → sends TOKEN_1
REFRESH TOKENS SENT: ['TOKEN_1', 'TOKEN_1']

Under rotation, the second call reuses a rotated-out token, so Salesforce revokes the whole family.

Suggestions: skip the file re-read when saving is disabled. Or, more generally, only take stored tokens when they're newer, e.g. save a token_saved_at timestamp or a counter with the tokens and compare.

[sf-brahm-gaur]

Fixed. With that keychain, reload_org no longer reads the file and org_lock doesn't lock, since the file is never written. Also, _load_latest_tokens now adopts stored tokens only if they differ from the token this config was loaded with or last saved, not from the token in memory. So an older file can't replace newer tokens in memory. Test: test_refresh_oauth_token__rotation_when_orgs_not_saved_to_files: refresh tokens sent are TOKEN_1, then TOKEN_2.

[sumeet-parmar]

[Fix before merge] Only processes that already have the rotation flag are protected here

The lock and merge in save() depend on the in-memory refresh_token_rotation. If a process loaded the org before rotation was ever detected, it has no flag and writes its out-of-date tokens with a plain set_org, without the lock.

Reproduced:

1. Process A loads the org (TOKEN_1, no flag)
2. Process B refreshes, rotates, and saves TOKEN_2 + refresh_token_rotation: True
3. A changes something and calls save() without refreshing first

STORED AFTER A.save(): TOKEN_1 None

The dead token is back in the file and the flag is gone, so the next command reuses TOKEN_1 and the family gets revoked.

Suggestion: also look at the flag in the stored file. Or always take the lock and merge tokens for orgs that use refresh tokens (it's cheap). Be careful not to break cci org connect re-connects, which should replace the stored tokens.

Related, but code can't fix it: an older cci version sharing ~/.cumulusci (e.g. a pipx install alongside a project venv) will always write back out-of-date tokens. That's worth a release note.

[sf-brahm-gaur]

Fixed. save() now locks and merges for any org with a refresh token, using the same "changed since loaded" check. Org configs that didn't come from the keychain (cci org connect) still replace the stored tokens. Tests: test_save__keeps_first_rotation_by_another_process, test_save__connected_org_replaces_stored_tokens. The older-cci-sharing-~/.cumulusci case goes in the release notes.

[sumeet-parmar]

[Fix before merge] Network calls run while the lock is held, with no timeout

This requests.get (and the token POST in _refresh_token) runs while the per-org file lock is held, and neither call has a timeout. If either one hangs (VPN drop, proxy, Salesforce maintenance), the lock stays held indefinitely. Every other cci process on that org then waits ORG_LOCK_TIMEOUT (60s) and fails with a raw filelock.Timeout.

filelock.Timeout isn't a CumulusCIException, so users see something like:

Error: The file lock '/Users/.../.cumulusci/<project>/dev.org.lock' could not be acquired.

Suggestions:

• Pass an explicit timeout= to both HTTP calls (ideally well under ORG_LOCK_TIMEOUT)
• Catch filelock.Timeout in _refresh_lock/org_lock and raise a CumulusCIException that says another cci process is refreshing org X and to retry

[sf-brahm-gaur]

Fixed. The userinfo GET times out after 20s (USERINFO_TIMEOUT). The refresh POST only gets a 10s connect timeout (TOKEN_REFRESH_TIMEOUT = (10, None)), because it isn't safe to retry under rotation: if Salesforce rotated the token and the response timed out, the next run would send the old token and revoke the whole family. In that rare case, waiting processes get the lock timeout instead, which is safe. filelock.Timeout becomes a CumulusCIException: "Timed out after 60 seconds waiting for another cci process to finish updating org X. Try again."

[sumeet-parmar]

[Fix before merge] Orgs that don't rotate now fail to refresh when the directory isn't writable

The lock creates <org>.org.lock next to the org file. If the directory isn't writable but the .org file is, which happens with some Docker/CI images and shared setups, every refresh now fails. That includes orgs that never rotate, which used to work:

PermissionError: [Errno 13] Permission denied: '.../.cumulusci/TestProject/test.org.lock'

Suggestion: catch OSError when creating or acquiring the lock, log at debug level, and fall back to nullcontext(). That's the same behavior as having no lock, which is what everyone has today.

[sf-brahm-gaur]

Fixed. An OSError while acquiring logs at debug level and continues without the lock. cci creates the lock file itself before acquiring, because filelock on Windows takes EACCES when creating it to mean the lock is held and retries until the timeout (and os.access(dir, W_OK) is always true for directories on Windows). The atomic write from #6 also falls back to writing in place. Tests: test_org_lock__directory_not_writable, test_org_lock__lock_file_cannot_be_created, test_save_org__directory_not_writable.

[sumeet-parmar]

[Should fix] The new message hides Salesforce's error_description

handle_exception in cli/cci.py prints only str(error); the chained from e goes only to the log file. invalid_grant isn't always an expired or revoked token. Salesforce also returns it for inactive user, ip restricted, user hasn't approved this consumer, authentication failure, and a sandbox/prod login URL mix-up. In those cases, telling the user to run cci org connect sends them the wrong way.

Suggestion: parse the response and include error_description, e.g.

The refresh token for org dev is no longer valid (Salesforce said: "expired access/refresh token"). ...

Small nit: matching on "invalid_grant" in str(e) is fragile. Exposing the status/JSON on OAuth2Error would make this sturdier.

[sf-brahm-gaur]

Fixed. OAuth2Error now carries response. The message is "Salesforce rejected the refresh token for org X: <error_description>. If the token expired from inactivity or was revoked, run cci org connect X...". It matches on the JSON error field, not on a substring.

[sumeet-parmar]

[Should fix] Org files aren't written atomically, and they're now re-read on every refresh

_save_org truncates the file (O_TRUNC) and then writes it, so the write isn't atomic. This PR makes reload_org run on every refresh, including orgs that don't rotate, but several writers don't take the lock: cci org connect (new OrgConfig with no flag), save() on orgs without the flag, and set_default_org. If a read lands in the middle of a write, it gets a truncated file and fails mid-flow with OrgCannotBeLoaded/KeychainKeyNotFound.

It's unlikely, but the fix is cheap: write to a temp file in the same directory and os.replace() it into place (keeping SERVICE_ORG_FILE_MODE).

[sf-brahm-gaur]

Fixed. cci writes a unique temp file next to the org file (tempfile.mkstemp, mode 0600), fsyncs it, and calls os.replace, following a symlinked org file. On PermissionError (directory not writable, or the file is open on Windows) or EBUSY (org file mounted on its own), it writes in place as before. Other errors (e.g. disk full) are raised without touching the org file. Tests: test_save_org__no_temp_file_left, test_save_org__file_in_use, test_save_org__write_fails, test_save_org__symlinked_org_file, test_save_org__org_file_mounted_on_its_own.

[sumeet-parmar]

[Should fix] Taking the lock twice in one process blocks for 60s

The comment above explains that set_org is used because save() would take the lock again. Each org_lock() call creates a new FileLock, and on Unix flock locks belong to the open file descriptor. So if one process takes the lock twice, the second attempt blocks for ORG_LOCK_TIMEOUT and then raises Timeout. Any later change that calls save() (or something that calls it) inside this block will hit that.

Suggestions:

• Use FileLock(..., is_singleton=True) (or cache one lock per path) so taking it twice in the same process is safe
• filelock is added to pyproject.toml with no version floor. Please add a minimum (e.g. the version where is_singleton was added) so older installs don't behave differently

[sf-brahm-gaur]

Fixed. FileLock(..., is_singleton=True), so the refresh path now calls save() directly. The pin is filelock>=3.15.2: is_singleton came in 3.13.0, but singleton fixes kept landing until 3.15.2.

[sumeet-parmar]

@sf-brahm-gaur thanks for this. The PKCE implementation looks correct (128-char verifier, S256, unpadded base64url, regenerated on every login, scoped to cci org connect only), and JWT, sf CLI/scratch orgs, and connected apps that don't rotate behave as before. All of the new tests pass locally; the only failures (test_org_list x2 and test_server_domain_from_repo_info) fail the same way on main.

I found a few rotation edge cases. In two of them, a rotated-out refresh token still gets reused, and Salesforce then revokes the whole token family. I reproduced these with throwaway tests against this branch. Inline comments below:

Should be fixed before merge

1. EnvironmentProjectKeychain mode: reload_org reads an out-of-date org file, so the next refresh reuses a dead token
2. save() only takes the lock and merges stored tokens when this process already has the rotation flag
3. The lock is held during network calls that have no timeout, and filelock.Timeout reaches the user as a raw error
4. Refresh now fails when the org directory isn't writable, even for orgs that don't rotate

Should fix
5. The new invalid_grant message hides Salesforce's error_description
6. Org files aren't written atomically, and reload_org now reads them on every refresh
7. Taking the lock twice in one process blocks, and filelock has no minimum version

[sf-brahm-gaur]

@sumeet-parmar thanks for the review. All 7 comments are addressed in 961ac98, replies inline. A few more fixes from further review are in the same commit:

• Snowfakery parallel workers no longer crash refreshing or saving an org (SubprocessKeychain gets no-op org_lock/reload_org).
• Stored tokens aren't adopted if the org was connected again as another org or user, and an unreadable org file is overwritten on save, as before.
• On Windows, cci creates the lock file itself, so an unwritable directory doesn't wait 60s for the lock.
• The token refresh has no read timeout, so a slow response can't lose a rotated token.
  Full suite and the manual tests against a rotating test app pass. Could you take another look?

[sumeet-parmar]

@sf-brahm-gaur please check this feedback:

1. Snowfakery workers can still revoke the tokens (reproduced). Workers use SubprocessKeychain, which never saves. If a worker's access token expires mid-load, the worker rotates the refresh token but doesn't save it. The parent then refreshes with TOKEN_1, and Salesforce revokes every token issued from it. Two workers refreshing at once would trigger this immediately. The developer's code comment admits it ("A refresh token rotated in a sub-process isn't saved either"), but the PR description and release notes don't mention it. Two options:

   • Have workers raise a clear error instead of refreshing an org that uses rotation.
   • Have workers take the org's file lock and save the new tokens.

2. The "same org/user" check compares the full identity URL (a cheap hardening). It treats https://login.salesforce.com/id/00D…/005… and https://<mydomain>/id/00D…/005… as different orgs. My test confirmed that a hostname difference alone puts the dead TOKEN_1 back in the file. I couldn't confirm whether Salesforce ever returns different hostnames between login and refresh. The developer's parallel-flow tests passed, so maybe not, but comparing just the trailing org ID and user ID would remove the risk entirely.

[sf-brahm-gaur]

@sumeet-parmar thanks, I looked into both.

1. Snowfakery workers. Agreed, it can still happen as you describe. It isn't new to this PR, though. On main, every worker refreshes the token when its load task starts, so once rotation is on, any run with two or more workers revokes the token family right away. With this PR, workers reuse the access token the parent just checked and only refresh if it expires mid-load, which makes it much less likely but doesn't rule it out.

• It will be called out as a known limitation in the 4.11.0 release notes, with cci org connect <org> to recover.
• I've reworded the limitation in the PR description so it says a worker refresh can revoke the token family, not just that the token isn't saved.
• The fix will come in a follow-up PR: workers raise a clear error instead of refreshing an org that uses rotation.

2. Identity check. Fixed in 4a314b6. It now compares only the org ID and user ID at the end of the identity URL, so a host change alone (login.salesforce.com vs. My Domain) keeps the rotated token. Connecting again as a different user or org is still detected. The new test test_save__identity_url_host_changed covers your scenario and fails with the old full-URL comparison. For what it's worth, on our rotating test app the stored identity URL kept the login host after many refreshes at the instance's token endpoint, so Salesforce doesn't seem to change it. Now we don't depend on that.

Full suite and the live tests against the rotating test app pass.

[sumeet-parmar]

looks good!

[mgrandhi]

LGTM