Skip to content

fix(auth): revoke privileges immediately on role change, deactivation, or deletion - #9360

Open
lstein wants to merge 26 commits into
invoke-ai:mainfrom
lstein:fix/multiuser-privilege-revocation
Open

fix(auth): revoke privileges immediately on role change, deactivation, or deletion#9360
lstein wants to merge 26 commits into
invoke-ai:mainfrom
lstein:fix/multiuser-privilege-revocation

Conversation

@lstein

@lstein lstein commented Jul 17, 2026

Copy link
Copy Markdown
Collaborator

Summary

Follow-on PR 1 from @JPPhoto's review of #9163 (the "Database role changes do not invalidate JWT privileges" / "Open sockets retain revoked account privileges" / "Deactivated users can continue queued execution" findings).

Note

Stacked on #9163 — this branch is based on the WAN video branch because the fixes build on machinery that only exists there (media cookie, video invocation-context authorization). The diff will show #9163's changes until it merges; only the top commit (960d607c76) is new. I'll rebase/retarget once #9163 lands.

1. JWT privileges now derive from the database on every request

  • All auth dependencies (get_current_user, get_current_user_or_default, the media-cookie validator) build the returned TokenData from the database record — the token proves identity only. A demoted administrator's old token gets 403 on admin endpoints immediately; a promoted user gains admin on their next request without re-login (this defines the promotion semantics JPPhoto asked to pin down).
  • The sliding-window refresh middleware refuses to refresh for missing/inactive users (no X-Refreshed-Token, no media cookie) and mints refreshed tokens from the database record, so a stale admin claim can never be renewed — the media cookie renewal path is closed with it.

2. Open sockets are re-authorized live

  • _handle_connect derives is_admin from the database, so a demoted admin reconnecting with an old token does not rejoin the admin room.
  • A new server-internal user_access_changed event (not registered with payload_schema, never emitted to clients — no typegen churn) is emitted by the user-management routes on role/status changes. The socket layer responds: demotion leaves the admin room (and the cached is_admin is corrected so subscribe_queue can't re-add it), promotion joins it, deactivation/deletion disconnects all the user's sockets.

3. Deactivated users' queued execution is revoked

Policy: pending items are rejected (canceled) at dequeue; running items are canceled immediately where possible and always stop before the next node.

  • The session processor cancels dequeued items whose owner is inactive before any invocation runs.
  • The session runner revalidates the owner between nodes and cancels mid-session.
  • The processor also listens for user_access_changed and cancels the currently running item immediately — this drives the existing cancel-event machinery, so step-callback nodes (e.g. denoising) stop mid-node rather than running to completion.
  • Invocation-context media reads and saves now require an active account (previously only user existence was checked), so no output can be saved on behalf of a revoked account even as defense in depth.
  • Single-user mode and the system user are exempt, per the review spec.

Tests (all per JPPhoto's specs)

  • tests/app/routers/test_privilege_revocation.py — demoted admin: 403 + no X-Refreshed-Token + no media cookie; same token denied reading another user's private image (positive pre-demotion, negative post-demotion); promoted user gains admin with old token; unchanged admin refresh carries is_admin=true; demoted user's allowed mutation refreshes with is_admin=false; event emission on demotion/deactivation/deletion and not on display-name changes.
  • tests/app/api/test_sliding_window_token.py — new multiuser class: demoted/promoted refresh carries DB role; deactivated/deleted users get no refresh; remember_me preserved. (Existing tests now run under an explicit single-user harness.)
  • tests/app/test_socket_privilege_revocation.py — reconnect-with-old-token after demotion; deactivated reconnect rejected; live demotion leaves admin room and can't re-subscribe into it; deactivation/deletion disconnects; promotion joins; other users' sockets untouched (the positive unchanged-admin case).
  • tests/app/services/session_processor/test_privilege_revocation.pyqueue_owner_is_active matrix; dequeue rejection (incl. concurrent-deletion race); immediate cancel of the running item on deactivation; multi-node session stops after node 1 when the owner is deactivated mid-run; positive active-user and single-user/system cases.
  • tests/app/services/shared/test_invocation_context_{images,videos}.py — inactive/deleted queue user denied reads and saves (even uncategorized); active user still saves.

370 tests pass across the affected areas (auth routes, multiuser authorization, sockets, session queue/processor, invocation context, videos multiuser); ruff clean.

🤖 Generated with Claude Code

@github-actions github-actions Bot added api python PRs that change python files Root invocations PRs that change invocations backend PRs that change backend files services PRs that change app services frontend PRs that change frontend files python-tests PRs that change python tests docs PRs that change docs python-deps PRs that change python dependencies labels Jul 17, 2026
@lstein lstein mentioned this pull request Jul 17, 2026
7 tasks
@lstein lstein added the 6.14.1 label Jul 19, 2026
@lstein
lstein force-pushed the fix/multiuser-privilege-revocation branch from 960d607 to efae7b4 Compare July 20, 2026 00:14
@lstein lstein changed the title fix(auth): revoke privileges immediately on role change, deactivation, or deletion fix(auth): revoke privileges immediately on role change, deactivation, or deletion (REBASE AFTER 9163 MERGES) Jul 20, 2026
@lstein
lstein force-pushed the fix/multiuser-privilege-revocation branch from 7ff3da0 to 90e1f03 Compare July 31, 2026 02:01
@lstein
lstein marked this pull request as ready for review July 31, 2026 02:01
@lstein lstein changed the title fix(auth): revoke privileges immediately on role change, deactivation, or deletion (REBASE AFTER 9163 MERGES) fix(auth): revoke privileges immediately on role change, deactivation, or deletion Jul 31, 2026
@lstein
lstein force-pushed the fix/multiuser-privilege-revocation branch from 90e1f03 to 0c95caa Compare July 31, 2026 02:18
lstein and others added 4 commits July 31, 2026 11:12
…, or deletion

Three related gaps let stale credentials outlive database changes in
multiuser mode:

1. JWT privileges survived demotion. Authentication verified the user
   exists and is active but kept trusting the token's is_admin claim, and
   the sliding-window middleware re-minted new tokens (and the media
   cookie) from those stale claims — so a demoted administrator kept admin
   rights indefinitely as long as they kept making requests. All auth
   dependencies now derive authorization fields from the database record
   on every request (the token proves identity only), and the middleware
   refuses to refresh for missing/inactive users and mints refreshed
   tokens from the database record. A promoted user symmetrically gains
   admin rights on their next request without re-login.

2. Open sockets retained revoked privileges. Socket room membership was
   established once at connect from the token's claims. Connect now
   derives is_admin from the database, and a new server-internal
   user_access_changed event (emitted by the user-management routes)
   re-authorizes live sockets: demotion leaves the admin room, promotion
   joins it, deactivation/deletion disconnects the user's sockets.

3. Deactivated users' queued work kept executing. The session processor
   now rejects (cancels) dequeued items whose owner is inactive, stops
   running sessions at the next node boundary, and cancels the current
   item immediately when its owner is deactivated (which also stops
   step-callback nodes mid-node via the existing cancel-event machinery).
   Invocation-context media reads and saves also require an active
   account. Single-user mode and the "system" user are exempt.

Tests cover: demoted-admin 403 with no token/cookie refresh, DB-derived
refresh claims, promoted-user semantics, socket reconnect-with-old-token,
live socket demotion/promotion/deactivation, dequeue rejection,
multi-node mid-session deactivation, and invocation-context read/save
denial for inactive accounts, plus positive cases for unchanged admins,
active users, and single-user mode.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…dmin test stubs

Two CI failures introduced by the privilege-revocation commit:

- `UserAccessChangedEvent` is dispatched only between server components, but
  `EventBase.get_events()` sweeps in every subclass carrying `__event_name__`,
  and its sole consumer is the OpenAPI generator. The event therefore leaked
  into `openapi.json`/`schema.ts`, failing openapi-checks and typegen-checks
  and contradicting the event's own documented contract. Events can now opt out
  with `__server_internal__ = True`.

- The pre-existing "rejects non-admin users" tests in `test_app_info.py` stubbed
  the user lookup with a bare `Mock(is_active=True)`. Authorization is now
  derived from the database record on every request, so `TokenData` validation
  rejected the Mock-valued fields. The stub now carries concrete values.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A JWT is self-contained: nothing in the database can make an already-issued
token stop verifying. Authorization fields are re-derived from the user record
on every request, so demotion and deactivation take effect immediately — but a
password change had no way to invalidate anything. A stolen token therefore
outlived the password rotation meant to evict the thief, and because the
sliding window renews on every mutating request, it survived indefinitely
rather than expiring after a day.

Adds `users.token_epoch`, stamped into every minted token and compared against
the record on each authenticated request. Rotating the password bumps it in
SQL (`token_epoch = token_epoch + 1`, so concurrent bumps can't lose each
other), invalidating every token issued before the change. This is the general
"revoke everything issued so far" primitive the auth layer was missing.

Details worth noting:

- The sliding-window middleware also checks the epoch. It runs after the route
  and refreshes on any 2xx mutation, so an unauthenticated route carrying a
  stale bearer header would otherwise launder a revoked token into a valid one.
- Changing your own password signs out your *other* sessions, not the tab you
  did it from: the route mints a replacement and returns it in
  X-Refreshed-Token. The middleware can't do this — it correctly refuses to
  refresh a stale-epoch token, and leaves an already-set header alone.
- Rejection is reported as an ordinary invalid/expired token, so a stolen
  token's holder isn't told the password was just rotated.
- Existing rows and existing tokens both start at 0, so upgrading logs nobody
  out; only a real bump revokes.

Tests cover cross-session revocation, the calling session surviving, admin
password reset revoking the target, the refresh-laundering path, non-password
updates not revoking, and pre-existing tokens staying valid.
…ecks

`queue_owner_is_active` exempted `user_id == "system"` on the stated grounds
that the system user "has no database record". That premise is wrong:
migration_27 creates a real, active `system` row that owns every board, image,
and workflow carried over from before multiuser support.

Because the row exists and is active, the exemption changed nothing in normal
operation — it only took effect when the row was missing or inactive, which is
exactly the case where this gate then disagreed with the save gates in
`invocation_context`, which have no such exemption. A system-owned item would
pass the gate that decides whether to spend GPU time, load models and denoise,
then fail at the first `context.images.save()`. That is the worst possible
ordering for two checks that disagree.

Drops the exemption so all three checks agree: the system user now passes on
its own merits, and if its row is ever gone the item is rejected at dequeue
instead of after generating.

Also protects the row, since orphaning it is what made the disagreement
reachable in the first place. Neither `delete_user` nor `update_user` guarded
it — `list_users` merely hides it from the UI, and the last-admin guard does
not apply because the system row is deliberately not an admin. Deleting or
deactivating it is now rejected; it is not a login account, so there is no
legitimate reason to do either.
lstein and others added 5 commits August 8, 2026 22:32
…n guard

Both layers of the last-admin guard — the route pre-check and the service backstop
inside the write transaction — enforce "at least one row with is_admin AND
is_active". The property that actually matters is "at least one administrator who
can log in", and the `system` row satisfies the first without the second: it is
active, but migration_27 gives it an empty password hash, so `verify_password`
always fails.

Nothing stopped an admin from promoting it, and neither guard objects to a
promotion — it is not a change that revokes anything:

    PATCH /auth/users/system   {"is_admin": true}    -> 200, count_admins() 1 -> 2
    PATCH /auth/users/{admin}  {"is_admin": false}   -> 200, both guards see 2
    POST  /auth/login  system@system.invokeai        -> 401, empty password hash

Zero usable administrators, `has_admin()` still true so `/auth/setup` stays closed,
and no authenticated path back — precisely the outcome the guard exists to prevent.
Setting a password on the system row was the same hole from the other end: it turns
the owner of every pre-multiuser board, image, workflow, and queue item into a
login account.

The system-user protection also lived only in the routes, so `invoke-userdel
--email system@system.invokeai --force` deleted it and reported success, and
`UserService.update` would deactivate it — the CLIs construct the service directly.
That is the same reasoning that moved the last-admin invariant into the service.

So: `SYSTEM_USER_ID` and the two refusal messages move to `users_common` beside
their errors, and `UserService.update`/`delete` refuse to delete, deactivate,
promote, or set a password on that row. Renaming it stays allowed — this is not a
blanket lock. `SystemUserProtectedError` subclasses `ValueError`, so the routes and
both CLIs map it to a friendly message with no call-site changes, exactly like
`LastAdministratorError`.

Two route contract fixes found in the same pass:

- `update_user` returned 400 for an unknown user id, contradicting its own
  docstring and both sibling routes, by falling through to the service's
  `ValueError("User ... not found")`. It already reads the record; it now 404s.
- The delete route said "Cannot delete the last administrator" while the service
  said "Cannot remove the last administrator", so losing the race between the
  pre-check and the backstop changed the wording of the same refusal on the same
  endpoint. Both now use `LAST_ADMIN_DETAIL`.
- An admin who deactivates themselves in the same request as a password change no
  longer receives a replacement token and media cookie for the account they just
  disabled.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… re-read fails

The re-read added in the previous commit borrowed `queue_owner_is_active`, and with it
its fail-to-active policy: a lookup that raises is treated as "still active". That
policy is right for the two gates it was written for, which re-run at the next node
boundary — but it is wrong here.

This handler is the only thing that stops a *single-node* graph. `_run_session_loop`
checks the owner before `run_node` and then only tests `session.is_complete()`, so a
one-node session is checked exactly once, before it starts. Skipping the cancel on a
busy-timeout therefore lets a deactivated account's node run to completion:
`worker.cancel_event` is deliberately not set here, so nothing else interrupts it.

The event is itself evidence of a committed deactivation. When the re-read cannot
contradict it — as opposed to actively reporting the account active again — the event
stands. A successful read showing an active account still spares the item, which is the
reactivation race the re-read exists for.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The service guard blocks future promotions but does nothing about a database where
`PATCH /auth/users/system {"is_admin": true}` already succeeded — which shipped builds
allow. That state is self-perpetuating: `count_admins()` is inflated by one forever, so
the last-administrator guard stays willing to demote the last real administrator, and
`has_admin()` stays true afterwards, which keeps `/auth/setup` closed. There is then no
authenticated way back, since the system row cannot log in.

A dated migration clears `is_admin` on that row. It is a no-op on a healthy database
(the row is seeded non-admin), restores the count's meaning on a poisoned one, and can
reopen `/auth/setup` for an instance already locked out. Nobody is signed out: no token
can carry `user_id="system"`, because authenticating as it is impossible.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The deactivation branch went straight to `disconnect(sid); continue` without indexing
`_socket_users`, so a mid-loop removal never raised there — only the active branch (the
epoch compare and the `is_admin` write) could. Verified against the pre-fix loop: the
deactivation scenario raised nothing and still reached the last socket, failing only on
the redundant second `disconnect`.

`test_demotion_survives_a_socket_dropping_mid_loop` is the test that reproduces the
KeyError. Rename and reword its neighbour so nobody reads it as covering that too.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
FastAPI publishes route docstrings as the operation description, so widening
`update_user`'s documented 400 to mention the protected system account changes both
generated artifacts. openapi-checks and typegen-checks diff them against the committed
copies.

Regenerated with the frontend's locked deps and CI's own commands; the only delta in
either file is that sentence.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@JPPhoto JPPhoto left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I took a look at this PR and the comments made in #9479 to prepare this.

Merge blockers:

  • invokeai/app/services/shared/sqlite_migrator/migrations/migration_2026_08_08_demote_system_user.py:26-28 only demotes system; it does not clear an existing password. auth.py:220-243 still permits that account to log in and access legacy assets. #9479 explicitly identified this password hole. Test: seed system with a bcrypt hash, run migration, assert login fails.

  • invokeai/app/util/user_management.py:218-248,492-527 mutates users directly, without emit_user_access_changed. Existing sockets therefore retain admin/user-room access after CLI demotion, deletion, or password reset; this contradicts the immediate-revocation claim. Test: connect a socket, run CLI mutation, emit a private/admin event, assert disconnect/no delivery.

  • invokeai/app/services/session_processor/session_processor_default.py:47-82,733-745 treats every owner lookup exception as “active.” A deleted/deactivated user can therefore run queued work during a DB/read failure, contradicting the “stop before the next node” policy. Test: make users.get() raise after deactivation; assert cancellation and zero node execution.

Other findings/issues:

  • docs/src/content/docs/features/Multi-User Mode/admin-guide.mdx:285-294 still describes tokens as expiry-only. It omits DB-derived role revocation, password epoch invalidation, socket disconnection, and queued-work cancellation. Test: reconcile this section with the new auth and password-change behavior.

Alternative implementation ideas:

  • Instead of only demoting system, clear its password hash in the migration and reject system authentication at the service boundary; this makes the internal-account invariant hold for damaged legacy rows too.

  • Instead of route-only invalidation events, use a shared mutation service plus DB-backed socket revalidation; this covers CLI and future direct-service callers.

  • Instead of treating lookup failures as authorization success, fail closed with a retryable queue state; this prevents unknown DB state from becoming executable work.

  • Instead of documenting JWT expiry alone, document epoch and live-revocation semantics; this prevents unsafe operator assumptions.

lstein and others added 6 commits August 9, 2026 13:13
The system row owns everything carried over from before multiuser support. It is
seeded unable to authenticate, but an administrator could once give it a password
through `PATCH /auth/users/system`. Refusing that from now on repairs nothing that
already happened, and the hole has three separate ends:

- the row itself, which may still carry a usable hash under a fixed, public email —
  the migration now clears `password_hash` alongside `is_admin`;
- a row damaged *after* the migration, by direct SQL or on a database that applied an
  earlier revision of the same migration id, since migrations run once —
  `UserService.authenticate` now refuses the account outright, whatever the row holds;
- a token *already issued*, which the migration cannot reach at all. The row is
  deliberately left active and its epoch untouched, so nothing else rejected it and
  sliding-window refresh would renew it forever — `resolve_authorized_user` now
  refuses the id, which covers REST, media, sockets and the video-upload gate in one
  place.

Single-user mode, where everything legitimately runs as `system`, never reaches
`resolve_authorized_user`: its dependencies synthesize the TokenData and return first.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…t be read

`queue_owner_is_active` treated any lookup exception as "active", which makes unknown
database state executable: the account may have been deactivated a moment earlier, and
this gate is what stands between that and GPU time spent on its behalf.

It now retries the read before refusing, so a transient error — a busy-timeout on the
shared SQLite connection under multi-GPU write contention, say — does not cost a valid
user their queued work. Only a database that is unreadable across every attempt refuses
the item, and that costs a cancellation, which is retryable. Both call sites run on a
worker thread, so the wait between attempts blocks nothing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`invoke-usermod` and `invoke-userdel` write to the database from their own process, so
no in-process event can be raised for them. Everything the server derives per request —
REST authorization, the dequeue and between-node gates, the media read and save gates —
already changes the instant those commands commit. Socket room membership does not: it
is established at connect time and refreshed only by `user_access_changed`, so a demoted
administrator's socket sat in the admin room, receiving other users' private events,
until it happened to reconnect.

A periodic sweep now re-derives each connected user from the database and publishes any
difference as the same event the routes emit, so sockets re-authorize and the session
processor cancels the user's running items through one code path rather than a second
copy that drifts from the first. Staleness is judged against every socket of the user,
not a representative one — a session that reconnected after a password change holds the
current epoch while the superseded session is still connected under the old one.

`_handle_user_access_changed` now re-reads the record and applies that, treating the
event as a trigger, the same way `_on_user_access_changed` already does. Handlers are
dispatched as independent tasks and the sweep's payload is a snapshot taken before an
await, so an event can arrive already superseded: applying it would re-grant the admin
room to someone just demoted, or disconnect the replacement session a password change
had just issued. A read that fails leaves the event standing.

The sweep is started and stopped from the app lifespan, in a `finally` so an abnormal
exit cannot leave it running against a half-torn-down process.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Session Management section described tokens as expiry-only, which invites the
operator assumption that a demotion or deactivation does not take effect until the
target's token runs out. Replace it with what actually happens: role changes derived
from the database per request, epoch invalidation on password change, socket
disconnection, queued-work cancellation, and the bounded staleness of a change made
with the CLIs.

Both limits are stated plainly too — the token stays cryptographically valid until it
expires, and a session in flight is stopped, not rewound.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@lstein

lstein commented Aug 9, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks — all three blockers were real. Fixed at 8a3d38d, along with the doc gap.

1. The system account's password

You were right, and it turned out to have three ends rather than one. Taking your alternative — clear the hash in the migration and reject system authentication at the service boundary — and then asking what each of those still misses:

  • The row itself. The migration now clears password_hash alongside is_admin.
  • A row damaged after the migration. Migrations run once, so direct SQL — or a database that applied an earlier revision of this same migration id — is out of its reach. UserService.authenticate now refuses the account outright, whatever the row holds.
  • A token already issued. This is the one that worried me most, and neither of the above touches it. On an instance that used the old hole and logged in, the JWT survives the upgrade: the row is deliberately left active and its epoch untouched, so resolve_authorized_user honored it and SlidingWindowTokenMiddleware would renew it indefinitely. That is a standing session over every pre-multiuser board, image, workflow and queue item. resolve_authorized_user now refuses the id, which covers REST, media, the socket handshake and the video-upload gate in one place.

I kept the migration id unchanged deliberately: get_migration_plan raises Database contains unknown applied migration IDs for any applied id it no longer recognizes, so renaming it would hard-error on every database that already ran the old version. That is exactly why the check had to be version-independent.

Single-user mode never reaches resolve_authorized_userget_current_user_or_default, get_current_media_user_or_default and _identify_video_upload_user all synthesize the system TokenData and return first — so this refuses only real minted tokens.

Tests: your suggested one (seed system with a bcrypt hash, run the migration, assert login fails), plus test_migration_2026_08_08_demote_system_user.py, plus two end-to-end ones asserting a minted system token is refused by /auth/me, by media, and with no X-Refreshed-Token — both verified to fail without the guard.

2. user_management.py does not emit emit_user_access_changed

Confirmed, though not fixable the way the finding implies: invoke-usermod / invoke-userdel are separate console-script processes, so there is no in-process event bus to emit onto. Your second alternative — DB-backed socket revalidation — is the part that actually applies.

Scoping what was really broken: everything the server derives per request already changes the instant those commands commit — REST authorization, the dequeue and between-node gates, and the media read/save gates all re-read the record. Only the socket layer caches connect-time state, which is precisely your symptom.

So SocketIO now runs a periodic sweep that re-derives each connected user from the database and publishes any difference as the same user_access_changed event the routes emit — so sockets re-authorize and the session processor cancels the user's running items through one code path, not a second copy that drifts. Bound is ~30 s; the CLIs now say so in their output.

Two things fell out of building it that are worth flagging:

  • Staleness has to be judged against every socket of the user, not a representative one. A session that reconnected after a password change holds the current epoch while the superseded session is still connected under the old one; sampling the first would find nothing to do and leave the revoked socket in place.
  • _handle_user_access_changed now re-reads the record and applies that, treating the event as a trigger — the same rule _on_user_access_changed already follows. Handlers are dispatched as independent tasks and the sweep's payload is a snapshot taken before an await, so an event can arrive already superseded: applying it would re-grant the admin room to someone just demoted, or disconnect the replacement session a password change had just issued, with nothing to correct either until the next sweep. A read that fails leaves the event standing.

One residual I chose to document rather than build around: the sweep only covers users with an open socket, so an out-of-process deletion of a user with no socket does not reach _on_user_access_changed, and a single-node graph of theirs already running — the one case no other gate re-checks — runs to completion. It cannot persist anything (the save gates re-read the record and raise PermissionError); the cost is the wasted node.

3. Owner lookup failures treated as "active"

Agreed, and taking your framing: unknown database state should not become executable work. queue_owner_is_active now retries the read and then fails closed. The retry is what makes that affordable — a busy-timeout under multi-GPU write contention shouldn't cost a valid user their queued work — and only a database unreadable across every attempt refuses the item. The resulting state is retryable: retry_items_by_id accepts canceled as well as failed, so no work is destroyed.

Tests cover the retry-then-refuse path, the transient-failure-then-succeed path, rejection at dequeue, and your case specifically: users.get() raising after deactivation, asserting cancellation and zero further node execution.

4. Admin guide

Rewritten. The section now covers DB-derived role revocation, epoch invalidation on password change, socket disconnection, queued-work cancellation, and the bounded staleness of a CLI change — plus the two limits an operator could otherwise assume away: the token stays cryptographically valid until it expires (anything accepting these tokens without consulting the database would still honor it), and a session in flight is stopped, not rewound.


Full tests/app run is green at 2327 passed / 8 skipped / 6 xfailed, and I put the diff through an adversarial pass before pushing — which is where the already-issued-token hole in (1) and the superseded-event race in (2) came from.

…d_user

The middleware carried its own copy of the exists/active/epoch checks, which is exactly
how a rule added later reaches every entry point but this one: the refusal of the
internal `system` id landed in `resolve_authorized_user`, so REST, media, the socket
handshake and the video-upload gate all stopped honoring those tokens while this kept
minting fresh ones for them. The `system` row is deliberately active with an untouched
epoch, so none of the local checks fired.

Nothing accepted the renewed token, so no access followed from it — but an indefinitely
renewed session is a hole waiting for one consumer that trusts a token without
re-checking the id. Deciding in one place is the point of that function.

The lookup still runs off the event loop, for the reason the old comment gave.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@lstein

lstein commented Aug 9, 2026

Copy link
Copy Markdown
Collaborator Author

Follow-up at 49d00b3, on my own point (1) above.

I said resolve_authorized_user now covers REST, media, the socket handshake and the video-upload gate. It did — but SlidingWindowTokenMiddleware was not among them: it carried its own copy of the exists/active/epoch checks rather than calling that function. The system row is deliberately active with an untouched epoch, so none of those local checks fired, and the middleware kept minting a fresh X-Refreshed-Token for a token every other entry point had just stopped honoring.

No access followed from it — nothing accepts the renewed token — but that is precisely the shape the function's own docstring warns about ("a check added to some copies but not others is indistinguishable from no check at all on the paths that were missed"), and an indefinitely renewed session is a hole waiting for one consumer that trusts a token without re-checking the id.

The middleware now decides through resolve_authorized_user, which collapses the three duplicated checks into the shared one. The lookup still runs off the event loop, for the reason the old comment gave. Test added and verified to fail without the guard; tests/app green at 2328.

@lstein
lstein requested a review from JPPhoto August 9, 2026 21:43

@JPPhoto JPPhoto left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Merge blockers:

  • invokeai/app/api/sockets.py:312-317,339-370 preserves cached privileges when DB reads fail. After CLI demotion, a stale admin socket can keep receiving private events. Test: commit demotion, force users.get to fail across sweeps, emit another user's event, assert the socket receives nothing.

  • invokeai/app/api/sockets.py:393-418,441-447 applies stale event authorization when revalidation fails. A superseded promotion event can re-add a demoted user to admin. Test: send promotion event after demotion, make the reread raise, assert enter_room(..., "admin") is never called.

Other findings/issues:

  • invokeai/app/api/sockets.py:319-324 and invokeai/app/services/session_processor/session_processor_default.py:163-178 leave a no-socket user's already-running single-node graph executing after CLI deletion; only save gates stop persistence. Docs at docs/src/content/docs/features/Multi-User Mode/admin-guide.mdx:298-303 overstate cancellation. Test: run a side-effecting one-node item, delete the owner with no socket, assert the node is not invoked.

  • invokeai/app/services/users/users_default.py:100-133 omits token_epoch from get_many(), so users whose passwords were changed receive DTOs reporting epoch 0. Test: bump an account epoch, call get_many([user_id]), assert the returned epoch matches get().

Suggestions:

  • Instead of preserving socket authorization on repeated DB errors, disconnect or remove privileged room membership after bounded failures; this fails closed and limits stale access.

  • Instead of sweeping only users with sockets, track authorization changes in the database or periodically revalidate every running queue owner; this closes the no-socket single-node gap.

  • Try using one shared row mapper/projection instead of manually maintaining each user SELECT; this prevents future fields such as token_epoch from silently becoming stale.

Pfannkuchensack added a commit to Pfannkuchensack/InvokeAI that referenced this pull request Aug 11, 2026
Converting the routes from `async def` to `def` removed an implicit guarantee:
a handler whose body contains no `await` could not be interleaved with another
request, because the event loop had no point at which to switch. Two handlers
relied on it.

`POST /auth/setup` did has_admin() then create_admin() in separate transactions.
Two concurrent requests both saw no admin and both created one, so the loser
ended up with a persistent admin account instead of the intended 400. The
condition now lives inside the INSERT's own transaction, behind BEGIN IMMEDIATE
so a second process (invoke-useradd --admin) cannot slip a write in either. This
mirrors what invoke-ai#9360 does for the update/delete last-admin invariant; create_admin
was the one path it does not cover.

Custom node install, uninstall and reload all mutate the same custom-nodes
directory, sys.modules and invocation registry. Interleaved, a failed install's
cleanup rmtree'd the directory a concurrent install had just cloned into. A
module-level lock restores the exclusion; the install and uninstall bodies moved
into helpers so the lock scope is visible rather than an 80-line reindent.

Both regression tests fail without their fix: the admin one creates two
administrators, the pack one loses the successful install's directory.

 perf(queue): render the queue list from summaries, with one sanitizer

The list fetched full queue items for every visible row, each carrying its
session graph and workflow — megabytes per screenful for fields no row draws.
The rows now render from SessionQueueItemSummary and the full item is fetched
only when a row is expanded, which is what the summary route added in this
branch was for; until now nothing consumed it.

The per-item summary query provides the same cache tags as getQueueItem, so
every existing invalidation path covers the list rows with nothing to wire up,
and the optimistic status write is mirrored so a row's status still flips
without a round trip. The range hook batches at the backend's 1000-id limit,
which a fast fling could otherwise exceed.

Both sanitizers are now one generic function over a single redaction table: the
summary and the full item are two projections of one row, and a field stripped
from the list but left on the detail view is leaked anyway. A test walks the
intersection of both models and asserts they redact it identically.

`device` is deliberately not redacted in either. It names the instance's GPU
rather than anything about the other user's work, and the list has always shown
it — redacting it here would have quietly changed what non-admins see.

parent_item_id joins the summary because the rows decide from it whether to
offer a retry.
Pfannkuchensack added a commit to Pfannkuchensack/InvokeAI that referenced this pull request Aug 11, 2026
Two review follow-ups landed together here.

Converting the routes from `async def` to `def` removed an implicit guarantee: a
handler whose body contains no `await` could not be interleaved with another
request, because the event loop had no point at which to switch.

`POST /auth/setup` did has_admin() then create_admin() in separate transactions.
Two concurrent requests both saw no admin and both created one, so the loser
ended up with a persistent admin account instead of the intended 400. The
condition now lives inside the INSERT's own transaction, behind BEGIN IMMEDIATE
so a second process (invoke-useradd --admin) cannot slip a write in either. This
mirrors what invoke-ai#9360 does for the update/delete last-admin invariant; create_admin
is the one path it does not cover.

Custom node install, uninstall and reload all mutate the same custom-nodes
directory, sys.modules and invocation registry. Interleaved, a failed install's
cleanup rmtree'd the directory a concurrent install had just cloned into. A
module-level lock restores the exclusion; the install and uninstall bodies moved
into helpers so the lock scope is visible rather than an 80-line reindent.

Both regression tests fail without their fix: the admin one creates two
administrators, the pack one loses the successful install's directory.

The route added earlier in this branch had none — the list still fetched full
queue items for every visible row, each carrying its session graph and workflow,
so the claimed saving was not being realised. The rows now render from
SessionQueueItemSummary and the full item is fetched only when a row is expanded.

Measured against the previous commit, same backend and same 396-item queue,
identical scroll (page load, queue tab, scroll to 60%):

  requests            62  ->  2
  payload (gzip)  262 KB  ->  1.3 KB   (30 items)
  server time       60ms  ->  4ms      (30 items)

The request count collapses because the old path was self-amplifying: the range
hook re-asks which ids are uncached on every range event, and at ~60ms per
response the cache had not filled yet, so overlapping fetches piled up.

A side effect worth knowing: `items_by_ids` silently skips items it cannot
deserialize, so a queue item whose graph references an unregistered node type
left its row permanently blank. Summaries never touch the graph, so the row now
renders and only the expanded detail is affected.

The per-item summary query provides the same cache tags as getQueueItem, so
every existing invalidation path covers the list rows with nothing to wire up;
the optimistic status write is mirrored so a row still flips without a round
trip. The range hook batches at the backend's 1000-id limit, which a fast fling
could otherwise exceed.

The summary and the full item are two projections of one row, and a field
stripped from the list but left on the detail view is leaked anyway. Both now go
through one generic function over a single redaction table; a test walks the
intersection of the two models and asserts they redact it identically.

`device` is deliberately not redacted in either. It names the instance's GPU
rather than anything about the other user's work, and the list has always shown
it — redacting it would have quietly changed what non-admins see.

parent_item_id joins the summary because the rows decide from it whether to
offer a retry.
Pfannkuchensack added a commit to Pfannkuchensack/InvokeAI that referenced this pull request Aug 11, 2026
Six follow-ups from @lstein's sweep that were left open.

`require_admin` and `require_admin_or_default` go back to `async def`. They only read
`is_admin` off already-resolved token data, so declaring them `def` bought a threadpool
round-trip per admin request and nothing else. The `users.get` that can block lives in
`get_current_user`, which stays synchronous — the docstrings now say why the two layers
differ.

The AST guard now inspects what it claims to. It walked only `tree.body`, so a handler
registered from inside a factory function or an `if` block was never seen, and
`_awaits_something` used `ast.walk`, which counts `await`s inside nested closures — a
handler could have passed by defining an inner async helper it never awaits. Both are
fixed and both now have their own tests, so the guard's behaviour is pinned rather than
asserted in a comment.

`convert_model` takes a lock non-blocking and answers 409 otherwise. Blocking would be
wrong: a conversion runs for minutes, and for the same key the second caller reads a
record the first is midway through replacing. Two conversions in flight also means two
models resident at once, which nothing bounds. The body moved into `_convert_model` so
the lock scope is visible. Tested for the 409 and for the lock surviving a failed
conversion rather than wedging the endpoint for the process's lifetime.

`do_hf_login` and `reset_hf_token` hold a lock across the write and the status read-back,
which otherwise could report a status belonging to a different token than the one just
written.

The blocking-work doc gains the bound it was missing: anyio's thread limiter holds 40
tokens, so past forty concurrent blocking requests the stall moves rather than vanishes —
and anything else needing a thread queues behind them, including the synchronous auth
dependency that runs before a handler is reached. Noted there too that
`test_event_loop_blocking.py` cannot show this, because its probe route has neither auth
nor database access.

`QueueItemDetail` tells a failed fetch apart from a pending one. A queue item the backend
cannot serve — one whose graph references a node type this build no longer registers —
previously read as "Loading" forever.

Left alone deliberately: the stale `old_is_public` in the workflow-updated event, which is
cosmetic and would need `workflow_records.update()` to return the previous row to fix
properly; and the `delete_user` / `update_user` last-admin invariants, which belong to
invoke-ai#9360.
Both of the second review round's blockers are about the same seam: what the
socket layer does when the database read it re-derives authorization from
fails. Round 1 made both paths re-read the record; neither said what to do
when the re-read itself does not answer.

`_handle_user_access_changed` fell back to the event payload. That payload can
be a superseded promotion — the sweep snapshots before an await, so a demotion
can commit while the event is in flight — and applying it re-granted the admin
room to a user who had just lost it, with no record consulted to catch it. The
event still stands in the direction that takes privileges away, since it is
evidence of a committed change the re-read could not contradict, but `is_admin`
is now forced False on that path: this handler can demote on an unreadable
database and never promote. A genuine promotion is deferred, not lost — the
sweep sees the record disagree with the cache and publishes it again.

`_revalidate_socket_users` retried forever, which makes an unreadable database
indistinguishable from a quiet one: a CLI demotion during an outage left the
socket in the admin room, reading every other user's private events, for as
long as the reads kept failing. It now retries a bounded number of times and
then drops the privilege rather than the connection. Disconnecting would be the
outage the sweep exists to avoid — `_handle_connect` fails closed, so the
clients could not get back in — whereas a demoted socket keeps working and
recovers through the ordinary staleness check once reads succeed.

The counter that bounds this is the new piece of state, and the two orderings
that get it wrong are both handled. The sweep's lookup awaits, so a user's last
socket can close inside that window; the disconnect clears their count and a
naive write would resurrect it for a user with no sockets, where nothing would
collect it again because the loop skips sweeps entirely while none are open. And
the count answers "how long have these sockets gone unchecked", so every
successful read of the record clears it, not just the sweep's own — otherwise a
socket admitted by a read that proved the user is an admin would be demoted by
the next single failure instead of by three.

The bound covers is_admin only. Deactivation, deletion and epoch revocations
still wait for a successful read, because acting on those means disconnecting;
that limit is now stated in the docstring and the admin guide rather than
implied away.

One existing test was pinning the bug: TestUserAccessChangedHandler never bound
ApiDependencies, so every test in it ran the handler's exception path, and the
promotion test passed only because of the fallback removed here. Same hazard in
the epoch test. Both now bind the record they claim to be testing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@lstein

lstein commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator Author

Both blockers are fixed at 592fee4. You were pointing at the same seam twice: round 1 made both paths re-read the record, and neither said what to do when the re-read itself doesn't answer.

sockets.py:393-418,441-447 — stale event authorization on a failed re-read. Confirmed, and it was the worse of the two: the handler fell back to the event payload, and a superseded promotion payload would call enter_room(sid, "admin") for a user who had just been demoted, with no record consulted to catch it.

is_admin is now forced False whenever the re-read raises. The event still stands in the direction that takes privileges away — it's evidence of a committed change the re-read couldn't contradict, so the deactivation and epoch branches still fire — but this handler can now demote on an unreadable database and never promote. A genuine promotion isn't lost, only deferred: the sweep sees the record disagree with the cache and publishes it again.

Your test passes — enter_room(..., "admin") is never called.

sockets.py:312-317,339-370 — preserved privileges across repeated failures. Confirmed. I'd argued in the docstring that failing closed here would be a self-inflicted outage, and I still think that's right about disconnecting, but it doesn't justify retrying forever: an unreadable database becomes indistinguishable from a quiet one, and a CLI-demoted socket sits in the admin room reading everyone's private events for as long as reads keep failing.

So it now retries SOCKET_REVALIDATION_FAILURE_LIMIT (3) times and then drops the privilege rather than the connection — the sockets leave the admin room and their cached is_admin goes False, but stay connected. That bounds stale admin access to ~90s without the outage, and _handle_connect failing closed means a disconnect-based policy would lock everyone out for the duration. Recovery rides the staleness check you already reviewed: cached False against a record saying True is a difference, so the next good sweep republishes and the room is rejoined.

Your test passes — after committed demotion with users.get failing across sweeps, the socket receives nothing from another user's event.

Scope, stated rather than implied. The bound covers is_admin only. Deactivation, deletion and epoch revocations still wait for a successful read, because acting on those means disconnecting. During an outage a deleted user's socket therefore stops seeing other users' events but keeps receiving its own, while every HTTP request it makes is already refused. That's in the docstring and the admin guide now. For the epoch case specifically your argument is stronger than mine — the replacement session is by definition already connected, so dropping the superseded socket isn't the outage I'm avoiding. Happy to bound that too if you want it; I left it as a separate policy call rather than fold it in here.


On the other two findings

users_default.py:100-133 — confirmed, get_many() was the only one of the four SELECTs in that file missing token_epoch. Taking that with your shared-row-mapper suggestion in a follow-up rather than this PR, since it's the fix that stops the next field from going stale the same way.

The no-socket single-node gap I'm leaving documented for now; DB-tracked auth changes or revalidating every running queue owner is a bigger change than this branch should carry.


One thing worth flagging, because it cuts against my own round-1 work. I ran a fresh-context adversarial pass over the fix before pushing, and it found that TestUserAccessChangedHandler never bound ApiDependencies — so every test in that class was running the handler's exception path, and test_promoted_user_sockets_join_admin_room was passing only because of the fail-open fallback you reported. The test that was supposed to protect promotion was pinning the vulnerability instead. Same hazard in TestTokenEpochOnSockets, where the whole re-read block could be deleted with the test still green. Both now bind the record they claim to test.

The same pass found two defects in the new failure counter, both fixed here: the sweep's lookup awaits, so a user's last socket can disconnect inside it and the failure write would resurrect a counter for a user with zero sockets that nothing ever collects; and the counter has to be cleared by every successful read of the record, not just the sweep's own, or a socket admitted as an admin by a read that succeeded gets demoted by the next single failure instead of by three.

New/changed tests are in tests/app/test_socket_privilege_revocation.py (42 total). Each fix was verified by reverting it and confirming exactly one targeted test fails.

@lstein
lstein requested a review from JPPhoto August 13, 2026 23:52

@JPPhoto JPPhoto left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Merge blockers:

  • invokeai/app/api/sockets.py:368-373 leaves socketless queue items unchecked. invokeai/app/services/session_processor/session_processor_default.py:163-178 checks owner only between nodes. CLI deletion emits no server event (invokeai/app/util/user_management.py:24-29). A deleted user’s running single-node graph can finish. Docs overpromise at docs/src/content/docs/features/Multi-User Mode/admin-guide.mdx:327. Test: run socketless one-node job; delete owner via CLI during node; verify cancellation before node completion.

Other findings/issues:

  • invokeai/app/api/sockets.py:503-507 sets cached is_admin=False before leave_room(). If room removal fails, later successful reads see no cache difference, so admin-room membership can persist while direct sends trust cache. Test: make leave_room() fail at retry limit; keep DB role demoted; run recovery sweep; verify room removal retries.

  • invokeai/app/api/sockets.py:547-586 only forces is_admin=False after failed reread. Stale is_active and token_epoch remain trusted, so stale epoch-0 event disconnects replacement epoch-1 socket. Contradicts replacement-session behavior documented at docs/src/content/docs/features/Multi-User Mode/admin-guide.mdx:328. Test: cache socket epoch 1; send active event epoch 0; make users.get() fail; verify socket remains connected.

  • invokeai/app/services/users/users_default.py:100-133 get_many() omits token_epoch; returned UserDTO silently defaults to 0 after password rotation. Test: bump epoch, compare get_many([id])[id].token_epoch with get(id).token_epoch.

Alternative implementation ideas:

  • Instead of tying revocation propagation to open sockets, persist a durable access-change version or recheck owners before and after every node; this closes socketless single-node gaps.

  • Instead of marking cached privilege false before room removal, reconcile desired room state and retry failed leave_room() calls; this prevents cache/room divergence.

  • Consider deferring active/epoch disconnects until successful reread, while retaining admin downgrade; this prevents stale event snapshots from killing replacement sessions.

  • Instead of separate SQL projections, share one user-row mapper and select list; this prevents future fields like token_epoch becoming stale in bulk reads.

…ing an unreadable record

Addresses the round-3 review.

Blocker: an out-of-process deletion of a user with no socket open raised no
event at all, so a graph of theirs already inside a node ran on. The
revalidation sweep now covers the owner of every executing queue item as well
as every connected user. Publishing for them routes the case through
`DefaultSessionProcessor._on_user_access_changed`, which cancels the item and
so sets the worker's cancel event, stopping a node with step callbacks
part-way. A graph whose last node finishes before the sweep notices is caught
by one more owner check after that node, so it is canceled rather than
recorded as completed.

That post-node check is deliberately narrow. Before a node, failing closed on
an unreadable record costs a retryable cancellation and may save a GPU; after
one there is no execution left to refuse, only a finished result to destroy —
and for a workflow-call child, its whole parent chain. So it runs only for a
session that completed its own work (an errored session is `is_complete()`
too, and a suspended workflow call is not), and it treats an unreadable record
as active. A genuinely revoked owner keeps nothing by that: the
`invocation_context` save gates re-read the record and already refused every
write.

`worker.queue_item` was only ever overwritten by the next dequeue, and pausing
blocks before that — a worker that finished an item and parked would have gone
on reporting its owner as running work, making the sweep republish forever. It
is now cleared at the top of the worker loop.

`_handle_user_access_changed` no longer disconnects on a record it could not
read. The payload cannot be trusted to close a connection for the same reason
it cannot be trusted to grant the admin room: it may already be superseded, and
a stale `is_active` or `token_epoch` would close the replacement session a
password change had just issued, with no way back in while `_handle_connect` is
also failing closed. The admin downgrade still applies; everything else waits
for a read that succeeds, which the sweep republishes.

Room membership and the cached `is_admin` can no longer diverge. The cache is
written before the room call and rolled back if it raises, so a change that did
not happen is never recorded as done — which matters because the sweep decides
there is nothing left to do by comparing the record against that cache, and
`_handle_sub_queue` re-derives room membership from it. A failed `disconnect`
no longer escapes the handler either: it used to abandon the user's remaining
sockets and, since every handler for an event shares one task, skip the queue
cancellation entirely.

`UserService.get_many` omitted `token_epoch`, so its DTOs reported the model
default of 0 — a revoked epoch indistinguishable from a fresh one. All five
queries that build a `UserDTO` now share one column list and one row mapper, so
the next field cannot go stale the same way.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@lstein

lstein commented Aug 17, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks — all four confirmed and fixed at 7c0032b77e. Nothing in this round was a false positive, and chasing the blocker turned up two more defects that I would not have found on my own; details below.

Blocker: socket-less queue items

Confirmed, and you were right that documenting it was not good enough. _revalidate_socket_users now sweeps the owner of every executing queue item as well as every connected user (SessionProcessorBase.get_running_queue_item_owners). They contribute no cached state of their own, so an active owner matches the staleness comparison vacuously and produces no event; what they add is the deleted/deactivated case, which reaches _on_user_access_changed, cancels the item, and so sets the worker's cancel event — a node with step callbacks stops part-way, which is the cancellation-before-node-completion your test asks for.

A node that finishes inside the sweep interval needed the other half, so there is now one more owner check after the graph's last node. But my first version of that check was worse than the gap it closed, and I want to be explicit about it, because it bears on where fail-closed belongs:

queue_owner_is_active fails closed, and the module's own comment names a busy-timeout under multi-GPU write contention as the routine cause. Before a node that is right — refusing costs a retryable cancellation and may save a GPU. After the last node it inverts: there is no execution left to refuse, only a finished result to destroy. A 0.5s blip would have canceled a valid user's completed generation; for a call_saved_workflow child, _cancel_parent_from_canceled_child would have taken the entire parent chain with it; and after a node error (is_complete() is true via has_error()) it would have overwritten the user's error with a cancellation, so _fail_parent_from_failed_child would then find the parent terminal and drop the error entirely.

So the post-node check is now scoped to a session that finished its own work cleanly — errored sessions and suspended workflow calls are excluded — and it passes unreadable_is_active=True rather than failing closed. A genuinely revoked owner loses nothing by that: the invocation_context save gates re-read the record independently and had already refused every write. Gating it on is_complete() also means one extra lookup per session rather than one per node.

Two related defects came out of the same pass:

  • worker.queue_item is only ever overwritten by the next dequeue(), and resume_event.wait() (pause) and the image-move maintenance continue both block before that. A worker that finished an item and then parked would have gone on reporting its owner as running work indefinitely — so with nobody connected, the sweep would have published a revocation and logged it every 30 seconds forever. It is now cleared at the top of the worker loop (not after run_queue_item, because the non-fatal error handler still needs it). Two older worker.queue_item and worker.queue_item.x sites are now bound to a local, since that clear widens the window in which the second read can see None.
  • I had added a terminal-item skip to _on_user_access_changed to suppress repeat cancellations. It was inert — _transition_queue_item_status already guards terminal states inside the transaction and emits nothing — and it was harmful, because cancel_queue_item walks the workflow-call chain, so skipping on a completed child left a waiting parent and its pending siblings alive. Removed.

is_admin=False before leave_room

Confirmed, and the reasoning was sharper than I had it. _set_socket_admin now writes the cache first and rolls it back if the room call raises, so the cache never records a change that did not happen. Your framing was about retrying; the part I had missed is that the sweep decides there is nothing left to do by comparing the record against that cache, so the stale write destroys the only signal that could have caught it. _handle_sub_queue re-deriving room membership from the same cache is a second consumer I had not accounted for. Rolling back leaves the socket visibly more privileged than its record, which is exactly what the sweep looks for — and past the failure limit every sweep re-attempts the drop, so the failed-lookup path retries too.

One honest caveat: with python-socketio's in-memory manager leave_room swallows KeyError and neither room call awaits anything, so this branch is only reachable via enter_room (which does raise for a departed sid) or under an external client manager. The ordering is written not to depend on that.

While in there: disconnect was still unguarded. It flushes a packet over a possibly half-closed transport, and an exception escaped into a bare task — stranding the user's other sockets and, because LocalHandler runs every handler for an event in one task with the socket handler first, skipping _on_user_access_changed and the queue cancellation entirely. Wrapped.

Stale is_active/token_epoch on a failed reread

Confirmed — this was the residual I flagged last round, and your argument for closing it is the right one. On an unreadable record the handler now applies only the admin downgrade and disconnects nothing. The payload cannot be trusted to close a connection for the same reason it cannot be trusted to grant the admin room, and this direction is the unrecoverable one: _handle_connect is failing closed at the same time, so a socket dropped on a stale is_active cannot get back in. Nothing is lost — a real deactivation or epoch bump still fails the sweep's staleness check on every subsequent tick, so it is republished until a readable record lets the handler act on it. Your test (cache epoch 1, event epoch 0, users.get() failing → socket stays connected) is in as test_an_unreadable_record_does_not_disconnect_on_a_stale_epoch.

The _revalidation_loop guard also short-circuits on single-user mode now, so it no longer reaches into the session processor every tick on a server with no records to check.

get_many() and the shared mapper

Confirmed. I took your alternative rather than just adding the column: all five queries that build a UserDTO now share _USER_DTO_COLUMNS and _user_dto_from_row, and authenticate appends password_hash after them instead of inserting it mid-list, so the positions stay stable. The only production caller today reads just display_name/email, so this was a latent trap rather than a live defect — which is the argument for fixing the duplication and not only the symptom.

Docs

admin-guide.mdx no longer promises more than the code does: the deactivation bullet now says a running item stops at the next node or is canceled rather than recorded as completed if it was already on its last one, and that a generation reporting progress stops part-way; the CLI paragraph now says the sweep covers a running generation whether or not its owner still has the browser open. The paragraph about closing a connection still waiting for a successful read stands, and now covers in-process events too.

Full suite green (2490 passed). Each fix was verified by reverting it and confirming its named test fails.

@JPPhoto
JPPhoto self-requested a review August 17, 2026 03:50

@JPPhoto JPPhoto left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is good to merge.

There are a few corner cases, unlikely to appear during normal usage and perhaps not worth dealing with:

  • invokeai/app/api/sockets.py:489-496: socketless running owners have revalidation failures discarded, so repeated DB-read failures never emit cancellation. A deleted user’s long-running single-node job can run to completion. Test: use only running_owners={"u"}, make users.get raise for three sweeps, assert no access-change event or cancellation occurs.

  • invokeai/app/services/session_processor/session_processor_default.py:114-119, 231-232: the final owner check treats an unreadable record as active. If revocation occurs during a one-node, no-save node, the queue item can still be recorded completed. Test: return active before execution, raise on all final lookups, run a no-save node, and assert complete_queue_item is called.

  • invokeai/app/api/sockets.py:651-691: when the access-change reread fails, stale sockets are demoted but not disconnected. A password-revoked or deleted socket remains in its user room and can receive private events until a DB read succeeds. Test: use an old-epoch socket, make users.get raise, dispatch the event, then emit to user:<id>; verify no disconnect occurs and the socket receives it.

  • invokeai/app/services/session_processor/session_processor_base.py:73-82: the new abstract method breaks downstream/custom SessionProcessorBase implementations with TypeError at instantiation. Test: instantiate an old subclass implementing the previous abstract methods only.

Suggestions:

  • Instead of clearing failures for socketless owners, track running-owner failures separately and cancel after the retry limit; this bounds revoked work during DB outages without fabricating socket state.

  • Instead of treating an unreadable final owner record as active, persist an owner authorization snapshot/epoch with the queue item; this distinguishes transient read failure from genuine revocation without false cancellation or false completion.

  • Consider serializing per-user revocations with a monotonic generation; this permits safe socket disconnects while ignoring stale access-change events.

  • Instead of adding a mandatory abstract method, provide a default empty implementation or optional capability; this preserves existing custom processors.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

6.14.1 api backend PRs that change backend files docs PRs that change docs frontend PRs that change frontend files invocations PRs that change invocations python PRs that change python files python-deps PRs that change python dependencies python-tests PRs that change python tests Root services PRs that change app services

Projects

Status: 6.14.1: Bug fixes to 6.14.0

Development

Successfully merging this pull request may close these issues.

2 participants