docs(platform): document the cache and coordination engine (Redis / Valkey)

docs/platform/advanced_setup.md

@@ -65,6 +65,108 @@ cd ../backend
 prisma migrate dev --schema postgres/schema.prisma
 ```
 
+## Cache and coordination engine
+
+Alongside PostgreSQL and RabbitMQ, the platform depends on a Redis-compatible engine for caching, distributed locking, rate limiting, spend and usage counters, session metadata, pending-message buffers, and the server-sent-event streams that carry agent output to the browser.
+
+Redis is the default. **Valkey is a tested alternative:** it is the engine inside the single-container image, which CI builds and smoke-tests on every change to the platform. Valkey forked from Redis 7.2 and speaks the same protocol — nothing the backend does distinguishes the two.
+
+| | Redis | Valkey |
+|---|---|---|
+| Docker Compose stack | Default (`redis:7`) | Opt-in, via a Compose override |
+| Single-container image | Not used | The engine it ships |
+| CI coverage | Every backend test leg | The single-container image build and smoke test |
+| Connection settings | `REDIS_HOST`, `REDIS_PORT`, `REDIS_PASSWORD` | Identical — nothing to change |
+
+There is no functional reason to prefer one over the other for this workload: both clear the [version floor](#engine-version-floor) below, and the platform uses no Redis modules. Choose on operational grounds — which engine your managed provider offers, and which licence terms you want. The two projects' licences differ and the Redis side has changed more than once, so check the licence of the specific tag you pin.
+
+### Cluster mode is required
+
+The backend always connects with a cluster client, so **a single standalone node will not work** — regardless of which engine you pick. Local development deliberately runs a real multi-shard cluster so that cross-slot bugs surface on a laptop rather than in production.
+
+The self-hosting distributions each bring up a three-shard, no-replica cluster on ports `17000`, `17001` and `17002` (cluster bus ports `27000`–`27002`):
+
+| Distribution | Engine | How the cluster is formed |
+|---|---|---|
+| Docker Compose stack (`autogpt_platform/docker-compose.platform.yml`) | `redis:7` by default | Three `redis-server` containers plus a one-shot `redis-init` sidecar that runs `redis-cli --cluster create`. Each shard announces its own Compose hostname. |
+| Single-container image (`autogpt_platform/single-container`) | Valkey | Three supervised `valkey-server` processes inside the container, formed by `valkey-cli --cluster create`. Each shard announces `127.0.0.1`. |
+
+{% hint style="info" %}
+`make start-core` brings up this cluster along with the platform's other dependencies — PostgreSQL, RabbitMQ, FalkorDB, ClamAV and the database migration job — not a single cache node.
+{% endhint %}
+
+FalkorDB is a separate service that also speaks the Redis protocol, but it is the CoPilot graph store and depends on the FalkorDB graph module. It is not part of the cache and coordination layer, and Redis or Valkey cannot serve it.
+
+### Switching the Compose stack to Valkey
+
+The shard image is set in `docker-compose.platform.yml`, which is a tracked file. To run the cluster on Valkey without editing it, override the image for all three shards and the init sidecar in `autogpt_platform/docker-compose.override.yml`:
+
+```yaml
+services:
+  redis-0:
+    image: valkey/valkey:8.1
+    user: "999:999"
+  redis-1:
+    image: valkey/valkey:8.1
+    user: "999:999"
+  redis-2:
+    image: valkey/valkey:8.1
+    user: "999:999"
+  redis-init:
+    image: valkey/valkey:8.1
+    user: "999:999"
+```
+
+Then bring the dependencies up as usual:
+
+```bash
+cd autogpt_platform/
+docker compose up -d deps
+```
+
+{% hint style="warning" %}
+`user: "999:999"` is not optional, and omitting it fails quietly. Valkey's entrypoint only drops privileges when it is invoked as `valkey-server`, and the Compose command lines call the `redis-server` symlink — so an override that sets `image:` alone runs the shards as **root**. Stock `redis:7` drops to uid 999 on its own, so this is a difference you only hit after switching engines.
+
+The one numeric value is correct for either engine: uid 999 is `redis` in `redis:7` and `valkey` in `valkey/valkey:8.1`, and it owns the `/data` workdir where `nodes.conf` is written.
+{% endhint %}
+
+Nothing else changes. Connection settings are engine-neutral — `REDIS_HOST`, `REDIS_PORT` and `REDIS_PASSWORD` mean the same thing to both engines. (`REDIS_CLUSTER_HOST` and `REDIS_CLUSTER_PORT` take precedence over the first two when they are set.) The service names stay `redis-0`/`redis-1`/`redis-2` with the `redis-init` sidecar, and the Valkey image ships `redis-server` and `redis-cli` as symlinks, so the cluster command lines and health checks in the Compose file work unaltered.
+
+### Using a managed or external deployment
+
+For a cluster you buy or run yourself — Amazon ElastiCache or Google Memorystore, both of which offer Redis- and Valkey-flavoured clusters, or a self-managed cluster of either engine — the deployment must provide:
+
+- **Cluster mode enabled.** A single-node or cluster-mode-disabled deployment cannot serve this platform, because the backend speaks only the cluster protocol.
+- **Sharded pub/sub** (`SPUBLISH`, `SSUBSCRIBE`, `SUNSUBSCRIBE`). Agent output streaming and websocket reconnection depend on it; it is not optional.
+- **Announced shard addresses that resolve from the platform**, together with `REDIS_USE_ANNOUNCED_ADDRESS=true`. Without that variable the backend rewrites every shard address to the seed host, keeping only the announced port. Managed clusters give each shard a distinct hostname on a shared port, so the rewrite collapses all shards onto one node and sharded pub/sub is pinned to the wrong shard. The Compose stack already sets this variable; set it yourself if you run the backend outside Compose.
+
+To point the Compose stack at an external cluster, change `REDIS_HOST` and `REDIS_PORT` in the `x-backend-env` block of `docker-compose.platform.yml`. The backend services take their values from there, so editing `backend/.env` alone will not move them. `REDIS_PASSWORD` is the exception: it is absent from that block, so `backend/.env` does set it.
+
+If you would rather not edit a tracked file, set the same variables per backend service in `docker-compose.override.yml` — a service-level `environment:` entry overrides the value merged in from `x-backend-env`. You have to list every backend service you run, which is why the block above is the shorter route.
+
+Then start the dependencies you still need directly instead of through `deps`, which always brings up the bundled shards:
+
+```bash
+docker compose up -d db rabbitmq clamav falkordb migrate
+```
+
+### Engine version floor
+
+The commands the backend issues imply these minimums, for either engine:
+
+| Requirement | Commands |
+|---|---|
+| Redis 7.0-equivalent semantics | Sharded pub/sub (`SPUBLISH`/`SSUBSCRIBE`/`SUNSUBSCRIBE`), `EXPIRE … NX` |
+| Redis 6.2-equivalent semantics | `LPOP` with a `count` argument, `GETEX` |
+
+Valkey 8.1 and Redis 7 both clear this floor.
+
+The cache and coordination layer uses **no Redis modules** — no `FT.*`, `JSON.*`, `BF.*` or `TS.*` commands appear in the backend — so there is no module bundle to install or license on either engine. It does rely on Redis Streams, server-side Lua (`EVAL`), and transactions within a single hash slot, which a thin protocol proxy may not implement in full.
+
+{% hint style="warning" %}
+This guidance covers self-hosting and local development. Behaviour under sustained production load, failover and persistence tuning depends on how you size and operate the deployment, and is outside the scope of these instructions.
+{% endhint %}
+
 ## AutoGPT Agent Server Advanced set up
 
 This guide walks you through a dockerized set up, with an external DB (postgres)

docs/platform/getting-started.md

@@ -121,7 +121,7 @@ Inside the `autogpt_platform` directory, you can use:
 | Command                | What it Does                                                                 |
 |------------------------|-------------------------------------------------------------------------------|
 | `make init-env`        | Create missing `.env` files from `.env.default` (`autogpt_platform`, `backend`, and `frontend`) |
-| `make start-core`      | Start just the core services (Postgres, Redis, RabbitMQ) in background        |
+| `make start-core`      | Start dependency services (PostgreSQL, cache cluster, RabbitMQ, FalkorDB, ClamAV) and run migrations, in background |
 | `make stop-core`       | Stop the core services                                                        |
 | `make logs-core`       | Tail the logs for core services                                               |
 | `make format`          | Format & lint backend (Python) and frontend (TypeScript) code                 |