diff --git a/docs/server/features/projections/README.md b/docs/server/features/projections/README.md index 68c3d9fbfff..be625f27671 100644 --- a/docs/server/features/projections/README.md +++ b/docs/server/features/projections/README.md @@ -50,8 +50,22 @@ meet the criteria. The output of all queries is a stream, you can listen to this There are two types of projections in KurrentDB: - [Built in (system) projections](system.md) -- [User-defined JavaScript projections](custom.md) which you create via the API or the admin - UI +- [User-defined JavaScript projections](custom.md), which you write yourself and create through the + API, a client library, or the embedded UI + +## Tooling + +Kurrent maintains tooling for developing user-defined projections: + +- **[gaffer](https://gaffer.kurrent.io)**, a command-line tool, runs a projection on your machine against + a file of sample events or subscribed to one of your databases. It deploys the projection to a server + when you are ready, and can run it with a debugger attached so you can step through a handler and watch + the state change. See [Your first projection](https://gaffer.kurrent.io/getting-started/first-projection/). +- **[The VS Code extension](https://gaffer.kurrent.io/extension/vs-code/)** drives the same runs and + breakpoints from the editor, and shows which of your projections are deployed where. Other DAP-aware + editors can attach to the debugger too. +- **[The testing library](https://gaffer.kurrent.io/testing/nodejs/)** feeds events to a projection from + a JavaScript test suite and hands back the state at each step for your tests to assert on. ## Performance impact diff --git a/docs/server/features/projections/custom.md b/docs/server/features/projections/custom.md index e79bfbefe52..3a5e4881fce 100644 --- a/docs/server/features/projections/custom.md +++ b/docs/server/features/projections/custom.md @@ -33,6 +33,11 @@ fromStream('account-1') .outputState() ``` +::: tip +Kurrent's [gaffer](https://gaffer.kurrent.io) tooling runs projections like this one on your machine and +deploys them to a server. +::: + ## Projections API Below, you can find the JavaScript API for user defined projections. @@ -110,9 +115,8 @@ handler. The event provided through the handler contains the following propertie ## Debugging -User projections have a bonus that debugging is easier -via any browser that ships with debugging capabilities. The screenshots in this document show the use of -Chrome, but we have tested debugging with all major browsers including Firefox, Edge and Safari. +A projection can log from inside a handler, and you can step through one before it goes anywhere near a +server. Both are covered below. ### Logging from a projection @@ -160,29 +164,13 @@ curl -i -d@stats-counter.json \ -u admin:changeit ``` -### Debugging your first projection - -Once the projection is running, open your browser and enable the developer tools. Once you have the developer -tools open, visit your projection URL and you should see a button labelled _Debug_. - -![Projections Debugging Part 1](images/projections_debugging_part_1.png) - -After clicking the projection "Debug" button, you see the debugging interface with the definition of the -projection and information about the events the projection is processing on the right-hand side. - -At the top there are a couple of buttons to take note of, specifically the _Run Step_ and _Update_ buttons. -You use _Run Step_ to step through the event waiting in the queue, placing you in projection debugging mode. -The _Update_ button provides you with a way to update the projection definition without having to go back to -the projection itself and leave the context of the debugger. - -![Projections Debugging Part 2](images/projections_debugging_part_2.png) - -If the _Run Step_ button is not greyed out, and you click it, the browser has hit a breakpoint. - -![Projections Debugging Part 3](images/projections_debugging_part_3.png) +### Stepping through a projection -You are now able to step through the projection, the important method to step into is -the `handler(state, eventEnvelope)` method. +The [gaffer](https://gaffer.kurrent.io) command line runs a projection on your machine with a debugger +attached, against a file of sample events or against one of your databases, so you can set breakpoints, +step through a handler and inspect the state as it changes without deploying anything. The +[VS Code extension](https://gaffer.kurrent.io/extension/vs-code/) drives it from the editor, and any +DAP-aware editor can attach to it. ## Configuring projections diff --git a/docs/server/features/projections/images/getting-started-create-projection.png b/docs/server/features/projections/images/getting-started-create-projection.png deleted file mode 100644 index ce983c878a8..00000000000 Binary files a/docs/server/features/projections/images/getting-started-create-projection.png and /dev/null differ diff --git a/docs/server/features/projections/images/projections_debugging_part_1.png b/docs/server/features/projections/images/projections_debugging_part_1.png deleted file mode 100644 index 07f7aa1259a..00000000000 Binary files a/docs/server/features/projections/images/projections_debugging_part_1.png and /dev/null differ diff --git a/docs/server/features/projections/images/projections_debugging_part_2.png b/docs/server/features/projections/images/projections_debugging_part_2.png deleted file mode 100644 index bb18d3d566d..00000000000 Binary files a/docs/server/features/projections/images/projections_debugging_part_2.png and /dev/null differ diff --git a/docs/server/features/projections/images/projections_debugging_part_3.png b/docs/server/features/projections/images/projections_debugging_part_3.png deleted file mode 100644 index a42683b3168..00000000000 Binary files a/docs/server/features/projections/images/projections_debugging_part_3.png and /dev/null differ diff --git a/docs/server/features/projections/tutorial.md b/docs/server/features/projections/tutorial.md index 3c410589f1a..31765e3956e 100644 --- a/docs/server/features/projections/tutorial.md +++ b/docs/server/features/projections/tutorial.md @@ -4,150 +4,331 @@ order: 3 # Projections tutorial -Follow the steps below to learn how to create a user defined projection. +In this tutorial you write a projection that counts how many Xbox One S consoles customers added to their shopping carts. You run it on your machine with no database involved, then deploy it to KurrentDB. -## Add sample data +You author it with [gaffer](https://gaffer.kurrent.io), a command-line tool for developing, testing and deploying projections. Gaffer is a standalone tool rather than a client library, and it runs the same JavaScript engine the server does, so what you see locally matches what the server does with the same events. -Download the following files that contain sample data used throughout this step of the getting started guide. +## Before you start -- [shoppingCart-b989fe21-9469-4017-8d71-9820b8dd1164.json](@httpapi/data/shoppingCart-b989fe21-9469-4017-8d71-9820b8dd1164.json) -- [shoppingCart-b989fe21-9469-4017-8d71-9820b8dd1165.json](@httpapi/data/shoppingCart-b989fe21-9469-4017-8d71-9820b8dd1165.json) -- [shoppingCart-b989fe21-9469-4017-8d71-9820b8dd1166.json](@httpapi/data/shoppingCart-b989fe21-9469-4017-8d71-9820b8dd1166.json) -- [shoppingCart-b989fe21-9469-4017-8d71-9820b8dd1167.json](@httpapi/data/shoppingCart-b989fe21-9469-4017-8d71-9820b8dd1167.json) +Install gaffer: -Add the sample data to four different streams: +```bash:no-line-numbers +npm install -g @kurrent/gaffer +``` -@[code](@samples/http-api/add-sample-data.sh) +It needs Node.js 22 or later. See [Install](https://gaffer.kurrent.io/getting-started/install/) for other options. -## Create your first projection +You also need Docker for the [Deploy to KurrentDB](#deploy-to-kurrentdb) section. Until then no database is involved: the projection runs entirely on your machine. -::: tip Next steps -Read [this guide](custom.md#projections-api) to find out more about the user defined projection's API. +::: tip +Gaffer collects anonymous usage telemetry and checks once a day for a newer release. It tells you so the first time you run it. Set `GAFFER_TELEMETRY_OPTOUT=1` to turn the telemetry off. ::: -The projection counts the number of 'XBox One S's that customers added to their shopping carts. - -A projection starts with a selector, in this case `fromAll()`. Another possibility is `fromCategory({category})` which this step discusses later, but for now, `fromAll` should do. - -The second part of a projection is a set of filters. There is a special filter called `$init` that sets up an initial state. You want to start a counter from 0 and each time KurrentDB observes an `ItemAdded` event for an 'Xbox One S,' increment the counter. - -Here is the projection code: - -@[code](@samples/http-api/xbox-one-s-counter.js) - -You create a projection by calling the projection API and providing it with the definition of the projection. Here you decide how to run the projection, declaring that you want the projection to start from the beginning and keep running. You can create a projection using the Admin UI by opening the _Projections_ tab, clicking the _New Projection_ button and filling in the details of your projection. - -![Creating a projection with the KurrentDB Admin UI](images/getting-started-create-projection.png) - -You can also create projections programmatically. Pass the projection JSON file as a parameter of your request, along with any other settings: - -@[code](@samples/http-api/projections/create-projection.sh) - -## Query for the projection state - -Now the projection is running, you can query the state of the projection. As this projection has a single state, query it with the following request: - -@[code](@samples/http-api/projections/query-state.sh) - -The server will send a response similar to this: - -@[code](@samples/http-api/projections/query-state.json) - -## Append to streams from projections - -The above gives you the correct result but requires you to poll for the state of a projection. What if you wanted KurrentDB to notify you about state updates via subscriptions? - -### Output state - -Update the projection to output the state to a stream by calling the `outputState()` method on the projection which by default produces a `$projections-{projection-name}-result` stream. - -Below is the updated projection: +::: tip +Every step below is a branch in the [companion project](https://github.com/kurrent-io/projections-tutorial), so you can start anywhere rather than typing along from the top: -@[code](@samples/http-api/xbox-one-s-counter-outputState.js) +```bash:no-line-numbers +git clone https://github.com/kurrent-io/projections-tutorial +cd projections-tutorial +git switch step-03-run +``` +::: -To update the projection, edit the projection definition in the Admin UI, or issue the following request: +## Create a project + +In an empty directory, create a project and scaffold your first projection: + +```bash:no-line-numbers +gaffer init +gaffer scaffold projections/xbox-one-s-counter.js --engine-version 1 --yes +``` + +`gaffer init` writes a `gaffer.toml`, the file that lists your projections and the databases you deploy them to. `gaffer scaffold` adds a projection to it and creates the JavaScript file: + +```toml +[[projection]] + name = "xbox-one-s-counter" + entry = "projections/xbox-one-s-counter.js" + engine_version = 1 +``` + +Drop the `--yes` and scaffold asks you about each of these instead - which events to read, whether to partition, whether the projection emits, and which engine to use. + +The engine matters here. This version of KurrentDB runs V1, so `--engine-version 1` is explicit because gaffer would otherwise pick V2, an experimental parallel-partition engine added in KurrentDB 26.1. + +The scaffolded projection is a working skeleton with no logic in it yet: + +```javascript +fromAll() + .when({ + $init() { + return {}; + }, + // Add your event handlers here + // EventType(state, event) { + // return state; + // } + }) +``` + +Two pieces to know: + +- **`fromAll()`** is the selector. It reads every event in the database. `fromStream`, `fromStreams` and `fromCategory` narrow that down, and you use one of them later in this tutorial. +- **`.when({...})`** is the handler map. `$init` returns the projection's starting state, and every other key names an event type. A handler receives the current state and the incoming event, and returns the new state. + +## Count the consoles + +Replace the handler map so the projection counts `ItemAdded` events whose description names an Xbox One S: + +```javascript +fromAll() + .when({ + $init() { + return { count: 0 }; + }, + ItemAdded(state, event) { + if (event.body.Description.indexOf("Xbox One S") >= 0) { + state.count += 1; + } + return state; + }, + }) +``` + +`event.body` is the event's JSON payload, already parsed. Events that are not `ItemAdded` have no handler, so the projection ignores them. + +## Run it without a database + +A projection is only interesting once events flow through it, and you do not need a running KurrentDB for that. Put some events in a JSON file - `fixtures/carts.json` - where each entry names the stream it belongs to: + +```json +[ + { + "eventType": "ItemAdded", + "streamId": "shoppingCart-b989fe21-9469-4017-8d71-9820b8dd1164", + "data": "{\"Description\": \"Xbox One S 1TB (Console)\"}" + }, + { + "eventType": "ItemAdded", + "streamId": "shoppingCart-b989fe21-9469-4017-8d71-9820b8dd1164", + "data": "{\"Description\": \"Gears of War 4\"}" + } +] +``` + +The companion project's file holds seven such events across four carts. Register it in `gaffer.toml` as a named fixture: + +```toml +[[projection]] + name = "xbox-one-s-counter" + entry = "projections/xbox-one-s-counter.js" + engine_version = 1 + fixtures.carts = "fixtures/carts.json" +``` + +Then run the projection against it: + +```bash:no-line-numbers +gaffer dev xbox-one-s-counter --fixture carts +``` + +Gaffer summarises what it found in your projection, then feeds each event through your handlers and shows the state after every one: + +```text:no-line-numbers +xbox-one-s-counter + Source: $all + Events: ItemAdded + Engine: v1 + Quirks: unversioned (matching all KurrentDB quirks) + +0@shoppingCart-b989fe21-9469-4017-8d71-9820b8dd1164 +│ type: ItemAdded +│ data: {"Description": "Xbox One S 1TB (Console)"} +╰ processed + state: {"count":1} + +1@shoppingCart-b989fe21-9469-4017-8d71-9820b8dd1164 +│ type: ItemAdded +│ data: {"Description": "Gears of War 4"} +╰ processed + state: {"count":1} + +7 events processed + + State: {"count":3} +``` + +Three of the seven items are Xbox One S consoles, and the second event leaves the count alone because a copy of Gears of War 4 is not a console. Edit the projection and run it again as often as you like: the loop takes seconds and touches no database. -@[code](@samples/http-api/xbox-one-s-counter-outputState.sh) +::: tip +The [VS Code extension](https://gaffer.kurrent.io/extension/vs-code/) runs the same thing with breakpoints, so you can step through a handler and inspect the state as it changes. +::: -Then reset the projection you created above: +## Count the items in each cart -@[code](@samples/http-api/projections/reset-projection.sh) +The projection so far keeps one number for the whole database. More often you want a number per stream - here, how many items are in each cart. -You should get a response similar to the one below: +Add a second projection, `projections/shopping-cart-counter.js`: -@[code](@samples/http-api/projections/reset-projection.json) +```javascript +fromCategory("shoppingCart") + .foreachStream() + .when({ + $init() { + return { count: 0 }; + }, + ItemAdded(state) { + state.count += 1; + return state; + }, + }) +``` -You can now read the events in the result stream by issuing a read request. +Two changes matter. `fromCategory("shoppingCart")` reads only the streams whose names start with `shoppingCart-`, and `.foreachStream()` gives each of those streams its own state, so the counters do not merge into one. -@[code](@samples/http-api/projections/read-projection-events.sh) +Register it the same way, then run it: -And you'll get a response like this: +```bash:no-line-numbers +gaffer dev shopping-cart-counter --fixture carts +``` -@[code](@samples/http-api/projections/reset-projection.json) +This time the result is one state per cart, printed in no particular order: -## The number of items per shopping cart +```text:no-line-numbers +7 events processed -The example in this step so far relied on a global state for the projection, but what if you wanted a count of the number of items in the shopping cart per shopping cart. +shoppingCart-b989fe21-9469-4017-8d71-9820b8dd1164 + state: {"count":2} +shoppingCart-b989fe21-9469-4017-8d71-9820b8dd1165 + state: {"count":2} +shoppingCart-b989fe21-9469-4017-8d71-9820b8dd1166 + state: {"count":2} +shoppingCart-b989fe21-9469-4017-8d71-9820b8dd1167 + state: {"count":1} +``` -KurrentDB has a built-in `$by_category` projection that lets you select events from a particular list of streams. Enable this projection with the following command. +::: tip +On a server, `fromCategory` reads the `$ce-shoppingCart` stream, which the [`$by_category`](system.md#by-category) system projection maintains. That projection must be running, which the database you start in the next section does for you. +::: -@[code](@samples/http-api/projections/enable-by-category.sh) +## Deploy to KurrentDB -The projection links events from existing streams to new streams by splitting the stream name by a separator. You can configure the projection to specify the position of where to split the stream `id` and provide a separator. +Start a database. From the companion project: -By default, the category splits the stream `id` by a dash. The category is the first string. +```bash:no-line-numbers +docker compose up -d +docker compose wait seed +``` -| Stream Name | Category | -|--------------------|----------------------------------------| -| shoppingCart-54 | shoppingCart | -| shoppingCart-v1-54 | shoppingCart | -| shoppingCart | _No category as there is no separator_ | +The first command starts a single insecure KurrentDB node on `localhost:2113` with projections enabled, alongside a one-shot service that appends the same seven events to it. The second waits for that seeding to finish, so the projections have the same data to work with as they did locally. -You want to define a projection that produces a count per stream for a category, but the state needs to be per stream. To do so, use `$by_category` and its `fromCategory` API method. +Tell gaffer about the database by adding an environment to `gaffer.toml`: -Below is the projection: +```toml +[env.local] +connection = "kurrentdb://localhost:2113?tls=false" +default = true +``` -@[code](@samples/http-api/projections/shopping-cart-counter.js) +Then deploy: -Create the projection with the following request: +```bash:no-line-numbers +gaffer deploy +``` -@[code](@samples/http-api/projections/shopping-cart-counter.sh) +Gaffer works out what needs to change and shows you the plan before writing anything: what it would create or update, with everything already in sync reported as a count rather than listed. Confirming applies it: -### Querying for the state of the projection by partition +```text:no-line-numbers + ✓ xbox-one-s-counter created + ✓ shopping-cart-counter created -Querying for the state of the projection is different due to the partitioning of the projection. You have to specify the partition and the name of the stream. +2 created · 0 updated · 0 skipped +``` -@[code](@samples/http-api/projections/read-state-partition.sh) +Deploying is safe to repeat. A projection whose definition has not changed is skipped rather than rewritten, so running `gaffer deploy` twice does nothing the second time. In a script, pass `--yes` to skip the confirmation. -The server then returns the state for the partition: +`gaffer status` then lists each projection with its state on the server, when it was last deployed, and whether what is deployed still matches your local files. -@[code](@samples/http-api/projections/read-state-partition.json) +Your projections are now running on the server, processing events as they arrive. Ask the server what a projection's state is - give it a moment after deploying, since a projection that has just started may still be working through the events: -## Configure projection properties +```bash:no-line-numbers +curl http://localhost:2113/projection/xbox-one-s-counter/state +``` -You can configure properties of the projection by updating values of the `options` object. For example, the following projection changes the name of the results stream: +```json +{"count":3} +``` -@[code{2}](@samples/http-api/projections/update-projection-options.js) +The same count you saw locally. For the partitioned projection, name the partition you want: -Then send the update to the projection: +```bash:no-line-numbers +curl "http://localhost:2113/projection/shopping-cart-counter/state?partition=shoppingCart-b989fe21-9469-4017-8d71-9820b8dd1164" +``` -@[code](@samples/http-api/projections/update-projection-options.sh) +```json +{"count":2} +``` ::: tip -You can find all the options available in the [projections API](custom.md#projections-api) documentation. +Polling for state is not the only option. A projection that calls `.outputState()` also publishes its state as events - on `$projections-{projection-name}-result`, or one stream per partition for a partitioned projection - and a client can subscribe to those the way it would any other stream. They are system streams, so a secured server may require additional permissions to read them. See [User-defined projections](custom.md#projections-api) for that and the rest of the API. ::: -Now you can read the result as above, but use the new stream name: - -@[code](@samples/http-api/projections/read-projection-events-renamed.sh) - -## Query running projections - -You can also query the state of all projections using the HTTP API. - -@[code](@samples/http-api/projections/list-all-projections.sh) - -The response is a list of all known projections and useful information about them. +## Change it and see the difference + +Projections evolve. Suppose you also want to know how many items were added in total, not only the consoles: + +```javascript + $init() { + return { count: 0, items: 0 }; + }, + ItemAdded(state, event) { + state.items += 1; + if (event.body.Description.indexOf("Xbox One S") >= 0) { + state.count += 1; + } + return state; + }, +``` + +Your local file and the deployed projection now disagree. Ask gaffer what changed: + +```bash:no-line-numbers +gaffer diff xbox-one-s-counter +``` + +```text:no-line-numbers +xbox-one-s-counter + Query: +2 -1 + Engine version: 1 + Emit: disabled + Drift: drifted + Last deploy: 2026-08-12 13:08 + + 1 1 fromAll() + 2 2 .when({ + 3 3 $init() { + 4 - return { count: 0 }; + 4 + return { count: 0, items: 0 }; + 5 5 }, + 6 6 ItemAdded(state, event) { + 7 + state.items += 1; + 7 8 if (event.body.Description.indexOf("Xbox One S") >= 0) { + 8 9 state.count += 1; + 9 10 } +10 11 return state; +11 12 }, +12 13 }) +``` + +Run it locally first if you want to check the new logic, then `gaffer deploy` again to update the server. + +::: warning +Changing a projection's logic does not change the state it has already built. By default the updated projection carries on from where it left off, so state accumulated by the old code is still there - here, `items` counts from the point you deployed rather than from the first event. Pass `--reset-on-logic-change` to `gaffer deploy` to reprocess from the beginning instead. +::: -@[code](@samples/http-api/projections/list-all-projections.json) +## Next steps +- [User-defined projections](custom.md) - the full JavaScript API: selectors, partitioning, emitting events, and the options you can set. +- [System projections](system.md) - the projections KurrentDB ships with, including the `$by_category` projection used above. +- [Gaffer documentation](https://gaffer.kurrent.io) - testing projections, deploying from CI, reviewing deploy history, and rolling back. +- Managing projections from your application: see the projections page for your [client SDK](https://docs.kurrent.io/clients/).