docs: audio playback position reports in audioEventsSystem

content/creator/sdk7/3d-essentials/sounds.md

@@ -79,7 +79,7 @@ The following properties can be set:
   {{< hint info >}}
   **💡 Tip**: To prevent a sound effect from becoming too repetitive during a game, it's useful to randomize some slight variations to the sound's pitch every time it plays.
   {{< /hint >}}
-- `currentTime`: _(optional)_ 0 by default. Set this value to avoid starting from the beginning of the sound file.
+- `currentTime`: _(optional)_ 0 by default. Set this value to avoid starting from the beginning of the sound file. This is a seek command: the renderer never writes the current position of the sound back into it. To know where a sound actually is, see [Playback position reports](#playback-position-reports).
 
 Each entity can only have a single `AudioSource` component, that can only play a single clip at a time. This limitation can be easily overcome by modifying the audio source at the time of playing a new sound, or by including multiple invisible child entities, each with their own sound.
 
@@ -184,6 +184,127 @@ To play a segment of a longer sound file, use the `playSoundSegment()` in the SD
 
 You can also achieve this by explicitly set the `currentTime` property on an `AudioSource` component, and then stopping it after waiting for a period of time.
 
+## Audio events
+
+The `AudioEvent` component is written by the renderer, not by your scene. It reports the state of the sound that an entity's `AudioSource` (or `AudioStream`) is playing: loading, ready, playing, paused, etc. Use the `audioEventsSystem` to react to these reports.
+
+Use `audioEventsSystem.registerAudioEventsEntity` to define a function that runs every time the audio state of an entity changes. Your function can check the new state and respond accordingly.
+
+```ts
+import { engine, AudioSource, audioEventsSystem, MediaState } from '@dcl/sdk/ecs'
+
+const sourceEntity = engine.addEntity()
+
+AudioSource.create(sourceEntity, {
+	audioClipUrl: 'sounds/music.mp3',
+	playing: true,
+})
+
+audioEventsSystem.registerAudioEventsEntity(sourceEntity, (audioEvent) => {
+	switch (audioEvent.state) {
+		case MediaState.MS_PLAYING:
+			console.log('audio started PLAYING')
+			break
+		case MediaState.MS_PAUSED:
+			console.log('audio is PAUSED')
+			break
+		case MediaState.MS_ERROR:
+			console.log('audio ERROR')
+			break
+	}
+})
+```
+
+The `audioEvent` object passed to the function contains the following properties:
+
+- `state`: The new state of the sound, expressed as a value of the `MediaState` enum. See [Stream state]({{< ref "/content/creator/sdk7/media/audio-streaming.md#stream-state" >}}) for the full list of possible values.
+- `timestamp` (_number_): A counter that the renderer increments on every report it writes for the entity. It is **not** a time value, only use it to tell reports apart or to order them.
+
+Query the latest report for an entity at any time with `audioEventsSystem.getAudioState()`. It returns `undefined` if the renderer hasn't written any report yet.
+
+To stop listening, use `audioEventsSystem.removeAudioEventsEntity()`. Use `audioEventsSystem.hasAudioEventsEntity()` to check if an entity is currently registered.
+
+### Playback position reports
+
+{{< hint warning >}}
+**📔 Note**: Playback position reports require an SDK version that includes them and an explorer that implements them. The DCL 2.0 desktop client is the first to do so. On explorers that don't, the properties described below stay `undefined`, and `getAudioPlayback()` always returns `undefined`.
+{{< /hint >}}
+
+While an `AudioSource` clip is playing, the renderer also writes reports of the playback position into the `AudioEvent` component, every time the playhead moves. These reports carry the following _optional_ properties, in addition to `state` and `timestamp`:
+
+- `tickNumber` (_number_): The scene tick in which the position was sampled. It matches the `tickNumber` of the [EngineInfo]({{< ref "/content/creator/sdk7/interactivity/runtime-data.md#the-engineinfo-component" >}}) component in that same tick.
+- `currentOffset` (_number_): The playback position of the clip, in seconds, at that tick.
+- `clipLength` (_number_): The total length of the clip, in seconds, when known.
+
+These reports are the only way to know what the player is actually hearing. The renderer starts a clip 100 to 250 milliseconds after your scene asks for it, and that delay is different every time. The `currentTime` property of the `AudioSource` component doesn't help either: it's a seek command that your scene writes and the renderer never updates, so reading it back only tells you what you last set. Rely on the reports instead whenever your gameplay needs to follow the sound: rhythm games, effects that fire on the beat, or aligning a sound to a video.
+
+Use `audioEventsSystem.registerAudioPlaybackEntity` to define a function that runs once per frame with the newest position report for that entity. It's skipped on frames where no new position arrived. Functions registered with `registerAudioEventsEntity` do **not** run for reports that only update the position.
+
+Your function receives the reading already worked out against your scene's clock, as `{ report, sceneTime, offset }`. `report` is the raw `AudioEvent` if you need `clipLength` or the state, `offset` is the position in seconds, and `sceneTime` is explained in the next section.
+
+```ts
+audioEventsSystem.registerAudioPlaybackEntity(sourceEntity, ({ report, offset }) => {
+	console.log(`clip at ${offset}s of ${report.clipLength ?? 'unknown'}s`)
+})
+```
+
+Use `audioEventsSystem.getAudioPlayback()` to read the latest report that carries a position, for example to draw a progress bar. It returns `undefined` until the renderer reports a position, so always check for that before using the value. To stop listening, use `audioEventsSystem.removeAudioPlaybackEntity()`.
+
+```ts
+engine.addSystem(() => {
+	const playback = audioEventsSystem.getAudioPlayback(sourceEntity)
+	if (!playback || playback.currentOffset === undefined || playback.clipLength === undefined) return
+
+	const progress = playback.currentOffset / playback.clipLength
+	// ... update a progress bar, etc
+})
+```
+
+### Sync gameplay to the sound
+
+A report reaches your scene a few frames after the renderer sampled it. Don't compare `currentOffset` to your clock at the moment your function runs, or you'll be off by however long the report took to arrive. It has to be compared against your clock *in the tick the position was sampled*.
+
+The SDK keeps that per-tick history for you, which is why `registerAudioPlaybackEntity` hands you `sceneTime`: the scene clock in seconds at the tick the reading was taken, paired with `offset`, the clip position at that same moment.
+
+Subtracting one from the other gives the moment the audible clip started. Keep that, and the clip's position at any later time is a single subtraction.
+
+```ts
+import { engine, AudioSource, audioEventsSystem, MediaState } from '@dcl/sdk/ecs'
+
+const sourceEntity = engine.addEntity()
+
+// Your own scene clock, in milliseconds
+let clockMs = 0
+engine.addSystem((dt) => {
+	clockMs += dt * 1000
+})
+
+// The moment the sound you can actually hear started, on that same clock
+let originMs: number | undefined
+
+audioEventsSystem.registerAudioPlaybackEntity(sourceEntity, ({ report, sceneTime, offset }) => {
+	if (report.state !== MediaState.MS_PLAYING) return
+
+	originMs = sceneTime * 1000 - offset * 1000
+})
+
+AudioSource.playSound(sourceEntity, 'sounds/music.mp3', true)
+
+// At any moment, the clip is at about (clockMs - originMs) milliseconds
+```
+
+`clockMs - originMs` is your best estimate of the clip's position at any moment between two reports. Use it to schedule effects on the beat, or to judge how well timed a player's input was. If the explorer doesn't send position reports, the callback never runs, `originMs` stays `undefined`, and your scene should fall back to its own clock.
+
+If you'd rather handle the raw reports, `audioEventsSystem.getSceneTimeAtTick(tickNumber)` gives you the same per-tick lookup on its own. It works for `VideoEvent` reports too.
+
+{{< hint warning >}}
+**📔 Note**: `currentOffset` is where the decoder is reading, which isn't exactly what reaches the speakers. The sound card and its buffers add a few tens of milliseconds on top, and no property reports that. It's roughly constant for a given device, so if you need accuracy finer than a tick, measure it once at the start and subtract it.
+{{< /hint >}}
+
+{{< hint info >}}
+**💡 Tip**: The `VideoEvent` component reports `tickNumber` and `currentOffset` for videos in the same way, see [Video events]({{< ref "/content/creator/sdk7/media/video-playing.md#video-events" >}}). Use both to keep a sound and a video aligned.
+{{< /hint >}}
+
 ## Audio streaming
 
 See [Audio streaming]({{< ref "/content/creator/sdk7/media/audio-streaming.md" >}}) to learn how you can play a live audio stream from an external source.