From 7282ff5ebca892685a303ab3d04e7d35ef46e0d4 Mon Sep 17 00:00:00 2001 From: LautaroPetaccio Date: Tue, 15 Sep 2026 20:31:55 -0300 Subject: [PATCH 01/10] feat: ADR-318 audio playback position reports --- ...ADR-318-audio-playback-position-reports.md | 97 +++++++++++++++++++ 1 file changed, 97 insertions(+) create mode 100644 content/ADR-318-audio-playback-position-reports.md diff --git a/content/ADR-318-audio-playback-position-reports.md b/content/ADR-318-audio-playback-position-reports.md new file mode 100644 index 00000000..2f307807 --- /dev/null +++ b/content/ADR-318-audio-playback-position-reports.md @@ -0,0 +1,97 @@ +--- +layout: adr +adr: 318 +title: Audio playback position reports +date: 2026-09-15 +status: Draft +type: RFC +spdx-license: CC0-1.0 +authors: + - LautaroPetaccio +--- + +# Abstract + +Scenes that need to line gameplay or visuals up with sound (rhythm games, beat-driven effects, video-synchronized audio) have no way to learn where an `AudioSource` clip's playhead actually is. This document extends the `AudioEvent` component so renderers report the clip position periodically while a clip plays, in addition to the media state changes they already report. The change is additive and backwards compatible. + +## Context and problem statement + +A scene controls audio through `PBAudioSource`: it sets `playing`, and optionally `current_time` as a seek target. Neither is a clock. `current_time` is a write-only command, and the renderer starts or resumes the clip some time after the component is applied: fetching, decoding and seeking a streamed clip all happen before the first sample plays. + +Measured on the Unity explorer with a rhythm-game scene that plays a 64-second song as four synchronized instrument stems and judges key presses against a note chart, the audible drums started 100 to 250 ms after the scene's song clock, with a different value on each start. That game accepts a press within 150 ms of the charted note, so a player who plays by ear is judged late, and the scene had no signal it could use to correct itself. + +The only feedback channel today is `PBAudioEvent`, which carries a `MediaState` and a monotonic counter. It says that playback started, but not when in scene time or from which clip position. Its video counterpart, `PBVideoEvent`, already reports `current_offset`, `video_length` and `tick_number`, so scenes can do for video exactly what they cannot do for audio. + +Workarounds are poor. Manual calibration (tapping to a click) covers a player's own input chain but not the per-start playback latency. Onset detection through `AudioAnalysis` works, and was used to measure the numbers above, but it is Unity-only, indirect, and blind to the last tens of milliseconds of the pipeline. + +## Specification + +### Protocol + +Three optional fields are added to `PBAudioEvent` (component id 1105): + +```protobuf +message PBAudioEvent { + common.MediaState state = 1; + uint32 timestamp = 2; // monotonic counter + + optional uint32 tick_number = 3; // scene tick in which the report was taken, equals EngineInfo.tick_number + optional float current_offset = 4; // playback position of the clip in seconds at that tick + optional float clip_length = 5; // total length of the clip in seconds, when known +} +``` + +### Renderer behaviour + +- Renderers keep appending an `AudioEvent` on every media state change, as today. +- While an `AudioSource` is in `MS_PLAYING`, renderers also append a report at least every 15 scene ticks (about twice a second at the reference tick rate), carrying `tick_number`, `current_offset` and `clip_length`. +- `tick_number` is the tick, as defined by ADR-148, in which the position was sampled. `current_offset` is the clip position at that same frame, so the pair can be compared with any scene-side clock that is also sampled per tick. State-change events written while a clip is attached carry the same fields. +- For `AudioStream` entities the fields may be omitted when the underlying player exposes no position. +- `AudioEvent` remains a grow-only value set with a bounded size; periodic reports evict the oldest entries like any other value. + +### SDK + +`audioEventsSystem` in `@dcl/ecs` gains: + +- `registerAudioPlaybackEntity(entity, callback)` and `removeAudioPlaybackEntity(entity)`: the callback runs for every report, position updates included. +- `getAudioPlayback(entity)`: the latest report that carries `current_offset`, or `undefined`. + +`registerAudioEventsEntity` keeps its current semantics and only fires on state changes, so existing scenes receive no extra callbacks from the periodic reports. + +### Scene usage + +A scene that keeps its own song clock records the wall-clock time at which it processed each tick, then aligns on every report: + +```ts +audioEventsSystem.registerAudioPlaybackEntity(drums, report => { + if (report.currentOffset === undefined || report.tickNumber === undefined) return + const heardAt = songTimeAtTick(report.tickNumber) // ms, from EngineInfo.tickNumber samples + const offset = heardAt - report.currentOffset * 1000 // > 0: audio runs behind the chart + applyOffset(offset) // shift the chart, seek once, or start earlier next time +}) +``` + +The result is exact to one tick, needs no analysis component, and works on every renderer that reports positions. + +## Alternatives considered + +- **Scheduled playback** (`play_at` in scene time): the most precise option for rhythm games, but it needs a clock shared by scene and renderer and a sample-accurate scheduling path in every renderer. It composes with this proposal and can follow it; the report is still needed to verify what was scheduled. +- **Making `current_time` readable**: it is a command with put semantics in a last-write-wins component; turning it into a clock would fight the scene's own writes on every tick. +- **Higher tick rates**: they improve resolution but do not expose the playhead at all. +- **Onset detection with `AudioAnalysis`**: the workaround used to gather the measurements; Unity-only and indirect. + +## Backwards compatibility + +All new fields are optional. Renderers that do not implement the reports keep writing the existing events and scenes see `undefined` positions. Scenes that ignore the fields see exactly the events they see today. No component id changes. + +## Implementation + +- `decentraland/protocol`: the field additions (branch `feat/audio-event-playback-position`). +- `decentraland/js-sdk-toolchain`: the `audioEventsSystem` additions with tests, on `main`; the commit cherry-picks cleanly onto `auth-server`. +- `decentraland/unity-explorer`: periodic reports in `AudioEventsSystem` with a test, pending the regenerated bindings. +- `decentraland/bevy-explorer`: pending; the playhead is available from the audio backend's playback state. + +## References + +- ADR-148: synchronization of CRDT messages between scenes and renderer (tick numbers). +- `PBVideoEvent`, the existing precedent for position reports. From d347d2c54a3c02a70507951dfc6fc4b891e33421 Mon Sep 17 00:00:00 2001 From: LautaroPetaccio Date: Wed, 16 Sep 2026 09:47:35 -0300 Subject: [PATCH 02/10] docs: ADR-318 clarify the VideoEvent precedent and why current_time stays write-only --- content/ADR-318-audio-playback-position-reports.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/content/ADR-318-audio-playback-position-reports.md b/content/ADR-318-audio-playback-position-reports.md index 2f307807..db320b35 100644 --- a/content/ADR-318-audio-playback-position-reports.md +++ b/content/ADR-318-audio-playback-position-reports.md @@ -12,7 +12,7 @@ authors: # Abstract -Scenes that need to line gameplay or visuals up with sound (rhythm games, beat-driven effects, video-synchronized audio) have no way to learn where an `AudioSource` clip's playhead actually is. This document extends the `AudioEvent` component so renderers report the clip position periodically while a clip plays, in addition to the media state changes they already report. The change is additive and backwards compatible. +Scenes that need to line gameplay or visuals up with sound (rhythm games, beat-driven effects, video-synchronized audio) have no way to learn where an `AudioSource` clip's playhead actually is. This document extends the `AudioEvent` component so renderers report the clip position periodically while a clip plays, in addition to the media state changes they already report. It mirrors what `PBVideoEvent` already does for video: the position travels in the renderer-owned event component, and the scene-owned `PBAudioSource` is not touched. The change is additive and backwards compatible. ## Context and problem statement @@ -20,7 +20,7 @@ A scene controls audio through `PBAudioSource`: it sets `playing`, and optionall Measured on the Unity explorer with a rhythm-game scene that plays a 64-second song as four synchronized instrument stems and judges key presses against a note chart, the audible drums started 100 to 250 ms after the scene's song clock, with a different value on each start. That game accepts a press within 150 ms of the charted note, so a player who plays by ear is judged late, and the scene had no signal it could use to correct itself. -The only feedback channel today is `PBAudioEvent`, which carries a `MediaState` and a monotonic counter. It says that playback started, but not when in scene time or from which clip position. Its video counterpart, `PBVideoEvent`, already reports `current_offset`, `video_length` and `tick_number`, so scenes can do for video exactly what they cannot do for audio. +The only feedback channel today is `PBAudioEvent`, which carries a `MediaState` and a monotonic counter. It says that playback started, but not when in scene time or from which clip position. Its video counterpart, `PBVideoEvent`, already reports `current_offset`, `video_length` and `tick_number`, written by the renderer at a fixed cadence while the video plays, so scenes can do for video exactly what they cannot do for audio. Workarounds are poor. Manual calibration (tapping to a click) covers a player's own input chain but not the per-start playback latency. Onset detection through `AudioAnalysis` works, and was used to measure the numbers above, but it is Unity-only, indirect, and blind to the last tens of milliseconds of the pipeline. @@ -44,7 +44,8 @@ message PBAudioEvent { ### Renderer behaviour - Renderers keep appending an `AudioEvent` on every media state change, as today. -- While an `AudioSource` is in `MS_PLAYING`, renderers also append a report at least every 15 scene ticks (about twice a second at the reference tick rate), carrying `tick_number`, `current_offset` and `clip_length`. +- While an `AudioSource` is in `MS_PLAYING`, renderers also append a report at least every 15 scene ticks (about twice a second at the reference tick rate), carrying `tick_number`, `current_offset` and `clip_length`. This is the same mechanism the Unity explorer uses for `PBVideoEvent` in [`VideoEventsSystem`](https://github.com/decentraland/unity-explorer/blob/fe6974465b0d2e3a70eeb1ba2da3cb87df27e654/Explorer/Assets/DCL/SDKComponents/MediaStream/Systems/VideoEventsSystem.cs#L63), applied to audio sources. +- Renderers never write `PBAudioSource`. That component stays scene-owned, and `current_time` keeps its meaning as a seek command. - `tick_number` is the tick, as defined by ADR-148, in which the position was sampled. `current_offset` is the clip position at that same frame, so the pair can be compared with any scene-side clock that is also sampled per tick. State-change events written while a clip is attached carry the same fields. - For `AudioStream` entities the fields may be omitted when the underlying player exposes no position. - `AudioEvent` remains a grow-only value set with a bounded size; periodic reports evict the oldest entries like any other value. @@ -76,7 +77,7 @@ The result is exact to one tick, needs no analysis component, and works on every ## Alternatives considered - **Scheduled playback** (`play_at` in scene time): the most precise option for rhythm games, but it needs a clock shared by scene and renderer and a sample-accurate scheduling path in every renderer. It composes with this proposal and can follow it; the report is still needed to verify what was scheduled. -- **Making `current_time` readable**: it is a command with put semantics in a last-write-wins component; turning it into a clock would fight the scene's own writes on every tick. +- **Making `current_time` readable**: rejected for two reasons. First, `PBAudioSource` is a last-write-wins component owned by the scene; if the renderer also wrote `current_time`, both sides would be updating the same property at the same time and each write would clobber the other. Second, a CRDT put carries the whole component, so when a scene changes any other property (for example `volume`) it re-sends `current_time` as well, and the renderer cannot tell a re-sent value from a real seek. The property was designed to tell the renderer where to start, and only that use survives if it stays write-only. Keeping the position in the renderer-owned event component avoids both problems, exactly as `PBVideoEvent` does. - **Higher tick rates**: they improve resolution but do not expose the playhead at all. - **Onset detection with `AudioAnalysis`**: the workaround used to gather the measurements; Unity-only and indirect. From b6438165e12c7cb421975e875e8e5bd983a41201 Mon Sep 17 00:00:00 2001 From: LautaroPetaccio Date: Thu, 17 Sep 2026 09:16:46 -0300 Subject: [PATCH 03/10] docs: ADR-318 explain why reports carry a tick number --- content/ADR-318-audio-playback-position-reports.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/content/ADR-318-audio-playback-position-reports.md b/content/ADR-318-audio-playback-position-reports.md index db320b35..d2285e97 100644 --- a/content/ADR-318-audio-playback-position-reports.md +++ b/content/ADR-318-audio-playback-position-reports.md @@ -41,6 +41,14 @@ message PBAudioEvent { } ``` +### Why a tick number + +A position report has two halves: where the clip is (`current_offset`) and when that reading was taken. The second half is what makes the first usable. A report travels through the CRDT queue and is processed by the scene some frames after the renderer sampled it, so comparing `current_offset` with the scene's clock at processing time would make every report look late by an unknown and variable amount. + +The scene and the renderer do not share a clock, so the sampling moment cannot be expressed as a wall-clock time either side would trust. They do share the tick: ADR-148 defines it as one round of the scene-to-renderer message exchange, and the renderer publishes the current one in `EngineInfo.tick_number` every frame. A report stamped with the tick lets a scene reason as follows: at tick N my clock read T, the renderer says the clip was at offset X in that same tick, therefore the audio runs T minus X behind my clock. The correlation costs the scene one lookup in a short history of its own clock per tick. + +This is the established convention for renderer-written results. `PBVideoEvent.tick_number` and `PBPointerEventsResult.tick_number` both carry "the tick in which the event was produced, equals to EngineInfo.tick_number", and `PBEngineInfo` documents its tick and frame numbers as correlation values. The Unity explorer fills `PBVideoEvent.TickNumber` from the scene's current tick at sampling time; the audio report is filled the same way. The monotonic `timestamp` field that already existed on `PBAudioEvent` stays what it was, a per-entity ordering counter, and is not a time. + ### Renderer behaviour - Renderers keep appending an `AudioEvent` on every media state change, as today. @@ -61,7 +69,7 @@ message PBAudioEvent { ### Scene usage -A scene that keeps its own song clock records the wall-clock time at which it processed each tick, then aligns on every report: +A scene that keeps its own song clock records that clock against `EngineInfo.tickNumber` each frame, keeping a short history so a report's tick can be looked up after the fact, then aligns on every report: ```ts audioEventsSystem.registerAudioPlaybackEntity(drums, report => { From 50ebc6420704305f311636561b5a7e8a07c2e2f7 Mon Sep 17 00:00:00 2001 From: LautaroPetaccio Date: Thu, 17 Sep 2026 12:26:45 -0300 Subject: [PATCH 04/10] docs: ADR-318 report on playhead change, resolve ticks in the SDK Address review: reports go out whenever the clip position changes, as VideoEventsSystem does, not on a fixed cadence; the scene-clock history that resolves a report's tick is part of the SDK, shown in full, and no round-trip estimation is involved because renderer and scene key on the same tick. --- ...ADR-318-audio-playback-position-reports.md | 41 ++++++++++++++----- 1 file changed, 31 insertions(+), 10 deletions(-) diff --git a/content/ADR-318-audio-playback-position-reports.md b/content/ADR-318-audio-playback-position-reports.md index d2285e97..9a6abe13 100644 --- a/content/ADR-318-audio-playback-position-reports.md +++ b/content/ADR-318-audio-playback-position-reports.md @@ -52,11 +52,11 @@ This is the established convention for renderer-written results. `PBVideoEvent.t ### Renderer behaviour - Renderers keep appending an `AudioEvent` on every media state change, as today. -- While an `AudioSource` is in `MS_PLAYING`, renderers also append a report at least every 15 scene ticks (about twice a second at the reference tick rate), carrying `tick_number`, `current_offset` and `clip_length`. This is the same mechanism the Unity explorer uses for `PBVideoEvent` in [`VideoEventsSystem`](https://github.com/decentraland/unity-explorer/blob/fe6974465b0d2e3a70eeb1ba2da3cb87df27e654/Explorer/Assets/DCL/SDKComponents/MediaStream/Systems/VideoEventsSystem.cs#L63), applied to audio sources. +- Renderers also append a report whenever the clip position has changed since the last report, carrying `tick_number`, `current_offset` and `clip_length`. For a playing clip that is every frame; a paused or stopped clip emits nothing until something changes. This is exactly the rule the Unity explorer applies to `PBVideoEvent` in [`VideoEventsSystem`](https://github.com/decentraland/unity-explorer/blob/fe6974465b0d2e3a70eeb1ba2da3cb87df27e654/Explorer/Assets/DCL/SDKComponents/MediaStream/Systems/VideoEventsSystem.cs#L52), which writes when the state or the current time differs from the last propagated value; no fixed cadence is involved. - Renderers never write `PBAudioSource`. That component stays scene-owned, and `current_time` keeps its meaning as a seek command. - `tick_number` is the tick, as defined by ADR-148, in which the position was sampled. `current_offset` is the clip position at that same frame, so the pair can be compared with any scene-side clock that is also sampled per tick. State-change events written while a clip is attached carry the same fields. - For `AudioStream` entities the fields may be omitted when the underlying player exposes no position. -- `AudioEvent` remains a grow-only value set with a bounded size; periodic reports evict the oldest entries like any other value. +- `AudioEvent` remains a grow-only value set with a bounded size; position reports evict the oldest entries like any other value. ### SDK @@ -65,22 +65,43 @@ This is the established convention for renderer-written results. `PBVideoEvent.t - `registerAudioPlaybackEntity(entity, callback)` and `removeAudioPlaybackEntity(entity)`: the callback runs for every report, position updates included. - `getAudioPlayback(entity)`: the latest report that carries `current_offset`, or `undefined`. -`registerAudioEventsEntity` keeps its current semantics and only fires on state changes, so existing scenes receive no extra callbacks from the periodic reports. +- `registerAudioPlaybackSampleEntity(entity, callback)` and `removeAudioPlaybackSampleEntity(entity)`: the callback receives each position report already resolved against the scene clock, as `{ report, sceneTime, offset }`, where `sceneTime` is the scene clock in the tick the renderer sampled the position. This is the form most scenes should use. +- `getSceneTimeAtTick(tickNumber)`: the scene clock recorded in a given tick, or `undefined` outside the history window. It resolves `PBVideoEvent` reports the same way. + +`registerAudioEventsEntity` keeps its current semantics and only fires on state changes, so existing scenes receive no extra callbacks from the position reports. + +The scene-clock history behind the last two functions lives in the SDK rather than in each scene. It is the one piece a scene could get wrong, and every scene that aligns anything with audio or video needs the same one: + +```ts +// Inside audioEventsSystem. The scene clock is the engine's accumulated delta time. +const sceneTimeByTick = new Map() // tick -> scene clock (s), ~128 ticks kept +let sceneTime = 0 +engine.addSystem((dt) => { + sceneTime += dt + const tick = EngineInfo.getOrNull(engine.RootEntity)?.tickNumber + if (tick === undefined) return + sceneTimeByTick.set(tick, sceneTime) + if (sceneTimeByTick.size > 128) sceneTimeByTick.delete(sceneTimeByTick.keys().next().value!) +}, SYSTEMS_REGULAR_PRIORITY + 1) // runs before reports are delivered in the same tick + +function getSceneTimeAtTick(tick: number) { return sceneTimeByTick.get(tick) } +``` + +Nothing in it estimates a round trip. The renderer stamps the report with the tick in which it read the position, and the scene records its clock under that same tick, so the transport delay between the two cancels out by construction: whether a report takes one tick or ten to arrive, `getSceneTimeAtTick(report.tickNumber)` returns the clock at the sampling moment. The remaining error is the width of one tick. ### Scene usage -A scene that keeps its own song clock records that clock against `EngineInfo.tickNumber` each frame, keeping a short history so a report's tick can be looked up after the fact, then aligns on every report: +A scene started its music at scene clock `songStart` (seconds). The lag between what is heard and the scene's idea of the song position is then one subtraction per report: ```ts -audioEventsSystem.registerAudioPlaybackEntity(drums, report => { - if (report.currentOffset === undefined || report.tickNumber === undefined) return - const heardAt = songTimeAtTick(report.tickNumber) // ms, from EngineInfo.tickNumber samples - const offset = heardAt - report.currentOffset * 1000 // > 0: audio runs behind the chart - applyOffset(offset) // shift the chart, seek once, or start earlier next time +audioEventsSystem.registerAudioPlaybackSampleEntity(drums, ({ sceneTime, offset }) => { + const expected = sceneTime - songStart // where the scene thought the clip was, at the sampling tick + const lag = expected - offset // > 0: the audible clip runs behind the scene clock + applyLag(lag) // shift the chart, seek once, or start earlier next time }) ``` -The result is exact to one tick, needs no analysis component, and works on every renderer that reports positions. +The result is exact to one tick, needs no analysis component, and works on every renderer that reports positions. A scene that prefers raw reports can still use `registerAudioPlaybackEntity` together with `getSceneTimeAtTick`. ## Alternatives considered From 574fca8ae4b35a42781d42f4c1a7f9bac63f75d4 Mon Sep 17 00:00:00 2001 From: LautaroPetaccio Date: Thu, 17 Sep 2026 14:05:18 -0300 Subject: [PATCH 05/10] docs: ADR-318 state the accuracy limits of a position report The text claimed the transport delay cancels out by construction and that the remaining error is one tick. The first half is what the tick stamp does; the second was wrong. Two terms survive it: the sampling granularity of the playhead, and the renderer's output latency, which is the gap between the decoder position a report carries and the moment a sample leaves the speaker. The second is tens of milliseconds and is precisely what a scene aligning to audible sound wants removed, so it is named rather than implied away. Also correct the SDK bullets: delivery is once per scene frame with the newest report, not one callback per appended report. --- .../ADR-318-audio-playback-position-reports.md | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/content/ADR-318-audio-playback-position-reports.md b/content/ADR-318-audio-playback-position-reports.md index 9a6abe13..3912fb0f 100644 --- a/content/ADR-318-audio-playback-position-reports.md +++ b/content/ADR-318-audio-playback-position-reports.md @@ -62,10 +62,10 @@ This is the established convention for renderer-written results. `PBVideoEvent.t `audioEventsSystem` in `@dcl/ecs` gains: -- `registerAudioPlaybackEntity(entity, callback)` and `removeAudioPlaybackEntity(entity)`: the callback runs for every report, position updates included. +- `registerAudioPlaybackEntity(entity, callback)` and `removeAudioPlaybackEntity(entity)`: the callback runs once per scene frame with the newest report for that entity, position updates included, and is skipped when nothing new arrived. A renderer sampling faster than the scene ticks will have appended several reports; the callback sees the freshest, which is the one a scene aligning to the playhead wants. - `getAudioPlayback(entity)`: the latest report that carries `current_offset`, or `undefined`. -- `registerAudioPlaybackSampleEntity(entity, callback)` and `removeAudioPlaybackSampleEntity(entity)`: the callback receives each position report already resolved against the scene clock, as `{ report, sceneTime, offset }`, where `sceneTime` is the scene clock in the tick the renderer sampled the position. This is the form most scenes should use. +- `registerAudioPlaybackSampleEntity(entity, callback)` and `removeAudioPlaybackSampleEntity(entity)`: the same delivery, already resolved against the scene clock, as `{ report, sceneTime, offset }`, where `sceneTime` is the scene clock in the tick the renderer sampled the position. This is the form most scenes should use. - `getSceneTimeAtTick(tickNumber)`: the scene clock recorded in a given tick, or `undefined` outside the history window. It resolves `PBVideoEvent` reports the same way. `registerAudioEventsEntity` keeps its current semantics and only fires on state changes, so existing scenes receive no extra callbacks from the position reports. @@ -87,7 +87,16 @@ engine.addSystem((dt) => { function getSceneTimeAtTick(tick: number) { return sceneTimeByTick.get(tick) } ``` -Nothing in it estimates a round trip. The renderer stamps the report with the tick in which it read the position, and the scene records its clock under that same tick, so the transport delay between the two cancels out by construction: whether a report takes one tick or ten to arrive, `getSceneTimeAtTick(report.tickNumber)` returns the clock at the sampling moment. The remaining error is the width of one tick. +The history does not estimate the round trip, and does not need to. The renderer stamps the report with the tick in which it read the position, and the scene records its clock under that same tick, so the comparison is made against the clock at the sampling moment instead of at processing time: whether a report takes one tick or ten to arrive, `getSceneTimeAtTick(report.tickNumber)` returns the same value. Removing the transport delay from the comparison is what the tick number is for. + +In practice a renderer publishes `EngineInfo.tick_number` and the reports sampled in that tick in the same batch, so the lookup usually resolves inside the frame that receives the report. The history still has to tolerate a tick it has not recorded, and a scene must not discard a report on a miss. + +Removing the transport delay does not make the result exact. Two terms remain, and a scene aligning to the audio it can actually hear is affected by both: + +- **Sampling granularity.** `current_offset` is read once per renderer frame, from a playhead that advances in audio-buffer steps. The reading is quantised to the coarser of the two. +- **Output latency.** The playhead a renderer exposes is the decoder's read position, not the moment a sample leaves the speaker. The mixer buffer, the driver and the device add a further delay, typically tens of milliseconds, in the same direction and roughly constant for a given machine and output device. Nothing in this report captures it. + +The sum of the two behaves as a per-session constant, so a scene that needs alignment finer than a tick can measure it once and subtract it. A renderer able to estimate its own output latency should expose it, and a later revision of this component may carry it as a field; until then the residual is the scene's to calibrate. ### Scene usage @@ -101,7 +110,7 @@ audioEventsSystem.registerAudioPlaybackSampleEntity(drums, ({ sceneTime, offset }) ``` -The result is exact to one tick, needs no analysis component, and works on every renderer that reports positions. A scene that prefers raw reports can still use `registerAudioPlaybackEntity` together with `getSceneTimeAtTick`. +The result needs no analysis component and works on every renderer that reports positions. Its accuracy is bounded by the sampling granularity and the output latency described above, so a scene needing finer alignment than a tick should calibrate that residual once and subtract it here. A scene that prefers raw reports can still use `registerAudioPlaybackEntity` together with `getSceneTimeAtTick`. ## Alternatives considered From a47abc7d028f2b7aacd56b943dcbf341fba14b1a Mon Sep 17 00:00:00 2001 From: LautaroPetaccio Date: Thu, 17 Sep 2026 14:12:44 -0300 Subject: [PATCH 06/10] docs: ADR-318 note that a finished clip reports a zero offset Players rewind the playhead on stop and on reaching the end, so the last report of a finished clip carries zero, not the clip length. A scene driving a progress bar from current_offset would see it snap back instead of complete. --- content/ADR-318-audio-playback-position-reports.md | 1 + 1 file changed, 1 insertion(+) diff --git a/content/ADR-318-audio-playback-position-reports.md b/content/ADR-318-audio-playback-position-reports.md index 3912fb0f..616a6792 100644 --- a/content/ADR-318-audio-playback-position-reports.md +++ b/content/ADR-318-audio-playback-position-reports.md @@ -56,6 +56,7 @@ This is the established convention for renderer-written results. `PBVideoEvent.t - Renderers never write `PBAudioSource`. That component stays scene-owned, and `current_time` keeps its meaning as a seek command. - `tick_number` is the tick, as defined by ADR-148, in which the position was sampled. `current_offset` is the clip position at that same frame, so the pair can be compared with any scene-side clock that is also sampled per tick. State-change events written while a clip is attached carry the same fields. - For `AudioStream` entities the fields may be omitted when the underlying player exposes no position. +- `current_offset` reports where the playhead is, not how much of the clip was consumed. Players commonly rewind the playhead to zero when a clip is stopped or reaches its end, so the last report of a finished clip carries zero rather than `clip_length`. A scene tracking completion should read the state transition out of `MsPlaying`, not wait for the offset to approach the length. - `AudioEvent` remains a grow-only value set with a bounded size; position reports evict the oldest entries like any other value. ### SDK From 4f8d1eae9318580535ec010ce3aaedd782f1b290 Mon Sep 17 00:00:00 2001 From: LautaroPetaccio Date: Thu, 17 Sep 2026 15:18:48 -0300 Subject: [PATCH 07/10] docs: ADR-318 join the split SDK bullet list --- content/ADR-318-audio-playback-position-reports.md | 1 - 1 file changed, 1 deletion(-) diff --git a/content/ADR-318-audio-playback-position-reports.md b/content/ADR-318-audio-playback-position-reports.md index 616a6792..bc54ad68 100644 --- a/content/ADR-318-audio-playback-position-reports.md +++ b/content/ADR-318-audio-playback-position-reports.md @@ -65,7 +65,6 @@ This is the established convention for renderer-written results. `PBVideoEvent.t - `registerAudioPlaybackEntity(entity, callback)` and `removeAudioPlaybackEntity(entity)`: the callback runs once per scene frame with the newest report for that entity, position updates included, and is skipped when nothing new arrived. A renderer sampling faster than the scene ticks will have appended several reports; the callback sees the freshest, which is the one a scene aligning to the playhead wants. - `getAudioPlayback(entity)`: the latest report that carries `current_offset`, or `undefined`. - - `registerAudioPlaybackSampleEntity(entity, callback)` and `removeAudioPlaybackSampleEntity(entity)`: the same delivery, already resolved against the scene clock, as `{ report, sceneTime, offset }`, where `sceneTime` is the scene clock in the tick the renderer sampled the position. This is the form most scenes should use. - `getSceneTimeAtTick(tickNumber)`: the scene clock recorded in a given tick, or `undefined` outside the history window. It resolves `PBVideoEvent` reports the same way. From 4c940f9cc29352d41207e8c79f25231f1ce7208e Mon Sep 17 00:00:00 2001 From: LautaroPetaccio Date: Thu, 17 Sep 2026 16:30:20 -0300 Subject: [PATCH 08/10] docs: ADR-318 offer one playback registration, not two The proposal listed a raw-report callback beside a resolved one. They were the same delivery; the resolved payload carries the raw report as a field, so the raw entry point added no reachable information and was the shorter name a scene author would pick, leading back to per-scene histories of clock snapshots. Record the choice and why the raw form was dropped: reports carrying no position are media-state changes, which registerAudioEventsEntity already delivers. --- content/ADR-318-audio-playback-position-reports.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/content/ADR-318-audio-playback-position-reports.md b/content/ADR-318-audio-playback-position-reports.md index bc54ad68..25336f20 100644 --- a/content/ADR-318-audio-playback-position-reports.md +++ b/content/ADR-318-audio-playback-position-reports.md @@ -63,14 +63,15 @@ This is the established convention for renderer-written results. `PBVideoEvent.t `audioEventsSystem` in `@dcl/ecs` gains: -- `registerAudioPlaybackEntity(entity, callback)` and `removeAudioPlaybackEntity(entity)`: the callback runs once per scene frame with the newest report for that entity, position updates included, and is skipped when nothing new arrived. A renderer sampling faster than the scene ticks will have appended several reports; the callback sees the freshest, which is the one a scene aligning to the playhead wants. +- `registerAudioPlaybackEntity(entity, callback)` and `removeAudioPlaybackEntity(entity)`: the callback runs once per scene frame with the newest position report for that entity, already resolved against the scene clock, as `{ report, sceneTime, offset }` where `sceneTime` is the scene clock in the tick the renderer sampled the position. It is skipped when no new position arrived. A renderer sampling faster than the scene ticks will have appended several reports; the callback sees the freshest, which is the one a scene aligning to the playhead wants. - `getAudioPlayback(entity)`: the latest report that carries `current_offset`, or `undefined`. -- `registerAudioPlaybackSampleEntity(entity, callback)` and `removeAudioPlaybackSampleEntity(entity)`: the same delivery, already resolved against the scene clock, as `{ report, sceneTime, offset }`, where `sceneTime` is the scene clock in the tick the renderer sampled the position. This is the form most scenes should use. - `getSceneTimeAtTick(tickNumber)`: the scene clock recorded in a given tick, or `undefined` outside the history window. It resolves `PBVideoEvent` reports the same way. +Only one registration is offered, and it hands over the resolved reading rather than the raw report. A second entry point delivering the raw report was considered and dropped: it would have been the shorter name and the simpler-looking signature, so it would have been the one scenes reached for, and it leads straight back to every scene keeping its own history of clock snapshots. Reports that carry no position are media-state changes, which `registerAudioEventsEntity` already delivers. + `registerAudioEventsEntity` keeps its current semantics and only fires on state changes, so existing scenes receive no extra callbacks from the position reports. -The scene-clock history behind the last two functions lives in the SDK rather than in each scene. It is the one piece a scene could get wrong, and every scene that aligns anything with audio or video needs the same one: +The scene-clock history behind those functions lives in the SDK rather than in each scene. It is the one piece a scene could get wrong, and every scene that aligns anything with audio or video needs the same one: ```ts // Inside audioEventsSystem. The scene clock is the engine's accumulated delta time. @@ -103,14 +104,14 @@ The sum of the two behaves as a per-session constant, so a scene that needs alig A scene started its music at scene clock `songStart` (seconds). The lag between what is heard and the scene's idea of the song position is then one subtraction per report: ```ts -audioEventsSystem.registerAudioPlaybackSampleEntity(drums, ({ sceneTime, offset }) => { +audioEventsSystem.registerAudioPlaybackEntity(drums, ({ sceneTime, offset }) => { const expected = sceneTime - songStart // where the scene thought the clip was, at the sampling tick const lag = expected - offset // > 0: the audible clip runs behind the scene clock applyLag(lag) // shift the chart, seek once, or start earlier next time }) ``` -The result needs no analysis component and works on every renderer that reports positions. Its accuracy is bounded by the sampling granularity and the output latency described above, so a scene needing finer alignment than a tick should calibrate that residual once and subtract it here. A scene that prefers raw reports can still use `registerAudioPlaybackEntity` together with `getSceneTimeAtTick`. +The result needs no analysis component and works on every renderer that reports positions. Its accuracy is bounded by the sampling granularity and the output latency described above, so a scene needing finer alignment than a tick should calibrate that residual once and subtract it here. A scene that prefers raw reports can read the `AudioEvent` values directly and resolve them with `getSceneTimeAtTick`. ## Alternatives considered From 58677ae86cf9c2d20a1cf64a2593c5eaafab4612 Mon Sep 17 00:00:00 2001 From: LautaroPetaccio Date: Thu, 17 Sep 2026 17:53:05 -0300 Subject: [PATCH 09/10] docs: ADR-318 correct the implementation status for the renderer --- content/ADR-318-audio-playback-position-reports.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/ADR-318-audio-playback-position-reports.md b/content/ADR-318-audio-playback-position-reports.md index 25336f20..1b73eb1d 100644 --- a/content/ADR-318-audio-playback-position-reports.md +++ b/content/ADR-318-audio-playback-position-reports.md @@ -128,7 +128,7 @@ All new fields are optional. Renderers that do not implement the reports keep wr - `decentraland/protocol`: the field additions (branch `feat/audio-event-playback-position`). - `decentraland/js-sdk-toolchain`: the `audioEventsSystem` additions with tests, on `main`; the commit cherry-picks cleanly onto `auth-server`. -- `decentraland/unity-explorer`: periodic reports in `AudioEventsSystem` with a test, pending the regenerated bindings. +- `decentraland/unity-explorer`: `AudioEventsSystem` reports on playhead movement, with the emit rule extracted so it is covered without an audio device; the protocol bindings are regenerated. - `decentraland/bevy-explorer`: pending; the playhead is available from the audio backend's playback state. ## References From 8e98c96511901da7c230411cd3da18584978d8bb Mon Sep 17 00:00:00 2001 From: LautaroPetaccio Date: Mon, 21 Sep 2026 16:00:24 -0300 Subject: [PATCH 10/10] docs: ADR-318 make the reports opt-in, and keep the spec protocol-only Adds PBAudioSource.report_playback_position. A position report is written whenever the playhead moves, far more often than a media state changes, and a scene can hold many more audio sources than video players, which are capped by a prioritisation mechanism. AudioEvent is also a grow-only set bounded per entity, so an unconditional reporter evicts its own state history within seconds. State changes are reported whatever the flag says. State a reporting rate rather than leaving it to each renderer: at most one report per tick, since the tick is the resolution of the stamp, and at least one per second of playback so a scene converges promptly. Existing video implementations already differ from one another inside that band, which is the reason to bound it here. Drop the SDK implementation and the scene usage example. What the SDK does with these fields is a client-library concern, and the code duplicated what the js-sdk-toolchain pull request already carries. What survives is the requirement: the tick-to-clock correlation belongs in the library, not in every scene, and a report must not be dropped when its tick is missing from the history. That requirement is also the protocol rationale for stamping a tick rather than a wall-clock time, so it belongs here. Move the accuracy limits out of the SDK section. Sampling granularity and output latency bound what any renderer can deliver through these fields, so they are protocol-level, not a property of one client library. --- ...ADR-318-audio-playback-position-reports.md | 99 ++++++++----------- 1 file changed, 43 insertions(+), 56 deletions(-) diff --git a/content/ADR-318-audio-playback-position-reports.md b/content/ADR-318-audio-playback-position-reports.md index 1b73eb1d..0195a994 100644 --- a/content/ADR-318-audio-playback-position-reports.md +++ b/content/ADR-318-audio-playback-position-reports.md @@ -12,7 +12,7 @@ authors: # Abstract -Scenes that need to line gameplay or visuals up with sound (rhythm games, beat-driven effects, video-synchronized audio) have no way to learn where an `AudioSource` clip's playhead actually is. This document extends the `AudioEvent` component so renderers report the clip position periodically while a clip plays, in addition to the media state changes they already report. It mirrors what `PBVideoEvent` already does for video: the position travels in the renderer-owned event component, and the scene-owned `PBAudioSource` is not touched. The change is additive and backwards compatible. +Scenes that need to line gameplay or visuals up with sound (rhythm games, beat-driven effects, video-synchronized audio) have no way to learn where an `AudioSource` clip's playhead actually is. This document extends the `AudioEvent` component so renderers report the clip position while a clip plays, in addition to the media state changes they already report, for sources that ask for it. It mirrors what `PBVideoEvent` already does for video: the position travels in the renderer-owned event component, and the renderer never writes the scene-owned `PBAudioSource`. The change is additive, opt-in and backwards compatible. ## Context and problem statement @@ -20,7 +20,7 @@ A scene controls audio through `PBAudioSource`: it sets `playing`, and optionall Measured on the Unity explorer with a rhythm-game scene that plays a 64-second song as four synchronized instrument stems and judges key presses against a note chart, the audible drums started 100 to 250 ms after the scene's song clock, with a different value on each start. That game accepts a press within 150 ms of the charted note, so a player who plays by ear is judged late, and the scene had no signal it could use to correct itself. -The only feedback channel today is `PBAudioEvent`, which carries a `MediaState` and a monotonic counter. It says that playback started, but not when in scene time or from which clip position. Its video counterpart, `PBVideoEvent`, already reports `current_offset`, `video_length` and `tick_number`, written by the renderer at a fixed cadence while the video plays, so scenes can do for video exactly what they cannot do for audio. +The only feedback channel today is `PBAudioEvent`, which carries a `MediaState` and a monotonic counter. It says that playback started, but not when in scene time or from which clip position. Its video counterpart, `PBVideoEvent`, already reports `current_offset`, `video_length` and `tick_number` while the video plays, so scenes can do for video exactly what they cannot do for audio. Workarounds are poor. Manual calibration (tapping to a click) covers a player's own input chain but not the per-start playback latency. Onset detection through `AudioAnalysis` works, and was used to measure the numbers above, but it is Unity-only, indirect, and blind to the last tens of milliseconds of the pipeline. @@ -41,77 +41,64 @@ message PBAudioEvent { } ``` -### Why a tick number +One field is added to `PBAudioSource` (component id 1020), by which a scene asks for the reports: -A position report has two halves: where the clip is (`current_offset`) and when that reading was taken. The second half is what makes the first usable. A report travels through the CRDT queue and is processed by the scene some frames after the renderer sampled it, so comparing `current_offset` with the scene's clock at processing time would make every report look late by an unknown and variable amount. +```protobuf +optional bool report_playback_position = 8; // default false +``` -The scene and the renderer do not share a clock, so the sampling moment cannot be expressed as a wall-clock time either side would trust. They do share the tick: ADR-148 defines it as one round of the scene-to-renderer message exchange, and the renderer publishes the current one in `EngineInfo.tick_number` every frame. A report stamped with the tick lets a scene reason as follows: at tick N my clock read T, the renderer says the clip was at offset X in that same tick, therefore the audio runs T minus X behind my clock. The correlation costs the scene one lookup in a short history of its own clock per tick. +### Why the reports are opt-in -This is the established convention for renderer-written results. `PBVideoEvent.tick_number` and `PBPointerEventsResult.tick_number` both carry "the tick in which the event was produced, equals to EngineInfo.tick_number", and `PBEngineInfo` documents its tick and frame numbers as correlation values. The Unity explorer fills `PBVideoEvent.TickNumber` from the scene's current tick at sampling time; the audio report is filled the same way. The monotonic `timestamp` field that already existed on `PBAudioEvent` stays what it was, a per-entity ordering counter, and is not a time. +A position report is written whenever the playhead moves, which is far more often than a media state changes, and a scene can hold many more audio sources than video players: video playback is capped by a prioritisation mechanism in the renderer, audio is not. Reporting unconditionally would make every fire-and-forget sound effect pay for a signal that only rhythm and synchronisation scenes read. -### Renderer behaviour +The cost is not only traffic. `AudioEvent` is a grow-only value set with a bounded size, and the bound is per entity, so a source reporting on every playhead movement fills its own window within seconds. A scene reading the component directly, rather than through the callbacks a client library offers, would find its state-change history evicted by position reports it never asked for. -- Renderers keep appending an `AudioEvent` on every media state change, as today. -- Renderers also append a report whenever the clip position has changed since the last report, carrying `tick_number`, `current_offset` and `clip_length`. For a playing clip that is every frame; a paused or stopped clip emits nothing until something changes. This is exactly the rule the Unity explorer applies to `PBVideoEvent` in [`VideoEventsSystem`](https://github.com/decentraland/unity-explorer/blob/fe6974465b0d2e3a70eeb1ba2da3cb87df27e654/Explorer/Assets/DCL/SDKComponents/MediaStream/Systems/VideoEventsSystem.cs#L52), which writes when the state or the current time differs from the last propagated value; no fixed cadence is involved. -- Renderers never write `PBAudioSource`. That component stays scene-owned, and `current_time` keeps its meaning as a seek command. -- `tick_number` is the tick, as defined by ADR-148, in which the position was sampled. `current_offset` is the clip position at that same frame, so the pair can be compared with any scene-side clock that is also sampled per tick. State-change events written while a clip is attached carry the same fields. -- For `AudioStream` entities the fields may be omitted when the underlying player exposes no position. -- `current_offset` reports where the playhead is, not how much of the clip was consumed. Players commonly rewind the playhead to zero when a clip is stopped or reaches its end, so the last report of a finished clip carries zero rather than `clip_length`. A scene tracking completion should read the state transition out of `MsPlaying`, not wait for the offset to approach the length. -- `AudioEvent` remains a grow-only value set with a bounded size; position reports evict the oldest entries like any other value. +Media state changes are reported whatever the flag says, so a scene that never sets it observes exactly what it observes today. -### SDK +### Why a tick number -`audioEventsSystem` in `@dcl/ecs` gains: +A position report has two halves: where the clip is (`current_offset`) and when that reading was taken. The second half is what makes the first usable. A report travels through the CRDT queue and is processed by the scene some frames after the renderer sampled it, so comparing `current_offset` with the scene's clock at processing time would make every report look late by an unknown and variable amount. -- `registerAudioPlaybackEntity(entity, callback)` and `removeAudioPlaybackEntity(entity)`: the callback runs once per scene frame with the newest position report for that entity, already resolved against the scene clock, as `{ report, sceneTime, offset }` where `sceneTime` is the scene clock in the tick the renderer sampled the position. It is skipped when no new position arrived. A renderer sampling faster than the scene ticks will have appended several reports; the callback sees the freshest, which is the one a scene aligning to the playhead wants. -- `getAudioPlayback(entity)`: the latest report that carries `current_offset`, or `undefined`. -- `getSceneTimeAtTick(tickNumber)`: the scene clock recorded in a given tick, or `undefined` outside the history window. It resolves `PBVideoEvent` reports the same way. +The scene and the renderer do not share a clock, so the sampling moment cannot be expressed as a wall-clock time either side would trust. They do share the tick: ADR-148 defines it as one round of the scene-to-renderer message exchange, and the renderer publishes the current one in `EngineInfo.tick_number` every frame. A report stamped with the tick lets a scene reason as follows: at tick N my clock read T, the renderer says the clip was at offset X in that same tick, therefore the audible clip began at T minus X on my own clock. The correlation costs one lookup in a short history of the scene's clock per tick, and whether the report took one tick or ten to arrive does not enter the result. Removing the transport delay from the comparison is what the tick number is for. -Only one registration is offered, and it hands over the resolved reading rather than the raw report. A second entry point delivering the raw report was considered and dropped: it would have been the shorter name and the simpler-looking signature, so it would have been the one scenes reached for, and it leads straight back to every scene keeping its own history of clock snapshots. Reports that carry no position are media-state changes, which `registerAudioEventsEntity` already delivers. +In practice a renderer publishes `EngineInfo.tick_number` and the reports sampled in that tick in the same batch, so the lookup usually resolves inside the frame that receives the report. It still has to tolerate being asked for a tick it has not recorded, and a report must not be discarded when that happens. -`registerAudioEventsEntity` keeps its current semantics and only fires on state changes, so existing scenes receive no extra callbacks from the position reports. +This is the established convention for renderer-written results. `PBVideoEvent.tick_number` and `PBPointerEventsResult.tick_number` both carry "the tick in which the event was produced, equals to EngineInfo.tick_number", and `PBEngineInfo` documents its tick and frame numbers as correlation values. The monotonic `timestamp` field that already existed on `PBAudioEvent` stays what it was, a per-entity ordering counter, and is not a time. -The scene-clock history behind those functions lives in the SDK rather than in each scene. It is the one piece a scene could get wrong, and every scene that aligns anything with audio or video needs the same one: +### Accuracy -```ts -// Inside audioEventsSystem. The scene clock is the engine's accumulated delta time. -const sceneTimeByTick = new Map() // tick -> scene clock (s), ~128 ticks kept -let sceneTime = 0 -engine.addSystem((dt) => { - sceneTime += dt - const tick = EngineInfo.getOrNull(engine.RootEntity)?.tickNumber - if (tick === undefined) return - sceneTimeByTick.set(tick, sceneTime) - if (sceneTimeByTick.size > 128) sceneTimeByTick.delete(sceneTimeByTick.keys().next().value!) -}, SYSTEMS_REGULAR_PRIORITY + 1) // runs before reports are delivered in the same tick +Correlating at the sampling tick removes the transport delay. It does not make the result exact, and a scene aligning to the audio it can actually hear is affected by two terms that remain: -function getSceneTimeAtTick(tick: number) { return sceneTimeByTick.get(tick) } -``` +- **Sampling granularity.** `current_offset` is read once per renderer frame, from a playhead that advances in audio-buffer steps. The reading is quantised to the coarser of the two. +- **Output latency.** The playhead a renderer exposes is the decoder's read position, not the moment a sample leaves the speaker. The mixer buffer, the driver and the device add a further delay, typically tens of milliseconds, in the same direction and roughly constant for a given machine and output device. Nothing in this report captures it. -The history does not estimate the round trip, and does not need to. The renderer stamps the report with the tick in which it read the position, and the scene records its clock under that same tick, so the comparison is made against the clock at the sampling moment instead of at processing time: whether a report takes one tick or ten to arrive, `getSceneTimeAtTick(report.tickNumber)` returns the same value. Removing the transport delay from the comparison is what the tick number is for. +The sum of the two behaves as a per-session constant, so a scene that needs alignment finer than a tick can measure it once and subtract it. A renderer able to estimate its own output latency should expose it, and a later revision of this component may carry it as a field; until then the residual is the scene's to calibrate. -In practice a renderer publishes `EngineInfo.tick_number` and the reports sampled in that tick in the same batch, so the lookup usually resolves inside the frame that receives the report. The history still has to tolerate a tick it has not recorded, and a scene must not discard a report on a miss. +### Renderer behaviour -Removing the transport delay does not make the result exact. Two terms remain, and a scene aligning to the audio it can actually hear is affected by both: +- Renderers keep appending an `AudioEvent` on every media state change, as today, whether or not the source opted in. +- While a source that set `report_playback_position` is playing, renderers append a report carrying `tick_number`, `current_offset` and `clip_length` whenever the clip position has moved since the last report. A source that did not opt in never carries those fields, and a paused or stopped clip emits nothing until something changes. +- **Reporting rate.** A renderer writes at most one position report per tick, because the tick is the resolution of the stamp and several reports inside one tick are indistinguishable to the scene. While a clip plays it writes at least one report per second of playback, so a scene converges promptly after a start or a seek. Any rate between those bounds is an implementation choice. Existing video implementations sit inside the same band and differ from one another, which is why the bounds are stated here rather than left to each renderer. +- A position jump the scene did not command, such as a loop wrapping, is reported at once rather than held until the next scheduled report. +- Renderers never write `PBAudioSource`. That component stays scene-owned, and `current_time` keeps its meaning as a seek command. +- `tick_number` is the tick, as defined by ADR-148, in which the position was sampled. `current_offset` is the clip position at that same moment, so the pair can be compared with any scene-side clock that is also sampled per tick. State-change events written while an opted-in clip is attached carry the same fields. +- For `AudioStream` entities the fields may be omitted when the underlying player exposes no position, which is the case for a live stream with no fixed beginning. +- `current_offset` reports where the playhead is, not how much of the clip was consumed. Players commonly rewind the playhead to zero when a clip is stopped or reaches its end, so the last report of a finished clip carries zero rather than `clip_length`. A scene tracking completion should read the state transition out of `MsPlaying`, not wait for the offset to approach the length. +- `AudioEvent` remains a grow-only value set with a bounded size; position reports evict the oldest entries like any other value. -- **Sampling granularity.** `current_offset` is read once per renderer frame, from a playhead that advances in audio-buffer steps. The reading is quantised to the coarser of the two. -- **Output latency.** The playhead a renderer exposes is the decoder's read position, not the moment a sample leaves the speaker. The mixer buffer, the driver and the device add a further delay, typically tens of milliseconds, in the same direction and roughly constant for a given machine and output device. Nothing in this report captures it. +### Client libraries -The sum of the two behaves as a per-session constant, so a scene that needs alignment finer than a tick can measure it once and subtract it. A renderer able to estimate its own output latency should expose it, and a later revision of this component may carry it as a field; until then the residual is the scene's to calibrate. +The tick-to-clock correlation belongs in the client library rather than in each scene. It is the one piece a scene can get wrong, and every scene aligning anything to audio or video needs the same one. -### Scene usage +A library keeps a bounded history of its own scene clock per tick, resolves each report against the clock at the tick the report names, and hands the scene the resolved pair rather than the raw tick. A tick absent from that history must not cause the report to be dropped. The same history resolves `PBVideoEvent` reports, so it is not audio-specific and should not be exposed as though it were. -A scene started its music at scene clock `songStart` (seconds). The lag between what is heard and the scene's idea of the song position is then one subtraction per report: +The reference implementation for `@dcl/ecs` is linked under Implementation. -```ts -audioEventsSystem.registerAudioPlaybackEntity(drums, ({ sceneTime, offset }) => { - const expected = sceneTime - songStart // where the scene thought the clip was, at the sampling tick - const lag = expected - offset // > 0: the audible clip runs behind the scene clock - applyLag(lag) // shift the chart, seek once, or start earlier next time -}) -``` +### Scene usage + +A scene sets `report_playback_position` on the source it wants to follow, then reads the resolved reports. Subtracting the offset from the scene clock at the sampling tick gives the moment the audible clip began, on the scene's own clock. Keeping that origin is enough: the clip's position at any later moment is the scene clock now minus that origin, and each further report corrects it for drift. -The result needs no analysis component and works on every renderer that reports positions. Its accuracy is bounded by the sampling granularity and the output latency described above, so a scene needing finer alignment than a tick should calibrate that residual once and subtract it here. A scene that prefers raw reports can read the `AudioEvent` values directly and resolve them with `getSceneTimeAtTick`. +Comparing that origin against the moment the scene believed it started the clip gives the start delay, which is what a rhythm scene shifts its chart by, or seeks once to remove. The result needs no analysis component and works on every renderer that reports positions, bounded by the accuracy described above. A scene that prefers raw reports can read the `AudioEvent` values directly and resolve them against the per-tick clock history the client library exposes. ## Alternatives considered @@ -122,14 +109,14 @@ The result needs no analysis component and works on every renderer that reports ## Backwards compatibility -All new fields are optional. Renderers that do not implement the reports keep writing the existing events and scenes see `undefined` positions. Scenes that ignore the fields see exactly the events they see today. No component id changes. +All new fields are optional and the reports are off unless a scene asks for them, so a scene that changes nothing observes exactly the events it observes today. Renderers that do not implement the reports keep writing the existing events, and a scene sees `undefined` positions there whether or not it set the flag, so scenes must treat a missing position as normal rather than as an error. No component id changes. ## Implementation -- `decentraland/protocol`: the field additions (branch `feat/audio-event-playback-position`). -- `decentraland/js-sdk-toolchain`: the `audioEventsSystem` additions with tests, on `main`; the commit cherry-picks cleanly onto `auth-server`. -- `decentraland/unity-explorer`: `AudioEventsSystem` reports on playhead movement, with the emit rule extracted so it is covered without an audio device; the protocol bindings are regenerated. -- `decentraland/bevy-explorer`: pending; the playhead is available from the audio backend's playback state. +- `decentraland/protocol`: the `PBAudioEvent` and `PBAudioSource` field additions (branch `feat/audio-event-playback-position`). +- `decentraland/js-sdk-toolchain`: the `audioEventsSystem` additions with tests, including the per-tick clock history and the resolved callback, on `main`; the commit cherry-picks cleanly onto `auth-server`. +- `decentraland/unity-explorer`: `AudioEventsSystem` reports on playhead movement for opted-in sources, with the emit rule extracted so it is covered without an audio device. +- `decentraland/bevy-explorer` and `decentraland/godot-explorer`: pending for audio. Both already report video positions on a threshold rather than every frame, which is the behaviour the reporting-rate bounds above are written to accommodate. ## References