From 6d453791ee4c7b86387e5b715a2791a697833f58 Mon Sep 17 00:00:00 2001 From: rmaddikery Date: Fri, 4 Sep 2026 09:10:39 +0200 Subject: [PATCH 1/4] docs: Adds DR-002 for DLT network transport - GTL adpotion - DLTv2 seam support - DLT header construction/transmission into a dedicated network-stack module --- .../DR-002-dlt-network-transport.rst | 127 ++++++++++++++++++ docs/features/logging/index.rst | 1 + 2 files changed, 128 insertions(+) create mode 100644 docs/features/logging/architecture/DR-002-dlt-network-transport.rst diff --git a/docs/features/logging/architecture/DR-002-dlt-network-transport.rst b/docs/features/logging/architecture/DR-002-dlt-network-transport.rst new file mode 100644 index 00000000..49bfc606 --- /dev/null +++ b/docs/features/logging/architecture/DR-002-dlt-network-transport.rst @@ -0,0 +1,127 @@ +.. + # ******************************************************************************* + # Copyright (c) 2026 Contributors to the Eclipse Foundation + # + # See the NOTICE file(s) distributed with this work for additional + # information regarding copyright ownership. + # + # This program and the accompanying materials are made available under the + # terms of the Apache License Version 2.0 which is available at + # https://www.apache.org/licenses/LICENSE-2.0 + # + # SPDX-License-Identifier: Apache-2.0 + # ******************************************************************************* + +DLT Network Transport Evolution +================================ + +.. dec_rec:: DLT Network Transport Evolution + :id: dec_rec__logging__dlt_transport_evolution + :status: proposed + :version: 1 + :context: See below. + :decision: TBA + + Today the remote/DLT path is split across two processes, as described in + :doc:`index` and :doc:`../../../components/datarouter/index`: + + - `mw::log` (application side) serialises log records and writes + them into a shared-memory buffer. + - `datarouter` (a separate process) reads that buffer, constructs the + DLT protocol headers, and transmits the resulting UDP/IPv4 multicast + packets using the standard BSD Socket API over the platform's default + network stack, shared with all other networked services. + + A second, feature-flagged client backend (`shm_dma_enabled`) + would allow forwarding of records through a GTL client into a + DMA-capable shared-memory region instead of the DataRouter + ring buffer, handing them to a DLT-aware daemon/plugin on the receiving + side. This is not the default today, and it does not by itself change + how that receiving daemon talks to the network stack, so the properties + below still apply regardless of which client backend feeds it. + + This works, but has three structural properties worth revisiting: + + - Every message crosses two IPC hops before it reaches the wire: + one between `mw::log` and `datarouter`, and a second one from + `datarouter` into the network stack itself, since the socket API + it uses is, on this class of platforms, implemented over IPC to a + separate network-stack process rather than executing inline. Each + hop also implies copying the message (application buffer into shared + memory, shared memory into a new buffer with headers prepended, and + again into the network stack's own send buffers). It is this + combination of copies and context switches across both hops, not a + single IPC call, that drives CPU load. + - Where that network-stack process is a single-threaded resource + manager (e.g. QNX's `io-pkt`), it serialises *all* socket traffic + on the system through one queue. This is a separate overhead from + the copies above: it is contention/scheduling cost, so a burst of + log and trace traffic can add latency for every other socket user of + that same instance, and vice versa. Newer, multithreaded stack + implementations (e.g. `io-sock`) reduce this specific contention, + but do not by themselves remove the two IPC hops and copies above. + - log and trace traffic shares the same network stack and send queue as + other service traffic (E.g. Someip communication), so there is no + structural isolation between the two; any queuing or scheduling + behaviour of one can influence the other. + + **Way Forward:** + + Part 1: A GTL-based client backend as the remote-logging path. + + Part 2: Provide a compile-time seam to select between the DLTv1 wire + format and the DLTv2 wire format, analogous to the existing + build-flag pattern for Part 1, rather than replacing one with the + other. Payload serialisation (verbose/non-verbose argument + encoding) is identical between DLTv1 and DLTv2 and does not need a + seam. The concrete DLTv2 protocol implementation behind + this seam is closed-source and maintained outside this repository; + this repository only needs to own the seam/interface, not the DLTv2 + implementation itself. + + Part 3: Move the DLT header-construction + and transmission stage (i.e. the network-writing responsibility + on the daemon receiving GTL records) into a module that is loaded + directly by the network stack, running on a second, dedicated + network-stack instance used exclusively for log and trace traffic: + + - Removes the second IPC hop (and its associated copy) between the + router logic and the network stack for the transmit path. + - Allows direct use of the network stack's native buffer/interface APIs + instead of the generic socket API, removing at least one further + copy and enabling zero-copy transmission where supported by the + driver. + - Structurally isolates log and trace traffic from other network traffic, + since it no longer shares a network-stack instance, queue, or + scheduling domain with it. + - Enables transport-level controls (e.g. egress traffic shaping) to be + applied specifically to the log and trace traffic instance without + affecting other traffic. + + .. uml:: _assets/lsm_plugin.puml + + Out of scope / unaffected: + + - The `mw::log` application-facing logging APIs are unaffected; only + the backend/transport selected underneath it changes. + - The DLT payload serialisation (verbose/non-verbose argument encoding) + is unaffected, since it is identical between DLTv1 and DLTv2. + - The DLTv2 protocol implementation is out of scope: this repository only + provides the compile-time seam to select it (Part 2); the + implementation behind that seam is closed-source and lives in a + separate, non-public `repository `_. + - Freedom-from-interference (FFI) guarantees is unaffected and already + provided by the existing mw::log infrastructure. + + **Trade-offs** + + - Both halves of this evolution targets DMA/zero-copy where the + target hardware happens to support it. Each target platform + needs to be verified and configured individually; where + DMA/zero-copy isn't available, the transport still works, + just without the associated performance benefit. + - Introduces a second network-stack instance that must be configured, + operated, and kept isolated from the default one. + - Requires a feasibility phase to confirm the target network stack + supports loadable modules with the required capabilities for the + supported target platforms. diff --git a/docs/features/logging/index.rst b/docs/features/logging/index.rst index 7538d834..f306a902 100644 --- a/docs/features/logging/index.rst +++ b/docs/features/logging/index.rst @@ -43,4 +43,5 @@ capture the safety and security constraints that any backend implementation must architecture/index.rst architecture/chklst_arc_inspection.rst architecture/DR-001-logging.rst + architecture/DR-002-dlt-network-transport.rst safety_planning/index.rst From db51c173e5ab9a53592a12a58a3881ab7f346450 Mon Sep 17 00:00:00 2001 From: rmaddikery Date: Fri, 4 Sep 2026 09:53:47 +0200 Subject: [PATCH 2/4] docs: Adds puml diagram for dr-002 --- .../architecture/_assets/lsm_plugin.puml | 102 ++++++++++++++++++ 1 file changed, 102 insertions(+) create mode 100644 docs/features/logging/architecture/_assets/lsm_plugin.puml diff --git a/docs/features/logging/architecture/_assets/lsm_plugin.puml b/docs/features/logging/architecture/_assets/lsm_plugin.puml new file mode 100644 index 00000000..37ced241 --- /dev/null +++ b/docs/features/logging/architecture/_assets/lsm_plugin.puml @@ -0,0 +1,102 @@ +' ******************************************************************************* +' Copyright (c) 2026 Contributors to the Eclipse Foundation +' +' See the NOTICE file(s) distributed with this work for additional +' information regarding copyright ownership. +' +' This program and the accompanying materials are made available under the +' terms of the Apache License Version 2.0 which is available at +' https://www.apache.org/licenses/LICENSE-2.0 +' +' SPDX-License-Identifier: Apache-2.0 +' ******************************************************************************* + +@startuml network_stack_plugin + +skinparam componentStyle rectangle +skinparam nodesep 40 +skinparam ranksep 60 +skinparam linetype ortho + +package "Network Stack (default instance)" <> { + component "devs-network-driver.so" as driver_default +} +component "mw::diag" as mwdiag +component "DLT Diagnostics" as dltdiag <> + +driver_default -r-> mwdiag +mwdiag -r-> dltdiag + +package "client" <> { + component "mw::log" as mwlog <> + component "GTL" as gtl <> +} + +file "Client\nConfiguration" as clientconfig +clientconfig -d-> mwlog + +interface "shm\n(payload)" as shmpayload +interface "shm\n(metadata)" as shmmeta +interface "shm\n(control block)" as shmctrl +interface "DLT QNX\nControl Channel" as ctrlchannel + +gtl -r-> shmpayload +gtl -r-> shmmeta +gtl -r-> shmctrl +mwlog -r-> ctrlchannel +dltdiag -r-> ctrlchannel + +package "Network Stack (log_and_trace instance)" <> { + package "DLTv2 plugin" { + component "GTL Backend" as gtlbackend + component "Core" as core + component "Statistics\nModule" as stats + component "DLTv2 Router" as router + component "DLTv2 File\nWriter" as filewriter + component "DLTv2 Network\nWriter" as netwriter + component "Network Stack\nNative APIs" as fbsdapi + component "devs-network-driver.so" as driver + } +} + +shmpayload -r-> gtlbackend : R/O +shmmeta -r-> gtlbackend : R/W +shmctrl -r-> core : R/W +ctrlchannel -r-> core + +' Hidden constraints only: force the intended left-to-right column order +' (diagnostics, client, shared memory, plugin) without drawing extra lines. +driver_default -[hidden]r-> gtl +gtl -[hidden]r-> gtlbackend + +gtlbackend -d-> core +core -d-> router +core -l-> stats +router -d-> filewriter +router -d-> netwriter +filewriter -d-> fbsdapi +netwriter -d-> fbsdapi +fbsdapi -d-> driver + +file "Global\nConfiguration" as globalconfig +globalconfig -d-> core + +file "Router\nConfiguration" as routerconfig +routerconfig -d-> router + +database "filesystem\n(devb-*, fs-qnx6.so)" as fsdb +filewriter -d-> fsdb + +' Optional/experimental time-sync path, greyed out in the source diagram +component "gPTP\nLogger Time" as gptp <> +interface "shm\n(logger time)" as shmtime <> +note right of shmtime + Optional, not yet available: read by + ""mw::time::hw_logger_time"" for use by ""mw::log"". +end note + +fbsdapi -d-> gptp +gptp -r-> shmtime + +@enduml + From 6750b80283e3c33986cf3fcb520d3eb0395d9fda Mon Sep 17 00:00:00 2001 From: rmaddikery Date: Tue, 8 Sep 2026 16:18:23 +0200 Subject: [PATCH 3/4] Updates lsm_plugin diagram with wire and GTL backend details - Adds dlt wire format selection seam - Shows how log records are forwarded to GTL backend via GTL - Fix mw::diag interface and shows an independant component translating diag commands via DLT qnx control channel - Added legend --- .../architecture/_assets/lsm_plugin.puml | 125 ++++++++++++------ 1 file changed, 84 insertions(+), 41 deletions(-) diff --git a/docs/features/logging/architecture/_assets/lsm_plugin.puml b/docs/features/logging/architecture/_assets/lsm_plugin.puml index 37ced241..ea3c0361 100644 --- a/docs/features/logging/architecture/_assets/lsm_plugin.puml +++ b/docs/features/logging/architecture/_assets/lsm_plugin.puml @@ -11,75 +11,108 @@ ' SPDX-License-Identifier: Apache-2.0 ' ******************************************************************************* -@startuml network_stack_plugin +@startuml lsm_plugin skinparam componentStyle rectangle skinparam nodesep 40 skinparam ranksep 60 skinparam linetype ortho -package "Network Stack (default instance)" <> { - component "devs-network-driver.so" as driver_default -} -component "mw::diag" as mwdiag -component "DLT Diagnostics" as dltdiag <> +interface "mw::diag" as mwdiag +component "DLT Diagnostics\nComponent" as dltdiag <> +dltdiag --> mwdiag -driver_default -r-> mwdiag -mwdiag -r-> dltdiag +note top of dltdiag + Independent component registered with the + Diagnostic Manager. +end note package "client" <> { component "mw::log" as mwlog <> component "GTL" as gtl <> + + mwlog -r-> gtl } +note bottom of mwlog + Forwards records to GTL via an owned + DltTraceBackend/ITraceLibrary client. +end note + file "Client\nConfiguration" as clientconfig clientconfig -d-> mwlog -interface "shm\n(payload)" as shmpayload -interface "shm\n(metadata)" as shmmeta -interface "shm\n(control block)" as shmctrl -interface "DLT QNX\nControl Channel" as ctrlchannel +'interface name left blank to avoid overlaps and instead highlighted via note below +interface " " as shmpayload +interface " " as shmmeta +interface " " as shmctrl +interface " " as ctrlchannel -gtl -r-> shmpayload -gtl -r-> shmmeta -gtl -r-> shmctrl -mwlog -r-> ctrlchannel -dltdiag -r-> ctrlchannel +note top of shmpayload #Wheat : shm\n(payload) +note top of shmmeta #Wheat : shm\n(metadata) +note bottom of shmctrl #Wheat : shm\n(control block) +note top of ctrlchannel #Wheat :DLT QNX\nControl Channel -package "Network Stack (log_and_trace instance)" <> { - package "DLTv2 plugin" { +gtl -r-> shmpayload: R/W +gtl -r-> shmmeta: R/W +mwlog -r-> shmctrl: R/W +dltdiag --> ctrlchannel + +package "Network Stack (log_and_trace instance)" <> as networkstackinstance { + package "DLT Plugin" { component "GTL Backend" as gtlbackend component "Core" as core component "Statistics\nModule" as stats - component "DLTv2 Router" as router - component "DLTv2 File\nWriter" as filewriter - component "DLTv2 Network\nWriter" as netwriter + component "DLT Router" as router + interface "DLT Wire Format\n(compile-time seam)" as wireformat + component "DLT File\nWriter" as filewriter + component "DLT Network\nWriter" as netwriter component "Network Stack\nNative APIs" as fbsdapi component "devs-network-driver.so" as driver + component "gPTP\nLogger Time" as gptp <> } } +package "DLT Wire Format Implementations" as wireformatpkg { + component "DLTv2 \n(closed-source)" as dltv2wire <> #OrangeRed + component "DLTv1 \n(this repository)" as dltv1wire +} + +' Optional/experimental time-sync path +package "mw::time" as mwtime <> { + component "shm\n(logger time)" as shmloggertime <> +} + +dltv2wire -[hidden]r-> dltv1wire + shmpayload -r-> gtlbackend : R/O -shmmeta -r-> gtlbackend : R/W -shmctrl -r-> core : R/W -ctrlchannel -r-> core +shmmeta -r-> gtlbackend : R/O -' Hidden constraints only: force the intended left-to-right column order -' (diagnostics, client, shared memory, plugin) without drawing extra lines. -driver_default -[hidden]r-> gtl -gtl -[hidden]r-> gtlbackend +shmctrl <-r-> core : R/W +ctrlchannel -u-> core gtlbackend -d-> core core -d-> router core -l-> stats router -d-> filewriter router -d-> netwriter + +wireformat -u-> filewriter +wireformat -u-> netwriter + +wireformatpkg ..|> wireformat + +note left of wireformat + Compile-time build-flag selection between + DLTv1 and DLTv2 wire formats. +end note + filewriter -d-> fbsdapi netwriter -d-> fbsdapi fbsdapi -d-> driver file "Global\nConfiguration" as globalconfig -globalconfig -d-> core +core <-d- globalconfig file "Router\nConfiguration" as routerconfig routerconfig -d-> router @@ -87,16 +120,26 @@ routerconfig -d-> router database "filesystem\n(devb-*, fs-qnx6.so)" as fsdb filewriter -d-> fsdb -' Optional/experimental time-sync path, greyed out in the source diagram -component "gPTP\nLogger Time" as gptp <> -interface "shm\n(logger time)" as shmtime <> -note right of shmtime - Optional, not yet available: read by - ""mw::time::hw_logger_time"" for use by ""mw::log"". -end note - -fbsdapi -d-> gptp -gptp -r-> shmtime +' Optional time-sync path +fbsdapi <-r- gptp +driver <-r- gptp +gptp <-r- mwtime +mwlog -r-> shmloggertime + +legend bottom + |= Acronym |= Meaning | + | DLT | Diagnostics Log and Trace | + | GTL | Generic Trace Library | + | QM | Quality Managed (non-ASIL) | + | ASIL | Automotive Safety Integrity Level | + | FFI | Freedom From Interference | + | shm | Shared Memory | + | R/O | Read-Only | + | R/W | Read/Write | + | gPTP | generalized Precision Time Protocol | + | mw::log | Middleware Logging (client-side logging API) | + | mw::diag | Middleware Diagnostics | + | mw::time | Middleware Time (time synchronisation) | +end legend @enduml - From a0fdb73daa05148a71aeb3f6f42c739e9fe9f12d Mon Sep 17 00:00:00 2001 From: rmaddikery Date: Tue, 8 Sep 2026 17:45:54 +0200 Subject: [PATCH 4/4] Rename DR-002-dlt-network-transport puml as dlt_plugin --- .../logging/architecture/DR-002-dlt-network-transport.rst | 2 +- .../architecture/_assets/{lsm_plugin.puml => dlt_plugin.puml} | 3 ++- 2 files changed, 3 insertions(+), 2 deletions(-) rename docs/features/logging/architecture/_assets/{lsm_plugin.puml => dlt_plugin.puml} (96%) diff --git a/docs/features/logging/architecture/DR-002-dlt-network-transport.rst b/docs/features/logging/architecture/DR-002-dlt-network-transport.rst index 49bfc606..9cb4669b 100644 --- a/docs/features/logging/architecture/DR-002-dlt-network-transport.rst +++ b/docs/features/logging/architecture/DR-002-dlt-network-transport.rst @@ -98,7 +98,7 @@ DLT Network Transport Evolution applied specifically to the log and trace traffic instance without affecting other traffic. - .. uml:: _assets/lsm_plugin.puml + .. uml:: _assets/dlt_plugin.puml Out of scope / unaffected: diff --git a/docs/features/logging/architecture/_assets/lsm_plugin.puml b/docs/features/logging/architecture/_assets/dlt_plugin.puml similarity index 96% rename from docs/features/logging/architecture/_assets/lsm_plugin.puml rename to docs/features/logging/architecture/_assets/dlt_plugin.puml index ea3c0361..e510f388 100644 --- a/docs/features/logging/architecture/_assets/lsm_plugin.puml +++ b/docs/features/logging/architecture/_assets/dlt_plugin.puml @@ -11,7 +11,8 @@ ' SPDX-License-Identifier: Apache-2.0 ' ******************************************************************************* -@startuml lsm_plugin +@startuml dlt_plugin +title DLT Daemon functionality as a \nPlugin / Loadable Shared Module (a second dedicated Network Stack instance) skinparam componentStyle rectangle skinparam nodesep 40