Repository navigation
Conversation
Add submanifests/tflm.yml defining submanifest entries for TensorFlow Lite Micro (TFLM) and associated neural network / math acceleration projects: - tensorflow/tflite-micro - thesofproject/flatbuffers - thesofproject/nnlib-hifi4 - google/gemmlowp - google/ruy These components provide TFLM kernels and acceleration libraries required by MicroWakeWord (MWW), TensorFlow Classifier (tflmcly), MFCC, and PCAN audio components. Co-developed-by: Liam Girdwood <liam.r.girdwood@linux.intel.com> Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com> Co-developed-by: Naveen Manohar <naveen.m@intel.com> Signed-off-by: Naveen Manohar <naveen.m@intel.com> Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
Replace hardcoded compiler versions and toolchain paths with dynamic toolchain discovery for MicroWakeWord (MWW): - Query target C compiler for libc.a and derive sysroot and include paths dynamically across GCC, Clang, and xt-clang. - Discover target C++ standard library headers under include/c++/*. - Add scripts/mww_shim_flatten.py to generate flattened libc shims for LLEXT linking. - Add -ffunction-sections and -fdata-sections to mww_tflm_lib to ensure literals are placed on a per-function basis in mirrored RAM. - Use XTENSA_CROSS_PREFIX gcc for LLEXT linking when available, allowing Clang Xtensa builds to link LLEXT modules without host ld incompatibility. - Add check=True to subprocess.run in llext_link_helper.py to propagate linker failure exit codes properly. Co-developed-by: Liam Girdwood <liam.r.girdwood@linux.intel.com> Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com> Co-developed-by: Naveen Manohar <naveen.m@intel.com> Signed-off-by: Naveen Manohar <naveen.m@intel.com> Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
The DP lock is taken by the LL thread on every tick in scheduler_dp_ll_tick() and by each DP thread after every run. It was a k_sem, which has no priority inheritance: when a DP thread holding it was preempted by other DP threads, the cooperative high priority LL thread blocked until the holder got CPU time again. With several busy DP modules (3 MFCC + 3 MWW on cAVS 2.5) single LL ticks took up to ~40 ms, overrunning the DMIC DMAs of unrelated streams. With a k_mutex the holder inherits the LL priority: LL max drops from ~1.5M to <30k timer cycles, with no LL overruns or DMA xruns. Co-developed-by: Liam Girdwood <liam.r.girdwood@linux.intel.com> Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com> Signed-off-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com>
CONFIG_ZEPHYR_DP_SCHEDULER can be enabled on cAVS, but only the ACE platform_init() registered the DP scheduler. Creating a DP module on cAVS then failed with "unable to bind scheduler to task (type 3)" and module init returned -ENODEV. Call scheduler_dp_init() after the LL scheduler is set up, as ACE does. Co-developed-by: Liam Girdwood <liam.r.girdwood@linux.intel.com> Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com> Signed-off-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com>
Add NOTIFIER_ID_D0IX_STATE to notify registered DSP components when the host changes the DSP D0 substate via SOF_IPC4_MOD_SET_D0IX. Components such as microWakeWord that keep running while the host sleeps in S0iX / D0i3 can use this event to adjust telemetry reporting or power characteristics. Co-developed-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com> Signed-off-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com> Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Support scheduling and execution for hostless pipelines on IPC4: - In ipc4/handler-user.c, support pipelines that have no host gateway copier (such as DMIC DAI capture feeding ECNS and KPB in hardware listening mode). - Handle COMP_STATE_PREPARE properly in pipeline triggering and state transitions. - Guard against NULL module_data in ipc4_comp_connect(). - In ipc-helper.c, reset component before freeing if not in ready state. Co-developed-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com> Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com> Co-developed-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com> Signed-off-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com> Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Support DP-to-DP component binding and terminal virtual detect sinks: - Allow attaching a single shared secondary ring_buffer on both input and output of a comp_buffer when CONFIG_DP_TO_DP_BIND is enabled. - Pass SIZE_MAX to audio_buffer_sync_secondary_buffer() during ring buffer synchronization. - Skip sinks without audio format (zero frame bytes or rate) during DP period calculation to prevent divide-by-zero for detection components. - In module_adapter_prepare(), set buffer params on DP consumers that leave them unset at bind, and allow virtual detect sinks without downstream consumers. - Declare sys_comp_ecns_init() and sys_comp_wov_arbiter_init() in component.h. Co-developed-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com> Signed-off-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com> Co-developed-by: Liam Girdwood <liam.r.girdwood@linux.intel.com> Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com> Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
Add PCAN (Per-Channel AGC Normalization) fixed-point math library for normalizing mel filterbank energies. PCAN estimates per-channel noise background, applies non-linear wide-dynamic-range AGC compression with fixed-point LUT gain lookup, and delivers scaled features for streaming keyword-spotting networks. Include ztest test suite with MATLAB reference models covering gain lookup, noise estimation, dynamic shrinking, stream processing, and corner cases. Co-developed-by: Liam Girdwood <liam.r.girdwood@linux.intel.com> Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com> Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
Extend MFCC (Mel-Frequency Cepstral Coefficients) audio component with 8-bit Per-Channel AGC Normalization (PCAN) feature extraction: - Add CONFIG_COMP_MFCC_PCAN Kconfig option and CMake configuration. - Implement 8-bit quantized PCAN feature extraction paths alongside standard 16-bit cepstral outputs. - Add Topology2 component definition macros for 40-mel and 80-mel PCAN compressed feature extraction. - Lower high-frequency PCAN tracing to debug level to prevent IPC trace buffer saturation during continuous listening. Co-developed-by: Liam Girdwood <liam.r.girdwood@linux.intel.com> Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com> Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
… fixes Enhance Keyphrase Buffer (KPB) component for multi-slot Wake-on-Voice: - Add multi-client downstream detector triggering, allowing a single buffered history stream to feed up to 3 concurrent keyword detectors. - Support 32-bit sample formats on history buffering and drain streaming. - Fix buffer parameter configuration and state machine race on drain resume. - Fix drain boundary copy calculations when wrapping history buffer pointers. - Allow continuous history circular buffering when the selector sink is full. - Disable synchronous draining mode on IPC4 to support asynchronous pre-roll drain delivery to host. Co-developed-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com> Signed-off-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com> Co-developed-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com> Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com> Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Introduce the Wake-on-Voice (WOV) arbiter component for multi-keyword detection: - Routes between up to 3 concurrent KPB drain client streams and a single host capture PCM copier. - Listens for keyword detection events from downstream MWW/detect components, determines the winning slot, routes the buffered KPB audio to the host, and notifies other detector slots to pause. - Exposes 'wov_active_slot' ALSA enum mixer control and automatically resets it back to 0 upon stream teardown. - Sends PHRASE_DETECTED notifications via AMS once per detection event, rate-limiting IPC traffic during active copy streaming. - Supports mixed sample formats (16-bit and 32-bit). Co-developed-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com> Signed-off-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com> Co-developed-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com> Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com> Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Introduce the Echo Cancellation and Noise Suppression (ECNS) Data
Processing (DP) component:
- Runs in the DP domain with a 20ms period to process 4 DMIC channels
(2 microphones + 2 echo references).
- Provides dual-rate output streams:
- Pin 0: Clean downmixed 16 kHz mono to KPB and keyword detectors with
integrated 100 Hz highpass filter (+20 dB gain).
- Pin 1: Clean 48 kHz stereo to Host Capture.
- Supports both 16-bit and 32-bit sample formats and single-producer fallback.
- Optimizes inner copy loops using audio_stream_frames_without_wrap() to
minimize buffer boundary wrapping overhead.
- Includes CONFIG_ECNS_TEST_SIGNAL Kconfig option to gate the monotonic
counter test signal generator on the KPB mono pin for automated CI
and continuity validation.
Co-developed-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
Co-developed-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com>
Signed-off-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com>
Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Integrate the MicroWakeWord (MWW) audio component with multi-slot keyword detection, D0i3 low-power listening, and PCAN feature extraction: - Implement multi-slot keyword detection architecture running up to 3 concurrent wake-word models (Hey Jarvis, OK Google, Alexa, and fruit evaluation models). - Support 8-bit quantized PCAN feature streams from upstream MFCC. - Use static storage for TFLM op resolver and interpreter instances to eliminate DSP heap fragmentation and memory leaks across stream lifecycles. - Track host and DSP power transitions via D0ix power state notifier to support low-power D0i3 background listening. - Add volatile 'wovdebug_10*' switch kcontrols armed with a 5000 ms countdown timer to trigger synthetic keyword detection during automated hardware S0 and D0i3 validation suites. - Align IPC4 slot identification with reverse DAPM widget instance allocation (2 - (IPC4_INST_ID % 3)). - Provide dataset preparation, training, and mtrace tuning scripts. Co-developed-by: Liam Girdwood <liam.r.girdwood@linux.intel.com> Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com> Co-developed-by: Naveen Manohar <naveen.m@intel.com> Signed-off-by: Naveen Manohar <naveen.m@intel.com> Co-developed-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com> Signed-off-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com> Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
…CAVS 2.5 Enable Wake-on-Voice, ECNS, KPB, MFCC, and MicroWakeWord across Intel Panther Lake (PTL), Wildcat Lake (WCL), and Tiger Lake (CAVS 2.5) board configurations: - Enable CONFIG_COMP_WOV_ARBITER, CONFIG_COMP_KPB, CONFIG_COMP_ECNS, CONFIG_COMP_MFCC, CONFIG_COMP_MFCC_PCAN, CONFIG_COMP_MWW, and CONFIG_ECNS_TEST_SIGNAL=y. - Increase Zephyr heap size to 512KB (0x80000) on WCL and PTL to support 32-bit KPB circular history buffering and multi-slot detector memory. - Include ecns.toml, wov_arbiter.toml, and mww.toml module descriptors in ptl.toml.h, wcl.toml.h, and tgl.toml.h rimage configurations. - Add app/wov-d0i3-fake-wake.conf test overlay. Co-developed-by: Naveen Manohar <naveen.m@intel.com> Signed-off-by: Naveen Manohar <naveen.m@intel.com> Co-developed-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com> Signed-off-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com> Co-developed-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com> Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com> Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Provide Topology2 component definition macros and widget structures for Echo Cancellation and Noise Suppression (ECNS) and Wake-on-Voice (WOV) Arbiter: - ecns.conf: ECNS processing component definition with dual output pins. - wov-arbiter.conf: Wake-on-Voice arbiter widget definition and active slot kcontrol enum mapping. - common_definitions.conf: Define default KPB_BUFF_TIME_MS. Co-developed-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com> Signed-off-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com> Co-developed-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com> Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com> Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Provide Topology2 component definition macros and widget structures for MicroWakeWord (MWW) neural network detection pipelines: - mww.conf: MicroWakeWord widget definition with dynamic input pin binding and volatile trigger controls. - mww/score_enum.conf: Detection probability score reporting enum definitions. Co-developed-by: Liam Girdwood <liam.r.girdwood@linux.intel.com> Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com> Co-developed-by: Naveen Manohar <naveen.m@intel.com> Signed-off-by: Naveen Manohar <naveen.m@intel.com> Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
…multi-slot WoV Implement the dual-rate DMIC topology architecture with multi-slot Wake-on-Voice: - Provide dual PDM DAI endpoints: - dmic01 on PDM0: 48 kHz stereo 16-bit capture. - dmic16k on PDM1: 16 kHz stereo 16-bit detection. - Feed ECNS directly from the DMIC DAI copiers, eliminating redundant mixins and mixouts between DMIC, ECNS, and KPB. - Connect ECNS output pin 0 (16k mono clean with internal 100 Hz HPF) to KPB and multi-slot MWW detectors (slots 1, 2, 3), feeding WoV Arbiter and Host Copier PCM 11 / PCM 12. - Connect ECNS output pin 1 (48k stereo clean) to Host Copier PCM 11. - Replace WoV mixouts with copiers and run ECNS, MFCC, and MWW in the DP domain. - Add multi-slot WoV manifests for Panther Lake (PTL), Wildcat Lake (WCL), and generic CAVS platforms. Co-developed-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com> Signed-off-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com> Co-developed-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com> Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com> Co-developed-by: Naveen Manohar <naveen.m@intel.com> Signed-off-by: Naveen Manohar <naveen.m@intel.com> Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Integrate multi-slot Wake-on-Voice pipelines and feature inclusion into SoundWire and HDA topologies: - Wire multi-slot WoV features into cavs-sdw.conf, cavs-rt5682.conf, sdw-dmic-generic.conf, and sof-hda-generic.conf. - Add sof-hda-generic-wov-manifest.conf manifest. - Add CMake topology build targets for Panther Lake (PTL) ACE 3.0, Wildcat Lake (WCL), and Tiger Lake (TGL) with multi-slot WoV SoundWire and HDA targets in production CMake files. Co-developed-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com> Signed-off-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com> Co-developed-by: Naveen Manohar <naveen.m@intel.com> Signed-off-by: Naveen Manohar <naveen.m@intel.com> Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
…PTL, TGL, WCL Provide prebuilt topology binary files for rapid bring-up and hardware verification: - build/sof-ptl-dmic-wov-multi-4ch.tplg - build/sof-tgl-dmic-wov-multi-4ch.tplg - build/sof-wcl-dmic-wov-multi-4ch.tplg Co-developed-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com> Signed-off-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com> Co-developed-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com> Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com> Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
…lidation suite Add hardware test utilities for validating multi-slot Wake-on-Voice capture and D0i3 power transitions: - wov_blocking_read.c: ALSA capture utility configuring software avail_min to ensure prompt poll() wakeups on no-period-wakeup streams. - wov_blocking_read_tinyalsa.c: TinyALSA variant with period wakeup handling. - run_wov_10x.py: Automated 20-run hardware test suite (10x S0 and 10x D0i3 idle delay wake tests) with mathematical sample continuity verification. - test_wov_no_irq.py, wov_capture_app.c, wov_daemon.py, wov_multi_slot_test.py. Co-developed-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com> Signed-off-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com> Co-developed-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com> Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com> Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
… Lake Add Panther Lake ACE 3.0 Intel OTC debug key modulus and exponent mapping in sof_ri_info.py to support manifest and signature decoding on PTL binaries. Co-developed-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com> Signed-off-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com> Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
…i3 verification Comprehensive documentation updates for multi-slot Wake-on-Voice and dual-rate DMIC: - tools/topology/topology2/README.md: - Document complete pipeline architecture and DMIC routing diagrams. - Document dual-rate DMIC PDM DAI structure (dmic01 @ 48k, dmic16k @ 16k). - Document ECNS 20ms DP processing and pin mappings. - Document KPB circular buffering, pre-roll drain, and WoV Arbiter slot routing. - Document multi-slot WoV detector pipeline (Hey Jarvis, OK Google, Alexa). - Document PCM endpoints: PCM 11 (48k stereo clean), PCM 12 (16k mono WoV). - Add build steps with alsatplg and verification runbooks. - doc/developer_guides/wov_ecns_integration_guide.md: - Integration guide for custom WoV and ECNS algorithms. - scripts/host-testbench.sh: - Unset GTK_PATH to avoid library conflict. Co-developed-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com> Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com> Co-developed-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com> Signed-off-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com> Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
…ty fixes Driver hardening and test sample support for Wake-on-Voice: - In copier_dai.c, dai-zephyr.c, dai.c, and dma.c: add NULL pointer guards during tear-down and free paths, preventing kernel panics on aborted pipelines. - In detect_test.c: support multi-slot keyword detection testing with DP thread batching. - Add src/include/sof/audio/ecns.h component header. - In agent.h: guard against NULL system agent in sa_set_panic_on_delay(). - In host-zephyr.c: throttle no-bytes warning logs. - In intel_adsp_cavs25_tgph.conf: enable multi-channel WoV and PCAN. - Update top-level README.md and drop obsolete tensorflow-clone.sh. Co-developed-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com> Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com> Co-developed-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com> Signed-off-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com> Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Export essential C++ symbols and configure DP-to-DP binding: - In src/lib/cpp_new_export.cpp: export operator new, operator delete, __cxa_pure_virtual, and __assert_no_args so LLEXT modules linking C++ code (such as TensorFlow Lite Micro) can resolve runtime support symbols against the base firmware image. - In zephyr/Kconfig: define CONFIG_DP_TO_DP_BIND to gate binding between two Data Processing (DP) scheduled components. - In zephyr/edf_schedule.c: cancel delayed work on task free. - In ipc4/handler-kernel.c: synchronize secondary core boot under userspace. Co-developed-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com> Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com> Co-developed-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com> Signed-off-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com> Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Support running multiple Micro Wake Word models to detect multiple wake key words with a pseudo ECNS/AEC frontend.
Still needs some integration updates so draft. Will update PR message when ready.
@ujfalusi @singalsu @naveen-manohar pls check, I have cleaned up and squash/rebased development tree into upstreamable patches. Pls make sure I have attributed your efforts correctly for each patch, if not I can easily fix before this is non draft.