audio: microwakeword: add mww component - #11135
Draft
singalsu wants to merge 23 commits into
Draft
Conversation
Previously binding two DP (Data Processing) scheduled components was rejected with IPC4_INVALID_REQUEST. This patch adds support for DP-to-DP binding by creating a dual ring buffer configuration where each DP module gets its own ring buffer on either side of the intermediate comp_buffer. Data flow for DP-to-DP: src_DP -> ring_buf_src -> comp_buffer -> ring_buf_sink -> sink_DP Changes in helper.c: - Remove the DP-to-DP bind rejection in ipc_comp_connect(). - Add src_is_dp, sink_is_dp, and dp_to_dp flags to detect the DP-to-DP case. - Create a second ring_buffer allocated from the source module's mod_alloc_ctx for the source side of the comp_buffer. - Refcount the DP vregion for each created ring_buffer via vregion_get(), with a NULL alloc guard. Changes in audio_buffer.c: - Change audio_buffer_attach_secondary_buffer() from a global rejection to per-side checks, allowing both secondary_buffer_sink and secondary_buffer_source to be set simultaneously. - Add a dual-secondary sync path in audio_buffer_sync_secondary_buffer() that cascades data through: input ring_buffer -> comp_buffer -> output ring_buffer, with rate-limiting applied on the output side. Changes in ring_buffer.c: - Release the DP vregion in ring_buffer_free() via vregion_put() and free the mod_alloc_ctx when the refcount reaches zero, matching the pattern used in comp_buffer_free(). Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
When CONFIG_LLEXT_TYPE_ELF_RELOCATABLE is active, bypass appending static address flags (-Ttext, --section-start, -Tdata) in the linker helper script. This keeps section base addresses at 0. Also adjust the offset calculator to avoid integer parsing errors when all section addresses are set to 0. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
…table modules Implement page-level virtual memory mapping using Zephyr's sys_bitarray utility over the library region. Compile section layout at load-time to allocate virtual addresses and rewrite section sh_addr headers in-place. This enables Zephyr LLEXT to naturally relocate references. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Enable CONFIG_LLEXT_EXPORT_BUILTINS_BY_SLID=y in llext_relocatable.conf to link relocatable LLEXT modules against build-time function signature hashing, providing load-time ABI mismatch protection. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
LLEXT modules linking C++ code (e.g. TensorFlow Lite Micro) can end up with undefined references to global operator new/delete and __cxa_pure_virtual even when built with -fno-exceptions -- some support code (e.g. TFLM's arena allocators) still emits calls to the sized deallocation form in generated destructors, and any TU referencing an abstract class's vtable needs __cxa_pure_virtual resolvable for its pure-virtual slots. zephyr/lib/cpp/minimal/cpp_new.cpp and cpp_virtual.c already define these, but neither is referenced anywhere in the base image build, so their definitions are never pulled into the link or exported. Add thin wrappers in src/lib/cpp_new_export.cpp and export them under the mangled names so LLEXT modules can resolve against the base image.
Without this flag, calls between GLOBAL-visibility functions defined in different translation units of the SAME llext module are emitted as PLT calls. Zephyr's llext_link_plt() only resolves PLT symbols against the base image's export table, this module's own .exported_sym table, or other already-loaded extensions -- never against symbols merely defined locally in this module's own .dynsym. Multi-TU C++ libraries (e.g. TensorFlow Lite Micro) call plenty of non-exported internal helpers across .cc files, so without this flag those calls fail to link at load time.
Remove the is_relocatable branch that skipped -Wl,-Ttext/--section-start placement and the custom llext_merge.ld section-merge script for CONFIG_LLEXT_TYPE_ELF_RELOCATABLE builds, and remove the clang --target=/--ld-path= hoisting logic. All LLEXT builds now go through the same fixed-section-address layout path unconditionally, and the strip-debug/remove-section objcopy flags applied afterward are dropped as well.
…ted-symbol segment llext_manager_layout_sections() rebases the addresses of recognized sections in place, directly in the raw ELF buffer, before llext_load() ever parses the file. Three problems with the pre-existing rebase and layout logic, all fixed together here: 1. Relocation/symbol staleness after rebase: two classes of data in the file still referenced the OLD (pre-rebase) addresses after this mutation. .rela.dyn/.rela.plt r_offset fields are byte-for-byte copies from the build-time ELF, so llext_link_plt()'s llext_file_offset() lookup failed (Offset not found) for any relocation whose target section moved. R_XTENSA_RELATIVE relocations are treated as a complete no-op by Zephyr's llext core whenever pre_located is set, under the assumption that the stored pointer value is already correct -- SOF's rebase step violates that assumption, so the delta has to be applied to the pointer value at each R_XTENSA_RELATIVE target directly. .symtab/.dynsym st_value fields are taken as final absolute addresses, unmodified, under the same pre_located assumption, feeding both llext_find_sym() lookups and exported symbol addresses. Track (old_addr, size, delta, shdr index) for every section actually rebased, then walk the ELF again applying all three fixups via the new llext_manager_fixup_rela() and llext_manager_fixup_symtab(). 2. Layout packing: .bss must remain immediately contiguous with writable DATA (they share a single VMA mapping, since .bss has no file backing and just extends DATA's mapped-and-zeroed tail), so it is now aliased into LLEXT_MEM_DATA for layout purposes instead of being treated as its own region. A region whose sections are split apart by an interleaving section of a different region (e.g. .exported_sym appearing between two .data-region members) now reuses that region's already-established address delta when it reappears, instead of being re-packed into the forward layout cursor as if it were a fresh region -- which previously inserted a spurious page-aligned gap and could desync .bss from the data region it must stay contiguous with. 3. .exported_sym is now tracked as its own real segment (LIB_MANAGER_EXPORT, new enum value), with its own copy-from-storage pass on load/unload -- some toolchains (GNU ld, unlike the Clang LLEXT overlay which merges it into .rodata) keep it as a distinct allocatable section. It also needs eager, synchronous population from the raw ELF buffer before llext_load() runs (llext_manager_load_sections_early()), because llext_load()'s llext_export_symbols() reads .exported_sym content itself during load, well before SOF's own on-demand per-module copy path (llext_manager_load_module()) would otherwise populate it. Also fixes llext_manager_add_library() to index module manifests by module_id + ctx->mod[i].start_idx instead of module_id + i, for correct indexing with multi-module libraries.
Export ams_send(), ams_helper_register_producer(), ams_helper_unregister_producer(), and ams_helper_prepare_payload() so an LLEXT module can act as an AMS message producer (e.g. a keyword-spotting component signaling KPB directly) without needing these calls statically linked into the base image.
Consume the lib_uuids dict already populated earlier in the script: when a library's final .bin file did not yet exist at UUID-collection time, its UUIDs were deferred into lib_uuids instead of being symlinked immediately. install_lib() now walks lib_uuids[key] and creates the deferred <uuid>.bin symlink/copy once the library is actually installed.
Update tensorflow-clone.sh to ensure all required TFLM dependency repositories are checked out to their specific target commits: - Add explicit checkout_commit() to switch to the required commit ID after cloning or fetching. - Make BASE_DIR default dynamically to the parent directory containing the SOF workspace, while supporting an optional directory argument. - Check current HEAD before checkout to ensure idempotent runs. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@gmail.com>
Call scheduler_dp_init() in platform_init() on cAVS platforms when CONFIG_ZEPHYR_DP_SCHEDULER is enabled so that DP tasks (such as MFCC and MWW in the Data Processing domain) can bind to the DP scheduler without failing with -ENODEV (-19). Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
Add microWakeWord (MWW) processing module for low-power keyword spotting: - Implement module adapter in mww.c with soft mel-log AGC, VAD gating, and KPB wake-on-voice notification. - Implement TFLM bridge in mww_model.cc with MixConv operator resolver, support for int8 dequantization, and streaming resource variable resets. - Add robust circular ring buffer boundary unwrap handling for MFCC hops. - Add 2-step debounce verification before triggering KPB drain. - Add LLEXT wrapper and build system integration for both static and dynamic module targets. - Include initial placeholder model interface in mww_model_data.h/cc. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com> Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
Document the mww component's architecture and data flow (modeled on src/audio/tensorflow/README.md), the LLEXT fixes required to build and load it as a real module on aphid, and build/deploy/test instructions. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com> Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
Update the embedded hey_jarvis wake-word model data array with the newly retrained causal streaming MixConv model. Streaming verification report (threshold 0.85): ================================================================= microWakeWord Streaming Verification Report (Threshold: 0.85) ================================================================= Class Role Files Detected Rate Mean Peak ------------------------------------------------------------- silence Negative 500 0 0.0% 0.001 unknown Negative 1500 33 2.2% 0.062 hey_jarvis Positive 3000 2999 100.0% 0.996 ------------------------------------------------------------- Overall Wake Word Recall (True Positive Rate) : 99.97% (2999/3000) Overall False Alarm Rate (False Positive Rate): 1.65% (33/2000) Precision : 98.91% ================================================================= Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
…hain Add an end-to-end offline training, quantization, and verification toolchain for microWakeWord streaming models using SOF host testbench MFCC features: - sof_mfcc_extract_features.sh: Batch-extract real SOF 40-bin mel spectrogram features from WAV datasets via sof-testbench4. - sof_mww_generate_keyword_dataset_piper_tts.sh: Synthesize keyword utterances across multiple Piper ONNX neural voices. - sof_mww_prepare_silence_unknown.sh: Prepare ambient background and non-target speech datasets. - sof_mww_dataset.py: Dataset loader with temporal jitter, silence pool background mixing, and hard negative fragment synthesis. - sof_mww_train.py: Causal streaming MixConv model training with quantization-aware calibration and C-array / topology export. - sof_mww_verify.py: Streaming temporal verification with configurable threshold and consecutive-detection debounce. - sof_mww_train_pipeline.sh: One-shot automation pipeline for training, export, and streaming verification. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
Add common ABI manifest inclusion and update KPB component configuration with standard IPC4 UUID definitions. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com> Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
…tend Add 40-bin mel filterbank configurations (mel40, mel40_compress, mel40_10ms, and mel40_10ms_compress) to setup_mfcc.m and generate the corresponding topology blobs for microWakeWord streaming frontends. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
Add build of testbench topologies to test the 40-bin 10ms-hop Mel spectrogram profile of the MFCC component (mel40_10ms.conf). The new testbench topologies are named: sof-hda-benchmark-mfccmel40_10ms16.tplg sof-hda-benchmark-mfccmel40_10ms24.tplg sof-hda-benchmark-mfccmel40_10ms32.tplg Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
…ines Add Topology2 configuration and pipeline graphs for microWakeWord (MWW) Wake-on-Voice with KPB: - Define MWW module component in mww.conf with default control definitions. - Add HDA Mic capture pipeline (host-gateway-src-mfcc-mww-capture.conf) and host-gateway-micsel-mfcc-mww-capture.conf. - Add SoundWire jack MWW branches and DMIC MFCC profiles. - Add sof-hda-mww.conf topology target for HDA and SoundWire platforms. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
Include audio/microwakeword/mww.toml when CONFIG_COMP_MWW is enabled across platform rimage manifest headers (tgl, tgl-h, mtl, lnl, ptl, wcl) so the MWW module UUID and entry are registered in base firmware images and loadable on target devices. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
Enable CONFIG_COMP_MWW=y, C++ runtime dependencies, and EDF stack sizing in intel_adsp_cavs25.conf so that microWakeWord is statically built into the firmware binary for cAVS 2.5 (Tiger Lake). Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
…atforms Enable CONFIG_COMP_MWW=m as an LLEXT loadable extension on Intel ADSP ACE 1.5 (Meteor Lake MTPM) and ACE 3.0 (Panther Lake PTL) targets: - Sizing LLEXT metadata heap for TFLM weak symbols. - Adjusting virtual heap and memory pool sizes for C++ TFLM arenas. - Enabling C++ runtime support for MWW dependencies. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.