# ESP-SR Speech Recognition Framework
Espressif [ESP-SR](https://github.com/espressif/esp-sr) helps users build AI speech solutions based on ESP32 or ESP32-S3 chips.
ESP-SR framework includes the following modules:
* [Audio Front-end AFE](https://docs.espressif.com/projects/esp-sr/en/latest/esp32s3/audio_front_end/README.html)
* [Wake Word Engine WakeNet](https://docs.espressif.com/projects/esp-sr/en/latest/esp32s3/wake_word_engine/README.html)
* [Speech Command Word Recognition MultiNet](https://docs.espressif.com/projects/esp-sr/en/latest/esp32s3/speech_command_recognition/README.html)
* [Speech Synthesis](https://docs.espressif.com/projects/esp-sr/en/latest/esp32s3/speech_synthesis/readme.html)
These algorithms are provided in the form of a component, so they can be integrated into your projects with minimum effort.
ESP32-S3 is recommended, which supports AI instructions and larger, high-speed octal SPI PSRAM.
The new algorithms will no longer support ESP32 chips.
## Wake Word Engine
Espressif wake word engine **WakeNet** is specially designed to provide a high performance and low memory footprint wake word detection algorithm for users, which enables devices always listen to wake words, such as “Alexa”, “Hi,lexin” and “Hi,ESP”.
Currently, Espressif has not only provided an official wake word "Hi,Lexin","Hi,ESP" to the public for free, but also allows customized wake words. For details on how to customize your own wake words, please see [Espressif Speech Wake Words Customization Process](https://docs.espressif.com/projects/esp-sr/en/latest/esp32s3/wake_word_engine/ESP_Wake_Words_Customization.html).
The following wake words are supported in esp-sr:
|wake words | ESP32 | ESP32-S3 |
|:--------------- | :------------------------------------:| :---------------------:|
|Hi,乐鑫 | wn5_hilexin, wn5_hilexinX3 | wn9_hilexin |
|你好小智 | wn5_nihaoxiaozhi,wn5_nihaoxiaozhiX3 | wn9_nihaoxiaozhi |
|Hi,ESP | | wn9_hiesp |
|Hi,M Five | | wn9_himfive |
|Alexa | | wn9_alexa |
|小爱同学 | | wn9_xiaoaitongxue |
|Jarvis | | wn9_jarvis_tts |
|Computer | | wn9_computer_tts |
*NOTE:* `_tts` suffix means this WakeNet model is trained by TTS samples.
## Speech Command Recognition
Espressif's speech command recognition model **MultiNet** is specially designed to provide a flexible off-line speech command recognition model. With this model, you can easily add your own speech commands, eliminating the need to train model again.
Currently, Espressif **MultiNet** supports up to 300 Chinese or English speech commands, such as “打开空调” (Turn on the air conditioner) and “打开卧室灯” (Turn on the bedroom light).
## Audio Front End
Espressif Audio Front-End **AFE** integrates AEC (Acoustic Echo Cancellation), VAD (Voice Activity Detection), BSS (Blind Source Separation) and NS (Noise Suppression).
Our two-mic Audio Front-End (AFE) have been qualified as a “Software Audio Front-End Solution” for [Amazon Alexa Built-in devices](https://developer.amazon.com/en-US/alexa/solution-providers/dev-kits#software-audio-front-end-dev-kits).
**In order to achieve optimal performance:**
* Please refer to software design [esp-skainet](https://github.com/espressif/esp-skainet).
# Change log for esp-sr
## Known issues:
- Available storage is less than the remaining flash space on IDF v5.0.
If you can not map model partition successfully, please check the left free storage by `spi_flash_mmap_get_free_pages(ESP_PARTITION_MMAP_DATA)` or update IDF to v5.1 or later.
- Add Chinese MultiNet7 models
- Add first Noise Suppression model: nsnet1
- Add wakenet model trained by TTS samples: Jarvis and Computer
- fix some bugs
- Reduce Internal RAM of multinet7
- Update benchmark
- Add ci build test for esp32
- Fix some bugs
- Add esp32c6 tts lib
- Return the volume of wake word audio when one wake word is detected
- Reduce MultiNet6 SRAM size from 48KB to 32 KB
- Add "Hi M Five" wake word model from M5Stack
- Remove all MultiNet4 models
- Update MultiNet7_en v2.0
- Fix some bugs in AFE
- Add Flite G2P module
- Reset timeout trigger of multinet6 when a new speech command is detected
- Allocate all beams from PSRAM
- Fix the wrong APIs of multinet2 when using ESP32 chip
- VAD can work fine when WakeNet is disabled
- Add ci tests to check the APIs of wakenet, multinet and AFE work fine
- Support to load and run two wakenet9 models at the same time in AFE
- Reduce the latency of multinet6
- Fix the bug of multinet5q8 whrn adding new speech commands
- Fix the crash of multinet5q8 when speech commands are triggered
- Update esp_mn_commands_update() in esp_process_sdkconfig.c
- Output text of ctc greedy search result when no command can be detected.
- Modify the default Chinese commands
- Remove __pycache__ in esp-sr/model folder
- Bugfix: remove all cxx11:string
- Bugfix: remove esp-partition for esp32s2 & esp32c3 on idf v4.4
- Update multinet API to add/modify/check new commands in the code
- Update documents to introduce how to use multinet API
- Update the partition APIs to keep compatible with both IDF v4.4 and IDF v5.0
- Add a new Chinese MultiNet6 mdoel for air conditioner controller
- Fix bugs in model loader
- Read all parameters sequentially, which reduces about 5x in model loading time.
- Use esp_partition_mmap to replace spiffs file system, which further reduces about 3x in model loading time
- Add WakeNet API unity test
- Add MultiNet API unity test
- ESP-DSP dependency is now installed from the component registry
- Add an English MultiNet6 model which is trained by RNNT and CTC
- Add a Chinese MultiNet6 model which is trained by RNNT and CTC
- Fixed CMake errors when esp-sr was installed from component registry
- Fixed the list of supported chips displayed in the component registry
- Support esp32c3 for Chinese TTS
- Update document of ESP-SR
- Add ESP-SR into Espressif component manager
- Add wakenet8 & wakenet9
- Add multinet5 to support English and Chinese speech command recognition
- Remove wakenet7
- Add AFE pipeline for voice communication
- support ESP32S3 chip
- add wakenet7 & update wakenet5 to support multi-channel detection
- remove wakenet6
- add AFE pipeline for speech recognition
- add chinese tts
- update noise suppression v2
- update AEC v3
- update multinet_cn_1.4 and add CONTINUOUS RECOGNITION mode
- improve hilexin wakeNet5X3 model(v5)
- support IDFv4.0 build system
- replace MAP algorithm with MASE(Mic Array Speech Enhancement) algorithm v1.0
- add multinet1 English model v1.0
- update multinet1 Chinese model v2.0
- add Mic Array Processing(MAP) algorithm
- Fix the bug of parsing speech command
- fix the bug of decoder
- add wakenet6
- support cmake
- add free wake word: hi jeson
- update wakenet5X3 wake word model(v2)
- add acoustic algorithm, include AEC, AGC, VAD ,NS
- add wakenet5X2 and wakenet5X3
- Initial commit, include wakenet4,wakenet5 and multinet1_cni