[![Component Registry](https://components.espressif.com/components/espressif/usb_stream/badge.svg)](https://components.espressif.com/components/espressif/usb_stream) ## USB Stream Component `usb_stream` is an USB `UVC` + `UAC` host driver for ESP32-S2/ESP32-S3, which supports read/write/control multimedia streaming from usb device. For example, at most one UVC + one Microphone + one Speaker streaming can be supported at the same time. Features: 1. Support video stream through UVC Stream interface, support both isochronous and bulk mode 2. Support microphone stream and speaker stream through the UAC Stream interface 3. Support volume, mute and other features control through the UAC Control interface 4. Support stream separately suspend and resume ### USB Stream User Guide Please refer: https://docs.espressif.com/projects/espressif-esp-iot-solution/en/latest/usb/usb_stream.html ### Add component to your project Please use the component manager command `add-dependency` to add the `usb_stream` to your project's dependency, during the `CMake` step the component will be downloaded automatically ``` idf.py add-dependency "espressif/usb_stream=*" ``` ### Examples Please use the component manager command `create-project-from-example` to create the project from example template * USB Camera WIFI picture transfer ``` idf.py create-project-from-example "espressif/usb_stream=*:usb_camera_mic_spk" ``` * USB Camera local display ``` idf.py create-project-from-example "espressif/usb_stream=*:usb_camera_lcd_display" ``` Then the example will be downloaded in current folder, you can check into it for build and flash. > Or you can download examples from esp-iot-solution repository: [USB Camera + Audio stream](https://github.com/espressif/esp-iot-solution/tree/master/examples/usb/host/usb_camera_mic_spk), [USB Camera LCD Display](https://github.com/espressif/esp-iot-solution/tree/master/examples/usb/host/usb_camera_lcd_display). ### Q&A Q1. I encountered the following problems when using the package manager ``` Executing action: create-project-from-example CMakeLists.txt not found in project directory /home/username ``` A1. This is because an older version package manager was used, please run `pip install -U idf-component-manager` in ESP-IDF environment to update.
# ChangeLog ## v1.0.0 - 2023-05-10 ### Release v1.0.0 ### Enhancements: * USB: * Refactor USB state machine, handle error and hot-plug more robustly. * Support user submit control transfer at any time (Better to support more control requests in future). * Support self-define initial state through `flags`, eg `FLAG_UAC_SPK_SUSPEND_AFTER_START` to make speaker suspend as startup state. * Support Stream Quick Start Mode (Skip the enum process to save at least 50 ms). * UVC: * Support get camera resolution list and current resolution. * Support dynamically config UVC frame rate and resolution. * UAC: * Support config UAC `*_ch_num`, `*_samples_frequence`, `*_bit_resolution` to `*_ANY` to adapter any available device. * Support get audio frame rate and resolution list. * Support dynamically config audio channel number, rate, and bits. * Test: * Add hardware test for usb_stream component. ### Bug Fixes: * Fix some hot-plug issues * Fix UAC control interface can not be 0 * Fix UAC stereo headset volume control failed ### Breaking Changes: * UAC: * Remove the configuration `mic_min_bytes` from `uac_config_t`, use KConfig option `CONFIG_UAC_MIC_CB_MIN_MS_DEFAULT` instead. The default value is 20 ms, which means the mic callback will be called every 20 ms. * Add the configuration `flags` to `uac_config_t` and `uvc_config_t`, users can use it to define the initial state of the device. The default value is 0. * Add the configuration `mic_ch_num` and `spk_ch_num` to `uac_config_t`, users can specify the number of the microphone and speaker channels. The default value is 0, which means support any number of channels. ## v0.4.13 - 2023-04-13 ### Enhancements * test_apps: * Update test_apps project ## v0.4.12 - 2023-04-12 ### Bug Fixes: * Mic: * Fix mic_min_bytes may exceed the limits ## v0.4.11 - 2023-03-22 ### Bug Fixes: * Mic: * Fix adaptive endpoint occasional assert "rem_len<=transfer->num_bytes" to crash ### Enhancements: * Mic: * Padding last frame data if packet size not as expected ## v0.4.10 - 2023-03-16 ### Enhancements: * Add example: usb_camera_lcd_display ## v0.4.9 - 2023-03-09 ### Enhancements: * Use cu_pkg_define_version to define the version of this component. ## v0.4.8 - 2023-03-02 ### Enhancements: * Add user guide for component manager ## v0.4.7 - 2023-02-20 ### Bug Fixes: * Camera: * Fix incorrect isoc buffer offset when transfer error occurs ## v0.4.6 - 2023-02-17 ### Enhancements: * Camera: * Add additional JPEG header check for bulk transfer # ChangeLog ## v0.4.5 - 2023-02-14 ### Bug Fixes: * Remove unnecessary header files "hal/usbh_ll.h" "hal/usb_hal.h" ### Enhancements: * Camera: * Support camera with length of 2, 6 bytes MJPEG UVC header * Support bulk camera with EOF bit set in each payload ## v0.4.4 - 2023-02-09 ### Enhancements: * Remove all assert, replace with error handler ### Bug Fixes: * Fix error message during disconnect ## v0.4.3 - 2023-02-07 ### Enhancements: * Remove some unnecessary vTaskDelay * Camera: * Support for bulk camera with suspend/resume control * Support force uvc format index if not found * Add support for bulk camera with MPS(Max Packet Size) equal to 512 bytes ### Bug Fixes: * Speaker: * fix some speaker resume control not work (should set freq after resume) ## v0.4.2 - 2023-01-30 ### Enhancements: * Camera: * Disable BULK_BYTES_PER_URB_SAME_AS_FRAME by default, support reassemble bulk package segment (will save one image frame size RAM) * Add config UVC_DROP_NO_EOF_FRAME, if disable(default) the frame with no EOF will be reserved (MJPEG may broken) * Add config UVC_DROP_OVERFLOW_FRAME, if enable(default) the overflow frame will be dropped ### Bug Fixes: * Camera * fix transfer size in GET_CUR Probe stage ## v0.4.1 - 2023-01-05 ### Enhancements: * Add IRAM_ATTR to packet parsing functions * Speaker: * padding zero packet if speaker buffer empty ## v0.4.0 - 2022-12-22 ### Enhancements: * Support parsing descriptors to check the device functions and capabilities * Self-adaption appropriate interfaces based on descriptors and user's configurations * Camera: * Support UVC config to any resolution, the first frame index will be used ### API Changes: * add `optional` label to some members in `uvc_config_t` and `uac_config_t`, users can optionally set a value. The self-adaption process will first use actual configs from descriptors, if no appropriate interface found, will use the params from user's configuration ### Bug Fixes: * USB: * Fix STALL (appear on some devices) during get short configuration descriptor * Resize endpoint 0 MPS based on device descriptor ## v0.3.3 - 2022-12-22 ### Enhancements: * print version message ### Bug Fixes: * Camera: * Fix not parse header only frames ## v0.3.2 - 2022-12-16 ### Enhancements: * Mic: * Padding last frame data when receive zero length packet ## v0.3.1 - 2022-12-8 ### Enhancements: * add development documentation https://docs.espressif.com/projects/espressif-esp-iot-solution/zh_CN/latest/usb/usb_stream.html ## v0.3.0 - 2022-12-05 ### Bug Fixes: * USB: * Fix memory leak during hot-plug
## USB Stream Lcd Display Example This routine demonstrates how to use the [usb_stream](https://components.espressif.com/components/espressif/usb_stream) component to acquire a USB camera image and display it adaptively on the RGB screen. * For better performance, please use ESP-IDF release/v5.0 or above versions. ## Hardware * An [ESP32-S3-LCD-EV-Board](https://docs.espressif.com/projects/espressif-esp-dev-kits/en/latest/esp32s3/esp32-s3-lcd-ev-board/index.html) development board with 800x480/480*480 LCD. * A USB camera (Can be connected to USB port or Expansion Connector) * An USB Type-C cable for Power supply and programming (Please connect to UART port instead of USB port) Note: Error message with `A fatal error occurred: Could not open /dev/ttyACM0, the port doesn't exist`: Please first make sure development board connected, then make board into "Download Boot" mode to upload by following steps: 1. keep pressing "BOOT(SW2)" button 2. short press "RST(SW1)" button 3. release "BOOT(SW2)". 4. upload program and reset ## IDF Patch The patch is intended to achieve best performance of RGB LCD by using **Octal PSRAM 120MHz** feature. Now there are two versions of patch under the patch folder that are used for the respective branch of ESP-IDF (master or release/v5.0). **Please make sure your IDF project is clean** (use `git status` to check), then the patch can be applied by following commands: ``` cd <root directory of IDF> git apply <path of the patch>/idf_psram_120m.patch # Nothing return if success git status # Check whether the operation is successful, the output should look like below: HEAD detached at f315986401 Changes not staged for commit: (use "git add <file>..." to update what will be committed) (use "git restore <file>..." to discard changes in working directory) modified: components/esp_hw_support/port/esp32s3/rtc_clk.c modified: components/esp_psram/esp32s3/Kconfig.spiram modified: components/hal/esp32s3/include/hal/spimem_flash_ll.h modified: components/spi_flash/esp32s3/mspi_timing_tuning_configs.h modified: components/spi_flash/esp32s3/spi_timing_config.c modified: components/spi_flash/esp32s3/spi_timing_config.h modified: components/spi_flash/spi_flash_timing_tuning.c Untracked files: (use "git add <file>..." to include in what will be committed) tools/test_apps/system/flash_psram/sdkconfig.ci.f8r8_120ddr tools/test_apps/system/flash_psram/sdkconfig.ci.f8r8_120ddr_120ddr ``` ### How To Use 1. First delete existing `build`, `sdkconfig` , `sdkconfig.old` ``` rm -rf build sdkconfig sdkconfig.old ``` 2. Use the command line to enable the relevant configuration ``` idf.py -D SDKCONFIG_DEFAULTS="sdkconfig.defaults;sdkconfig.psram_octal_120m" set-target esp32s3 ``` 3. Compile and burn ``` idf.py build flash monitor ```
## USB Stream Example This example demonstrates how to use `usb_stream` component with an ESP device. Example does the following steps: 1. Config a UVC function with specified frame resolution and frame rate, register frame callback 2. Config a UAC function with one microphone and one speaker stream, register mic frame callback 3. Start the USB streaming 4. In image frame callback, if `ENABLE_UVC_WIFI_XFER` is set to `1`, the real-time image can be fetched through ESP32Sx's Wi-Fi softAP (ssid: ESP32S3-UVC, http: 192.168.4.1), else will just print the image message 5. In mic callback, if `ENABLE_UAC_MIC_SPK_LOOPBACK` is set to `1`, the mic data will be write back to usb speaker, else will just print mic data message 6. For speaker, if `ENABLE_UAC_MIC_SPK_LOOPBACK` is set to `0`, the default sound will be played back ## Hardware * Development board 1. Any `ESP32-S2`,` ESP32-S3` board with USB Host port can be used. 2. Please note that the `esp32-sx-devkitC` board can not output 5V through USB port, If the OTG conversion cable is used directly, the device cannot be powered. 3. For `esp32s3-usb-otg` board, please enable the USB Host power domain to power the device * Connection ||USB_DP|USB_DM| |--|--|--| |ESP32-S2/S3|GPIO20|GPIO19| * Camera module * please refer the README of `usb_stream` for the Camera requirement ## Build and Flash Build the project and flash it to the board, then run the monitor tool to view the serial output: * Run `. ./export.sh` to set IDF environment * Run `idf.py set-target esp32s3` to set target chip * Run `pip install "idf-component-manager~=1.1.4"` to upgrade your component manager if any error happends during last step * Run `idf.py -p PORT flash monitor` to build, flash and monitor the project (To exit the serial monitor, type `Ctrl-]`.) See the Getting Started Guide for all the steps to configure and use the ESP-IDF to build projects. ## Example Output ``` I (439) UVC_STREAM: line 486 HCD_PORT_EVENT_CONNECTION I (489) UVC_STREAM: Resetting Port I (549) UVC_STREAM: Setting Port FIFO I (549) UVC_STREAM: Getting Port Speed I (549) UVC_STREAM: Port speed = 1 I (549) UVC_STREAM: USB Speed: full-speed I (550) UVC_STREAM: Set Device Addr = 1 I (550) UVC_STREAM: Set Device Addr Done I (551) UVC_STREAM: get device desc I (551) UVC_STREAM: get device desc, actual_num_bytes:26 *** Device descriptor *** bcdUSB 2.00 bDeviceClass 0xef bDeviceSubClass 0x2 bDeviceProtocol 0x1 bMaxPacketSize0 64 idVendor 0x1871 idProduct 0xff50 bNumConfigurations 1 I (552) UVC_STREAM: get short config desc I (553) UVC_STREAM: get config desc, actual_num_bytes:16 I (553) UVC_STREAM: get full config desc I (561) UVC_STREAM: get full config desc, actual_num_bytes:397 *** Configuration descriptor *** wTotalLength 389 bNumInterfaces 2 bConfigurationValue 1 *** Interface Association Descriptor: Video *** *** Interface descriptor *** bInterfaceNumber 0 bAlternateSetting 0 bNumEndpoints 1 bInterfaceClass 0xe (Video) bInterfaceSubClass 0x1 *** Endpoint descriptor *** bEndpointAddress 0x83 EP 3 IN bmAttributes 0x3 INT wMaxPacketSize 16 bInterval 6 *** Interface descriptor *** bInterfaceNumber 1 bAlternateSetting 0 bNumEndpoints 0 bInterfaceClass 0xe (Video) bInterfaceSubClass 0x2 *** Class-specific VS Interface Descriptor *** bNumFormats 2 *** VS Format MJPEG Descriptor *** bFormatIndex 0x2 bNumFrameDescriptors 4 bDefaultFrameIndex 1 *** VS MJPEG Frame Descriptor *** bFrameIndex 0x1 wWidth 640 wHeigh 480 FrameInterval[0] 666666 *** VS MJPEG Frame Descriptor *** bFrameIndex 0x2 wWidth 480 wHeigh 320 FrameInterval[0] 666666 FrameInterval[1] 1000000 FrameInterval[2] 2000000 *** VS MJPEG Frame Descriptor *** bFrameIndex 0x3 wWidth 352 wHeigh 288 FrameInterval[0] 666666 *** VS MJPEG Frame Descriptor *** bFrameIndex 0x4 wWidth 320 wHeigh 240 FrameInterval[0] 666666 *** Interface descriptor *** bInterfaceNumber 1 bAlternateSetting 1 bNumEndpoints 1 bInterfaceClass 0xe (Video) bInterfaceSubClass 0x2 *** Endpoint descriptor *** bEndpointAddress 0x81 EP 1 IN bmAttributes 0x5 ISOC wMaxPacketSize 956 bInterval 1 W (568) UVC_STREAM: VS Interface(MPS < 600) NOT found W (569) UVC_STREAM: Try with first alt-interface config I (569) UVC_STREAM: Actual MJPEG format index = 2, contains 4 frames I (569) UVC_STREAM: Actual MJPEG width*heigh: 320*240, frame index = 4 I (570) UVC_STREAM: UVC Streaming Config Succeed W (570) UVC_STREAM: UAC 1.0 TYPE1 NOT found I (570) UVC_STREAM: Set Device Configuration = 1 I (571) UVC_STREAM: Set Device Configuration Done I (571) UVC_STREAM: SET_CUR Probe I (575) UVC_STREAM: SET_CUR Probe Done I (575) UVC_STREAM: GET_CUR Probe I (576) UVC_STREAM: GET_CUR Probe Done, actual_num_bytes:34 I (576) UVC_STREAM: SET_CUR COMMIT I (580) UVC_STREAM: SET_CUR COMMIT Done I (580) UVC_STREAM: Set Device Interface = 1, Alt = 1 I (930) UVC_STREAM: Set Device Interface Done I (930) UVC_STREAM: usb stream task start I (930) UVC_STREAM: uvc stream urb ready I (931) UVC_STREAM: Sample processing task started I (931) UVC_STREAM: Creating uvc in(isoc) pipe itf = 1-1, ep = 0x81 I (931) UVC_STREAM: uvc streaming... I (1492) uvc_mic_spk_demo: uvc callback! frame_format = 7, seq = 1, width = 320, height = 240, length = 8752, ptr = 0 I (1557) uvc_mic_spk_demo: uvc callback! frame_format = 7, seq = 2, width = 320, height = 240, length = 9496, ptr = 0 I (1624) uvc_mic_spk_demo: uvc callback! frame_format = 7, seq = 3, width = 320, height = 240, length = 7920, ptr = 0 I (1688) uvc_mic_spk_demo: uvc callback! frame_format = 7, seq = 4, width = 320, height = 240, length = 7880, ptr = 0 I (1752) uvc_mic_spk_demo: uvc callback! frame_format = 7, seq = 5, width = 320, height = 240, length = 8616, ptr = 0 I (1820) uvc_mic_spk_demo: uvc callback! frame_format = 7, seq = 6, width = 320, height = 240, length = 8808, ptr = 0 I (1885) uvc_mic_spk_demo: uvc callback! frame_format = 7, seq = 7, width = 320, height = 240, length = 8912, ptr = 0 I (1952) uvc_mic_spk_demo: uvc callback! frame_format = 7, seq = 8, width = 320, height = 240, length = 8832, ptr = 0 I (2017) uvc_mic_spk_demo: uvc callback! frame_format = 7, seq = 9, width = 320, height = 240, length = 8816, ptr = 0 ```
idf.py add-dependency "espressif/usb_stream^1.0.0"
To create a project from this example, run:
idf.py create-project-from-example "espressif/usb_stream^1.0.0:usb_camera_lcd_display"
To create a project from this example, run:
idf.py create-project-from-example "espressif/usb_stream^1.0.0:usb_camera_mic_spk"