espressif/usb_stream

readme

[![Component Registry](https://components.espressif.com/components/espressif/usb_stream/badge.svg)](https://components.espressif.com/components/espressif/usb_stream)

## USB Stream Component

`usb_stream` is an USB `UVC` + `UAC` host driver for ESP32-S2/ESP32-S3, which supports read/write/control multimedia streaming from usb device. For example, at most one UVC + one Microphone + one Speaker streaming can be supported at the same time.

Features:

1. Support video stream through UVC Stream interface, support both isochronous and bulk mode
2. Support microphone stream and speaker stream through the UAC Stream interface
3. Support volume, mute and other features control through the UAC Control interface
4. Support stream separately suspend and resume

### USB Stream User Guide

Please refer: https://docs.espressif.com/projects/espressif-esp-iot-solution/en/latest/usb/usb_stream.html

### Add component to your project

Please use the component manager command `add-dependency` to add the `usb_stream` to your project's dependency, during the `CMake` step the component will be downloaded automatically

```
idf.py add-dependency "espressif/usb_stream=*"
```

### Examples

Please use the component manager command `create-project-from-example` to create the project from example template

* USB Camera WIFI picture transfer
```
idf.py create-project-from-example "espressif/usb_stream=*:usb_camera_mic_spk"
```

* USB Camera local display
```
idf.py create-project-from-example "espressif/usb_stream=*:usb_camera_lcd_display"
```

Then the example will be downloaded in current folder, you can check into it for build and flash.

> Or you can download examples from esp-iot-solution repository: [USB Camera + Audio stream](https://github.com/espressif/esp-iot-solution/tree/master/examples/usb/host/usb_camera_mic_spk), [USB Camera LCD Display](https://github.com/espressif/esp-iot-solution/tree/master/examples/usb/host/usb_camera_lcd_display).

### Q&A

Q1. I encountered the following problems when using the package manager

```
Executing action: create-project-from-example
CMakeLists.txt not found in project directory /home/username
```

A1. This is because an older version package manager was used, please run `pip install -U idf-component-manager` in ESP-IDF environment to update.

changelog

# ChangeLog

## v0.4.10 - 2023-03-16

### Enhancements:

* Add example: usb_camera_lcd_display

## v0.4.9 - 2023-03-09

### Enhancements:

* Use cu_pkg_define_version to define the version of this component.

## v0.4.8 - 2023-03-02

### Enhancements:

* Add user guide for component manager

## v0.4.7 - 2023-02-20

### Bug Fixes:

* Camera:
  * Fix incorrect isoc buffer offset when transfer error occurs

## v0.4.6 - 2023-02-17

### Enhancements:

* Camera:
  * Add additional JPEG header check for bulk transfer

# ChangeLog

## v0.4.5 - 2023-02-14

### Bug Fixes:

* Remove unnecessary header files "hal/usbh_ll.h" "hal/usb_hal.h"

### Enhancements:

* Camera:
  * Support camera with length of 2, 6 bytes MJPEG UVC header
  * Support bulk camera with EOF bit set in each payload

## v0.4.4 - 2023-02-09

### Enhancements:

* Remove all assert, replace with error handler

### Bug Fixes:

* Fix error message during disconnect

## v0.4.3 - 2023-02-07

### Enhancements:

* Remove some unnecessary vTaskDelay
* Camera:
  * Support for bulk camera with suspend/resume control
  * Support force uvc format index if not found
  * Add support for bulk camera with MPS(Max Packet Size) equal to 512 bytes

### Bug Fixes:

* Speaker: 
  * fix some speaker resume control not work (should set freq after resume)

## v0.4.2 - 2023-01-30

### Enhancements:

* Camera:
  * Disable BULK_BYTES_PER_URB_SAME_AS_FRAME by default, support reassemble bulk package segment (will save one image frame size RAM)
  * Add config UVC_DROP_NO_EOF_FRAME, if disable(default) the frame with no EOF will be reserved (MJPEG may broken)
  * Add config UVC_DROP_OVERFLOW_FRAME, if enable(default) the overflow frame will be dropped

### Bug Fixes:

* Camera
  * fix transfer size in GET_CUR Probe stage


## v0.4.1 - 2023-01-05

### Enhancements:

* Add IRAM_ATTR to packet parsing functions
* Speaker:
  * padding zero packet if speaker buffer empty


## v0.4.0 - 2022-12-22

### Enhancements:

* Support parsing descriptors to check the device functions and capabilities
* Self-adaption appropriate interfaces based on descriptors and user's configurations
* Camera:
  * Support UVC config to any resolution, the first frame index will be used

### API Changes:

* add `optional` label to some members in `uvc_config_t` and `uac_config_t`, users can optionally set a value. The self-adaption process will first use actual configs from descriptors, if no appropriate interface found, will use the params from user's configuration

### Bug Fixes:

* USB:
  * Fix STALL (appear on some devices) during get short configuration descriptor
  * Resize endpoint 0 MPS based on device descriptor

## v0.3.3 - 2022-12-22

### Enhancements:

* print version message

### Bug Fixes:

* Camera:
  * Fix not parse header only frames

## v0.3.2 - 2022-12-16

### Enhancements:

* Mic:
  * Padding last frame data when receive zero length packet

## v0.3.1 - 2022-12-8

### Enhancements:

* add development documentation https://docs.espressif.com/projects/espressif-esp-iot-solution/zh_CN/latest/usb/usb_stream.html

## v0.3.0 - 2022-12-05

### Bug Fixes:

* USB:
  * Fix memory leak during hot-plug

                                        
                                        ## USB Stream Lcd Display Example

This routine demonstrates how to use the [usb_stream](https://components.espressif.com/components/espressif/usb_stream) component to acquire a USB camera image and display it adaptively on the RGB screen.

* For better performance, please use ESP-IDF release/v5.0 or above versions.

## Hardware

* An [ESP32-S3-LCD-EV-Board](https://docs.espressif.com/projects/espressif-esp-dev-kits/en/latest/esp32s3/esp32-s3-lcd-ev-board/index.html) development board with 800x480/480*480 LCD.
* A USB camera (Can be connected to USB port or Expansion Connector)
* An USB Type-C cable for Power supply and programming (Please connect to UART port instead of USB port)

Note:
  Error message with `A fatal error occurred: Could not open /dev/ttyACM0, the port doesn't exist`: Please first make sure development board connected, then make board into "Download Boot" mode to upload by following steps:
  1. keep pressing  "BOOT(SW2)" button
  2. short press "RST(SW1)" button
  3. release "BOOT(SW2)".
  4. upload program and reset


## IDF Patch

The patch is intended to achieve best performance of RGB LCD by using **Octal PSRAM 120MHz** feature. Now there are two versions of patch under the patch folder that are used for the respective branch of ESP-IDF (master or release/v5.0). 

**Please make sure your IDF project is clean** (use `git status` to check), then the patch can be applied by following commands:

```
cd <root directory of IDF>
git apply <path of the patch>/idf_psram_120m.patch # Nothing return if success
git status      # Check whether the operation is successful, the output should look like below:

HEAD detached at f315986401
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   components/esp_hw_support/port/esp32s3/rtc_clk.c
        modified:   components/esp_psram/esp32s3/Kconfig.spiram
        modified:   components/hal/esp32s3/include/hal/spimem_flash_ll.h
        modified:   components/spi_flash/esp32s3/mspi_timing_tuning_configs.h
        modified:   components/spi_flash/esp32s3/spi_timing_config.c
        modified:   components/spi_flash/esp32s3/spi_timing_config.h
        modified:   components/spi_flash/spi_flash_timing_tuning.c

Untracked files:
  (use "git add <file>..." to include in what will be committed)
        tools/test_apps/system/flash_psram/sdkconfig.ci.f8r8_120ddr
        tools/test_apps/system/flash_psram/sdkconfig.ci.f8r8_120ddr_120ddr
```

### How To Use 
1. First delete existing `build`, `sdkconfig` , `sdkconfig.old`
```
rm -rf build sdkconfig sdkconfig.old
```

2. Use the command line to enable the relevant configuration
```
idf.py -D SDKCONFIG_DEFAULTS="sdkconfig.defaults;sdkconfig.psram_octal_120m" set-target esp32s3
```

3. Compile and burn
```
idf.py build flash monitor
```
                                    

                                        
                                        ## USB Stream Example

This example demonstrates how to use `usb_stream` component with an ESP device. Example does the following steps:

1. Config a UVC function with specified frame resolution and frame rate, register frame callback
2. Config a UAC function with one microphone and one speaker stream, register mic frame callback
3. Start the USB streaming
4. In image frame callback, if `ENABLE_UVC_WIFI_XFER` is set to `1`, the real-time image can be fetched through ESP32Sx's Wi-Fi softAP (ssid: ESP32S3-UVC, http: 192.168.4.1), else will just print the image message
5. In mic callback, if `ENABLE_UAC_MIC_SPK_LOOPBACK` is set to `1`, the mic data will be write back to usb speaker, else will just print mic data message
6. For speaker, if `ENABLE_UAC_MIC_SPK_LOOPBACK` is set to `0`, the default sound will be played back

## Hardware

* Development board

  1. Any `ESP32-S2`,` ESP32-S3` board with USB Host port can be used.
  2. Please note that the `esp32-sx-devkitC` board can not output 5V through USB port, If the OTG conversion cable is used directly, the device cannot be powered.
  3. For `esp32s3-usb-otg` board, please enable the USB Host power domain to power the device

* Connection

    ||USB_DP|USB_DM|
    |--|--|--|
    |ESP32-S2/S3|GPIO20|GPIO19|

* Camera module
  * please refer the README of `usb_stream` for the Camera requirement


## Build and Flash

Build the project and flash it to the board, then run the monitor tool to view the serial output:

* Run `. ./export.sh` to set IDF environment
* Run `idf.py set-target esp32s3` to set target chip
* Run `pip install "idf-component-manager~=1.1.4"` to upgrade your component manager if any error happends during last step
* Run `idf.py -p PORT flash monitor` to build, flash and monitor the project

(To exit the serial monitor, type `Ctrl-]`.)

See the Getting Started Guide for all the steps to configure and use the ESP-IDF to build projects.

## Example Output

```
I (439) UVC_STREAM: line 486 HCD_PORT_EVENT_CONNECTION
I (489) UVC_STREAM: Resetting Port
I (549) UVC_STREAM: Setting Port FIFO
I (549) UVC_STREAM: Getting Port Speed
I (549) UVC_STREAM: Port speed = 1
I (549) UVC_STREAM: USB Speed: full-speed
I (550) UVC_STREAM: Set Device Addr = 1
I (550) UVC_STREAM: Set Device Addr Done
I (551) UVC_STREAM: get device desc
I (551) UVC_STREAM: get device desc, actual_num_bytes:26
*** Device descriptor ***
bcdUSB 2.00
bDeviceClass 0xef
bDeviceSubClass 0x2
bDeviceProtocol 0x1
bMaxPacketSize0 64
idVendor 0x1871
idProduct 0xff50
bNumConfigurations 1
I (552) UVC_STREAM: get short config desc
I (553) UVC_STREAM: get config desc, actual_num_bytes:16
I (553) UVC_STREAM: get full config desc
I (561) UVC_STREAM: get full config desc, actual_num_bytes:397
*** Configuration descriptor ***
wTotalLength 389
bNumInterfaces 2
bConfigurationValue 1
*** Interface Association Descriptor: Video ***
        *** Interface descriptor ***
        bInterfaceNumber 0
        bAlternateSetting 0
        bNumEndpoints 1
        bInterfaceClass 0xe (Video)
        bInterfaceSubClass 0x1
                *** Endpoint descriptor ***
                bEndpointAddress 0x83   EP 3 IN
                bmAttributes 0x3        INT
                wMaxPacketSize 16
                bInterval 6
        *** Interface descriptor ***
        bInterfaceNumber 1
        bAlternateSetting 0
        bNumEndpoints 0
        bInterfaceClass 0xe (Video)
        bInterfaceSubClass 0x2
        *** Class-specific VS Interface Descriptor ***
        bNumFormats 2
        *** VS Format MJPEG Descriptor ***
        bFormatIndex 0x2
        bNumFrameDescriptors 4
        bDefaultFrameIndex 1
        *** VS MJPEG Frame Descriptor ***
        bFrameIndex 0x1
        wWidth 640
        wHeigh 480
        FrameInterval[0] 666666
        *** VS MJPEG Frame Descriptor ***
        bFrameIndex 0x2
        wWidth 480
        wHeigh 320
        FrameInterval[0] 666666
        FrameInterval[1] 1000000
        FrameInterval[2] 2000000
        *** VS MJPEG Frame Descriptor ***
        bFrameIndex 0x3
        wWidth 352
        wHeigh 288
        FrameInterval[0] 666666
        *** VS MJPEG Frame Descriptor ***
        bFrameIndex 0x4
        wWidth 320
        wHeigh 240
        FrameInterval[0] 666666
        *** Interface descriptor ***
        bInterfaceNumber 1
        bAlternateSetting 1
        bNumEndpoints 1
        bInterfaceClass 0xe (Video)
        bInterfaceSubClass 0x2
                *** Endpoint descriptor ***
                bEndpointAddress 0x81   EP 1 IN
                bmAttributes 0x5        ISOC
                wMaxPacketSize 956
                bInterval 1
W (568) UVC_STREAM: VS Interface(MPS < 600) NOT found
W (569) UVC_STREAM: Try with first alt-interface config
I (569) UVC_STREAM: Actual MJPEG format index = 2, contains 4 frames
I (569) UVC_STREAM: Actual MJPEG width*heigh: 320*240, frame index = 4
I (570) UVC_STREAM: UVC Streaming Config Succeed
W (570) UVC_STREAM: UAC 1.0 TYPE1 NOT found
I (570) UVC_STREAM: Set Device Configuration = 1
I (571) UVC_STREAM: Set Device Configuration Done
I (571) UVC_STREAM: SET_CUR Probe
I (575) UVC_STREAM: SET_CUR Probe Done
I (575) UVC_STREAM: GET_CUR Probe
I (576) UVC_STREAM: GET_CUR Probe Done, actual_num_bytes:34
I (576) UVC_STREAM: SET_CUR COMMIT
I (580) UVC_STREAM: SET_CUR COMMIT Done
I (580) UVC_STREAM: Set Device Interface = 1, Alt = 1
I (930) UVC_STREAM: Set Device Interface Done
I (930) UVC_STREAM: usb stream task start
I (930) UVC_STREAM: uvc stream urb ready
I (931) UVC_STREAM: Sample processing task started
I (931) UVC_STREAM: Creating uvc in(isoc) pipe itf = 1-1, ep = 0x81
I (931) UVC_STREAM: uvc streaming...
I (1492) uvc_mic_spk_demo: uvc callback! frame_format = 7, seq = 1, width = 320, height = 240, length = 8752, ptr = 0
I (1557) uvc_mic_spk_demo: uvc callback! frame_format = 7, seq = 2, width = 320, height = 240, length = 9496, ptr = 0
I (1624) uvc_mic_spk_demo: uvc callback! frame_format = 7, seq = 3, width = 320, height = 240, length = 7920, ptr = 0
I (1688) uvc_mic_spk_demo: uvc callback! frame_format = 7, seq = 4, width = 320, height = 240, length = 7880, ptr = 0
I (1752) uvc_mic_spk_demo: uvc callback! frame_format = 7, seq = 5, width = 320, height = 240, length = 8616, ptr = 0
I (1820) uvc_mic_spk_demo: uvc callback! frame_format = 7, seq = 6, width = 320, height = 240, length = 8808, ptr = 0
I (1885) uvc_mic_spk_demo: uvc callback! frame_format = 7, seq = 7, width = 320, height = 240, length = 8912, ptr = 0
I (1952) uvc_mic_spk_demo: uvc callback! frame_format = 7, seq = 8, width = 320, height = 240, length = 8832, ptr = 0
I (2017) uvc_mic_spk_demo: uvc callback! frame_format = 7, seq = 9, width = 320, height = 240, length = 8816, ptr = 0

```