espressif/esp_h264

# ESP_H264 [![Component Registry](https://components.espressif.com/components/espressif/esp_h264/badge.svg)](https://components.espressif.com/components/espressif/esp_h264) [[中文]](./README_CN.md) ESP_H264 is Espressif's lightweight H.264 encoder and decoder component, offering both hardware and software implementations. The hardware encoder (HW encoder) is designed specifically for the ESP32-P4 chip, achieving a frame rate greater than 30fps for 1080P resolution images. The software encoder (SW encoder) is sourced from v2.2.0 of [openh264](https://github.com/cisco/openh264), while the decoder is obtained from [tinyH264](https://github.com/udevbe/tinyh264). Both the software encoder and decoder are optimized for memory and CPU usage, ensuring optimal performance on Espressif chips. ## Term | Term Abbr | Full Name | Description | | ---------- | ----------------------- | --------------------------------------------------------------------------------------- | | QP | Quantization parameter | The higher the QP, the higher the compression rate and the worse the image quality. | | | | QP is within the range of 0 to 51. | | FPS | Frames per second | It is related to the smoothness of the video. It can be set to 24 in the general video. | | I-frame | Intra frame | Frames that can be encoded without reference to other frames. | | IDR-frame | Instantaneous Decoder | Special I-frame. | | | Refresh frame | Decoder can start decoding at this frame. | | P-frame | Predicted frame | Frames that must be decoded referring to other frames. | | GOP | Group of pictures | One I-frame plus the number of frames between two I-frames. | | | | GOP is usually set to the encoder output FPS. | | Resolution | Resolution | Width and height of the picture. | | MB | Macro block | For picture luma, MB size is 16x16. And for picture chrominance, MB size is 8x8. | | | | `mb_width` = (`width` + 15) >> 4. `mb_height` = (`height` + 15) >> 4 | | Slice | Slice | Multiple macroblocks form a slice | | MV | Motion vector | The horizontal and vertical displacement of the current MB relative | | | | to the best matching MB in the previous frame. | | ROI | Region of interest | An area of interest within a picture. | | | | This area can be set with different QP to make it clearer or blurrier. | | | | The quantization unit is the size of a luma MB. | | SPS | Sequence parameter set | Required to start decoding. | | PPS | Picture parameter set | Required to start decoding. | | PTS | Presentation time stamp | In baseline profile, the PTS is same as DTS. | | DTS | Decoding time stamp | In baseline profile, the DTS is same as PTS. | | RC | Rate control | The size of the output stream approaching the target stream is controlled. | ## Supported chip | ESP_H264 Version | ESP32-S3 | ESP32-S31 | ESP32-P4 | | ---------------- | --------- | --------- | --------- | | v1.4.0 | Supported | Supported | Supported | ## Features ### encoder | Feature | HW encoder | SW encoder | | ------------------- | ------------------------------------------------------------------- | ------------------------------------------- | | profile | Support baseline profile | Support baseline profile | | width | Supported range is 80 to 1920. | Supported range is greater than or equal to 16. | | height | Supported range is 80 to 2032. | Supported range is greater than or equal to 16. | | QP | Supported all | Supported all | | FPS | Supported FPS range is 1 to 255. | Supported FPS range is 1 to 255. | | GOP | Supported GOP range is 1 to 255. | Supported GOP range is 1 to 255. | | Force IDR | Supported via `esp_h264_enc_force_idr()`. | Unsupported | | SPS | Supported SPS for all IDR frames | Supported SPS for all IDR frames | | PPS | Supported PPS for all IDR frames | Supported PPS for all IDR frames | | unencoded data type | Supported ESP_H264_RAW_FMT_O_UYY_E_VYY/VUY/UYVY | Supported ESP_H264_RAW_FMT_YUYV | | | Supported ESP_H264_RAW_FMT_BGR888/RGB565_LE | Supported ESP_H264_RAW_FMT_I420 | | RC | Supported | Supported | | de-blocking filter | Supported | Supported | | Single stream | Supported | Supported | | Dual stream | Each stream supports different parameter configurations except GOP. | Unsupported | | ROI | Supported ROI region number is not greater than 8. | Unsupported | | | Each region supports fixed QP or delta QP. | Unsupported | | | Each non-ROI region supports delta QP. | Unsupported | | MV | Supported output MV data | Unsupported | ### decoder | Feature | SW decoder | | ------------------------------------------ | -------------------------------------------------------- | | profile | Support constrained baseline profile | | width | Supported range is greater than or equal to 16. | | height | Supported range is greater than or equal to 16. | | slice group | Support 1 slice group | | QP | Supported all | | FPS | Supported | | GOP | Supported | | SPS | Supported | | PPS | Supported | | unencoded data type | Supported ESP_H264_RAW_FMT_I420 | | long term reference (LTR) frames | Supported | | memory management control operation (MMCO) | Supported | | reference picture list modification | Supported | | dual task decoder | Supported, config task core and priority with menuconfig | ## Performance ### Test on chip ESP32-S3R8 #### SW ENCODER | Resolution | Raw Format | Memory (Byte) | Frame Per Second(fps) | | ---------- | ----------------------- | ------------- | --------------------- | | 320 * 192 | ESP_H264_RAW_FMT_I420 | 1 M | 17.48 | | 320 * 240 | ESP_H264_RAW_FMT_YUYV | 1 M | 11.23 | #### DECODER Note: the memory consumption is strongly dependent on the resolution of H264 stream and the encoded data. For the **mono task** decoder implementation, the performance is as follows: | Resolution | Raw Format | Memory (Byte) | Frame Per Second(fps) | | ---------- | ----------------------- | ------------- | --------------------- | | 640 * 480 | ESP_H264_RAW_FMT_I420 | 2.5 M | 9 | | 320 * 192 | ESP_H264_RAW_FMT_I420 | 1.0 M | 23 | For the **dual task** decoder implementation, the performance is as follows: | Resolution | Raw Format | Memory (Byte) | Frame Per Second(fps) | | ---------- | ----------------------- | ------------- | --------------------- | | 640 * 480 | ESP_H264_RAW_FMT_I420 | 2.5 M | 11 | | 320 * 192 | ESP_H264_RAW_FMT_I420 | 1.0 M | 27 | ### Test on ESP32-S31 #### SW Encoder | Resolution | Raw Format | Memory (Byte) | FPS | | ---------- | ----------------------- | ------------- | ---- | | 320 × 240 | ESP_H264_RAW_FMT_I420 | 1 M | 14.2 | | 320 × 240 | ESP_H264_RAW_FMT_YUYV | 1 M | 13.6 | #### Decoder Memory consumption strongly depends on the H.264 stream resolution and encoded data. Single-task decoder: | Resolution | Raw Format | Memory (Byte) | FPS (C) | FPS (Assembly) | | ----------- | ----------------------- | ------------- | ------- | -------------- | | 1280 × 720 | ESP_H264_RAW_FMT_I420 | 6.2 M | 4.6 | 5.6 | | 640 × 480 | ESP_H264_RAW_FMT_I420 | 2.5 M | 12.3 | 15.4 | Dual-task decoder: | Resolution | Raw Format | Memory (Byte) | FPS (C) | | ----------- | ----------------------- | ------------- | ------- | | 1280 × 720 | ESP_H264_RAW_FMT_I420 | 6.2 M | 6.4 | | 640 × 480 | ESP_H264_RAW_FMT_I420 | 2.5 M | 16.9 | ### Test on chip ESP32-P4 #### HW ENCODER | Resolution | Raw Format | Memory (Byte) | Frame Per Second(fps) | | ----------- | ----------------------------- | ------------- | --------------------- | | 1920 * 1080 | ESP_H264_RAW_FMT_O_UYY_E_VYY | 140k | 30 | Approximately: The frame rate for a given resolution can be estimated as: fps_cur ≈ fps_1080p × (current_resolution_pixels × bytes_per_pixel) ÷ (1920 × 1080 × bytes_per_pixel_1080p) Where: - fps_cur is the approximate frame rate for the current resolution - fps_1080p is the frame rate for 1080p resolution - current_resolution_pixels is the total number of pixels for the current resolution - bytes_per_pixel is the number of bytes per pixel for the current resolution - 1920 × 1080 is the number of pixels for 1080p resolution - bytes_per_pixel_1080p is the number of bytes per pixel for 1080p #### DECODER Note: the memory consumption is strongly dependent on the resolution of H264 stream and the encoded data. For the **mono task** decoder implementation, the performance is as follows: | Resolution | Raw Format | Memory (Byte) | Frame Per Second(fps) | | ---------- | ----------------------- | ------------- | --------------------- | | 1280 * 720 | ESP_H264_RAW_FMT_I420 | 6.2 M | 7 | | 640 * 480 | ESP_H264_RAW_FMT_I420 | 2.5 M | 25 | For the **dual task** decoder implementation, the performance is as follows: | Resolution | Raw Format | Memory (Byte) | Frame Per Second(fps) | | ---------- | ----------------------- | ------------- | --------------------- | | 1280 * 720 | ESP_H264_RAW_FMT_I420 | 6.2 M | 10 | | 640 * 480 | ESP_H264_RAW_FMT_I420 | 2.5 M | 31 | ## Example Please refer to the files test_apps/esp_h264\_\*\_test.c and test_apps/esp_h264\_\*\_test.h for more details on API usage. ## Prebuilt Library Selection When using ESP32-S31, esp_h264 provides two prebuilt libraries and uses the C implementation by default. Enabling `CONFIG_ESP_H264_S31_USE_ASM` selects the assembly-optimized prebuilt library for improved performance, but consumes part of CPU Core 1 processing resources. Do not use the dual-task decoder when this option is enabled. # FAQ ## Performance Issues ### Q: Why is decoding speed slow on ESP32-P4? **A:** Decoding performance depends on several factors: - **Resolution**: Higher resolutions require more processing power - **Task configuration**: Use dual-task decoder for better performance - **Memory allocation**: Ensure sufficient SPIRAM is available - **Stream complexity**: Complex H.264 streams take more time to decode **Performance optimization tips:** - Use dual-task decoder implementation (configure task core and priority via menuconfig) ``` ESP_H264_DECODER_IRAM=1 ESP_H264_DUAL_TASK=1 ``` - Lower resolution if possible (e.g., 640x480 instead of 1280x720) - Ensure adequate memory allocation - Check if the input H.264 stream uses the supported profile (constrained baseline) ## Usage Under Encryption ### Q: How do I use the hardware encoder on ESP32-P4 with PSRAM encryption enabled? **A:** The ESP-H264 hardware encoder uses DMA-related video buffers. When PSRAM encryption is enabled, encoder input and output buffers located in PSRAM must be allocated from an unencrypted memory region; otherwise, hardware encoding may fail. The component provides the following support for encryption: - When `CONFIG_SPIRAM_ENC_EXEMPT` is enabled, `ESP_H264_MEM_SPIRAM` includes the `MALLOC_CAP_SPIRAM_NO_ENC` capability and allocates buffers from the unencrypted PSRAM region. - `esp_h264_aligned_malloc()` and `esp_h264_aligned_calloc()` round the allocation size to cache-line alignment and add the `MALLOC_CAP_CACHE_ALIGNED` capability to the buffer. When using the component: 1. Allocate encoder buffers through `esp_h264_aligned_malloc()` or `esp_h264_aligned_calloc()` with `ESP_H264_MEM_SPIRAM` as the capability argument to ensure the correct memory region and cache alignment. 2. `CONFIG_SPIRAM_ENC_EXEMPT_SIZE` defines the size of the unencrypted PSRAM region. Buffer size depends on the maximum processed resolution, so configure it for the total buffers required by the component. Allocation fails if the region is insufficient. The unencrypted region is disabled if this value is greater than or equal to the physical PSRAM capacity.

Readme

Links

Targets

Tags

Stats

Badge