esp_llm_baidu_agent_app

# ESP LLM Baidu Agent Application

- [中文版](./README_CN.md)
- Complex Example: ⭐⭐⭐

## Example Brief

> **Positioning**: This is a **Baidu RTC agent** example built on `esp_llm_baidu`. It supports duplex voice by default and optional **camera understanding, cloud image generation, and local LCD preview** by scene. For audio only, set **Agent scene** to **Voice chat** in menuconfig. If the board has no camera or LCD, vision features are skipped and voice still runs.

- Joins the Baidu agent platform over RTC for duplex speech; image understanding, generation, and multi-modal behavior are selected by **Agent scene**.
- Board audio, display, and camera use lazy `esp_board_manager` init; `esp_board_manager_check_name()` probes camera/LCD before the graph is built—missing hardware skips vision and keeps voice.
- Configure App ID, User ID, Workflow, Licence Key, and Server URL before running.

Official documentation: [Baidu RTC](https://cloud.baidu.com/doc/RTC/index.html).

### Features

Every capability below is turned on in `idf.py menuconfig`:

| Capability | What it does | Configured by | Default |
| --- | --- | --- | --- |
| Voice conversation | Duplex speech with the agent workflow | `Agent scene` set to voice chat | On |
| Vision understanding | Uploads camera images so the agent can answer about them | `Agent scene` set to image understanding or multi-modal | Off |
| Image generation | Draws images that the agent generates on the LCD | `Agent scene` set to image generation or multi-modal | Off |
| Local camera preview | Shows the camera in a corner of the LCD during a camera scene | `Local LCD camera preview` | On in camera scenes |
| Wake word | Plays a prompt tone, cancels the current reply, and tells the agent to start listening | `Enable WakeNet wake-up` | Off |
| Speech barge-in | Lets you interrupt the agent by talking over it | `ESP LLM Baidu` server voice interrupt | On, level 80 |
| Voice volume control | The agent changes playback volume through an `adjust_volume` FunctionCall | Always built in | On |
| Music playback | Plays an HTTP music URL that the agent asks for | Always built in | On |

### Typical Scenarios

- Voice assistant speaker driven by a Baidu agent workflow
- Screen device that asks the agent about what the camera sees
- Screen device that displays images generated by the agent
- Multi-modal device that combines camera uplink, generated images, and a local preview

### Media Capabilities

| Item | Support in this example |
| --- | --- |
| Audio uplink | OPUS, 16 kHz, mono; PCM when the component selects `ESP_LLM_BAIDU_AGENT_AUDIO_CODEC_PCM` |
| Audio downlink | Same codec and rate as the uplink |
| Video uplink | Enabled in the image-understanding and multi-modal scenes; camera JPEG on `VIDEO_UP` |
| Video downlink | Enabled in the image-generation and multi-modal scenes; agent images on `VIDEO_DN` to the LCD player |
| Camera | `BAIDU_CAMERA_WIDTH/HEIGHT/FPS`, 320x240 at 10 fps by default and 640x480 at 15 fps on ESP32-S31, independent of the LCD size |
| LCD preview | Picture-in-picture in the LCD bottom-right corner, 8 px from each edge; RGB565 at half the camera size, so 160x120 by default, at 5 fps |
| Audio front end | AEC and VAD are always on; NS is off by default (`BAIDU_AUDIO_NS`) |
| Wake word | WakeNet, off by default, enabled by `BAIDU_AUDIO_WN`, with an MP3 prompt from SPIFFS |
| Uplink gating | None; the audio uplink stays open even when WakeNet is enabled |
| Player streams | Stream 0 carries agent TTS, stream 1 mixes an HTTP or file URL, stream 2 plays the wake-up prompt |

Which of these video items is active depends on the scene, and so does the owner of the LCD. See [Video Paths](#video-paths) for the three paths and how they share the screen.

### Agent Scenes

`Agent scene` selects which media streams the firmware enables:

| Scene | Kconfig | `VIDEO_UP` | `VIDEO_DN` | What the LCD shows |
| --- | --- | --- | --- | --- |
| Voice chat | `BAIDU_APP_SCENE_VOICE_CHAT` | Off | Off | Nothing; no display is created |
| Image understanding | `BAIDU_APP_SCENE_IMAGE_UNDERSTANDING` | On | Off | The camera preview, or nothing when the preview is off |
| Image generation | `BAIDU_APP_SCENE_IMAGE_GENERATION` | Off | On | Images generated by the agent |
| Multi-modal | `BAIDU_APP_SCENE_MULTI_MODAL` | On | On | Agent images, with the camera preview on top |

Voice chat is the default and needs no camera or display. In the two camera scenes, `Vision uplink mode` chooses between uploading one JPEG when the cloud requests vision (`BAIDU_VISION_MODE_IMAGE`) and sending every captured frame (`BAIDU_VISION_MODE_STREAM`).

### Video Paths

This example has the widest video behavior of the four AI agent examples, because the LCD belongs to the camera in one scene and to the agent in another. Three independent paths are involved:

| Path | Direction | How it travels | Active in |
| --- | --- | --- | --- |
| Camera uplink | Camera to cloud | Capture stream 0, JPEG at the configured camera size, linked to `VIDEO_UP` | Image understanding, multi-modal |
| Agent images | Cloud to LCD | `VIDEO_DN`, drawn by the video player | Image generation, multi-modal |
| Local preview | Camera to LCD | Capture stream 1, RGB565, drawn directly and never sent anywhere | Camera scenes with the preview on |

The display device is created only when something needs to draw on it, which means when agent images or the local preview are active. In every other case the example builds an audio-only player and never touches the LCD, so image understanding with the preview turned off runs headless even though a camera is in use.

In the multi-modal scene both paths draw at the same time. Agent images form the base layer and the camera preview sits above them as a picture-in-picture at z-order 1. The panel refresh rate then follows the agent image stream, 3 fps by default, rather than the 5 fps preview rate.

Both downlink tracks share a single link. The example links only `AUDIO_DN` to the player, because the LLM service exports one shared provider for audio and video downlink, so agent images arrive over that same link without a second link call.

### File Structure

```
esp_llm_baidu_agent_app/
├── main/
│   ├── app_main.c            Platform init, SPIFFS, Wi-Fi, and application lifecycle
│   ├── app_agent.c           Baidu configuration, scene streams, AFE mask, and events
│   ├── app_agent.h           Agent create / start / destroy API
│   ├── app_agent_utils.c     adjust_volume FunctionCall JSON parsing
│   ├── app_agent_utils.h     Volume-adjust helper declaration
│   ├── Kconfig.projbuild     Application menuconfig options
│   └── idf_component.yml
├── components/
│   └── baidu_app_media/      Board devices, capture, player, and LCD preview
├── spiffs/
│   └── lai_lo.mp3            Wake-up prompt played from the spiffs_data partition
├── partitions.csv
├── sdkconfig.defaults
├── sdkconfig.defaults.esp32p4
├── sdkconfig.defaults.esp32s3
├── sdkconfig.defaults.esp32s31
├── README.md
└── README_CN.md
```

### Code Map

`components/baidu_app_media/baidu_app_media.c` turns the selected scene into a media graph. The public API exposes only audio capture and player getters; video uplink, generated-image downlink, and local preview are linked inside `baidu_app_media_start()` without video service handles in the application. Compile-time switches at the top of that file decide the whole shape of the graph:

| Switch | Meaning |
| --- | --- |
| `BAIDU_APP_MEDIA_CAMERA` | Camera capture exists; set by the image-understanding and multi-modal scenes |
| `BAIDU_APP_MEDIA_VIDEO_DN` | Agent images are expected; set by the image-generation and multi-modal scenes |
| `BAIDU_APP_MEDIA_PREVIEW` | Local preview is on; follows `BAIDU_VIDEO_PREVIEW` |
| `BAIDU_APP_MEDIA_LCD` | Either of the two above needs the screen, so the player is built as a video player |

The functions worth reading first, in the order they run:

| Function | Responsibility |
| --- | --- |
| `create_video_capture()` | Opens capture stream 0 for the JPEG uplink and stream 1 for the RGB565 preview |
| `create_video_player()` | Builds the player with the display attached, used when `BAIDU_APP_MEDIA_LCD` is set |
| `create_audio_player()` | Builds the audio-only player used when no screen is needed |
| `create_preview_stream()` | Borrows the player render, opens the preview stream, and computes the picture-in-picture rectangle |
| `preview_task()` | Drains capture stream 1 and writes each frame straight to the preview render stream |
| `link_services()` | Links the audio uplink, the camera uplink, and the shared downlink to the player |

`main/app_agent.c` is the other half: it maps the scene onto `cfg.service.streams[]`, builds the audio feature mask, and handles the agent events, including the wake-up sequence and the `adjust_volume` FunctionCall.

## Environment Setup

### Hardware Requirements

- A board that provides `audio_adc` and `audio_dac` devices
- Speaker or headphones connected to the board audio output
- For the vision scenes, the board must also provide `camera` and `display_lcd` devices
- PSRAM, and 16 MB flash on ESP32-S3 and ESP32-S31 or 8 MB flash on ESP32-P4

The partition table uses a 6 MB `factory` application partition, a 512 KB `model` partition for the WakeNet model, and a 128 KB `spiffs_data` partition for the wake-up prompt.

### Supported IDF Branch

This example supports IDF release/v5.4 (>= v5.4.3) and release/v5.5 (>= v5.5.2).

### Software Requirements

- A Baidu application ID, user ID, agent platform URL, workflow, and instance ID
- A license key when the account requires one
- A Wi-Fi network that reaches the configured Baidu agent platform URL

Set App ID, User ID, Workflow, Licence Key, Server URL, instance ID, and Wi-Fi in `idf.py menuconfig` before running. CI injects those values from environment variables through `sdkconfig.ci`.

## Build and Flash

### Build Preparation

Make sure ESP-IDF is configured before building this example. If it is already set up, enter the project directory directly. Otherwise, run the following scripts in the ESP-IDF root directory. See the [ESP-IDF Programming Guide](https://docs.espressif.com/projects/esp-idf/en/latest/) for the target chip you use.

```
./install.sh
. ./export.sh
```

- Enter the example directory:

```
cd adf_examples/ai_agent/esp_llm_baidu_agent_app
```

- Select the chip, then the board:

```bash
idf.py set-target esp32s3
idf.py bmgr -b esp32_s3_korvo_2_3
```

Run `idf.py set-target` before `idf.py bmgr`, because it loads `sdkconfig.defaults` together with the matching `sdkconfig.defaults.<chip>` file that carries the camera defaults. This example uses [ESP Board Manager](https://github.com/espressif/esp-board-manager); `idf.py bmgr -l` lists the boards that are currently visible, and `idf.py bmgr -b <board_index|board_name>` selects one.

### Project Configuration

```bash
idf.py menuconfig
```

Set the following under `Baidu LLM agent application`:

- `Baidu application ID`, `Baidu user ID`, `Baidu license key`, `Baidu agent platform URL`, `Baidu workflow`, and `Baidu instance ID`: the account and workflow the device joins
- `Agent scene`: voice chat, image understanding, image generation, or multi-modal
- `Vision uplink mode`: snapshot on request, or continuous frames; shown only in the camera scenes
- `Camera JPEG width` / `height` / `Camera capture FPS`: camera scenes only; the size must match an esp-video JPEG sensor mode
- `Local LCD camera preview`: draws a second RGB565 stream on the LCD; this is local preview, not `VIDEO_DN`
- `Audio front-end features`: only `Enable WakeNet wake-up` is exposed, because AEC, NS, and VAD are always on
- `Audio output volume`: 0 through 100, default 80
- `Wi-Fi`: SSID, password, and connection timeout in milliseconds
- `Run time before clean shutdown`: 0 returns from `app_main` and leaves the agent running; a non-zero value exercises the full stop, unlink, and destroy path

Credentials are compiled into the firmware, so do not commit real values. The Wi-Fi profile is persisted to NVS by `esp_wifi_service_request_connect()`.

### Build and Flash

- Build the example

```
idf.py build
```

- Flash and monitor (replace PORT with your serial port):

```
idf.py -p PORT flash monitor
```

- Press `Ctrl-]` to exit the monitor

## How to Use the Example

### Features and Usage

1. After boot the device mounts SPIFFS, connects to Wi-Fi, then joins the agent platform
2. The log reports that the Baidu agent is ready once the room is joined
3. In voice chat, speak at any time; the agent workflow decides when a query ends
4. With WakeNet enabled, the wake word plays `lai_lo.mp3` on player stream 2, cancels the current response, and sends `[B]:[BEGIN]:0` so the agent starts listening. The microphone is never muted, so speech before the wake word still reaches the cloud
5. In the image-understanding and multi-modal scenes the camera feeds `VIDEO_UP`, and the LCD shows the local preview when it is enabled
6. In the image-generation and multi-modal scenes the agent's generated images arrive on `VIDEO_DN` and are drawn by the video player
7. The `adjust_volume` FunctionCall changes the player volume and then confirms it out loud through `esp_llm_baidu_send_text_to_tts()`
8. Backend `AUDIO_PLAYER_START` and `AUDIO_PLAYER_STOP` events play and stop an HTTP music URL on player stream 1
9. Set `Run time before clean shutdown` to a non-zero value to watch the complete teardown sequence

## Troubleshooting

### The application stops before joining the room

Check the log for the failing step. An empty Wi-Fi SSID stops the network startup, and incomplete Baidu account fields are rejected before the agent engine is created.

### Board device names are missing

Build or startup errors about unknown device handles mean the board metadata was not generated for the selected chip. Rerun `idf.py bmgr -b <board>` with the board that matches the target.

### Joining the room times out

Check network reachability, the agent platform URL, the account values, the workflow name, and the Wi-Fi connection timeout.

### `V4L2_SRC: Fail to open device`

Either the camera defaults for the selected chip were not applied, or the sensor fitted on the board is not the one those defaults describe. Recreate `sdkconfig` with `idf.py set-target` so the chip file is loaded. The chip files select one specific sensor and mode: OV3660 over DVP at JPEG 320x240 on ESP32-S3, SC101IOT over DVP at YUV422 YUYV 640x480 on ESP32-S31, and SC2336 over MIPI CSI at RAW8 1024x600 on ESP32-P4. If the board carries a different sensor, enable that sensor in menuconfig instead and set `Camera JPEG width` and `height` to one of its real modes.

### The wake-up prompt does not play

The `spiffs_data` partition was not flashed or `lai_lo.mp3` is missing from `spiffs/`. Reflash the full image so the SPIFFS partition is written.

### The application does not fit in flash

The `factory` partition is 6 MB. Enabling several scenes together with WakeNet grows the image; disable the local preview or the unused scene to recover space.

## Technical Support

- Forum: [esp32.com](https://esp32.com/viewforum.php?f=20)
- Issues: [GitHub issue](https://github.com/espressif/esp-adf/issues)

We will get back to you as soon as possible.