esp_llm_tongyi_agent_app

# ESP LLM Tongyi Agent

- [中文版](./README_CN.md)
- Complex Example: ⭐⭐⭐

## Example Brief

> **Positioning**: DashScope **multimodal WebSocket agent** on `esp_llm_tongyi`. **Dialog** is duplex voice only; **Vision** adds LCD preview and `snap` JPEG upload when the board has camera and LCD. Missing hardware skips vision; voice still runs.

- WebSocket to a DashScope app for duplex speech.
- Vision: live preview plus `snap` / `snap <prompt>`; if `check_name` finds no camera or LCD, vision is skipped and voice continues.
- Set API Key, Workspace ID, and Application ID (vision needs an `mm_` app).

Official documentation: [DashScope multimodal WebSocket](https://help.aliyun.com/zh/model-studio/websocket-for-multimodal-dialogue).

### Features

Every capability below is turned on in `idf.py menuconfig`:

| Capability | What it does | Configured by | Default |
| --- | --- | --- | --- |
| Voice conversation | Duplex speech with the DashScope application | Available in both application modes | On |
| Camera preview | Shows a live camera image on the LCD | `Application mode` set to vision | On in the shipped defconfig |
| Ask about a photo | `snap <prompt>` captures one JPEG and asks the model about that image | `Application mode` set to vision | On in vision |
| Silent image update | `snap` with no argument refreshes the session image without asking for a reply | `Default snapshot prompt` left empty | On in vision |
| Wake word | Keeps the microphone closed until the wake word, then cancels the current reply | `Dialog mode` set to wake word then conversation | Off |
| Echo cancellation and NS | AEC and NS are always enabled | Built in | On |
| Voice activity detection | Detects when speech starts and stops | `Enable voice activity detection` | On |

### Typical Scenarios

- Duplex voice assistant backed by a DashScope voice application
- Screen device that shows a live camera preview and asks the model about what it sees
- Wake-word gated conversation that keeps the microphone closed until WakeNet fires

### Media Capabilities

| Item | Support in this example |
| --- | --- |
| Audio uplink | OPUS, 16 kHz, mono, 24000 bit/s |
| Audio downlink | OPUS, 16 kHz, mono, 32 kbit/s |
| Video uplink | No continuous video track; exactly one JPEG is sent per `snap` command |
| Video downlink | Not supported |
| Camera | Vision mode only, MJPEG at `TONGYI_VISION_CAMERA_WIDTH/HEIGHT`, 320x240 by default and 640x480 on ESP32-S31 |
| LCD preview | Vision mode only, full screen. RGB565 captured at the full camera size and scaled to the panel, at `TONGYI_VISION_CAMERA_FPS`, default 15 |
| Audio front end | AEC and NS always on; VAD via `TONGYI_AUDIO_VAD`, on by default |
| Wake word | WakeNet, added only when `Dialog mode` is set to `Wake word then conversation` |
| Uplink gating | In wake-word dialog mode, the audio uplink stays closed until the wake word |
| Player streams | Stream 0 carries Tongyi TTS; stream 1 is free for a URL or prompt tone |

Vision uses **single JPEG snapshots**, not a continuous `VIDEO_UP` uplink. Preview is RGB565 to the LCD; dual-stream capture and performance notes are in the [FAQ](#faq).

### Application Modes

`Application mode` selects what the firmware builds:

| Mode | Kconfig | Behaviour |
| --- | --- | --- |
| Dialog | `TONGYI_APP_MODE_DIALOG` | Duplex voice only; no camera or LCD |
| Vision | `TONGYI_APP_MODE_VISION` | Adds camera capture, LCD preview, and the `snap` CLI command |

The Kconfig default is Dialog, but the checked-in `sdkconfig.defaults` sets `CONFIG_TONGYI_APP_MODE_VISION=y`. A fresh `idf.py set-target` therefore produces a Vision build. Change `Application mode` in menuconfig to build the Dialog variant.

The two modes need different DashScope applications. Dialog needs a voice-only application ID, while the `mm_` default shipped for Vision is a multimodal application that answers `images must not be empty` when no camera frame is ever sent.

### File Structure

```
esp_llm_tongyi_agent/
├── main/
│   ├── app_main.c            Platform init, Wi-Fi, SNTP, and application lifecycle
│   ├── app_agent.c           Tongyi configuration, AFE mask, and events
│   ├── app_agent.h           Agent create / start / destroy API
│   ├── app_cli.c             UART CLI (`esp_cli_service`); Vision adds `snap`
│   ├── app_cli.h             CLI start / stop API
│   ├── libc_compat.c         stderr compatibility shim for older IDF releases
│   ├── Kconfig.projbuild     Application menuconfig options
│   └── idf_component.yml
├── components/
│   ├── tongyi_app_media/     Board devices, capture, player, and LCD preview
│   └── gen_bmgr_codes/       Board metadata generated by esp_board_manager
├── partitions.csv
├── sdkconfig.defaults
├── sdkconfig.defaults.esp32p4
├── sdkconfig.defaults.esp32s3
├── sdkconfig.defaults.esp32s31
├── README.md
└── README_CN.md
```

## Environment Setup

### Hardware Requirements

- A board that provides `audio_adc` and `audio_dac` devices
- Speaker or headphones connected to the board audio output
- For Vision mode, the board must also provide `camera` and `display_lcd` devices
- PSRAM, and 16 MB flash on ESP32-S3 and ESP32-S31 or 8 MB flash on ESP32-P4

| Target | Board | Camera default |
| --- | --- | --- |
| `esp32s3` | `esp32_s3_korvo_2_3` | OV3660 DVP JPEG 320x240 @ 25 fps |
| `esp32s31` | `esp32_s31_korvo_1` | SC101IOT DVP YUV422 YUYV 640x480 @ 15 fps |
| `esp32p4` | `esp32_p4_function_ev_board` | SC2336 MIPI CSI RAW8 1024x600 @ 30 fps |

The firmware uses a custom partition table with a 6 MB `factory` application partition.

### Supported IDF Branch

This example supports IDF release/v5.4 (>= v5.4.3) and release/v5.5 (>= v5.5.2).

### Software Requirements

- A DashScope API key, a workspace ID, and an application ID
- A Wi-Fi network that reaches the DashScope endpoints and an NTP server

`Kconfig.projbuild` ships with an internal lab API key and workspace ID so the example builds without edits. Replace both with your own DashScope credentials before using the example outside that lab account.

## Build and Flash

### Build Preparation

Make sure ESP-IDF is configured before building this example. If it is already set up, enter the project directory directly. Otherwise, run the following scripts in the ESP-IDF root directory. See the [ESP-IDF Programming Guide](https://docs.espressif.com/projects/esp-idf/en/latest/) for the target chip you use.

```
./install.sh
. ./export.sh
```

- Enter the example directory:

```
cd adf_examples/ai_agent/esp_llm_tongyi_agent
```

- Select the chip, then the board:

```bash
idf.py set-target esp32s3
idf.py bmgr -b esp32_s3_korvo_2_3
```

Run `idf.py set-target` before `idf.py bmgr`, because it loads `sdkconfig.defaults` together with the matching `sdkconfig.defaults.<chip>` file that carries the camera defaults. This example uses [ESP Board Manager](https://github.com/espressif/esp-board-manager); `idf.py bmgr -l` lists the boards that are currently visible, and `idf.py bmgr -b <board_index|board_name>` selects one.

### Project Configuration

```bash
idf.py menuconfig
```

Set the following under `Tongyi LLM agent application`:

- `DashScope API key`, `Tongyi workspace ID`, and `Tongyi application ID`: the DashScope account and application to talk to
- `Application mode`: Dialog for audio only, Vision for camera preview plus the `snap` command
- `Camera JPEG width` / `height` / `Camera / LCD preview FPS`: Vision only; the size must match an esp-video JPEG sensor mode
- `Default snapshot prompt`: Vision only; used when `snap` is typed without an argument
- `Dialog mode`: direct conversation, or wake word then conversation
- `Audio front-end features`: AEC and NS built in; VAD enabled
- `Audio output volume`: 0 through 100, default 80
- `Wi-Fi`: SSID, password, and connection timeout in milliseconds
- `Run time before clean shutdown`: 0 runs forever; a non-zero value exercises the full stop, unlink, and destroy path

Credentials are compiled into the firmware, so do not commit real values. The Wi-Fi profile is persisted to NVS by `esp_wifi_service_request_connect()`.

### Build and Flash

- Build the example

```
idf.py build
```

- Flash and monitor (replace PORT with your serial port):

```
idf.py -p PORT flash monitor
```

- Press `Ctrl-]` to exit the monitor

## How to Use the Example

### Features and Usage

1. After boot the application connects to Wi-Fi and waits for the SNTP sync to complete
2. The log then reports that the Tongyi agent is ready, meaning the device can talk
3. In direct dialog mode, speak at any time; the server decides when a query ends
4. In wake-word dialog mode, say the wake word first. The microphone opens on wake-up and the reply in progress is cancelled
5. Replies arrive as TTS on player stream 0; spoken text is also reported through the `RESPONDING_CONTENT` event and dialog transitions through `DIALOG_STATE`
6. The UART CLI prompt is `tongyi>` (`esp_cli_service`). Dialog has `help` / `sys_heap`. Vision also has `snap`

```text
snap
snap what is in front of the camera
```

`snap` with no argument sends the cached JPEG as `UpdateInfo`, which updates the session image without asking for a reply. `snap <prompt>` sends `RequestToRespond`, which asks the model to answer about that image. When `Default snapshot prompt` is set, a bare `snap` uses it and therefore also requests a reply.

## Troubleshooting

### The application stops before connecting

Check the log for the failing step. An empty Wi-Fi SSID reports `Start network failed: Wi-Fi SSID is empty`, and a missing API key, workspace ID, or application ID is rejected before the session is created.

### Board device names are missing

Build or startup errors about unknown device handles mean the board metadata was not generated for the selected chip. Rerun `idf.py bmgr -b <board>` with the board that matches the target.

### The WebSocket connection fails

Confirm that the network reaches the DashScope endpoints, that the API key and workspace ID are valid, and that the clock was synchronized. This example raises `CONFIG_MBEDTLS_SSL_IN_CONTENT_LEN` to 16384 and requires PSRAM for the TLS buffers.

### The model answers `images must not be empty`

A multimodal `mm_` application ID was used in Dialog mode, so no image is ever sent. Either switch `Application mode` to Vision, or set a voice-only application ID.

### `V4L2_SRC: Fail to open device`

Either the camera defaults for the selected chip were not applied, or the sensor fitted on the board is not the one those defaults describe. Recreate `sdkconfig` with `idf.py set-target` so the chip file is loaded. The chip files select one specific sensor and mode: OV3660 over DVP at JPEG 320x240 on ESP32-S3, SC101IOT over DVP at YUV422 YUYV 640x480 on ESP32-S31, and SC2336 over MIPI CSI at RAW8 1024x600 on ESP32-P4. If the board carries a different sensor, enable that sensor in menuconfig instead and keep `TONGYI_VISION_CAMERA_WIDTH` and `TONGYI_VISION_CAMERA_HEIGHT` aligned with one of its real modes.

## FAQ

### Why is the LCD preview expensive when the screen is always on but JPEG upload is rare?

That is the intended product shape: the LCD stays up while `snap` uploads a JPEG occasionally. The cheaper path would be a camera RGB565 stream at panel size written straight to the LCD, encoding JPEG only inside `tongyi_app_media_capture_jpeg()`.

Korvo-2 uses the OV3660 over DVP, whose official modes at 320x240, the same size as the ILI9341 panel, are JPEG only; RGB565 exists at 240x240 and 640x480, but not at 320x240. Korvo-1 uses the SC101IOT, which has no JPEG mode at all and only outputs YUV422. Neither sensor can serve both formats natively, and this example does not add a custom sensor table, so `vision_open()` keeps two capture streams:

1. MJPEG, cached for the next `snap`; taken straight from the OV3660, encoded from YUV422 on the SC101IOT
2. RGB565, written to the LCD; decoded from JPEG on the OV3660, converted from YUV422 on the SC101IOT

ESP32-S3 has no hardware JPEG decoder, so on Korvo-2 every preview frame is decoded in software, and on Korvo-1 the color conversion runs per frame instead. Either way it is a sensor-mode limit, not a missing player decoder.

### What changes if the camera can output RGB at LCD size?

Stay inside `tongyi_app_media.c` and do not patch `esp_cam_sensor`.

1. Enable the sensor's official RGB565 or RGB565_BE mode in `sdkconfig`, and set `TONGYI_VISION_CAMERA_WIDTH` and `TONGYI_VISION_CAMERA_HEIGHT` to that mode
2. In `vision_open()`, one capture stream is enough: `ESP_CAPTURE_FMT_ID_RGB565` or `ESP_CAPTURE_FMT_ID_RGB565_BE` at panel size. Drop the MJPEG stream, the JPEG drain in `vision_preview_task()`, and the JPEG cache
3. Keep writing RGB through `esp_video_render_stream_write()`. Match the LCD endianness (`RGB565_BE` on Korvo-2) so render skips the color conversion
4. In `tongyi_app_media_capture_jpeg()`, encode the last RGB frame with `esp_jpeg_enc` using `JPEG_PIXEL_FORMAT_RGB565_LE` or `RGB565_BE`. The `esp_video_enc` software MJPEG encoder does not accept RGB565
5. `esp_video_dec_register_default()` is only needed while capture still transcodes JPEG to RGB, so an RGB-only sensor can drop it

The ESP32-P4 SC2336 emits RAW rather than packed RGB, so that board still needs an ISP or conversion path and is outside this answer.

## Technical Support

- Forum: [esp32.com](https://esp32.com/viewforum.php?f=20)
- Issues: [GitHub issue](https://github.com/espressif/esp-adf/issues)

We will get back to you as soon as possible.