espressif/esp_media_service

Readme

# ESP Media Service

- [![Component Registry](https://components.espressif.com/components/espressif/esp_media_service/badge.svg)](https://components.espressif.com/components/espressif/esp_media_service)
- [中文版](./README_CN.md)

`esp_media_service` provides a common media interface for audio and video services in ESP-ADF.

Applications create services, configure their media streams, and link source streams to sink streams. After services are linked and started, media frames flow through the shared provider and track APIs instead of being forwarded manually by application code.

## Features

- Unified audio and video service model
- Source, sink, and source-sink service roles
- Stream-based media endpoints (`esp_media_stream_id_t`)
- Link-time request negotiation between sinks and sources
- Provider read APIs and track-manager write APIs
- Default in-memory track manager for queued media frames
- Consistent service lifecycle through `esp_service`

## Agent Guidelines

For implementation and review guidance, see [agent.md](agent.md). It covers service model details, link flow, track manager behavior, stop/abort rules, and a review checklist.

## Service Roles

A media service can be one of these roles:

- `ESP_MEDIA_ROLE_SRC`: produces media frames.
- `ESP_MEDIA_ROLE_SINK`: consumes media frames.
- `ESP_MEDIA_ROLE_SRC_SINK`: consumes and produces media frames.

Services can implement `get_role()` to let `esp_media_service_link()` validate that the selected source and sink are compatible.

## Streams

Services communicate through stream IDs:

```c
typedef uint16_t esp_media_stream_id_t;
#define ESP_MEDIA_DEFAULT_STREAM  ((esp_media_stream_id_t)0)
```

One stream is a series of tracks. Use `ESP_MEDIA_DEFAULT_STREAM` for the first stream:

```c
esp_media_stream_id_t stream = ESP_MEDIA_DEFAULT_STREAM;
esp_media_service_link(src_service, stream, sink_service, stream);
```

## Typical Usage

Link a source service to a sink service, then start both services:

```c
esp_media_stream_id_t stream = ESP_MEDIA_DEFAULT_STREAM;

esp_media_service_link(src_service, stream, sink_service, stream);

esp_service_start(sink_service);
esp_service_start(src_service);
```

After the services are linked and started, media data flows from the source to the sink through the provider supplied by the source.

Typical lifecycle:

```mermaid
flowchart TD
    A[Create services] --> B[Configure services]
    B --> C[Link source and sink streams]
    C --> D[Start services]
    D --> E[Transfer media frames]
    E --> F[Stop services]
    F --> G[Unlink and destroy services]
```

## Provider And Track APIs

Media data uses two interfaces:

- `esp_media_provider_t` (read): queries tracks, receives events, acquires or reads frames, releases frames, aborts blocking reads. Defined in `esp_media_provider.h`.
- `esp_media_track_mngr_t` (write): manages track queues. Write APIs are in `esp_media_track.h`.

A sink receives an `esp_media_provider_t` from an upstream source during link. A source writes frames through `esp_media_track_write_frame()`.

Frames acquired with `esp_media_provider_acquire_frame()` must always be released with `esp_media_provider_release_frame()`. Do not use `frame.data` after the frame is released.

## Track Manager

`esp_media_track_mngr_t` is the default in-memory frame store. It exposes one provider handle and manages queues for one or more tracks.

It supports two payload ownership modes:

- `ESP_MEDIA_TRACK_CACHE_INTERNAL`: the manager copies and owns frame payload data.
- `ESP_MEDIA_TRACK_CACHE_USER`: the manager queues frame metadata only; user-owned payloads are returned through `frame_release`.

It also supports global cache, where all tracks share one arrival-order queue. This is useful for interleaved audio/video transports such as RTMP. Enable global cache before adding tracks.

## Implementing A Source Service

A source usually owns an `esp_media_track_mngr_t`:

```c
esp_media_track_mngr_cfg_t cfg = {
    .max_track_num = 2,
};

esp_media_track_mngr_create(&cfg, &svc->mngr);
esp_media_track_mngr_add_track(svc->mngr, &audio_track);
esp_media_track_mngr_add_track(svc->mngr, &video_track);
esp_media_track_mngr_get_provider(svc->mngr, &svc->provider);
```

The source implements `get_provider()` and writes produced frames with `esp_media_track_write_frame()`.

## Implementing A Sink Service

A sink implements `set_provider()` and stores the upstream provider:

```c
static esp_err_t my_sink_set_provider(esp_service_t *service,
                                      esp_media_stream_id_t stream,
                                      const esp_media_provider_t *provider)
{
    my_sink_t *sink = (my_sink_t *)service;

    if (provider == NULL) {
        sink->provider.ops = NULL;
        sink->provider.ctx = NULL;
        return ESP_OK;
    }

    sink->provider = *provider;
    return esp_media_provider_set_event_cb(&sink->provider, my_event_cb, sink);
}
```

The sink task reads frames and releases every acquired frame:

```c
while (!sink->stop) {
    esp_media_frame_t frame = {0};
    if (esp_media_provider_acquire_frame(&sink->provider, &frame, timeout_ms) == ESP_OK) {
        process_frame(&frame);
        esp_media_provider_release_frame(&sink->provider, &frame);
    }
}
```

## Stop And Abort

The media interfaces are designed so stop order is not fragile:

- Source stop should call `esp_media_track_write_abort()` to notify downstream providers with `ESP_MEDIA_PROVIDER_EVENT_TRACKS_ABORT`.
- Sink stop should set a local stop flag, call `esp_media_provider_abort()`, wait for tasks to exit, release acquired frames, and unlink.
- Unlink services before resetting or destroying a track manager that was shared through a link.
- Do not reset or destroy a track manager while another task may still hold an acquired frame or block on its queue.

## Debug Tips

Use `esp_media_dummy_service` to simulate a camera/source or a display/consumer without real hardware. Enable it with:

- `CONFIG_ESP_MEDIA_DUMMY_SERVICE_SRC_SUPPORT` — dummy source (pattern producer)
- `CONFIG_ESP_MEDIA_DUMMY_SERVICE_SINK_SUPPORT` — dummy sink (frame consumer + stats)

Typical uses:

- **Dummy source**: replace a camera/mic while bringing up capture, encode, mux, or link pipelines.
- **Dummy sink**: replace a display/renderer and verify frame flow with `esp_media_dummy_service_get_stats()`.

Supported pattern codecs:

| Kind | Codecs |
| --- | --- |
| Audio | PCM, AAC, OPUS, MP3 |
| Video | H264, MJPEG, RGB565, RGB888, YUV420 |

Encoded tracks only need the codec field in `add_track()`; raw PCM / raw video need full track metadata.

## MCP Tools

When `CONFIG_ESP_MEDIA_SERVICE_MCP_ENABLE=y` (depends on `CONFIG_ESP_MCP_ENABLE`):

- Global tools `esp_media_service_link` / `esp_media_service_unlink` resolve services by name from a persistent table filled on `esp_media_service_init()`.
- Call `esp_media_service_mcp_register(mgr)` once to bind those tools to a service manager (idempotent, not unbound on service destroy).
- Dummy sink tools include `esp_media_dummy_service_start` / `stop` / `get_stats` for PC verification of linked capture flows.

Capture examples under `esp_audio_capture_service` and `esp_video_capture_service` show UART MCP wiring.

## Implement

Built-on services that use `esp_media_service`. Start with the board-aware recorders for typical apps; call lower layers directly when you need custom sources or pipeline control.

| Service | Role | Features | Board / notes |
| --- | --- | --- | --- |
| [`esp_audio_capture_service`](../esp_audio_capture_service/README.md) | Audio recorder | Multi-stream, AI (AEC/VAD/WakeNet), muxer/storage | ADC via board manager |
| [`esp_video_capture_service`](../esp_video_capture_service/README.md) | A/V recorder | Multi-stream, A/V sync, muxer/storage | Camera via board manager; audio via `esp_audio_capture_service` |
| [`esp_capture_service`](../esp_capture_service/README.md) | Capture base | Common capture API (audio/video), no board binding | Advanced usage: call directly with your own sources, overlay etc |

## Technical Support

For technical support, use the links below:

- Technical support: [esp32.com](https://esp32.com/viewforum.php?f=20) forum
- Issue reports and feature requests: [GitHub issue](https://github.com/espressif/esp-adf/issues)

We will reply as soon as possible.