# lowzag on ESP-IDF
lowzag decodes progressive (SOF2) JPEG at 1/1, 1/2, 1/4 or 1/8 in a single
forward pass over the file. A progressive decoder cannot finish any block
until the last scan has been read, so it must hold coefficient storage for the
whole frame, and every general-purpose decoder surveyed sizes that storage at
the full 64 coefficients per block no matter which scale you asked for. lowzag
keeps only the low-frequency corner that a reduced-scale IDCT will read. On a
12 MP 4:2:0 progressive file its peak is 23.0 / 8.9 / 3.6 MB at 1/2 / 1/4 /
1/8, against libjpeg-turbo's ~36 MB coefficient array at every scale.
Storage still scales with the *source* image, not with the output: an 800x480
render of a 48 MP photograph costs more than an 800x480 render of a 12 MP one.
What the scale reduces is the per-source-block constant — measured for 4:2:0,
81.6 / 31.4 / 12.7 bytes per source block at 1/2 / 1/4 / 1/8, against
libjpeg's fixed 128-byte JBLOCK at every scale. Budget from the source
dimensions in the file header, not from the size you want on screen. Nothing
here fits in internal SRAM at megapixel sizes; PSRAM is the assumption.
If 1/8 is all you need, look at [bitbank2/JPEGDEC][jpegdec] first. Its DC-only
progressive path does 1/8 in about 18 KB of memory that is *constant* — it
does not grow with the image, where lowzag's does. On that 12 MP file its
18 KB is 200x less than lowzag's 3.6 MB, and on a big enough file the ratio
is unbounded. The catch is that the path is forced to 1/8 for any requested
scale, so 1/1, 1/2 and 1/4 progressive return success with silently wrong
output. Those three scales, within a bounded budget, are what lowzag adds.
[jpegdec]: https://github.com/bitbank2/JPEGDEC
## Depending on lowzag
lowzag is on the ESP Component Registry as `defensiblelogic/lowzag`:
```sh
idf.py add-dependency "defensiblelogic/lowzag^2.2.1"
```
which writes it into your project's `main/idf_component.yml`, where you can
equally write it by hand (create that file if you don't have one):
```yaml
dependencies:
idf: ">=5.0"
defensiblelogic/lowzag: "^2.2.1"
```
`idf.py build` unpacks it into `managed_components/defensiblelogic__lowzag/`,
reports it under `NOTICE: Processing N dependencies`, and records the version
it solved for in `dependencies.lock` so later builds are reproducible. `^2.2.1`
is a semver range — any 2.x from 2.2.1 up, newest that fits; `"=2.2.1"` if you
want exactly one version and nothing else.
Nothing goes into a `CMakeLists.txt`. The component manager adds the dependency
to the requirements of the component whose manifest declared it, so
`#include "lowzag.h"` works from `main` with no further edit — the
`main/CMakeLists.txt` here names only `esp_timer`, which is the one requirement
no manifest declares. Do not add `lowzag` to `REQUIRES` yourself: under a
registry install the build-system name is `defensiblelogic__lowzag`, so the
short name names nothing that is in the build. It does not fail today — the
component manager rewrites a bare `lowzag` to the namespaced name, checked on
ESP-IDF 5.4.1 — but it is a redundant line that depends on that rewrite, and
the requirement it duplicates is already there.
### From git instead
To track a branch, or to build a commit that has no release, name the
repository rather than the registry:
```yaml
dependencies:
idf: ">=5.0"
lowzag:
git: https://gitlab.defensiblelogic.com/pub/lowzag.git
version: "main"
```
That clones into `managed_components/lowzag/` — no namespace on this route, so
the directory and the build-system name are both `lowzag` — and records the
commit the ref resolved to in `dependencies.lock`. The requirement is injected
here too; there is still nothing to add to `REQUIRES`.
For a git dependency, `version` is a raw git ref, not a version range. The
component manager hands the string to git as the thing to check out, so a tag,
a branch name or a commit SHA all work and the semver syntax that applies to
registry dependencies does not:
```yaml
version: "v2.2.1" # a tag — a release, the long way round
version: "main" # track a branch, re-resolved on every solve
version: "0123456789abcdef0123456789abcdef01234567" # pin a commit
```
`"^1.0"` and `"~1.0.0"` are rejected outright — `^` and `~` are not legal in a
git ref name. `">=1.0.0"` is worse, because it *is* a syntactically legal ref
name: it passes manifest validation and then fails at checkout with `Git
reference ">=1.0.0" doesn't exist in the repository`. `"1.0.0"` fails the same
way unless a tag is literally named `1.0.0`; lowzag's tags carry the `v`
prefix. There is no version solving here — whatever ref you name is what you
get, and the lock file pins the commit it resolved to.
The contrast is with a registry dependency in the same file, where `version`
*is* a range and does get solved:
```yaml
dependencies:
espressif/esp_jpeg: "^1.3.1" # registry: semver range
lowzag:
git: https://gitlab.defensiblelogic.com/pub/lowzag.git
version: "main" # git: a ref, verbatim
```
`defensiblelogic/lowzag: "^2.2.1"` at the top of this page is the first of
those two forms, which is why it is the shorter one.
Two other keys:
```yaml
lowzag:
git: https://gitlab.defensiblelogic.com/pub/lowzag.git
version: "main"
path: "." # subdirectory holding the component; "." is the
# default and is correct for lowzag
override_path: "../lowzag" # ignore the source above and use a local
# checkout, for working on both at once
```
`override_path` is not specific to git dependencies — it overrides a registry
dependency the same way, and `main/idf_component.yml` in this example uses it
against `defensiblelogic/lowzag`. Building this example, below, is that case.
## Building this example
```
idf.py set-target esp32s3
idf.py build
idf.py -p PORT flash monitor
```
It goes through the component manager like any other project. The one
difference is that `main/idf_component.yml` here carries an `override_path`
alongside the version range:
```yaml
defensiblelogic/lowzag:
version: "^2.2.1"
override_path: "../../../"
```
so an in-tree build resolves `defensiblelogic/lowzag` to the checkout this file
is sitting in rather than downloading a release. The registry strips
`override_path` from the copy of the example it serves, so the same project
taken from the component page resolves the dependency out of the registry
instead, with nothing to edit. `main/CMakeLists.txt` is the same file either
way, because the dependency is declared in one manifest and named in no
`CMakeLists.txt` at all.
ESP-IDF takes a component's name from its directory, so the in-tree route needs
this checkout to be called `lowzag` (or `defensiblelogic__lowzag`); cloned
under some other name, `override_path` points at a component the build then
cannot match to the dependency. That applies to building from the tree and
nowhere else — a consumer gets the name from the registry.
The decoder is portable C99 with no target-specific code, so this builds for
any ESP-IDF target; esp32s3 is the one it was developed against. See the
memory note below before choosing.
## What it does
`main/sample.jpg` is a 640x480 4:2:0 progressive JPEG, 13895 bytes, linked
into the firmware with `EMBED_FILES`. It is a test card — a colour gradient,
two discs, a rotated quad and a strip of patches — drawn by
`main/gen_sample.py` beside it, so it is original to this project and
reproducible from the tree; `test/check_images.py` regenerates it in CI and
compares byte for byte. It carries a small EXIF APP1 saying orientation 6 —
not because a test card has a right way up, but because orientation is the one
piece of metadata almost every caller of the marker passthrough wants, and the
example is better off reading a well-formed EXIF tag than an invented marker.
`main/gen_sample.py` splices those 36 bytes in and writes out what they are.
`app_main()` calls `lowzag_probe()` on the header to confirm the file is
progressive, then decodes it against a 320x240 target and logs the source and
output dimensions with the scale chosen, `peak_alloc` against the budget it
was given, the row count with the decode time, the worst gap between
cancellation polls, and the orientation. The probe is the dispatch point where
a real application would hand a baseline file to a baseline decoder instead.
Then it decodes the same file a second time over a stream that stops two
thirds of the way in — an SD card pulled mid-read, a connection dropped —
with `cfg.allow_partial` set, which renders the scans that arrived instead of
returning `LOWZAG_ERR_TRUNCATED` and nothing. Both features are 2.2.0
additions and both are zero in a config that predates them.
The EXIF capture is worth reading before writing your own. The buffer is 512
bytes, fixed, and the callback returns `LOWZAG_MARKER_SKIP` once it is full:
a camera APP1 runs to tens of kilobytes, most of it a thumbnail, and this part
does not have that to spare. Orientation lives in IFD0 at the front of the
block, so keeping the front of it is enough — and if it is not, the answer is
0 and the picture is shown as it is. Nothing in that path allocates.
On a CrowPanel ESP32-S3 with no PSRAM configured, ESP-IDF v5.4.1:
```
I (279) lowzag: progressive JPEG 640x480, 13895 bytes of file
I (489) lowzag: 640x480 -> 160x120 at 1/4, 3 component(s)
I (489) lowzag: peak_alloc 224 KB of a 299 KB budget; state 17144 B internal
I (489) lowzag: 120 rows in 212 ms (byte sum 7415438)
I (499) lowzag: 483 cancel polls, worst gap 2.400 ms
I (499) lowzag: EXIF orientation 6 (APP1 32 bytes, kept 32)
I (649) lowzag: short stream: 120 of 120 rows... all of them, from 7 scans of 10, 9263 of 13895 bytes
I (649) lowzag: DC coverage 100% (no empty blocks, just softer), 139 ms
```
1/4 rather than the 1/2 the 320x240 target asks for, because without PSRAM the
internal heap has 299 KB to give and 1/2 wants 596 KB — the step-down
described under Memory below, happening. The short-stream decode got seven of
the file's ten scans out of two thirds of its bytes, and every one of the 120
rows: a partial decode gives up detail, never rows. Its DC coverage is 100%,
so no block is empty and the picture is only softer. Cut the stream further
and that number falls, and the bottom of the image goes flat grey.
There is no display code. The row callback counts rows and sums bytes; a real
application would blit each row to a panel. What the example does show is the
allocation split, the reason lowzag takes callbacks at all:
- **Working state** — Huffman and colour tables and the read buffer, touched on
every block. `lowzag_state_size()` bytes, about 17 KB — call it on your own build rather than copying a number. `main.c` places
it by hand with `heap_caps_calloc(1, ..., MALLOC_CAP_INTERNAL | MALLOC_CAP_8BIT)`
so it stays in internal RAM. `MALLOC_CAP_8BIT` matters: internal RAM alone
can hand back a region that is only 32-bit addressable, and the state holds
byte arrays. `calloc` and not `malloc` matters too: lowzag reads the
reentrancy sentinel out of that buffer before it writes to it, so it has to
start initialised. Once is enough — the same buffer then feeds decode after
decode.
- **Everything else** — coefficient storage and the row/strip buffers, the
allocations that scale with the source image. These come from `cfg.alloc`,
which the example points at `MALLOC_CAP_SPIRAM`. They are large and cold;
external RAM is where they belong.
Leaving `cfg.state` NULL is legal: the working state then comes from
`cfg.alloc()` along with everything else. On a PSRAM part that costs
inner-loop speed for no reason.
## Memory
`cfg.mem_limit` is not only a failure threshold. Under automatic scale
selection, a chosen scale that would exceed the limit makes lowzag drop to the
next scale down and decode a smaller image rather than fail. Set it too low
and you silently lose resolution — check `info.scale`. Set it too high and it
defeats the mechanism, because lowzag believes the frame fits and then dies on
a real allocation. The example asks the heap what is actually free and
subtracts a reserve, which is the honest version of this.
`lowzag_info_t.peak_alloc` for `main/sample.jpg`, measured with the host test
driver (`test/lowzag_cli.c`), in bytes:
| scale | output | peak_alloc |
|-------|---------|-----------:|
| 1/1 | 640x480 | 942,720 |
| 1/2 | 320x240 | 596,160 |
| 1/4 | 160x120 | 228,960 |
| 1/8 | 80x60 | 92,400 |
At the 320x240 target the example asks for, automatic selection picks 1/2, so
plan on roughly 600 KB. An ESP32-S3 without PSRAM has nowhere near that free
in one piece: without `CONFIG_SPIRAM` the example falls back to
`MALLOC_CAP_DEFAULT`, and the budget it computes from the internal heap will
not cover 1/2, so expect it to step down to 1/4 — or to report
`LOWZAG_ERR_NOMEM` if the internal heap is too fragmented to hand out the
blocks even then. It builds either way; enable PSRAM if you want the scale you
asked for. For real camera JPEGs at several megapixels PSRAM is not optional —
read the source dimensions from the header first and decide whether you can
afford the decode at all.
## Limits
Progressive only. `lowzag_decode()` returns `LOWZAG_ERR_NOT_PROGRESSIVE` for a
baseline file; use `lowzag_probe()` to route those to a baseline decoder — the
`espressif/esp_jpeg` registry component, or TJpgDec in ROM. 8-bit precision,
1 or 3 components, sampling factors 1 and 2, restart markers supported.
Arithmetic coding, 12-bit, CMYK/Adobe and lossless JPEG are out of scope.
Progressive is the more expensive format to parse — every scan is entropy
decoded, and none can be skipped, because refinement scans depend on the
nonzero history of the ones before them. Measured on an ESP32-S3 at 240 MHz
targeting an 800x480 panel, against the ROM TJpgDec baseline decoder on the
same image re-encoded as baseline, both given the same power-of-two reduction:
0.63 s vs 0.46 s from an 800x480 source at 1/1, 1.75 s vs 1.95 s from
1600x1200 at 1/2, and 5.7 s vs 9.1 s from 4000x3000 at 1/4. The sources are
DualCrow's test cards at those three sizes. Baseline wins at panel size; they
cross around a megapixel. Single measurements from one board and one set of
test cards, not a benchmark suite.
To create a project from this example, run:
idf.py create-project-from-example "defensiblelogic/lowzag=2.2.1:esp32"