jkent/frogfs

readme

# About FrogFS

FrogFS (Fast Read-Only General-purpose File System) is a read-only filesystem
designed for embedded use. It can be easily used with a CMake project —
including [ESP-IDF](https://github.com/espressif/esp-idf). It has built-in
filters to save space. Files in `frogfs-clockwise-demo` are reduced by about
half using the default filters. Here are links to the example repositories:

  * [frogfs-standalone-demo](https://github.com/jkent/frogfs-standalone-demo)
  * [frogfs-clockwise-demo](https://github.com/jkent/frogfs-clockwise-demo)

Transforms include:
  * babel-convert
  * babel-minify
  * html-minifier
  * terminate
  * uglify-js
  * uglifycss

Compression options include:
  * none
  * [zlib deflate](https://www.zlib.net/) (best compression)
  * [heatshrink](https://github.com/atomicobject/heatshrink) (best speed)

For an HTTP server, deflate compressed files can even be passed through
untouched! This saves both processing time and bandwidth.

# Getting started with ESP-IDF

To use this component with ESP-IDF, within your projects directory run

    idf.py add-dependency jkent/frogfs

## Embedding a FrogFS image

Embed FrogFS within your project binary with the folowing CMake function:

    target_add_frogfs(<target> [CONFIG yaml] [NAME name])

If **CONFIG** is not specified, `frogfs.yaml` will be used. If **NAME** is not
specified, it will default to `frogfs`.

As an example for ESP-IDF, in your project's toplevel CMakeLists.txt:

```cmake
cmake_minimum_required(VERSION 3.16)

include($ENV{IDF_PATH}/tools/cmake/project.cmake)
project(my_project)

target_add_frogfs(my_project.elf)
```

Where **target** `my_project.elf` must match your project's name and
**path** `files` is either an absolute path or a path relative to your
project's root. It should contain the files to embed in the resulting frogfs
binary. In C, this results in these two global symbols being available to your
application:

```C
extern const uint8_t frogfs_bin[];
extern const size_t frogfs_bin_len;
```

## Making a FrogFS binary and flashing it

You have the option of creating a binary without linking it with your
application. A CMake function is provided to output a binary with target
`generate_${name}`.

    declare_frogfs_bin(path [CONFIG yaml] [NAME name])

If **CONFIG** is not specified, `frogfs.yaml` is used. If **NAME** is not
specifed, `frogfs` is used.

Here's an example of what you can add to your toplevel CMakeLists.txt:

```cmake
set(FROGFS_NAME frogfs)
declare_frogfs_bin(NAME ${FROGFS_NAME})

idf_component_get_property(main_args esptool_py FLASH_ARGS)
idf_component_get_property(sub_args esptool_py FLASH_SUB_ARGS)
esptool_py_flash_target(${FROGFS_NAME}-flash "${main_args}" "${sub_args}" ALWAYS_PLAINTEXT)

esptool_py_flash_to_partition(${FROGFS_NAME}-flash storage ${BUILD_DIR}/CMakeFiles/${FROGFS_NAME}.bin)
add_dependencies(${FROGFS_NAME}-flash generate_${FROGFS_NAME}_bin)
```

In this case, **files** is the source directory to build the file system from,
**frogfs** is the target prefix and binary filename (without the .bin) and
**storage** is the name of the partition where the binary is flashed. You can
invoke the flash process by running `idf.py frogfs-flash`.

## Configuration

FrogFS expects a yaml configuration file.  There are 3 different sections:
define, collect and filter. All but collect is optional.

Define is a list or dict of variable definitions. There are 2 predefined
variables: `$cwd` and `$frogfs`. You can also reference environment variables
with the `${ENV:varname}` syntax.

Collect 'gathers' up files and directories and places them in the frogfs root.
Glob patterns are allowed in the 'basename' component of the path. There are 3
ways to specify sources; they cn be a string, list, or dictionary. If it's a
string, the path(s) become the root directory. If a list, the paths are merged
in order and become the root directory. If a dict is used, the paths are
merged into the destination of choice; empty string being the root directory.
Variables are expanded for both source and destination.

Filter allows you to do post-processing on the files before they are
integrated. Filter is a list or dict of dicts; with a glob pattern to a list
of verbs. Varibales are expanded and all patterns are evaluated for each file
or directory, top down. Transforms are applied first, then an optional final
compression before caching the file.

Verbs are applied in descending order. You can prefix a transforms or the
`compress` verb with `no` to disable it. There are a couple of special verbs:
`discard` which prevents inclusion and `cache` (default), which caches the
file in the build cache. See `frogfs_example.yaml` for example usage.

## Usage

Two interfaces are available: the [bare API](#bare-api) or when using IDF
there is the [VFS interface](#vfs-interface) which builds on top of the bare
API. You should use the VFS interface in IDF projects, as it uses the portable
and familiar `posix` and `stdio` C functions with it. There is nothing
preventing you from mix and matching both at the same time, however.

### Shared initialization

Configuration requries defining a `frogfs_config_t` structure and passing it
to `frogfs_init`. Two different ways to specify the filesystem:

  1. a memory address using the `addr` variable:

```C
frogfs_config_t frogfs_config = {
    .addr = frogfs_bin,
};
```

 2. a partition name using the `part_label` string:

```C
frogfs_config_t frogfs_config = {
    .part_label = "storage",
};
```

Then it is just a matter of passing the `frogfs_config` to `frogfs_init`
function and checking its return variable:

```C
frogfs_fs_t *fs = frogfs_init(&frogfs_config);
assert(fs != NULL);
```

When done, and all file handles are closed, you can call `frogfs_deinit`:

```C
frogfs_deinit(fs);
```

### VFS interface

The VFS interface has a similar method of initialization; you define a
`frogfs_vfs_conf_t` structure:

  * **base_path** - path to mount FrogFS
  * **fs** - a `frogfs_fs_t` instance
  * **max_files** - max number of files that can be open at a time

```C
frogfs_vfs_conf_t frogfs_vfs_conf = {
    .base_path = "/frogfs",
    .fs = fs,
    .max_files = 5,
};
frogfs_vfs_register(&frogfs_vfs_conf);
```

### Bare API

#### Filesystem functions:

  * frogfs_fs_t *[frogfs_init](https://frogfs.readthedocs.io/en/latest/api-reference/bare.html#c.frogfs_init)(const frogfs_config_t *conf)
  * void [frogfs_deinit](https://frogfs.readthedocs.io/en/latest/api-reference/bare.html#c.frogfs_deinit)(frogfs_fs_t *fs)

#### Object functions:

  * const frogfs_entry_t *[frogfs_get_entry](https://frogfs.readthedocs.io/en/latest/api-reference/bare.html#c.frogfs_get_entry)(const frogfs_fs_t *fs, const char *path)
  * char *[frogfs_get_name](https://frogfs.readthedocs.io/en/latest/api-reference/bare.html#c.frogfs_get_name)(const frogfs_entry_t *entry)
  * char *[frogfs_get_path](https://frogfs.readthedocs.io/en/latest/api-reference/bare.html#c.frogfs_get_path)(const frogfs_fs_t *fs, const frogfs_entry_t *entry)
  * int [frogfs_is_dir](https://frogfs.readthedocs.io/en/latest/api-reference/bare.html#c.frogfs_is_dir)(const frogfs_entry_t *entry)
  * int [frogfs_is_file](https://frogfs.readthedocs.io/en/latest/api-reference/bare.html#c.frogfs_is_file)(const frogfs_entry_t *entry)
  * void [frogfs_stat](https://frogfs.readthedocs.io/en/latest/api-reference/bare.html#c.frogfs_stat)(const frogfs_fs_t *fs, const frogfs_entry_t *entry, frogfs_stat_t *st)
  * frogfs_fh_t *[frogfs_open](https://frogfs.readthedocs.io/en/latest/api-reference/bare.html#c.frogfs_open)(const frogfs_fs_t *fs, const frogfs_entry_t *entry, unsigned int flags)
  * void [frogfs_close](https://frogfs.readthedocs.io/en/latest/api-reference/bare.html#c.frogfs_close)(frogfs_fh_t *fh)
  * int [frogfs_is_raw](https://frogfs.readthedocs.io/en/latest/api-reference/bare.html#c.frogfs_is_raw)(frogfs_fh_t *fh)
  * size_t [frogfs_read](https://frogfs.readthedocs.io/en/latest/api-reference/bare.html#c.frogfs_read)(frogfs_fh_t *fh, void *buf, size_t len)
  * ssize_t [frogfs_seek](https://frogfs.readthedocs.io/en/latest/api-reference/bare.html#c.frogfs_seek)(frogfs_fh_t *fh, long offset, int mode)
  * size_t [frogfs_tell](https://frogfs.readthedocs.io/en/latest/api-reference/bare.html#c.frogfs_tell)(frogfs_fh_t *fh)
  * size_t [frogfs_access](https://frogfs.readthedocs.io/en/latest/api-reference/bare.html#c.frogfs_access)(frogfs_fh_t *fh, void **buf)

#### Directory Functions:

  * frogfs_dh_t *[frogfs_opendir](https://frogfs.readthedocs.io/en/latest/api-reference/bare.html#c.frogfs_opendir)(frogfs_fs_t *fs, const frogfs_entry_t *entry)
  * void [frogfs_closedir](https://frogfs.readthedocs.io/en/latest/api-reference/bare.html#c.frogfs_closedir)(frogfs_dh_t *dh)
  * const frogfs_entry_t *[frogfs_readdir](https://frogfs.readthedocs.io/en/latest/api-reference/bare.html#c.frogfs_readdir)(frogfs_dh_t *dh)
  * void [frogfs_seekdir](https://frogfs.readthedocs.io/en/latest/api-reference/bare.html#c.frogfs_seekdir)(frogfs_dh_t *dh, long loc)
  * long [frogfs_telldir](https://frogfs.readthedocs.io/en/latest/api-reference/bare.html#c.frogfs_telldir)(frogfs_dh_t *dh)

# How it works

Under the hood there is a hash table consisting of djb2 path hashes to entry
offsets, which allow for fast lookups using a binary search algorithm. All
entries except the root entry have a parent locator offset. Directory entries
have a sorted list of offsets to child entries.

FrogFS binaries can be either embedded in your application, or accessed using
memory mapped I/O. It is not possible (at this time) to use FrogFS without the
file system binary existing in data address space.

Creation of a FrogFS filesystem is handled by a single tool,
`tools/mkfrogfs.py`. It uses transforms in the `tools` directory, or you can
add your own transforms by creating a `tools` directory in your projects root
directory, with a filename starting with `transform-` and ending with `.js` or
`.py`. Transform tools take data on stdin and produce output on stdout.

Both transform and compresors can accept arguments. See `frogfs_example.yaml`
for an example.

# History and Acknowledgements

FrogFS was split off of Chris Morgan (chmorgan)'s
[libesphttpd](https://github.com/chmorgan/libesphttpd/) project (MPL 2.0),
which is a fork of Jeroen Domburg (Sprite_tm)'s
[libesphttpd](https://github.com/spritetm/libesphttpd/) (BEER-WARE). This
project would never have existed without them.

Thank you to all the contributors to this project!