Packages

Fast Avro binary encoding/decoding for Erlang, implemented as a C++ NIF

Current section

Files

Jump to
erlav README.md
Raw

README.md

erlav
=====

High performance avro encoding/decoding NIF-based library

Build
-----

    $ rebar3 compile
    $ rebar3 eunit

Usage
-----

```erlang
SchemaId = erlav_nif:erlav_init(<<"priv/tschema2.avsc">>),
Term = #{
        <<"intField">> => 789,
        <<"longField">> => 2989898111,
        <<"doubleField">> => 11.2345,
        <<"floatField">> => 23.12,
        <<"boolField">> => true,
        <<"stringField">> => <<"asdadasdasdasd3453534dfgdgd123456789">>,
        <<"bytesField">> => <<1,99,57,127,0,56>>
},

% Encode Term to avro format
Ret = erlav_nif:erlav_encode(SchemaId, Term),

% Decode data in avro format to erlang term
DecodedTerm = erlav_nif:erlav_decode_fast(SchemaId, Ret)

```

### Negative block-count encoding

Avro allows array/map blocks to be encoded either as a plain positive item
count, or as a negative count followed by the block's byte length (used by
readers to skip a block without decoding it). `erlav_decode`/
`erlav_decode_fast` always understand both forms, so bytes produced by other
Avro implementations (e.g. `erlavro`, which emits the negative form by
default) decode correctly out of the box.

`erlav_encode` emits the plain positive-count form by default. To make it
emit the negative form instead -- e.g. to match `erlavro`'s byte output, or
for interop with a reader that expects it -- pass
`use_negative_block_count` when initializing the schema:

```erlang
SchemaId = erlav_nif:erlav_init(<<"priv/tschema2.avsc">>, [use_negative_block_count]),

% Every array/map field encoded with this SchemaId now uses the negative
% block-count form; decoding is unaffected either way.
Ret = erlav_nif:erlav_encode(SchemaId, Term),
DecodedTerm = erlav_nif:erlav_decode_fast(SchemaId, Ret)
```

Plain `erlav_init/1` (no options) keeps emitting today's positive-count
form, unchanged.

### Decoding long strings without copying: `decode_sub_binaries`

By default `erlav_decode_fast/2` copies every decoded string and `bytes` value
into a new binary. For values of 65 bytes or more the BEAM has to allocate a
reference-counted binary for each copy, which is most of the decode time for
string-heavy messages. Pass `decode_sub_binaries` to return those values as
sub-binaries of the input instead (no copy, no allocation):

```erlang
SchemaId = erlav_nif:erlav_init(<<"priv/tschema2.avsc">>, [decode_sub_binaries]),
Decoded = erlav_nif:erlav_decode_fast(SchemaId, Binary)
```

Measured on `array<string>` (100 strings per array), time per
`erlav_decode_fast/2` call (range over two runs):

| String length | Copy (default) | `decode_sub_binaries` | Speedup |
|---|---|---|---|
| 32 B | 4.7-5.2 us | 4.8-5.4 us | none (both copy) |
| 64 B | 6.6-6.8 us | 6.7-7.0 us | none (both copy) |
| 65 B | 16-17 us | 4.0-4.2 us | 3.9-4.2x |
| 136 B | 16.5-18.6 us | ~3.9 us | 4.3-4.7x |
| 500 B | 17.5-19 us | ~4.0 us | 4.4-4.8x |
| 2000 B | 25-26 us | 4.7 us | 5.3-5.7x |

Strings of 65 bytes or more decode about 4-5x faster, and the decode time stays
flat as strings get longer. Strings under 65 bytes are unchanged, because they
are still copied (a heap binary is cheaper than a sub-binary reference). This
is close to a best case, since the benchmark is almost pure string decoding;
messages that mix many other field types gain less. The decoded values compare
equal either way.

**The trade-off: memory.** A sub-binary keeps the *entire input binary* alive for
as long as any decoded value is referenced. If you keep one 136-byte string out
of a 1.4 MB message, the process holds on to the whole 1.4 MB until that string
is dropped. If your code stores slices of decoded messages long-term (a cache,
a long-lived process state), either leave this option off or copy what you keep
with `binary:copy/1`. (`erlavro`'s decoder behaves like the option: it returns
sub-binaries.) It is off by default for this reason.

Options are part of a schema's identity: the same file initialised with
different options gets a different `SchemaId`. `decode_sub_binaries` affects
`erlav_decode_fast/2` only; the legacy `erlav_decode/2` does not decode strings.


Performance
-----

run erlang shell

```bash
erl -pa _build/default/lib/*/ebin
```

run performance test ( it will generate report at the end )

```erlang
erlav_perf:all_tests().          %% encode and decode, one table
erlav_perf:all_encode_tests(10000, 50, null).
erlav_perf:all_decode_tests(10000, 50, null).
```

Every row times erlavro and erlav on the same schema and data and checks that
both produce the same result ( `Equal` ). Encode rows compare erlavro's
encoder with `erlav_encode/2`; `decode_*` rows compare erlavro's decoder with
`erlav_decode_fast/2` on binaries produced by erlavro. The CI `perf` job
(`ci/perf_check.sh`) runs the whole table and fails if any speedup drops below
a floor or any `Equal` is false.

Numbers below are one run on one machine ( Linux, GCC 14, 5000 iterations ) and
vary a lot between runs and machines; the ratio is what is meaningful.

```bash

+------------------------------+---------+--------------+--------------+----------+
| Test                         | Equal   | Erlavro us   | Erlav us     | Speedup  |
+------------------------------+---------+--------------+--------------+----------+
| erlav_perf_tst2              | true    |      1051.28 |       190.85 |    5.51x |
| erlav_perf_tst3              | true    |      1700.04 |       114.33 |   14.87x |
| map_perf_tst1                | true    |       174.90 |        53.39 |    3.28x |
| map_perf_tst2                | true    |       168.76 |        43.71 |    3.86x |
| array_int_perf_tst           | true    |        24.92 |         2.74 |    9.10x |
| array_str_perf_tst           | true    |       175.31 |        28.86 |    6.07x |
| array_map_perf_tst           | true    |       451.78 |        94.88 |    4.76x |
| decode_perf_tst2             | true    |       555.57 |        93.43 |    5.95x |
| decode_perf_tst3             | true    |      1127.38 |       169.36 |    6.66x |
| decode_map_perf_tst1         | true    |        60.08 |        17.01 |    3.53x |
| decode_map_perf_tst2         | true    |        81.91 |        32.46 |    2.52x |
| decode_array_int_perf_tst    | true    |        26.73 |         2.22 |   12.03x |
| decode_array_str_perf_tst    | true    |        42.11 |        23.24 |    1.81x |
| decode_array_map_perf_tst    | true    |       171.52 |        70.62 |    2.43x |
+------------------------------+---------+--------------+--------------+----------+

```