rocksdb

The Erlang Rocksdb binding provides access to most of the key/value API of the rocksdb library. 

## Installation

Download the sources from our [Github repository](https://github.com/EnkiMultimedia/erlang-rocksdb)

To build the application simply run 'rebar3 compile'. 

> Note: `cmake>=3.12` is required to install `erlang-rocksdb`.

To run tests run 'rebar3 eunit'. To generate doc, run 'rebar3 ex_doc'.

Or add it to your rebar config


```erlang
{deps, [
    ....
    {rocksdb, "3.1.2"}
]}.
```

Or your mix config file:

```
{:rocksdb, "~> 3.1"}
```

## Basic operations


### Open a database

```erlang

Path = "/tmp/erocksdb.fold.test",
Options = [{create_if_missing, true}],
{ok, DB} = rocksdb:open(Path, Options).

```

This gives you a nif resource, the database handle.

> Keep the handle reachable for as long as you need the database: hold it in a live process, in ETS or in `persistent_term`. Once nothing references it any more, it is garbage collected, which closes the database and everything tied to it (iterators, column families). See [OTP Integration](otp_integration.md) for where to keep it in an application.


### Closing a database

When you are done with a database run the following function

```erlang
ok = rocksdb:close(DB).
```

> The database is also closed for you once nothing holds the handle any more.

### Read and Writes

You modify and query the database with `put`, `get` and `delete`. Each of them takes read or write options as its last argument, so pass `[]` when you have none. This writes a key, reads it back, then deletes it:

```erlang
ok = rocksdb:put(Db, <<"my key">>, <<"my value">>, []),
case rocksdb:get(Db, <<"my key">>, []) of
  {ok, Value} -> io:format("retrieved value ~p~n", [Value]);
  not_found -> io:format("value not found~n", []);
  Error -> io:format("operational problem encountered: ~p~n", [Error])
end,
ok = rocksdb:delete(Db, <<"my key">>, []).
```

### Batch Reads with multi_get

When you need to retrieve multiple keys at once, `multi_get/3` is more efficient than calling `get/3` multiple times. It retrieves all keys in a single operation:

```erlang
%% Insert some data
ok = rocksdb:put(Db, <<"key1">>, <<"value1">>, []),
ok = rocksdb:put(Db, <<"key2">>, <<"value2">>, []),
ok = rocksdb:put(Db, <<"key3">>, <<"value3">>, []),

%% Retrieve multiple keys at once
Keys = [<<"key1">>, <<"key2">>, <<"key3">>, <<"missing">>],
Results = rocksdb:multi_get(Db, Keys, []),
%% Results = [{ok, <<"value1">>}, {ok, <<"value2">>}, {ok, <<"value3">>}, not_found]
```

The results are returned in the same order as the input keys. Each result is either:
- `{ok, Value}` - the key was found
- `not_found` - the key does not exist
- `{error, Reason}` - an error occurred

For column families, use `multi_get/4`:

```erlang
Results = rocksdb:multi_get(Db, ColumnFamily, Keys, []).
```

You can also use snapshots with `multi_get` to get a consistent view:

```erlang
{ok, Snapshot} = rocksdb:snapshot(Db),
Results = rocksdb:multi_get(Db, Keys, [{snapshot, Snapshot}]),
rocksdb:release_snapshot(Snapshot).
```

### Atomic Updates

If your process dies between two writes, you can be left with the same value stored under several keys. Collect the writes in a batch and apply them in one atomic operation instead:

```erlang
{ok, Batch} = rocksdb:batch(),
ok = rocksdb:batch_put(Batch, <<"key1">>, <<"value1">>),
ok = rocksdb:batch_delete(Batch, <<"keytodelete">>),
ok = rocksdb:write_batch(Db, Batch, []),
ok = rocksdb:release_batch(Batch).
```

See the [Batch API](batch_api.md) guide for savepoints, rollbacks and the other batch operations.

### Synchronous writes

Using the setting `{sync, true}` in any write operations will makes sure to store synchronously on the filesystem your data. 


### Iterate your data

You can start an iterator using the function `rocksdb:iterator/2` :

```erlang
ItrOptions = [],
{ok, Itr} = rocksdb:iterator(DB, ItrOptions)
```

You close the iterator using the function `rocksdb:iterator_close/1`:
```erlang
rocksdb:iterator_close(Itr)
```

You move the iterator with `rocksdb:iterator_move/2`:

* move to the next key: `rocksdb:iterator_move(Itr, next)`
* move to the previous key: `rocksdb:iterator_move(Itr, prev)`
* start at the first key: `rocksdb:iterator_move(Itr, first)`
* start at the last key: `rocksdb:iterator_move(Itr, last)`
* move to a key: `rocksdb:iterator_move(Itr, Key)` or  `rocksdb:iterator_move(Itr,{seek, Key})`
* move to the previous key of: `rocksdb:iterator_move(Itr,{seek_for_prev, Key})`

> An iterator doesn't support parallel operations, so make sure to use it accordingly.

You can optimize prefix seeks by putting your database in "prefix seek" mode with the `prefix_extractor` option on the database or the column family. See [Prefix Seek](prefix_seek.html) for more information.

### Wide-Column Entities

Starting in version 2.0.0, Erlang RocksDB supports RocksDB's Wide-Column Entity API. Entities allow storing structured data with multiple named columns per key, providing a more flexible schema than simple key-value pairs.

```erlang
%% Store an entity with multiple columns
Key = <<"user:123">>,
Columns = [
    {<<"name">>, <<"Alice">>},
    {<<"email">>, <<"alice@example.com">>},
    {<<"age">>, <<"30">>}
],
ok = rocksdb:put_entity(Db, Key, Columns, []),

%% Retrieve entity columns as a proplist
{ok, Result} = rocksdb:get_entity(Db, Key, []),
Name = proplists:get_value(<<"name">>, Result),  %% <<"Alice">>

%% Delete entity (same as regular delete)
ok = rocksdb:delete_entity(Db, Key, []).
```

When iterating, you can access columns for any entry:

```erlang
{ok, Itr} = rocksdb:iterator(Db, []),
{ok, Key, _Value} = rocksdb:iterator_move(Itr, first),
{ok, Columns} = rocksdb:iterator_columns(Itr),
%% For regular key-values, returns [{<<>>, Value}] (single default column)
%% For entities, returns all stored columns
rocksdb:iterator_close(Itr).
```

See [Wide-Column Entities](wide_column_entities.html) for more details.

### Snapshots

Snapshots give you a consistent read-only view over the entire state of the key-value store. Set `{snapshot, Snapshot}` in the read options to make a read operate on that version of the database state.


```erlang
{ok, Snapshot} = rocksdb:snapshot(Db),
ReadOptions = [{snapshot, Snapshot}],
{ok, Itr} = rocksdb:iterator(Db, ReadOptions),
... read using iter to view the state when the snapshot was created ...
rocksdb:iterator_close(Itr),
rocksdb:release_snapshot(Snapshot)
```

> Note that when a snapshot is no longer needed, it should be released using the `rocksdb:release_snapshot/1` function. This allows the implementation to get rid of state that was being maintained just to support reading as of that snapshot.


### Column Families

[Column Families](column_families.html) provide a way to logically partition the database. Users can provide atomic writes of multiple keys across multiple column families and read a consistent view from them.

### Backup and Checkpoint

[Backup](how_to_backup_rocksdb.html) allows users to create periodic incremental backups in a remote file system (think about HDFS or S3) and recover from any of them.

[Checkpoints](checkpoints.html) provides the ability to take a snapshot of a running RocksDB database in a separate directory. Files are hardlinked, rather than copied, if possible, so it is a relatively lightweight operation.

### Merge Operator

Starting in version 0.21.0, Erlang Rocksdb supports a [Merge Operator](erlang_merge_operator.html) for Erlang data types. A [Bitset Merge Operator](bitset_merge_operator.html) and a [Counter Merge Operator](counter_merge_operator.html) are also available to change a bit at a position in a string or maintain a simple counter.

### Transactions

Erlang RocksDB has preliminary support of multi-operation transactions. See [Transactions](transactions.html)

### SST File Operations

You can work directly with SST (Sorted String Table) files, the storage format RocksDB uses, for bulk data loading, offline inspection and data migration.

#### Creating SST Files

Use `sst_file_writer` to create SST files outside the database:

```erlang
%% Create an SST file
{ok, Writer} = rocksdb:sst_file_writer_open([], "/tmp/data.sst"),

%% Add key-value pairs (MUST be in sorted order)
ok = rocksdb:sst_file_writer_put(Writer, <<"key1">>, <<"value1">>),
ok = rocksdb:sst_file_writer_put(Writer, <<"key2">>, <<"value2">>),
ok = rocksdb:sst_file_writer_put(Writer, <<"key3">>, <<"value3">>),

%% Finish and get file info
{ok, FileInfo} = rocksdb:sst_file_writer_finish(Writer, with_file_info),
ok = rocksdb:release_sst_file_writer(Writer),

%% FileInfo contains: file_path, smallest_key, largest_key,
%% file_size, num_entries, sequence_number
```

#### Ingesting SST Files

Load SST files directly into a database:

```erlang
{ok, Db} = rocksdb:open("/tmp/mydb", [{create_if_missing, true}]),

%% Ingest SST files into the database
ok = rocksdb:ingest_external_file(Db, ["/tmp/data.sst"], []),

%% Data is now queryable
{ok, <<"value1">>} = rocksdb:get(Db, <<"key1">>, []),

%% Ingest into a specific column family
ok = rocksdb:ingest_external_file(Db, ColumnFamily, ["/tmp/cf_data.sst"], []),

rocksdb:close(Db).
```

#### Reading SST Files

Inspect SST files without loading them into a database:

```erlang
%% Open an SST file for reading
{ok, Reader} = rocksdb:sst_file_reader_open([], "/tmp/data.sst"),

%% Get table properties (metadata)
{ok, Props} = rocksdb:sst_file_reader_get_table_properties(Reader),
io:format("Entries: ~p, Size: ~p bytes~n",
    [maps:get(num_entries, Props), maps:get(data_size, Props)]),

%% Iterate through the file
{ok, Itr} = rocksdb:sst_file_reader_iterator(Reader, []),
{ok, Key1, Value1} = rocksdb:sst_file_reader_iterator_move(Itr, first),
{ok, Key2, Value2} = rocksdb:sst_file_reader_iterator_move(Itr, next),

%% Seek to a specific key
{ok, Key, Value} = rocksdb:sst_file_reader_iterator_move(Itr, {seek, <<"key2">>}),

%% Verify file integrity
ok = rocksdb:sst_file_reader_verify_checksum(Reader),

%% Cleanup
ok = rocksdb:sst_file_reader_iterator_close(Itr),
ok = rocksdb:release_sst_file_reader(Reader).
```

See [SST Files Guide](sst_files.html) for more details including options, best practices, and advanced use cases.