Packages
The datamodel document and what can be decided from it: the typed three-scope declaration, its named record and shape types, the path/type index, the read check, compatibility and coverage - pure functions, no dependency on the block editor, the compiler or the UI
Current section
Files
Jump to
Current section
Files
statifier_datamodel
README.md
README.md
# StatifierDatamodel
[](https://github.com/riddler/statifier_datamodel/actions/workflows/ci.yml)
[](https://hex.pm/packages/statifier_datamodel)
[](https://hex.pm/packages/statifier_datamodel)
[](https://hexdocs.pm/statifier_datamodel/)
[](https://github.com/riddler/statifier_datamodel/blob/main/LICENSE)
The datamodel document and what can be decided from it, with no dependency on
the block editor, the compiler, or the UI.
A datamodel document is a host's typed description of the data universe an
author writes conditions against. It has three scopes - `global`, `local`,
`event` - each carrying entries with a `name`, an absolute dotted `path`, a
`type` and a `label`; and a `types` key of named record and shape
declarations, each with ordered `fields` carrying `name`, `type` and
`required?`. This package is the reader of that document, and the home of
every question that can be answered from the document alone.
The contract is
[ADR-0001](https://github.com/riddler/statifier_datamodel/blob/main/docs/adr/0001-datamodel-document.md);
the document shape and the index are re-homed here from statifier_blocks'
ADR-0006, and ADR-0001 is the record of the re-homing.
## Installation
```elixir
def deps do
[
{:statifier_datamodel, "~> 0.4"}
]
end
```
## What it answers
Every example below is a doctest: `test/readme_test.exs` runs this file, so a
snippet that stops being true fails the suite. The worked domain is
credit-card processing, as it is everywhere in this family.
### The index
`StatifierDatamodel.Index.index/1` admits a decoded document and returns the
path/type index over it. Every entry contributes its own path at every
nesting depth, in document order, and nothing else does. Admission is a total
normalizer: a map carrying a list under `"scopes"` is a document, and
anything else is `nil` - so *not a document* stays distinguishable from *a
document declaring nothing*.
iex> alias StatifierDatamodel.Index
iex> document = %{
...> "version" => 1,
...> "scopes" => [
...> %{"scope" => "local", "label" => "Chart-local", "entries" => [
...> %{"name" => "amount_cents", "path" => "amount_cents",
...> "type" => "integer", "label" => "Amount (minor units)"},
...> %{"name" => "card", "path" => "card", "type" => "object",
...> "label" => "Card", "fields" => [
...> %{"name" => "brand", "path" => "card.brand", "type" => "string",
...> "label" => "Brand",
...> "one_of" => ["visa", "mastercard", "amex"]},
...> %{"name" => "expires_on", "path" => "card.expires_on",
...> "type" => "date", "label" => "Expires on"}]}]}]}
iex> index = Index.index(document)
iex> index.order
["amount_cents", "card", "card.brand", "card.expires_on"]
iex> Index.type(index, "card.expires_on")
:date
iex> Index.declared?(index, "card.cvv")
false
iex> Index.index(["amount_cents"])
nil
### The declared paths
`StatifierDatamodel.Document` is the shorthand for a consumer that holds a
document rather than an index: it admits its argument through the index and
answers totally either way. `declared_paths/1` is the projection an editor's
undeclared-path advisory reads; an empty document projects to an empty set,
and only a non-document is `nil`.
iex> alias StatifierDatamodel.Document
iex> document = %{"scopes" => [%{"scope" => "local", "entries" => [
...> %{"name" => "card", "path" => "card", "type" => "object", "fields" => [
...> %{"name" => "last4", "path" => "card.last4", "type" => "string"},
...> %{"name" => "brand", "path" => "card.brand", "type" => "string",
...> "one_of" => ["visa", "mastercard", "amex"]}]}]}]}
iex> Document.declared_paths(document)
MapSet.new(["card", "card.brand", "card.last4"])
iex> Document.candidates_under(document, "card")
["card.last4", "card.brand"]
iex> Document.declared_values(document)
%{"card.brand" => ["visa", "mastercard", "amex"]}
iex> Document.declared_paths(%{"scopes" => []})
MapSet.new([])
iex> Document.declared_paths(["card"])
nil
### The declared types and the read check
A **record** is a fact about what a write puts at a path; a **shape** is a
constraint a read places on one. `StatifierDatamodel.Declarations` indexes
the document's `types` key to `name -> declaration`, and
`StatifierDatamodel.Types` decides the read check over it: unknown is
permissive both ways, then identity, then a record read as a shape is
admitted when the record's fields cover the shape's required set. Covering a
required shape field takes a record field of the same name that the record
also declares required: an optional field has not promised the value, so it
does not cover. Identity is nominal - there is no structural widening between
two records. `satisfies/3` returns the reason a consumer renders;
`satisfies?/3` is the same check as a boolean.
iex> alias StatifierDatamodel.{Declarations, Types}
iex> declarations = Declarations.from_document(%{"types" => [
...> %{"name" => "cards.credit_txn", "kind" => "record",
...> "label" => "Credit transaction", "fields" => [
...> %{"name" => "amount_cents", "type" => "integer", "required?" => true},
...> %{"name" => "currency", "type" => "string", "required?" => true},
...> %{"name" => "authorized_at", "type" => "datetime", "required?" => true}]},
...> %{"name" => "Settleable", "kind" => "shape", "label" => "Settleable",
...> "fields" => [
...> %{"name" => "amount_cents", "type" => "integer", "required?" => true},
...> %{"name" => "currency", "type" => "string", "required?" => true},
...> %{"name" => "authorized_at", "type" => "datetime", "required?" => true}]},
...> %{"name" => "Refundable", "kind" => "shape", "label" => "Refundable",
...> "fields" => [
...> %{"name" => "amount_cents", "type" => "integer", "required?" => true},
...> %{"name" => "settled_at", "type" => "datetime", "required?" => true}]}]})
iex> Types.satisfies(declarations, {:declared, "cards.credit_txn"}, {:declared, "Settleable"})
:covers
iex> Types.satisfies(declarations, {:declared, "cards.credit_txn"}, {:declared, "Refundable"})
{:missing, ["settled_at"]}
iex> Types.satisfies?(declarations, {:declared, "cards.credit_txn"}, {:declared, "Refundable"})
false
iex> Types.satisfies(declarations, :date, :date)
:identical
Drop the `required?` on the record's `authorized_at` and the same read stops
being satisfied, naming that field the way an absent one is named. The fix in
the document is one key: mark the field required where the record does
promise the value.
iex> alias StatifierDatamodel.{Declarations, Types}
iex> loose = Declarations.from_document(%{"types" => [
...> %{"name" => "cards.credit_txn", "kind" => "record",
...> "label" => "Credit transaction", "fields" => [
...> %{"name" => "amount_cents", "type" => "integer", "required?" => true},
...> %{"name" => "currency", "type" => "string", "required?" => true},
...> %{"name" => "authorized_at", "type" => "datetime"}]},
...> %{"name" => "Settleable", "kind" => "shape", "label" => "Settleable",
...> "fields" => [
...> %{"name" => "amount_cents", "type" => "integer", "required?" => true},
...> %{"name" => "currency", "type" => "string", "required?" => true},
...> %{"name" => "authorized_at", "type" => "datetime", "required?" => true}]}]})
iex> Types.satisfies(loose, {:declared, "cards.credit_txn"}, {:declared, "Settleable"})
{:missing, ["authorized_at"]}
### An inline, unnamed shape
A consumer holding a value the host never declared - a fan-out's collected
envelope, a block's computed summary - says what it holds with
`{:shape, members}`, where each member carries `name`, `type` and
`required?`. It is read the same way a declaration is: a record covers it
member-wise, it covers a declared shape, and two inline shapes compare
structurally. It never satisfies a declared **record**, whose identity is
nominal, and a declared shape held covers nothing but itself. Identity for
the arm is member-set-wise, so member order is a rendering decision and not
a difference. There is no document spelling for one: `parse/2` reads only
strings, and an inline shape enters as an argument to the read check.
iex> alias StatifierDatamodel.{Declarations, Types}
iex> declarations = Declarations.from_document(%{"types" => [
...> %{"name" => "ChunkSummary", "kind" => "shape", "label" => "Chunk summary",
...> "fields" => [
...> %{"name" => "authorized_count", "type" => "integer", "required?" => true},
...> %{"name" => "declined_count", "type" => "integer", "required?" => true}]}]})
iex> summary = {:shape, [
...> %{name: "authorized_count", type: :integer, required?: true},
...> %{name: "declined_count", type: :integer, required?: true}]}
iex> envelope = {:shape, [
...> %{name: "index", type: :integer, required?: true},
...> %{name: "status", type: :string, required?: true},
...> %{name: "donedata", type: summary, required?: false}]}
iex> Types.satisfies(declarations, summary, {:declared, "ChunkSummary"})
:covers
iex> Types.satisfies(declarations, envelope, {:declared, "ChunkSummary"})
{:missing, ["authorized_count", "declined_count"]}
iex> Types.to_string(summary)
"{authorized_count: integer, declined_count: integer}"
The envelope's own members are `index`, `status` and `donedata`: the summary
is one member down, and this package widens nothing to find it.
### An entry typed by a declaration
An entry's `type`, and a `list` entry's `item_type`, may name a declaration
instead of one of the nine types. The reference is nominal - the index
carries `{:declared, name}`, not a copy of the fields - and the entry then
contributes the declaration's fields beneath its own path, exactly as an
inlined `object` entry contributes its `fields`. A name the `types` key does
not declare stays unknown, as it always has.
iex> alias StatifierDatamodel.Index
iex> index = Index.index(%{"scopes" => [
...> %{"scope" => "local", "entries" => [
...> %{"name" => "txn", "path" => "txn", "type" => "cards.credit_txn",
...> "label" => "Transaction"},
...> %{"name" => "note", "path" => "note", "type" => "cards.nothing"}]}],
...> "types" => [
...> %{"name" => "cards.credit_txn", "kind" => "record",
...> "label" => "Credit transaction", "fields" => [
...> %{"name" => "amount_cents", "type" => "integer", "required?" => true},
...> %{"name" => "card", "type" => "cards.card", "required?" => true}]},
...> %{"name" => "cards.card", "kind" => "record", "label" => "Card",
...> "fields" => [
...> %{"name" => "brand", "type" => "string",
...> "one_of" => ["visa", "mastercard", "amex"]}]}]})
iex> index.order
["txn", "txn.amount_cents", "txn.card", "txn.card.brand", "note"]
iex> Index.type(index, "txn.card")
{:declared, "cards.card"}
iex> Index.type(index, "note")
nil
iex> Index.path_types(index)
%{
"txn.amount_cents" => :number,
"txn.card.brand" => {:one_of, ["visa", "mastercard", "amex"]}
}
`txn` and `txn.card` are absent from the value kinds for the same reason an
`object` entry is: neither is a value the expression language has a kind
for. A cycle between declarations discharges rather than recurring, so
`index/1` stays total over any document a host can write.
### Compatibility of a redefined declaration
`StatifierDatamodel.Compatibility.breaks/2` is given the declaration a name
had and the declaration replacing it, and lists every way the new one narrows
the old one - every reason a read that held under the old might not hold
under the new - ordered by field name. The list is empty when the
redefinition takes nothing away; a name declared on neither side is `:error`,
never an empty list.
iex> alias StatifierDatamodel.{Compatibility, Declarations}
iex> was = Declarations.from_document(%{"types" => [
...> %{"name" => "cards.credit_txn", "kind" => "record",
...> "label" => "Credit transaction", "fields" => [
...> %{"name" => "amount_cents", "type" => "integer", "required?" => true},
...> %{"name" => "currency", "type" => "string", "required?" => true},
...> %{"name" => "authorized_at", "type" => "datetime", "required?" => true}]}]})
iex> now = Declarations.from_document(%{"types" => [
...> %{"name" => "cards.credit_txn", "kind" => "record",
...> "label" => "Credit transaction", "fields" => [
...> %{"name" => "amount_cents", "type" => "integer", "required?" => true},
...> %{"name" => "currency", "type" => "string", "required?" => false}]}]})
iex> {:ok, old} = Declarations.fetch(was, "cards.credit_txn")
iex> {:ok, new} = Declarations.fetch(now, "cards.credit_txn")
iex> Compatibility.breaks(old, new)
[{:field_removed, "authorized_at"}, {:made_optional, "currency"}]
iex> Compatibility.breaks(old, old)
[]
iex> Compatibility.breaks(nil, nil)
:error
Each break names its kind first and the field second. The kind is the row of
the record's table the break came from - `:field_removed`, `:type_changed`,
`:made_required`, `:required_added` or `:made_optional`, one per breaking row
and no others - so a host words a warning by matching on it rather than by
comparing the two declarations again.
Both changes break: dropping `authorized_at` takes the field away, and
relaxing `currency` takes away the promise that the value is there, which
the read check has read as *not covered* since decision 8 was amended. What
does not break is a widening - an optional field added - and a `one_of`,
which is a completion hint and not a constraint however it is edited.
iex> alias StatifierDatamodel.{Compatibility, Declarations}
iex> hinted = fn one_of -> Declarations.from_document(%{"types" => [
...> %{"name" => "cards.card", "kind" => "record", "label" => "Card",
...> "fields" => [
...> %{"name" => "brand", "type" => "string", "required?" => true,
...> "one_of" => one_of}]}]}) end
iex> {:ok, old} = Declarations.fetch(hinted.(["visa", "mastercard", "amex"]), "cards.card")
iex> {:ok, new} = Declarations.fetch(hinted.(["visa"]), "cards.card")
iex> Compatibility.breaks(old, new)
[]
This answers *may a reader keep reading*, not *did anything change*.
### Coverage of a map against a shape
`StatifierDatamodel.Coverage.missing/3` returns the `name` of every required
field of a shape that a map does not fill, in declaration order. "Fill" is
presence of the key with a non-`nil` value; the value's type is the
expression language's question, not this one's. A name that is not declared,
or that is a record rather than a shape, is `:error`.
iex> alias StatifierDatamodel.{Coverage, Declarations}
iex> declarations = Declarations.from_document(%{"types" => [
...> %{"name" => "Settleable", "kind" => "shape", "label" => "Settleable",
...> "fields" => [
...> %{"name" => "amount_cents", "type" => "integer", "required?" => true},
...> %{"name" => "currency", "type" => "string", "required?" => true},
...> %{"name" => "authorized_at", "type" => "datetime", "required?" => true}]},
...> %{"name" => "cards.card", "kind" => "record", "label" => "Card",
...> "fields" => [
...> %{"name" => "last4", "type" => "string", "required?" => true}]}]})
iex> Coverage.missing(declarations, "Settleable", %{"amount_cents" => 4200, "currency" => "USD"})
{:ok, ["authorized_at"]}
iex> Coverage.missing(declarations, "Settleable", %{"currency" => nil})
{:ok, ["amount_cents", "currency", "authorized_at"]}
iex> Coverage.missing(declarations, "cards.card", %{})
:error
### Value kinds for an expression editor
`StatifierDatamodel.Index.path_types/1` projects the index to
`%{path => kind | {:list, kind} | {:one_of, values}}`, in the expression
language's own vocabulary: `integer` and `decimal` are both `:number`, a
`list` with a scalar `item_type` is `{:list, kind}`, and a drawable `one_of`
wins over the kind. An `object`, a `list` with no usable `item_type` and an
unknown type are **absent** from the map - absence means unknown, not wrong,
and an editor handed the map treats a path it does not contain exactly as it
treats every path today.
iex> alias StatifierDatamodel.Index
iex> %{"scopes" => [%{"scope" => "local", "entries" => [
...> %{"path" => "amount_cents", "type" => "integer"},
...> %{"path" => "risk_reasons", "type" => "list", "item_type" => "string"},
...> %{"path" => "card", "type" => "object", "fields" => [
...> %{"path" => "card.brand", "type" => "string",
...> "one_of" => ["visa", "mastercard", "amex"]},
...> %{"path" => "card.expires_on", "type" => "date"}]}]}]}
...> |> Index.index()
...> |> Index.path_types()
%{
"amount_cents" => :number,
"risk_reasons" => {:list, :string},
"card.brand" => {:one_of, ["visa", "mastercard", "amex"]},
"card.expires_on" => :date
}
## What is not here
Every function is pure and total over an admitted document, and returns a
fact - a set, an index, a boolean, a list of names, a map. Nothing here
produces a finding, a severity or a verdict: an undeclared path is unknown,
not wrong, and an unsatisfied read is something a consumer assigns a severity
to in its own record.
What is deliberately elsewhere: the environment walk over a block document,
which needs the block tree and stays in
[statifier_blocks](https://github.com/riddler/statifier_blocks); anything
that renders, which is
[statifier_ui](https://github.com/riddler/statifier-ui)'s; and any runtime
enforcement, which no record in the family has asked for.
## Who takes this
statifier_blocks reads the document through this package today, for the typed
environment's read check. statifier_ui takes `path_types/1`'s map as the
expression editor's assign in a later release. Neither takes the other for
the document, and this package depends on nothing in the family - that
property is the whole reason it exists, and a runtime dependency added here
is a decision to record.
## License
MIT - see
[LICENSE](https://github.com/riddler/statifier_datamodel/blob/main/LICENSE).