Packages
localize_mf2_treesitter
0.1.0
Elixir bindings to the tree-sitter-mf2 grammar. Incremental, error-recovering CST parser for ICU MessageFormat 2 (MF2) messages suitable for editor tooling and LSP use.
Current section
Files
Jump to
Current section
Files
lib/localize/mf2/tree_sitter.ex
defmodule Localize.Mf2.TreeSitter do
@moduledoc """
Elixir bindings to the tree-sitter-mf2 grammar.
Produces a concrete, position-aware CST for ICU MessageFormat 2 (MF2)
source. Complements the NimbleParsec parser in
[`localize`](https://hexdocs.pm/localize/) — that parser is strict,
fast, and aborts on the first error; this one is resilient and
error-recovering, which is what editor tooling and an LSP need.
The NIF embeds a vendored copy of the tree-sitter C runtime and the
generated `tree-sitter-mf2` parser, so no Rust toolchain and no
system `libtree-sitter` are required to build.
### Primary API
* `parse/1` — parse a source binary into a `Tree`.
* `root/1` — obtain the root `Node`.
Walk from the root using the functions in
`Localize.Mf2.TreeSitter.Node`.
"""
alias Localize.Mf2.TreeSitter.{Edit, Nif, Node, Tree}
@doc """
Parse an MF2 source binary.
Tree-sitter is error-recovering: invalid input still yields a tree,
with `ERROR` or `MISSING` nodes placed at the failure points. Use
`Localize.Mf2.TreeSitter.Node.has_error?/1` on the root to detect
whether recovery was needed.
### Arguments
* `source` is a UTF-8 binary containing MF2 source.
### Returns
* `{:ok, tree}` on success.
* `{:error, :parse_failed}` if the underlying parser could not be
constructed (out-of-memory or library misconfiguration). Invalid
MF2 input does **not** produce this — it produces an `{:ok, tree}`
with `ERROR` nodes.
### Examples
iex> {:ok, tree} = Localize.Mf2.TreeSitter.parse("hello {$name}")
iex> root = Localize.Mf2.TreeSitter.root(tree)
iex> Localize.Mf2.TreeSitter.Node.type(root)
"source_file"
"""
@spec parse(binary()) :: {:ok, Tree.t()} | {:error, :parse_failed}
def parse(source) when is_binary(source) do
case Nif.parse(source) do
{:ok, ref} -> {:ok, %Tree{ref: ref, source: source}}
{:error, reason} -> {:error, reason}
end
end
@doc """
Return the root node of a parse tree.
### Arguments
* `tree` is a `Localize.Mf2.TreeSitter.Tree` returned by `parse/1`.
### Returns
* A `Localize.Mf2.TreeSitter.Node` struct.
### Examples
iex> {:ok, tree} = Localize.Mf2.TreeSitter.parse("")
iex> Localize.Mf2.TreeSitter.Node.type(Localize.Mf2.TreeSitter.root(tree))
"source_file"
"""
@spec root(Tree.t()) :: Node.t()
def root(%Tree{ref: ref, source: source}) do
%Node{handle: Nif.tree_root_node(ref), source: source}
end
@doc """
Reparse a source binary incrementally, reusing subtrees from a
previous parse.
This is the LSP / editor case: the user made a small text change,
and we want to produce a new tree without redoing the full parse.
The old tree is not mutated — internally a copy is taken, each
edit is applied to that copy, and the parser is fed the
edited-old-tree plus the new source. Unchanged subtrees are reused
byte-for-byte.
### Arguments
* `old_tree` is a `Localize.Mf2.TreeSitter.Tree` from a previous
`parse/1` or `parse_incremental/3` call.
* `edits` is a list of `Localize.Mf2.TreeSitter.Edit` structs
describing the text change(s) from the old source to the new.
* `new_source` is the updated UTF-8 binary.
### Returns
* `{:ok, new_tree}` with the new tree wrapping `new_source`.
* `{:error, :parse_failed}` on allocation or language failure.
### Examples
iex> {:ok, old} = Localize.Mf2.TreeSitter.parse("hello {$name}")
iex> # user appended "!" at the end
iex> edit = %Localize.Mf2.TreeSitter.Edit{
...> start_byte: byte_size("hello {$name}"),
...> old_end_byte: byte_size("hello {$name}"),
...> new_end_byte: byte_size("hello {$name}!"),
...> start_point: {0, byte_size("hello {$name}")},
...> old_end_point: {0, byte_size("hello {$name}")},
...> new_end_point: {0, byte_size("hello {$name}!")}
...> }
iex> {:ok, new} = Localize.Mf2.TreeSitter.parse_incremental(old, [edit], "hello {$name}!")
iex> Localize.Mf2.TreeSitter.Node.text(Localize.Mf2.TreeSitter.root(new))
"hello {$name}!"
"""
@spec parse_incremental(Tree.t(), [Edit.t()], binary()) ::
{:ok, Tree.t()} | {:error, :parse_failed}
def parse_incremental(%Tree{ref: old_ref}, edits, new_source)
when is_list(edits) and is_binary(new_source) do
edit_tuples = Enum.map(edits, &Edit.to_tuple/1)
case Nif.parse_incremental(old_ref, edit_tuples, new_source) do
{:ok, ref} -> {:ok, %Tree{ref: ref, source: new_source}}
{:error, reason} -> {:error, reason}
end
end
@doc """
Return the byte/point ranges that differ between two trees.
Intended to be called with a previous tree and the tree returned by
`parse_incremental/3` for the same source sequence. The result is a
list of `{start_byte, end_byte, start_point, end_point}` ranges that
a consumer (editor highlight pass, LSP semantic-tokens) needs to
redraw.
### Arguments
* `old_tree` and `new_tree` are `Localize.Mf2.TreeSitter.Tree`
structs.
### Returns
* A list of `{start_byte, end_byte, start_point, end_point}` tuples.
### Examples
iex> {:ok, old} = Localize.Mf2.TreeSitter.parse("hello")
iex> {:ok, new} = Localize.Mf2.TreeSitter.parse("hello")
iex> Localize.Mf2.TreeSitter.changed_ranges(old, new)
[]
"""
@spec changed_ranges(Tree.t(), Tree.t()) :: [
{non_neg_integer(), non_neg_integer(), Node.point(), Node.point()}
]
def changed_ranges(%Tree{ref: old}, %Tree{ref: new}) do
Nif.tree_get_changed_ranges(old, new)
end
@doc """
Walk a tree and return every `ERROR` or `MISSING` node as a
diagnostic descriptor.
Each diagnostic is `{kind, node}` where `kind` is `:error` or
`:missing`. Read the range with `Localize.Mf2.TreeSitter.Node.start_byte/1`,
`end_byte/1`, `start_point/1`, `end_point/1` to render a squiggle
or emit an LSP `textDocument/publishDiagnostics` notification.
The traversal is depth-first, pre-order; diagnostics appear in
source order. Well-formed trees return `[]`.
### Arguments
* `tree` is a `Localize.Mf2.TreeSitter.Tree`.
### Returns
* A list of `{:error | :missing, Localize.Mf2.TreeSitter.Node.t()}`
tuples in source order, possibly empty.
### Examples
iex> {:ok, tree} = Localize.Mf2.TreeSitter.parse("hello {$")
iex> [{kind, _node} | _] = Localize.Mf2.TreeSitter.diagnostics(tree)
iex> kind in [:error, :missing]
true
iex> {:ok, tree} = Localize.Mf2.TreeSitter.parse("hello")
iex> Localize.Mf2.TreeSitter.diagnostics(tree)
[]
"""
@spec diagnostics(Tree.t()) :: [{:error | :missing, Node.t()}]
def diagnostics(%Tree{} = tree) do
root = root(tree)
if Node.has_error?(root) do
root |> collect_diagnostics([]) |> Enum.reverse()
else
[]
end
end
# Depth-first pre-order walk, collecting ERROR and MISSING nodes.
# Accumulates in reverse (cons-onto-head) and the public function
# reverses once at the end. Descent is pruned at subtrees that have
# no errors anywhere beneath them.
defp collect_diagnostics(%Node{} = node, acc) do
acc =
cond do
Node.missing?(node) -> [{:missing, node} | acc]
Node.error?(node) -> [{:error, node} | acc]
true -> acc
end
if Node.has_error?(node) do
node
|> Node.children()
|> Enum.reduce(acc, &collect_diagnostics/2)
else
acc
end
end
end