Packages

elixir_proto

0.1.3
0.2.0 0.1.6 0.1.4 0.1.3 0.1.2 0.1.1 0.1.0

A compact serialization library for Elixir using context-scoped schema registries. Eliminates global index collisions while maintaining wire format compatibility.

HexDocs

Current section

7 Versions

Jump to
Readme
7 Versions
0 Dependencies
Dependants
Files
Activity
Hide whitespace

Compare versions

8 files changed
+757 additions
-253 deletions
  @@ -1,5 +1,20 @@
1 1 # Changelog
2 2
3 + ## [0.1.3] - 2025-09-14
4 +
5 + ### Added
6 + - **Nested struct serialization support** with unlimited nesting depth
7 + - Automatic detection and encoding of nested ElixirProto structs using `{:ep, schema_index, values_tuple}` format
8 + - Recursive encoding/decoding with `encode_field_value/1` and `decode_field_value/1` helper functions
9 + - Graceful handling of mixed data types (ElixirProto structs alongside regular Elixir structs)
10 + - Comprehensive test suite with 11 test cases covering 2-level nesting, 3-level nesting, edge cases, and performance validation
11 + - Error resilience for edge cases including literal `{:ep, index, tuple}` data that looks like nested format
12 +
13 + ### Changed
14 + - Enhanced `ElixirProto.encode/1` to recursively detect and encode nested ElixirProto structs
15 + - Enhanced `ElixirProto.decode/1` to recursively reconstruct nested struct hierarchies
16 + - Test modules moved into test module scope to avoid global namespace pollution
17 +
3 18 ## [0.1.2] - 2025-09-14
4 19
5 20 ### Breaking Changes
  @@ -119,6 +119,114 @@ ElixirProto.SchemaRegistry.get_name(1)
119 119 # => "myapp.ctx.user"
120 120 ```
121 121
122 + ## Nested Struct Serialization
123 +
124 + ElixirProto supports automatic nested struct serialization with unlimited nesting depth. When a struct field contains another ElixirProto struct, it's automatically encoded using a special compact format.
125 +
126 + ### Basic Nested Structures
127 +
128 + ```elixir
129 + defmodule Country do
130 + use ElixirProto.Schema, name: "myapp.country", index: 3
131 + defschema [:name, :code]
132 + end
133 +
134 + defmodule Address do
135 + use ElixirProto.Schema, name: "myapp.address", index: 4
136 + defschema [:street, :city, :country] # country field will hold a Country struct
137 + end
138 +
139 + defmodule User do
140 + use ElixirProto.Schema, name: "myapp.user", index: 5
141 + defschema [:id, :name, :address] # address field will hold an Address struct
142 + end
143 +
144 + # Create nested structure
145 + country = %Country{name: "USA", code: "US"}
146 + address = %Address{street: "123 Main St", city: "Portland", country: country}
147 + user = %User{id: 1, name: "Alice", address: address}
148 +
149 + # Serialize and deserialize - nesting handled automatically
150 + encoded = ElixirProto.encode(user)
151 + decoded = ElixirProto.decode(encoded)
152 +
153 + # Access nested data
154 + decoded.address.country.name # => "USA"
155 + ```
156 +
157 + ### Multi-Level Nesting
158 +
159 + ElixirProto supports unlimited nesting depth:
160 +
161 + ```elixir
162 + defmodule Company do
163 + use ElixirProto.Schema, name: "myapp.company", index: 6
164 + defschema [:name, :address, :ceo] # ceo field holds a User struct
165 + end
166 +
167 + # Three levels deep: Company -> User -> Address -> Country
168 + company = %Company{name: "ACME Corp", address: address, ceo: user}
169 +
170 + encoded = ElixirProto.encode(company)
171 + decoded = ElixirProto.decode(encoded)
172 +
173 + # Access deeply nested data
174 + decoded.ceo.address.country.code # => "US"
175 + ```
176 +
177 + ### Nested Encoding Format
178 +
179 + Nested ElixirProto structs use the internal format `{:ep, schema_index, values_tuple}`:
180 +
181 + ```elixir
182 + # When encoding this nested structure:
183 + %User{id: 1, name: "Alice", address: %Address{street: "123 Main St", city: "Portland", country: nil}}
184 +
185 + # The internal format becomes:
186 + {5, {1, "Alice", {:ep, 4, {"123 Main St", "Portland", nil}}}}
187 + # ^user index ^address as {:ep, address_index, address_values}
188 + ```
189 +
190 + ### Mixed Data Types
191 +
192 + ElixirProto gracefully handles mixed scenarios:
193 +
194 + ```elixir
195 + defmodule RegularStruct do
196 + defstruct [:field1, :field2] # Regular Elixir struct (not ElixirProto)
197 + end
198 +
199 + defmodule MixedUser do
200 + use ElixirProto.Schema, name: "myapp.mixed_user", index: 7
201 + defschema [:id, :name, :regular_data, :proto_address]
202 + end
203 +
204 + # Mix regular structs and ElixirProto structs
205 + mixed = %MixedUser{
206 + id: 1,
207 + name: "Bob",
208 + regular_data: %RegularStruct{field1: "value1", field2: "value2"}, # Preserved as-is
209 + proto_address: address # ElixirProto struct, gets nested encoding
210 + }
211 +
212 + encoded = ElixirProto.encode(mixed)
213 + decoded = ElixirProto.decode(encoded)
214 +
215 + # Regular struct preserved unchanged
216 + decoded.regular_data.__struct__ # => RegularStruct
217 +
218 + # ElixirProto struct properly nested
219 + decoded.proto_address.__struct__ # => Address
220 + ```
221 +
222 + ### Key Features
223 +
224 + - **Automatic Detection**: ElixirProto automatically detects nested ElixirProto structs
225 + - **Unlimited Depth**: Supports arbitrarily deep nesting
226 + - **Mixed Compatibility**: Works alongside regular Elixir structs
227 + - **Space Efficiency**: Nested structs use compact schema indices instead of full module names
228 + - **Error Resilience**: Gracefully handles edge cases like literal `{:ep, index, tuple}` data
229 +
122 230 ## Implementation Architecture
123 231
124 232 ### Core Components
  @@ -5,7 +5,7 @@
5 5 <<"https://github.com/maxohq/elixir_proto/blob/main/CHANGELOG.md">>},
6 6 {<<"GitHub">>,<<"https://github.com/maxohq/elixir_proto">>}]}.
7 7 {<<"name">>,<<"elixir_proto">>}.
8 - {<<"version">>,<<"0.1.2">>}.
8 + {<<"version">>,<<"0.1.3">>}.
9 9 {<<"description">>,
10 10 <<"A compact serialization library for Elixir that uses schema indices and fixed tuples\nfor space-efficient binary serialization with schema evolution support. Inspired by Protobuf.">>}.
11 11 {<<"elixir">>,<<"~> 1.14">>}.
  @@ -14,9 +14,11 @@
14 14 <<"lib/elixir_proto">>,<<"lib/elixir_proto/schema_test.exs">>,
15 15 <<"lib/elixir_proto/schema_registry.ex">>,
16 16 <<"lib/elixir_proto/schema_registry_test.exs">>,
17 - <<"lib/elixir_proto/schema.ex">>,<<"test">>,<<"test/test_helper.exs">>,
18 - <<"test/elixir_proto_test.exs">>,<<"README.md">>,<<"CHANGELOG.md">>,
19 - <<"LICENSE">>,<<"mix.exs">>]}.
17 + <<"lib/elixir_proto/schema.ex">>,
18 + <<"lib/elixir_proto/nested_struct_test.exs">>,
19 + <<"lib/elixir_proto/elixir_proto_test.exs">>,<<"test">>,
20 + <<"test/test_helper.exs">>,<<"README.md">>,<<"CHANGELOG.md">>,<<"LICENSE">>,
21 + <<"mix.exs">>]}.
20 22 {<<"app">>,<<"elixir_proto">>}.
21 23 {<<"licenses">>,[<<"MIT">>]}.
22 24 {<<"requirements">>,[]}.
  @@ -50,11 +50,12 @@ defmodule ElixirProto do
50 50 raise ArgumentError, "Schema index not found for '#{schema_name}'. Make sure the schema is registered with an explicit index."
51 51 end
52 52
53 - # Convert to fixed tuple format (most space efficient)
53 + # Convert to fixed tuple format with nested struct support
54 54 values = Enum.map(1..max_fields, fn i ->
55 55 field_name = Map.get(schema.index_fields, i)
56 56 if field_name do
57 - Map.get(Map.from_struct(struct), field_name)
57 + field_value = Map.get(Map.from_struct(struct), field_name)
58 + encode_field_value(field_value)
58 59 else
59 60 nil
60 61 end
  @@ -124,9 +125,83 @@ defmodule ElixirProto do
124 125 |> Enum.with_index()
125 126 |> Enum.reduce(%{}, fn {field_name, index}, acc ->
126 127 value = Enum.at(values_list, index)
127 - Map.put(acc, field_name, value)
128 + decoded_value = decode_field_value(value) # Handle nested structs
129 + Map.put(acc, field_name, decoded_value)
128 130 end)
129 131
130 132 struct(module, field_map)
131 133 end
134 +
135 + @doc false
136 + # Helper function to encode field values, detecting nested ElixirProto structs
137 + defp encode_field_value(%module{} = nested_struct) do
138 + case Registry.get_schema_by_module(module) do
139 + nil ->
140 + # Not an ElixirProto struct, keep as-is
141 + nested_struct
142 +
143 + schema ->
144 + # This is a nested ElixirProto struct - encode it compactly
145 + schema_name = schema.module.__schema__(:name)
146 + nested_schema_index = SchemaRegistry.get_index(schema_name)
147 +
148 + if nested_schema_index == nil do
149 + # Schema not registered, keep as regular struct
150 + nested_struct
151 + else
152 + # Encode nested struct values recursively
153 + max_fields = length(schema.fields)
154 + nested_values = Enum.map(1..max_fields, fn i ->
155 + field_name = Map.get(schema.index_fields, i)
156 + if field_name do
157 + field_value = Map.get(Map.from_struct(nested_struct), field_name)
158 + encode_field_value(field_value) # Recursive for deeper nesting
159 + else
160 + nil
161 + end
162 + end)
163 +
164 + # Return nested format: {:ep, schema_index, values_tuple}
165 + {:ep, nested_schema_index, List.to_tuple(nested_values)}
166 + end
167 + end
168 + end
169 +
170 + defp encode_field_value(other_value), do: other_value
171 +
172 + @doc false
173 + # Helper function to decode field values, detecting nested ElixirProto markers
174 + defp decode_field_value({:ep, schema_index, values_tuple}) do
175 + case SchemaRegistry.get_name(schema_index) do
176 + nil ->
177 + # Invalid schema index - treat as literal tuple data
178 + {:ep, schema_index, values_tuple}
179 +
180 + schema_name ->
181 + case Registry.get_schema(schema_name) do
182 + nil ->
183 + # Schema not found - treat as literal tuple data
184 + {:ep, schema_index, values_tuple}
185 +
186 + schema ->
187 + # Valid nested ElixirProto struct - reconstruct it
188 + module = schema.module
189 + fields = schema.fields
190 +
191 + # Convert tuple back to field map, recursively decoding nested values
192 + values_list = Tuple.to_list(values_tuple)
193 + field_map = fields
194 + |> Enum.with_index()
195 + |> Enum.reduce(%{}, fn {field_name, index}, acc ->
196 + value = Enum.at(values_list, index)
197 + decoded_value = decode_field_value(value) # Recursive for deeper nesting
198 + Map.put(acc, field_name, decoded_value)
199 + end)
200 +
201 + struct(module, field_map)
202 + end
203 + end
204 + end
205 +
206 + defp decode_field_value(other_value), do: other_value
132 207 end
  @@ -0,0 +1,245 @@
1 + defmodule ElixirProtoTest do
2 + use ExUnit.Case, async: false
3 +
4 + setup do
5 + # Reset registry for clean tests but re-register test modules
6 + ElixirProto.SchemaRegistry.reset!()
7 +
8 + # Manually register test schemas since @after_compile already ran
9 + ElixirProto.SchemaRegistry.force_register_index("myapp.ctx.user", 1)
10 + ElixirProto.SchemaRegistry.force_register_index("myapp.ctx.post", 2)
11 +
12 + # Re-register in the main schema registry too
13 + registry = %{
14 + "myapp.ctx.user" => %{
15 + module: ElixirProtoTest.User,
16 + fields: [:id, :name, :email, :age, :active],
17 + field_indices: %{id: 1, name: 2, email: 3, age: 4, active: 5},
18 + index_fields: %{1 => :id, 2 => :name, 3 => :email, 4 => :age, 5 => :active}
19 + },
20 + "myapp.ctx.post" => %{
21 + module: ElixirProtoTest.Post,
22 + fields: [:id, :title, :content, :author_id, :created_at],
23 + field_indices: %{id: 1, title: 2, content: 3, author_id: 4, created_at: 5},
24 + index_fields: %{1 => :id, 2 => :title, 3 => :content, 4 => :author_id, 5 => :created_at}
25 + }
26 + }
27 +
28 + :persistent_term.put({ElixirProto.Schema.Registry, :schemas}, registry)
29 +
30 + :ok
31 + end
32 +
33 + # Test schemas
34 + defmodule User do
35 + use ElixirProto.Schema, name: "myapp.ctx.user", index: 1
36 + defschema [:id, :name, :email, :age, :active]
37 + end
38 +
39 + defmodule Post do
40 + use ElixirProto.Schema, name: "myapp.ctx.post", index: 2
41 + defschema [:id, :title, :content, :author_id, :created_at]
42 + end
43 +
44 + # Regular struct without ElixirProto schema (for testing error cases)
45 + defmodule UnknownStruct do
46 + defstruct [:field]
47 + end
48 +
49 + describe "encode/1" do
50 + test "encodes struct to binary" do
51 + user = %User{id: 1, name: "Alice", email: "alice@example.com", age: 30, active: true}
52 + encoded = ElixirProto.encode(user)
53 +
54 + assert is_binary(encoded)
55 + assert byte_size(encoded) > 0
56 + end
57 +
58 + test "skips nil fields for space efficiency" do
59 + user_full = %User{id: 1, name: "Alice", email: "alice@example.com", age: 30, active: true}
60 + user_partial = %User{id: 1, name: "Alice"}
61 +
62 + encoded_full = ElixirProto.encode(user_full)
63 + encoded_partial = ElixirProto.encode(user_partial)
64 +
65 + # Partial should be smaller since it skips nil fields
66 + assert byte_size(encoded_partial) < byte_size(encoded_full)
67 + end
68 +
69 + test "raises error for unknown schema" do
70 + unknown = %UnknownStruct{field: "value"}
71 +
72 + assert_raise ArgumentError, ~r/Schema not found for module/, fn ->
73 + ElixirProto.encode(unknown)
74 + end
75 + end
76 +
77 + test "handles various data types" do
78 + post = %Post{
79 + id: 42,
80 + title: "Test Post",
81 + content: "This is a test post with various data types",
82 + author_id: 1,
83 + created_at: ~D[2023-01-01]
84 + }
85 +
86 + encoded = ElixirProto.encode(post)
87 + assert is_binary(encoded)
88 + end
89 + end
90 +
91 + describe "decode/1" do
92 + test "decodes binary back to original struct" do
93 + original = %User{id: 1, name: "Alice", email: "alice@example.com", age: 30, active: true}
94 + encoded = ElixirProto.encode(original)
95 + decoded = ElixirProto.decode(encoded)
96 +
97 + assert decoded == original
98 + assert decoded.id == 1
99 + assert decoded.name == "Alice"
100 + assert decoded.email == "alice@example.com"
101 + assert decoded.age == 30
102 + assert decoded.active == true
103 + end
104 +
105 + test "handles partial structs with nil fields" do
106 + original = %User{id: 1, name: "Alice"}
107 + encoded = ElixirProto.encode(original)
108 + decoded = ElixirProto.decode(encoded)
109 +
110 + assert decoded.id == 1
111 + assert decoded.name == "Alice"
112 + assert decoded.email == nil
113 + assert decoded.age == nil
114 + assert decoded.active == nil
115 + end
116 +
117 + test "raises error for unknown schema in encoded data" do
118 + # Manually create encoded data with unknown schema index
119 + fake_data = {999, {"test"}} # Use invalid schema index
120 + encoded = fake_data |> :erlang.term_to_binary() |> :zlib.compress()
121 +
122 + assert_raise ArgumentError, ~r/Schema index 999 not found/, fn ->
123 + ElixirProto.decode(encoded)
124 + end
125 + end
126 +
127 + test "handles various data types correctly" do
128 + original = %Post{
129 + id: 42,
130 + title: "Test Post",
131 + content: "This is a test post",
132 + author_id: 1,
133 + created_at: ~D[2023-01-01]
134 + }
135 +
136 + encoded = ElixirProto.encode(original)
137 + decoded = ElixirProto.decode(encoded)
138 +
139 + assert decoded == original
140 + assert decoded.id == 42
141 + assert decoded.title == "Test Post"
142 + assert decoded.content == "This is a test post"
143 + assert decoded.author_id == 1
144 + assert decoded.created_at == ~D[2023-01-01]
145 + end
146 + end
147 +
148 + describe "round-trip encoding" do
149 + test "multiple encode/decode cycles preserve data" do
150 + original = %User{id: 1, name: "Alice", email: "alice@example.com", age: 30, active: true}
151 +
152 + # Multiple round trips
153 + result1 = original |> ElixirProto.encode() |> ElixirProto.decode()
154 + result2 = result1 |> ElixirProto.encode() |> ElixirProto.decode()
155 + result3 = result2 |> ElixirProto.encode() |> ElixirProto.decode()
156 +
157 + assert result1 == original
158 + assert result2 == original
159 + assert result3 == original
160 + end
161 +
162 + test "works with complex nested data" do
163 + post = %Post{
164 + id: 1,
165 + title: "Complex Post",
166 + content: %{
167 + "text" => "Hello world",
168 + "metadata" => %{"tags" => ["elixir", "proto"], "count" => 42}
169 + },
170 + author_id: 123,
171 + created_at: DateTime.utc_now()
172 + }
173 +
174 + encoded = ElixirProto.encode(post)
175 + decoded = ElixirProto.decode(encoded)
176 +
177 + assert decoded == post
178 + assert decoded.content["text"] == "Hello world"
179 + assert decoded.content["metadata"]["tags"] == ["elixir", "proto"]
180 + end
181 + end
182 +
183 + describe "space efficiency" do
184 + test "encoded size is reasonable" do
185 + user = %User{id: 1, name: "Alice", email: "alice@example.com", age: 30, active: true}
186 +
187 + # Compare with raw term_to_binary
188 + proto_encoded = ElixirProto.encode(user)
189 + raw_encoded = :erlang.term_to_binary(user)
190 +
191 + # ElixirProto should be reasonably sized (compression helps)
192 + assert byte_size(proto_encoded) > 0
193 + assert byte_size(raw_encoded) > 0 # Just ensure both work
194 + end
195 +
196 + test "nil field omission saves space" do
197 + user_with_nils = %User{id: 1, name: "Alice"} # email, age, active are nil
198 + user_without_nils = %User{id: 1, name: "Alice", email: "", age: 0, active: false}
199 +
200 + encoded_with_nils = ElixirProto.encode(user_with_nils)
201 + encoded_without_nils = ElixirProto.encode(user_without_nils)
202 +
203 + # Version with nils should be smaller
204 + assert byte_size(encoded_with_nils) < byte_size(encoded_without_nils)
205 + end
206 + end
207 +
208 + describe "demonstration" do
209 + test "README example usage works correctly" do
210 + # Create test data as shown in README
211 + user = %User{id: 1, name: "Alice", email: "alice@example.com", age: 30, active: true}
212 +
213 + # Serialize
214 + encoded = ElixirProto.encode(user)
215 + assert is_binary(encoded)
216 + assert byte_size(encoded) > 0
217 +
218 + # Deserialize
219 + decoded = ElixirProto.decode(encoded)
220 + assert decoded == user
221 +
222 + # Test the specific example from README
223 + assert decoded.id == 1
224 + assert decoded.name == "Alice"
225 + assert decoded.email == "alice@example.com"
226 + assert decoded.age == 30
227 + assert decoded.active == true
228 +
229 + # Demonstrate space efficiency with nil omission
230 + user_partial = %User{id: 1, name: "Alice"} # Only id and name set
231 + encoded_partial = ElixirProto.encode(user_partial)
232 + decoded_partial = ElixirProto.decode(encoded_partial)
233 +
234 + # Partial user should decode correctly with nil for missing fields
235 + assert decoded_partial.id == 1
236 + assert decoded_partial.name == "Alice"
237 + assert decoded_partial.email == nil
238 + assert decoded_partial.age == nil
239 + assert decoded_partial.active == nil
240 +
241 + # Partial encoding should be smaller
242 + assert byte_size(encoded_partial) < byte_size(encoded)
243 + end
244 + end
245 + end
Loading more files…