Current section
20 Versions
Jump to
Current section
20 Versions
Compare versions
42
files changed
+1897
additions
-899
deletions
| @@ -1,5 +1,49 @@ | |
| 1 1 | # Changelog |
| 2 2 | |
| 3 | + ## v0.7.0 (2026-02-16) |
| 4 | + |
| 5 | + ### Enhancements |
| 6 | + |
| 7 | + * A Repo can be configured to be locked to a single tenant by specifying `:tenant_id` config. |
| 8 | + * `:key_limit` is a valid Repo option. When specified, the underlying FDB GetRange will be limited to the |
| 9 | + specified number of keys. Note that there is not always a 1-to-1 mapping of keys to objects, due to |
| 10 | + large objects being split across multiple keys. |
| 11 | + * Metadata retrieval is now skipped when querying a schema by its primary key only. e.g. `Repo.all(User, ...)` |
| 12 | + * Added `FDB.stream_range/4`, currently undocumented, that wraps the `:erlfdb_range_iterator` in a `Stream` for |
| 13 | + easier use in Elixir apps. |
| 14 | + * When querying with no where clause and an order_by, an index will now be selected if one exists. |
| 15 | + * Improved error messages for unsupported order_by + limit interactions. |
| 16 | + |
| 17 | + ### Bug fixes |
| 18 | + |
| 19 | + * A `:limit` in `Ecto.Query` will now work as expected when encountering objects split across multiple keys. |
| 20 | + * When reading SchemaMigration versions, only the maximum version number is retrieved, which will result in |
| 21 | + a small performance benefit when opening a Tenant. |
| 22 | + |
| 23 | + ### Relevant internal changes |
| 24 | + |
| 25 | + Internally, much of the Query and Future interfaces have been refactored to support the Query limit bug fix. |
| 26 | + In particular, we are now using `:erlfdb_range_iterator`, which provides more control over the paged retrieval |
| 27 | + of key-values from the database server. We hope these paths are now simpler and easier to maintain over time. |
| 28 | + |
| 29 | + ### New documentation |
| 30 | + |
| 31 | + * [Serialization Design](serialization.html): Describes how your data is serialized to FDB values. |
| 32 | + * Improvements to implementation traces in `fdb_api_counting_test.exs` |
| 33 | + |
| 34 | + ### Dependencies |
| 35 | + |
| 36 | + * `erlfdb ~> 0.3.4` |
| 37 | + * Added support for Elixir 1.19 and Erlang 28 |
| 38 | + * Updated CI FDB to 7.3.69 |
| 39 | + * Updated all outdated deps |
| 40 | + |
| 41 | + ## v0.6.3 (2025-01-12) |
| 42 | + |
| 43 | + ### Bug fixes |
| 44 | + |
| 45 | + * Support limit clause from Ecto.Query in addition to limit option on Repo calls. |
| 46 | + |
| 3 47 | ## v0.6.2 (2025-01-10) |
| 4 48 | |
| 5 49 | ### Bug Fixes |
| @@ -67,7 +111,7 @@ transactions, we have chosen to deprecate `Repo.transaction` in favor of `Repo.t | |
| 67 111 | |
| 68 112 | ### New documentation |
| 69 113 | |
| 70 | - * [Guide for Operators](operators_manual.html): Describes how to use the `EctoFoundationDB.CLI` functions to rename a field while guaraneeting that all |
| 114 | + * [Guide for Operators](operators_manual.html): Describes how to use the `EctoFoundationDB.CLI` functions to rename a field while guaranteeing that all |
| 71 115 | concurrent queries in your distributed application are successful. |
| 72 116 | * [Metadata Design](metadata.html): Describes how index metadata is managed and cached in EctoFDB. |
| 73 117 | * [Sync Engine Part I - Single Object](watches.livemd): Revamped Livebook that demonstrates how to create a Sync Engine for a single object (for syncing reads) |
| @@ -2,7 +2,7 @@ | |
| 2 2 | [{<<"GitHub">>, |
| 3 3 | <<"https://github.com/foundationdb-beam/ecto_foundationdb">>}]}. |
| 4 4 | {<<"name">>,<<"ecto_foundationdb">>}. |
| 5 | - {<<"version">>,<<"0.6.2">>}. |
| 5 | + {<<"version">>,<<"0.7.0">>}. |
| 6 6 | {<<"description">>,<<"FoundationDB adapter for Ecto">>}. |
| 7 7 | {<<"elixir">>,<<"~> 1.15">>}. |
| 8 8 | {<<"app">>,<<"ecto_foundationdb">>}. |
| @@ -11,7 +11,7 @@ | |
| 11 11 | [[{<<"name">>,<<"erlfdb">>}, |
| 12 12 | {<<"app">>,<<"erlfdb">>}, |
| 13 13 | {<<"optional">>,false}, |
| 14 | - {<<"requirement">>,<<"~> 0.3.3">>}, |
| 14 | + {<<"requirement">>,<<"~> 0.3.4">>}, |
| 15 15 | {<<"repository">>,<<"hexpm">>}], |
| 16 16 | [{<<"name">>,<<"ecto">>}, |
| 17 17 | {<<"app">>,<<"ecto">>}, |
| @@ -44,7 +44,10 @@ | |
| 44 44 | <<"lib/ecto_foundationdb/options.ex">>,<<"lib/ecto_foundationdb/sync.ex">>, |
| 45 45 | <<"lib/ecto_foundationdb/tenant.ex">>, |
| 46 46 | <<"lib/ecto_foundationdb/migrations_pj.ex">>, |
| 47 | + <<"lib/ecto_foundationdb/single_tenant_repo.ex">>, |
| 47 48 | <<"lib/ecto_foundationdb/cli">>,<<"lib/ecto_foundationdb/cli/internal.ex">>, |
| 49 | + <<"lib/ecto_foundationdb/assert">>, |
| 50 | + <<"lib/ecto_foundationdb/assert/correct_tenancy.ex">>, |
| 48 51 | <<"lib/ecto_foundationdb/index.ex">>,<<"lib/ecto_foundationdb/future.ex">>, |
| 49 52 | <<"lib/ecto_foundationdb/progressive_job.ex">>, |
| 50 53 | <<"lib/ecto_foundationdb/migration.ex">>,<<"lib/ecto_foundationdb/layer">>, |
| @@ -58,7 +61,7 @@ | |
| 58 61 | <<"lib/ecto_foundationdb/layer/decoded_kv.ex">>, |
| 59 62 | <<"lib/ecto_foundationdb/layer/tx.ex">>, |
| 60 63 | <<"lib/ecto_foundationdb/layer/primary_kv_codec">>, |
| 61 | - <<"lib/ecto_foundationdb/layer/primary_kv_codec/stream_decoder.ex">>, |
| 64 | + <<"lib/ecto_foundationdb/layer/primary_kv_codec/iterator.ex">>, |
| 62 65 | <<"lib/ecto_foundationdb/layer/metadata_version.ex">>, |
| 63 66 | <<"lib/ecto_foundationdb/layer/metadata">>, |
| 64 67 | <<"lib/ecto_foundationdb/layer/metadata/cache.ex">>, |
| @@ -93,6 +96,8 @@ | |
| 93 96 | <<"lib/ecto/adapters/foundationdb/ecto_adapter_assigns.ex">>, |
| 94 97 | <<"lib/ecto/adapters/foundationdb/ecto_adapter_transaction.ex">>, |
| 95 98 | <<"lib/ecto/adapters/foundationdb/ecto_adapter_queryable.ex">>, |
| 96 | - <<"lib/ecto/adapters/foundationdb/ecto_adapter.ex">>,<<".formatter.exs">>, |
| 97 | - <<"mix.exs">>,<<"README.md">>,<<"LICENSE.md">>,<<"CHANGELOG.md">>]}. |
| 99 | + <<"lib/ecto/adapters/foundationdb/ecto_adapter.ex">>,<<"lib/fdb.ex">>, |
| 100 | + <<"lib/fdb">>,<<"lib/fdb/stream.ex">>,<<"lib/fdb/lazy_range_iterator.ex">>, |
| 101 | + <<".formatter.exs">>,<<"mix.exs">>,<<"README.md">>,<<"LICENSE.md">>, |
| 102 | + <<"CHANGELOG.md">>]}. |
| 98 103 | {<<"build_tools">>,[<<"mix">>]}. |
| @@ -26,7 +26,7 @@ defmodule Ecto.Adapters.FoundationDB do | |
| 26 26 | ```elixir |
| 27 27 | defp deps do |
| 28 28 | [ |
| 29 | - {:ecto_foundationdb, "~> 0.6"} |
| 29 | + {:ecto_foundationdb, "~> 0.7"} |
| 30 30 | ] |
| 31 31 | end |
| 32 32 | ``` |
| @@ -78,6 +78,28 @@ defmodule Ecto.Adapters.FoundationDB do | |
| 78 78 | viewed as a logically partitioned database. See `EctoFoundationDB.Tenant` for a discussion |
| 79 79 | on the mechanism used to achieve this in FDB. |
| 80 80 | |
| 81 | + #### Single-Tenant Repo |
| 82 | + |
| 83 | + A particular Repo can be configured to be locked to a single tenant by specifying a `:tenant_id` on |
| 84 | + the Repo config. |
| 85 | + |
| 86 | + ```elixir |
| 87 | + config :my_app, MyApp.SystemRepo, |
| 88 | + cluster_file: "/etc/foundationdb/fdb.cluster", |
| 89 | + tenant_id: "my-system-tenant" |
| 90 | + ``` |
| 91 | + |
| 92 | + When `:tenant_id` is defined, EctoFoundationDB will automatically open the tenant for you, |
| 93 | + removing the requirement to specify a `:prefix` on Repo operations. Things to consider before |
| 94 | + using a single-tenant Repo: |
| 95 | + |
| 96 | + 1. EctoFDB opens the tenant for you and permanently stores it in memory. |
| 97 | + 2. You must restart your application to apply migrations. |
| 98 | + 3. To convert a Repo from single-tenant to multi-tenant, first remove the `:tenant_id` config. |
| 99 | + Then, modify all operations on the Repo to include the `:prefix`. |
| 100 | + 4. To convert a Repo from multi-tenant to single-tenant, first define the `:tenant_id` in config. |
| 101 | + Then, modify all operations on the Repo to remove the `:prefix`. |
| 102 | + |
| 81 103 | ### Creating a schema |
| 82 104 | |
| 83 105 | This is a standard `Ecto.Schema` that will be used in the examples of this documentation. |
| @@ -211,7 +233,7 @@ defmodule Ecto.Adapters.FoundationDB do | |
| 211 233 | in a well-defined order, the order of the fields in your index determines the Between queries you can perform. |
| 212 234 | |
| 213 235 | > **_RULE:_** For any Query, there can be 0 or 1 Between clauses, and if one exists for a field, then there must not |
| 214 | - exist an Equal clause for ay field that follows, according to the list of fields in the index. Moreover, the Between |
| 236 | + exist an Equal clause for any field that follows, according to the list of fields in the index. Moreover, the Between |
| 215 237 | field must be of a [Data Type](#module-data-types) supported for Between queries. |
| 216 238 | |
| 217 239 | Above, we demonstrated a nontrivial query with 2 Equal constraints and 1 Between constraint. Compare that with an invalid |
| @@ -236,7 +258,7 @@ defmodule Ecto.Adapters.FoundationDB do | |
| 236 258 | 1. Purposefully over-extract the data using Stream, and do the filtering in Elixir: |
| 237 259 | |
| 238 260 | ```elixir |
| 239 | - from(u in User, where: u.birthyear >= ^1992 and u.birthyear < 1995 and) |
| 261 | + from(u in User, where: u.birthyear >= ^1992 and u.birthyear < ^1995) |
| 240 262 | |> MyApp.Repo.stream(prefix: tenant) |
| 241 263 | |> Stream.filter(&((&1).department == "Engineering")) |
| 242 264 | |> Enum.to_list() |
| @@ -288,13 +310,13 @@ defmodule Ecto.Adapters.FoundationDB do | |
| 288 310 | compared to SQL databases. But you can develop with confidence that your queries will always be |
| 289 311 | efficient. |
| 290 312 | |
| 291 | - #### Using `order_by` and `limit` in the same query |
| 313 | + #### Using `order_by` and `key_limit` in the same query |
| 292 314 | |
| 293 | - EctoFDB will always attempt to return query result that has the ordering applied first and then the limit. If it's |
| 294 | - unable to do so, then an exception is raised. Remember: a key design decision of our Layer is that a query is valid |
| 315 | + EctoFDB will always attempt to return query result that has the ordering applied first and then the key_limit. If it's |
| 316 | + unable to do so, then an exception is raised. Remember: a central design decision of our Layer is that a query is valid |
| 295 317 | only if it can be achieved with a single Get or GetRange. |
| 296 318 | |
| 297 | - `order_by` and `limit` will work as expected as long as one of the following conditions is true: |
| 319 | + `order_by` and `key_limit` will work as expected as long as one of the following conditions is true: |
| 298 320 | |
| 299 321 | 1. An index is not being queried AND the `order_by` is defined on the primary key field. |
| 300 322 | 2. The fields of the selected index match with the fields in the `order_by` clause. |
| @@ -302,7 +324,7 @@ defmodule Ecto.Adapters.FoundationDB do | |
| 302 324 | Examples: |
| 303 325 | |
| 304 326 | ```elixir |
| 305 | - opts = [prefix: tenant, limit: 10] |
| 327 | + opts = [prefix: tenant, key_limit: 10] |
| 306 328 | |
| 307 329 | # order_by primary key field |
| 308 330 | Repo.all( |
| @@ -323,6 +345,37 @@ defmodule Ecto.Adapters.FoundationDB do | |
| 323 345 | ) |
| 324 346 | ``` |
| 325 347 | |
| 348 | + #### Query limit and option key_limit |
| 349 | + |
| 350 | + EctoFDB supports two types of limits. |
| 351 | + |
| 352 | + 1. **Query limit**: A `limit: n` clause provided to an `Ecto.Query` |
| 353 | + 2. **Option key_limit**: A `key_limit: n` option provided to a `Repo.all` call |
| 354 | + |
| 355 | + In almost all cases, the two are equivalent. However, if any objects surpass the `:max_single_value_size` |
| 356 | + and are split into multiple key-values, then the distinction becomes important. The Option key_imit constrains |
| 357 | + the total number of keys retrieved from FoundationDB. The Query Limit constrains the total number of items |
| 358 | + in the returned list. Therefore, in rare cases it's valid and reasonable to provide two separate limits |
| 359 | + in a function call. |
| 360 | + |
| 361 | + When only the Query limit is provided, keys will be retrieved to satisfy the limit. EctoFDB may retrieve at-most |
| 362 | + one extra page from the internal GetRange operation. |
| 363 | + |
| 364 | + ```elixir |
| 365 | + Repo.all( |
| 366 | + from(p in Post, limit: 10), #=> a maximum of 10 %Post{} will be returned |
| 367 | + prefix: tenant) |
| 368 | + ``` |
| 369 | + |
| 370 | + When the Option key_limit is provided, EctoFDB guarantees that no more than that number of keys will be retrieved |
| 371 | + by the internal GetRange operation. Any Query limit is executed after this fact. |
| 372 | + |
| 373 | + ```elixir |
| 374 | + Repo.all( |
| 375 | + from(p in Post, limit: 10), #=> a maximum of 10 %Post{} will be **returned** |
| 376 | + prefix: tenant, key_limit: 100) #=> a maximum of 100 key-values will be **retrieved** |
| 377 | + ``` |
| 378 | + |
| 326 379 | ### Custom indexes |
| 327 380 | |
| 328 381 | As data is inserted, updated, deleted, and queried, the Indexer callbacks (via behaviour `EctoFoundationDB.Indexer`) |
| @@ -418,7 +471,7 @@ defmodule Ecto.Adapters.FoundationDB do | |
| 418 471 | [ |
| 419 472 | {0, MyApp.IndexUserByDepartment} |
| 420 473 | # At a later date, we my add a new index with a corresponding addition |
| 421 | - # tho the migrations list: |
| 474 | + # to the migrations list: |
| 422 475 | # {1, MyApp.IndexUserByRole} |
| 423 476 | ] |
| 424 477 | end |
| @@ -561,7 +614,7 @@ defmodule Ecto.Adapters.FoundationDB do | |
| 561 614 | Note: |
| 562 615 | |
| 563 616 | - The approach to inserting above is guaranteed to have zero conflicts with other keys in the database. |
| 564 | - - A group of records inserted in a single transaction **will** have versionatmps that are guaranteed to increment by 1. |
| 617 | + - A group of records inserted in a single transaction **will** have versionstamps that are guaranteed to increment by 1. |
| 565 618 | - Records inserted in different transactions **will not** have a predictable versionstamp distance from each other. That distance |
| 566 619 | is very unlikely to ever be 1. |
| 567 620 | |
| @@ -733,8 +786,8 @@ defmodule Ecto.Adapters.FoundationDB do | |
| 733 786 | behavior. You should only consider its use if one of the following is true: |
| 734 787 | |
| 735 788 | - (a) your schema has no indexes |
| 736 | - - (b) you know apriori that any indexed fields are unchanged at the time of upsert, or |
| 737 | - - (c) you know apirori that the primary key you're inserting doesn't already exist. |
| 789 | + - (b) you know a priori that any indexed fields are unchanged at the time of upsert, or |
| 790 | + - (c) you know a priori that the primary key you're inserting doesn't already exist. |
| 738 791 | |
| 739 792 | For these use cases, this option may speed up the loading of initial data at scale. |
| 740 793 | |
| @@ -798,12 +851,12 @@ defmodule Ecto.Adapters.FoundationDB do | |
| 798 851 | Conceptually, only 1 network round-trip is required, so this will be faster than |
| 799 852 | waiting on each in sequence. |
| 800 853 | |
| 801 | - ### Pipelining on range queries (interleaving waits) |
| 854 | + ### Pipelining on range queries (interleaved waits) |
| 802 855 | |
| 803 856 | The functions `Repo.async_all/1` and `Repo.async_all/2` work a little bit different on the await. When the DB must |
| 804 857 | return a list of results, if that list ends up being larger than `:erlfdb`'s configured `:target_bytes`, |
| 805 858 | then the result set is split into multiple round-trips to the database. If your application awaits multiple |
| 806 | - such queries, then `:erlfdb`'s "interleaving waits" feature is used. |
| 859 | + such queries, waited on pages of data are pipelined for optimal throughput. |
| 807 860 | |
| 808 861 | Consider this example, where we're retrieving all Users and all Posts in the same transaction. |
| 809 862 | |
| @@ -828,8 +881,7 @@ defmodule Ecto.Adapters.FoundationDB do | |
| 828 881 | Server-->>Client: {10 users, done} |
| 829 882 | ``` |
| 830 883 | |
| 831 | - With `:erlfdb`'s interleaving waits, we send as many get_range requests as possible with pipelining |
| 832 | - until all are exhausted. |
| 884 | + We send as many get_range requests as possible with pipelining until all are exhausted. |
| 833 885 | |
| 834 886 | ```mermaid |
| 835 887 | sequenceDiagram |
| @@ -880,6 +932,8 @@ defmodule Ecto.Adapters.FoundationDB do | |
| 880 932 | keys. Please remember that FDB's transaction size limit of 10MB still applies. |
| 881 933 | The maximum number of key-value pairs for a given encoded struct, not counting |
| 882 934 | indexes, is computed by `1 + ceil(:max_value_size / :max_single_value_size)`. |
| 935 | + To disable splitting objects across multiple keys, set this value to match |
| 936 | + `:max_single_value_size`, which is `100_000` by default. |
| 883 937 | |
| 884 938 | ## Optimizing for throughput and latency |
| @@ -16,7 +16,7 @@ defmodule Ecto.Adapters.FoundationDB.EctoAdapter do | |
| 16 16 | telemetry_prefix = Keyword.fetch!(config, :telemetry_prefix) |
| 17 17 | telemetry = {config[:repo], log, telemetry_prefix ++ [:query]} |
| 18 18 | |
| 19 | - {:ok, Ecto.Adapters.FoundationDB.Supervisor.child_spec([]), |
| 19 | + {:ok, Ecto.Adapters.FoundationDB.Supervisor.child_spec([config]), |
| 20 20 | %{telemetry: telemetry, stacktrace: stacktrace, opts: config}} |
| 21 21 | end |
| @@ -105,7 +105,7 @@ defmodule Ecto.Adapters.FoundationDB.EctoAdapterAssigns do | |
| 105 105 | Tx.transactional(options[:prefix], fn _tx -> |
| 106 106 | assign_future = |
| 107 107 | repo.async_get(schema, id, options) |
| 108 | - |> Future.apply(fn struct_or_nil -> |
| 108 | + |> Future.then(fn struct_or_nil -> |
| 109 109 | struct_or_nil = usetenant(struct_or_nil, tenant) |
| 110 110 | new_future = maybe_new_watch(struct_or_nil, watch_options, options, new_watch_fn) |
| 111 111 | |
| @@ -132,7 +132,7 @@ defmodule Ecto.Adapters.FoundationDB.EctoAdapterAssigns do | |
| 132 132 | end |
| 133 133 | |
| 134 134 | assign_future = |
| 135 | - Future.apply(assign_future, fn result -> |
| 135 | + Future.then(assign_future, fn result -> |
| 136 136 | result = usetenant(result, tenant) |
| 137 137 | new_future = maybe_new_watch(result, watch_options, options, new_watch_fn) |
Loading more files…