Current section
20 Versions
Jump to
Current section
20 Versions
Compare versions
41
files changed
+3192
additions
-1140
deletions
| @@ -1,6 +1,44 @@ | |
| 1 | - # Changelog for v0.x |
| 1 | + # Changelog |
| 2 2 | |
| 3 | - ## v0.4 (2024-01-16) |
| 3 | + ## v0.5.0 (2025-07-08) |
| 4 | + |
| 5 | + ### Enhancements |
| 6 | + |
| 7 | + * `EctoFoundationDB.CLI` [[doc](operators_manual.md)]: Functions for an operator to use to safely manage |
| 8 | + data migrations that cannot be done automatically. |
| 9 | + * Index Metadata [[doc](metadata.html)] now makes use of FoundationDB's `\xff/metadataVersion` key, |
| 10 | + which allows the client to cache metadata and maintain transactional isolation |
| 11 | + without having to wait on any keys. Also, the cache is shared across all open tenants of the same id on a given node. |
| 12 | + * `EctoFoundationDB.Versionstamp` [[doc](Ecto.Adapters.FoundationDB.html#module-versionstamps-autoincrement)]: Added the ability to insert objects with a monotonically increasing integer id, via FoundationDB's versionstamp. |
| 13 | + * `SchemaMetadata` [[doc](Ecto.Adapters.FoundationDB.html#module-schema-metadata)]: This is a new built-in Indexer that allows your app to watch and sync collection of objects in a tenant. |
| 14 | + * Added ability to drop an index. |
| 15 | + |
| 16 | + ### Breaking changes |
| 17 | + |
| 18 | + * Key construction has changed, making databases created on <= 0.4.x incompatible with >=0.5. Specifically, a binary, atom, or |
| 19 | + number primary key is now encoded in the FDB key with the Tuple layer. All other types remain encoded with term_to_binary. |
| 20 | + If you need help upgrading your database, please put in a GitHub Issue. We strive for a stable v1.0. |
| 21 | + |
| 22 | + ### Bug fixes |
| 23 | + |
| 24 | + * Fixed a bug where index creation was failing for multikey objects |
| 25 | + * (#57) Fixed a bug where index management was failing while a new index was being created |
| 26 | + |
| 27 | + ### Deprecations |
| 28 | + |
| 29 | + * Ecto has deprecated `Repo.transaction` in favor of `Repo.transact`. Since this decision doesn't align with FoundationDB's view of |
| 30 | + transactions, we have chosen to deprecate `Repo.transaction` in favor of `Repo.transactional`. This terminology better aligns with |
| 31 | + `:erlfdb` and provides a distinction from RDBMS transactions, and allows us to avoid future deprecations. |
| 32 | + |
| 33 | + ### New documentation |
| 34 | + |
| 35 | + * [Guide for Operators](operators_manual.html): Describes how to use the `EctoFoundationDB.CLI` functions to rename a field while guaraneeting that all |
| 36 | + concurrent queries in your distributed application are successful. |
| 37 | + * [Metadata Design](metadata.html): Describes how index metadata is managed and cached in EctoFDB. |
| 38 | + * [Sync Engine Part I - Single Object](watches.livemd): Revamped Livebook that demonstrates how to create a Sync Engine for a single object (for syncing reads) |
| 39 | + * [Sync Engine Part II - Collections](collection_syncing.livemd): New Livebook that demonstrates a Sync Engine for a collection of objects in a tenant (still for reads) |
| 40 | + |
| 41 | + ## v0.4.0 (2025-01-16) |
| 4 42 | |
| 5 43 | ### Enhancements |
| 6 44 | |
| @@ -24,7 +62,7 @@ | |
| 24 62 | |
| 25 63 | * Added `fdb_api_counting_text.exs` which tests and documents the `:erlfdb` operations that our Layer is expected to make. |
| 26 64 | |
| 27 | - ## v0.3 (2024-10-20) |
| 65 | + ## v0.3.0 (2024-10-20) |
| 28 66 | |
| 29 67 | ### \*\* Major breaking changes \*\* |
| 30 68 | |
| @@ -61,5 +99,29 @@ to discuss the upgrade path. | |
| 61 99 | |
| 62 100 | ### New Documentation |
| 63 101 | |
| 64 | - * [TESTING.md](testing.html): Document to describe how to set up a Sandbox |
| 102 | + * [testing.md](testing.html): Document to describe how to set up a Sandbox |
| 65 103 | * [CHANGELOG.md](changelog.html): This file! |
| 104 | + |
| 105 | + ## v0.1.2 (2024-08-31) |
| 106 | + |
| 107 | + ### Bug fixes |
| 108 | + |
| 109 | + * Upgrade erlfdb |
| 110 | + |
| 111 | + ## v0.1.1 (2024-08-25) |
| 112 | + |
| 113 | + ### Enhancements |
| 114 | + |
| 115 | + * Upgrade to Ecto 3.12 |
| 116 | + |
| 117 | + ### New documentation |
| 118 | + |
| 119 | + * [Livebook | Getting Started](introduction.livemd): How to get started with EctoFoundationDB. |
| 120 | + |
| 121 | + ## v0.1.0 (2024-04-07) |
| 122 | + |
| 123 | + ### Features |
| 124 | + |
| 125 | + * Multitenancy |
| 126 | + * Basic CRUD operations |
| 127 | + * Indexes |
| @@ -2,7 +2,7 @@ | |
| 2 2 | [{<<"GitHub">>, |
| 3 3 | <<"https://github.com/foundationdb-beam/ecto_foundationdb">>}]}. |
| 4 4 | {<<"name">>,<<"ecto_foundationdb">>}. |
| 5 | - {<<"version">>,<<"0.4.0">>}. |
| 5 | + {<<"version">>,<<"0.5.0">>}. |
| 6 6 | {<<"description">>,<<"FoundationDB adapter for Ecto">>}. |
| 7 7 | {<<"elixir">>,<<"~> 1.15">>}. |
| 8 8 | {<<"app">>,<<"ecto_foundationdb">>}. |
| @@ -11,7 +11,7 @@ | |
| 11 11 | [[{<<"name">>,<<"erlfdb">>}, |
| 12 12 | {<<"app">>,<<"erlfdb">>}, |
| 13 13 | {<<"optional">>,false}, |
| 14 | - {<<"requirement">>,<<"~> 0.2">>}, |
| 14 | + {<<"requirement">>,<<"~> 0.3">>}, |
| 15 15 | {<<"repository">>,<<"hexpm">>}], |
| 16 16 | [{<<"name">>,<<"ecto">>}, |
| 17 17 | {<<"app">>,<<"ecto">>}, |
| @@ -34,30 +34,38 @@ | |
| 34 34 | <<"lib/ecto_foundationdb/options.ex">>, |
| 35 35 | <<"lib/ecto_foundationdb/tenant.ex">>, |
| 36 36 | <<"lib/ecto_foundationdb/migrations_pj.ex">>, |
| 37 | + <<"lib/ecto_foundationdb/cli">>,<<"lib/ecto_foundationdb/cli/internal.ex">>, |
| 37 38 | <<"lib/ecto_foundationdb/index.ex">>,<<"lib/ecto_foundationdb/future.ex">>, |
| 38 39 | <<"lib/ecto_foundationdb/progressive_job.ex">>, |
| 39 40 | <<"lib/ecto_foundationdb/migration.ex">>,<<"lib/ecto_foundationdb/layer">>, |
| 40 41 | <<"lib/ecto_foundationdb/layer/primary_kv_codec.ex">>, |
| 41 42 | <<"lib/ecto_foundationdb/layer/tx_insert.ex">>, |
| 42 43 | <<"lib/ecto_foundationdb/layer/internal_metadata.ex">>, |
| 43 | - <<"lib/ecto_foundationdb/layer/index_inventory.ex">>, |
| 44 44 | <<"lib/ecto_foundationdb/layer/pack.ex">>, |
| 45 | + <<"lib/ecto_foundationdb/layer/metadata.ex">>, |
| 45 46 | <<"lib/ecto_foundationdb/layer/fields.ex">>, |
| 46 47 | <<"lib/ecto_foundationdb/layer/query.ex">>, |
| 47 48 | <<"lib/ecto_foundationdb/layer/decoded_kv.ex">>, |
| 48 49 | <<"lib/ecto_foundationdb/layer/tx.ex">>, |
| 50 | + <<"lib/ecto_foundationdb/layer/primary_kv_codec">>, |
| 51 | + <<"lib/ecto_foundationdb/layer/primary_kv_codec/stream_decoder.ex">>, |
| 52 | + <<"lib/ecto_foundationdb/layer/metadata_version.ex">>, |
| 53 | + <<"lib/ecto_foundationdb/layer/metadata">>, |
| 54 | + <<"lib/ecto_foundationdb/layer/metadata/cache.ex">>, |
| 49 55 | <<"lib/ecto_foundationdb/layer/ordering.ex">>, |
| 50 56 | <<"lib/ecto_foundationdb/migrator.ex">>,<<"lib/ecto_foundationdb/sandbox">>, |
| 51 57 | <<"lib/ecto_foundationdb/sandbox/sandboxer.ex">>, |
| 52 58 | <<"lib/ecto_foundationdb/schema.ex">>, |
| 53 59 | <<"lib/ecto_foundationdb/schema_migration.ex">>, |
| 54 | - <<"lib/ecto_foundationdb/sandbox.ex">>, |
| 60 | + <<"lib/ecto_foundationdb/sandbox.ex">>,<<"lib/ecto_foundationdb/cli.ex">>, |
| 61 | + <<"lib/ecto_foundationdb/versionstamp.ex">>, |
| 55 62 | <<"lib/ecto_foundationdb/indexer.ex">>, |
| 56 63 | <<"lib/ecto_foundationdb/exception">>, |
| 57 64 | <<"lib/ecto_foundationdb/exception/unsupported.ex">>, |
| 58 65 | <<"lib/ecto_foundationdb/exception/incorrect_tenancy.ex">>, |
| 59 66 | <<"lib/ecto_foundationdb/indexer">>, |
| 60 | - <<"lib/ecto_foundationdb/indexer/max_value.ex">>, |
| 67 | + <<"lib/ecto_foundationdb/indexer/schema_metadata.ex">>, |
| 68 | + <<"lib/ecto_foundationdb/indexer/mdv_app_version.ex">>, |
| 61 69 | <<"lib/ecto_foundationdb/indexer/default.ex">>, |
| 62 70 | <<"lib/ecto_foundationdb/layer.ex">>,<<"lib/ecto">>,<<"lib/ecto/adapters">>, |
| 63 71 | <<"lib/ecto/adapters/foundationdb.ex">>, |
| @@ -26,7 +26,7 @@ defmodule Ecto.Adapters.FoundationDB do | |
| 26 26 | ```elixir |
| 27 27 | defp deps do |
| 28 28 | [ |
| 29 | - {:ecto_foundationdb, "~> 0.4"} |
| 29 | + {:ecto_foundationdb, "~> 0.5"} |
| 30 30 | ] |
| 31 31 | end |
| 32 32 | ``` |
| @@ -40,6 +40,8 @@ defmodule Ecto.Adapters.FoundationDB do | |
| 40 40 | use Ecto.Repo, otp_app: :my_app, adapter: Ecto.Adapters.FoundationDB |
| 41 41 | |
| 42 42 | use EctoFoundationDB.Migrator |
| 43 | + |
| 44 | + @impl true |
| 43 45 | def migrations(), do: [] |
| 44 46 | end |
| 45 47 | ``` |
| @@ -72,6 +74,10 @@ defmodule Ecto.Adapters.FoundationDB do | |
| 72 74 | Once you have opened a tenant, you'll use the Ecto `:prefix` option to provide the tenant |
| 73 75 | to each Repo operation. |
| 74 76 | |
| 77 | + Retrieving data from a tenant yields no execution overhead. In effect, each tenant can be |
| 78 | + viewed as a logically partitioned database. See `EctoFoundationDB.Tenant` for a discussion |
| 79 | + on the mechanism used to achieve this in FDB. |
| 80 | + |
| 75 81 | ### Creating a schema |
| 76 82 | |
| 77 83 | This is a standard `Ecto.Schema` that will be used in the examples of this documentation. |
| @@ -121,25 +127,23 @@ defmodule Ecto.Adapters.FoundationDB do | |
| 121 127 | MyApp.Repo.get!(User, alice.id, prefix: tenant) |
| 122 128 | ``` |
| 123 129 | |
| 124 | - ### Transactions |
| 130 | + ### Transaction basics |
| 125 131 | |
| 126 | - Multiple calls to Repo functions can be grouped inside a single transaction, which is |
| 127 | - [ACID](https://en.wikipedia.org/wiki/ACID) and [globally serializable](https://en.wikipedia.org/wiki/Global_serializability). |
| 128 | - |
| 129 | - Here's a simple example. Transactions can be arbitrarily complex, but there is a [limit to the size and runtime](#module-foundationdb). |
| 132 | + Here's a simple example of transaction usage. Notice that the calls to Repo inside the transaction function |
| 133 | + do not include the `:prefix` option. See [Transactions](#module-transactions) for more information. |
| 130 134 | |
| 131 135 | ```elixir |
| 132 | - MyApp.Repo.transaction(fn -> |
| 136 | + MyApp.Repo.transactional(tenant, fn -> |
| 133 137 | alice = MyApp.Repo.get!(User, alice.id) |
| 134 138 | if alice.balance > 0 do |
| 135 139 | MyApp.Repo.update(User.changeset(alice, %{balance: alice.balance - 1})) |
| 136 140 | else |
| 137 141 | raise "Overdraft" |
| 138 142 | end |
| 139 | - end, prefix: tenant) |
| 143 | + end) |
| 140 144 | ``` |
| 141 145 | |
| 142 | - ## Indexes and Queries |
| 146 | + ## Indexes and queries |
| 143 147 | |
| 144 148 | The implication of the [FoundationDB Layer Concept](https://apple.github.io/foundationdb/layer-concept.html) |
| 145 149 | is that the manner in which data can be stored and accessed is the responsibility of the client |
| @@ -166,12 +170,12 @@ defmodule Ecto.Adapters.FoundationDB do | |
| 166 170 | ``` |
| 167 171 | |
| 168 172 | In fact, the index value can be any Elixir term. All types support Equal queries (of the form `x.field == "value"`). |
| 169 | - However, certain Ecto Schema types support Between queries (of the form `x.field >= 10 and x.field < 20`). |
| 173 | + Certain Ecto Schema types support Between queries (of the form `x.field >= 10 and x.field < 20`). |
| 170 174 | See the [Data Types](#module-data-types) section for the list. |
| 171 175 | |
| 172 176 | See the [Migrations](#module-migrations) section for further details about managing indexes in your application. |
| 173 177 | |
| 174 | - ### Index Query Examples |
| 178 | + ### Index query examples |
| 175 179 | |
| 176 180 | Retrieve all users in the Engineering department (with an Equal constraint): |
| 177 181 | |
| @@ -224,8 +228,8 @@ defmodule Ecto.Adapters.FoundationDB do | |
| 224 228 | |
| 225 229 | **Why is this query unsupported by EctoFDB?** With the index defined as-is, EctoFDB is unable to extract a minimal |
| 226 230 | dataset from the database in one operation. Instead of over-extracting data and then applying a filter, we've chosen |
| 227 | - to reject the query. We hope this encourages correct index management and removes worry about execution time of |
| 228 | - individual Repo function calls. |
| 231 | + to reject the query. We hope this encourages correct index management and gives the developer confidence about the |
| 232 | + expected execution time for any given call to Repo. |
| 229 233 | |
| 230 234 | **Yeah that's cool, but how do I run the query I want to run?** You have 2 options. |
| 231 235 | |
| @@ -247,32 +251,33 @@ defmodule Ecto.Adapters.FoundationDB do | |
| 247 251 | ``` |
| 248 252 | |
| 249 253 | We encourage you to test execution times of your expected workload before deciding to create an index. |
| 254 | + Remember: since your data is already partitioned into tenants, you're effectively getting 1 index for free. |
| 250 255 | |
| 251 | - ### Custom Indexes |
| 256 | + ### Custom indexes |
| 252 257 | |
| 253 258 | As data is inserted, updated, deleted, and queried, the Indexer callbacks (via behaviour `EctoFoundationDB.Indexer`) |
| 254 259 | are executed. Your Indexer module does all the work necessary to maintain the index within the transaction |
| 255 260 | that the data itself is being manipulated. Many different types of indexes are possible. You are in control! |
| 256 261 | |
| 257 | - For an example, please see the [MaxValue implementation](https://github.com/foundationdb-beam/ecto_foundationdb/blob/main/lib/ecto_foundationdb/indexer/max_value.ex) |
| 258 | - which is used internally by EctoFoundationDB to keep track of the maximum value of a field in a Schema. |
| 262 | + For an example, please see the [Default implementation](https://github.com/foundationdb-beam/ecto_foundationdb/blob/main/lib/ecto_foundationdb/indexer/default.ex). |
| 259 263 | |
| 260 264 | ## Transactions |
| 261 265 | |
| 262 | - The Repo functions will automatically use FoundationDB transactions behind the scenes. EctoFoundationDB also |
| 263 | - exposes the Ecto.Repo transaction function, allowing you to group operations together with transactional isolation: |
| 266 | + All work involving FDB happens in a transaction, including each Repo function call (e.g. `Repo.get/1`). EctoFoundationDB also |
| 267 | + exposes the `Repo.transactional/2` function, allowing you to group operations together with transactional isolation ([ACID](https://en.wikipedia.org/wiki/ACID) |
| 268 | + and [globally serializable](https://en.wikipedia.org/wiki/Global_serializability)) |
| 264 269 | |
| 265 270 | ```elixir |
| 266 | - MyApp.Repo.transaction(fn -> |
| 271 | + MyApp.Repo.transactional(tenant, fn -> |
| 267 272 | # Ecto work |
| 268 | - end, prefix: tenant) |
| 273 | + end) |
| 269 274 | ``` |
| 270 275 | |
| 271 276 | Please read the [FoundationDB Developer Guide on transactions](https://apple.github.io/foundationdb/developer-guide.html#transaction-basics) |
| 272 277 | for more information about how to develop with transactions. |
| 273 278 | |
| 274 279 | It's important to remember that even though the database gives ACID guarantees about the |
| 275 | - keys updated in a transaction, the Elixir function passed into `Repo.transaction/1` |
| 280 | + keys updated in a transaction, the Elixir function passed into `Repo.transactional/2` |
| 276 281 | may be executed more than once when any conflicts |
| 277 282 | are encountered. For this reason, your function must not have any side effects other than |
| 278 283 | the database operations. For example, do not publish any messages to other processes |
| @@ -280,23 +285,23 @@ defmodule Ecto.Adapters.FoundationDB do | |
| 280 285 | |
| 281 286 | ```elixir |
| 282 287 | # Do not do this! |
| 283 | - MyApp.Repo.transaction(fn -> |
| 288 | + MyApp.Repo.transactional(tenant, fn -> |
| 284 289 | MyApp.Repo.insert(%User{name: "Alice"}) |
| 285 290 | ... |
| 286 291 | Phoenix.PubSub.broadcast("my_topic", "new_user") # Not safe! :( |
| 287 | - end, prefix: tenant) |
| 292 | + end) |
| 288 293 | ``` |
| 289 294 | |
| 290 295 | ```elixir |
| 291 296 | # Instead, do this: |
| 292 | - MyApp.Repo.transaction(fn -> |
| 297 | + MyApp.Repo.transactional(tenant, fn -> |
| 293 298 | MyApp.Repo.insert(%User{name: "Alice"}) |
| 294 299 | ... |
| 295 | - end, prefix: tenant) |
| 300 | + end) |
| 296 301 | Phoenix.PubSub.broadcast("my_topic", "new_user") # Safe :) |
| 297 302 | ``` |
| 298 303 | |
| 299 | - If you're looking for PubSub-like functionality at the transaction-level, please see |
| 304 | + Note: If you're looking for PubSub-like functionality pushed from the database itself, please see |
| 300 305 | [Watches](#module-watches). |
| 301 306 | |
| 302 307 | ## Migrations |
| @@ -313,12 +318,23 @@ defmodule Ecto.Adapters.FoundationDB do | |
| 313 318 | When you add a field to your Schema, any read requests on old objects will return `nil` in |
| 314 319 | that field. |
| 315 320 | |
| 321 | + ### Online migrations |
| 322 | + |
| 316 323 | As tenants are opened during your application runtime, migrations will be executed |
| 317 | - automatically. This distributes the migration across a potentially long period of time, |
| 324 | + automatically in a "lazy" fashion. This distributes the migration across a potentially long period of time, |
| 318 325 | as migrations will not be executed unless the tenant is opened. Specifically, a call to |
| 319 326 | `Tenant.open/2` will block until the migration for that tenant completes. Other tenants |
| 320 327 | will not be affected. |
| 321 328 | |
| 329 | + For example, imagine you have 2 nodes running your application. Node 1 has opened tenants `"a"` |
| 330 | + and `"b"`. You deploy a new version of your app to Node 2 that includes a new index, and open |
| 331 | + tenant `"a"` there. On Node 2 only, that call to `Tenant.open/2` will block until the migration |
| 332 | + completes. All operations on Node 1 will remain working. When `Tenant.open/2` completes on Node 2 |
| 333 | + for tenant `"a"`, the index is available to all `"a"` tenants across all nodes. Any `"b"` tenants |
| 334 | + remain unaffected throughout this process. |
| 335 | + |
| 336 | + ### Configuring your app |
| 337 | + |
| 322 338 | The Migrator is a module in your application runtime that provides the full list of |
| 323 339 | ordered migrations. These are the migrations that will be executed when a tenant is opened. |
| 324 340 | If you leave out a migration from the list, it will not be applied. |
| @@ -346,6 +362,8 @@ defmodule Ecto.Adapters.FoundationDB do | |
| 346 362 | ```elixir |
| 347 363 | defmodule MyApp.IndexUserByDepartment do |
| 348 364 | use EctoFoundationDB.Migration |
| 365 | + |
| 366 | + @impl true |
| 349 367 | def change() do |
| 350 368 | [ |
| 351 369 | create index(User, [:department]) |
| @@ -355,17 +373,49 @@ defmodule Ecto.Adapters.FoundationDB do | |
| 355 373 | ``` |
| 356 374 | |
| 357 375 | Your migrator and `MyApp.<Migration>` modules are part of your codebase. They must be |
| 358 | - included in your application release. |
| 376 | + included in your application release. This differs from traditional Ecto migrations (`:ecto_sql`), which |
| 377 | + are typically managed in `.exs` scripts executed outside the production application runtime. |
| 378 | + |
| 379 | + ### Creating and dropping indexes |
| 380 | + |
| 381 | + In your Migration, to create an index use `create index(table, columns)` and to drop an index use `drop index(table, columns)`. |
| 382 | + |
| 383 | + Indexes are created and dropped in a manner that is safe for your application transactions. |
| 384 | + |
| 385 | + ```elixir |
| 386 | + defmodule MyApp.ExampleMigration do |
| 387 | + use EctoFoundationDB.Migration |
| 388 | + |
| 389 | + @impl true |
| 390 | + def change() do |
| 391 | + [ |
| 392 | + create index(User, [:role]), |
| 393 | + drop index(User, [:department]), |
| 394 | + ] |
| 395 | + end |
| 396 | + end |
| 397 | + ``` |
| 398 | + |
| 399 | + ### Executing all migrations immediately |
| 359 400 | |
| 360 401 | Migrations can be completed in full at any time with a call to |
| 361 | - `EctoFoundationDB.Migrator.up_all/1`. Depending on how many tenants you have in |
| 362 | - your database, and the size of the data for each tenant, this operation might take a long |
| 363 | - time and make heavy use of system resources. It is safe to interrupt this operation, as |
| 364 | - migrations are performed transactionally. However, if you interrupt a migration, the next |
| 365 | - attempt for that tenant may have a brief delay (~5 sec); the migrator must ensure that the |
| 402 | + `EctoFoundationDB.CLI.migrate!/1`, which accepts your repo module (e.g. `MyApp.Repo`). |
| 403 | + |
| 404 | + ```elixir |
| 405 | + # May take a while to complete |
| 406 | + EctoFoundationDB.CLI.migrate!(MyApp.Repo) |
| 407 | + ``` |
| 408 | + |
| 409 | + Depending on how many tenants you have in your database, and the size of the data for each tenant, |
| 410 | + this operation might take a long time and make heavy use of system resources. It is safe to interrupt |
| 411 | + this operation, as migrations are performed transactionally. However, if you interrupt a migration, the next |
| 412 | + attempt for that tenant may have a brief delay (~5 sec) while the migrator ensures that the |
| 366 413 | previous migration has indeed stopped executing. |
| 367 414 | |
| 368 | - ## Data Types |
| 415 | + Please refer to the documentation of `EctoFoundationDB.CLI` for other actions to be carried out |
| 416 | + by a human operator. |
| 417 | + |
| 418 | + ## Data types |
| 369 419 | |
| 370 420 | EctoFoundationDB stores your struct's data using `:erlang.term_to_binary/1`, and |
| 371 421 | retrieves it with `:erlang.binary_to_term/1`. As such, there is no data type |
| @@ -374,7 +424,9 @@ defmodule Ecto.Adapters.FoundationDB do | |
| 374 424 | |
| 375 425 | Data types *are* used by EctoFoundationDB for the creation and querying of indexes. |
| 376 426 | Certain Ecto types are encoded into the FDB key, which allows you to formulate |
| 377 | - Between queries on indexes of these types: |
| 427 | + Between queries on indexes of these types. |
| 428 | + |
| 429 | + ** Between queries are supported for these types:** |
| 378 430 | |
| 379 431 | - `:id` |
| 380 432 | - `:binary_id` |
| @@ -393,6 +445,56 @@ defmodule Ecto.Adapters.FoundationDB do | |
| 393 445 | |
| 394 446 | See [Indexes and Queries](#module-indexes-and-queries) for more information. |
| 395 447 | |
| 448 | + ### Versionstamps (autoincrement) |
| 449 | + |
| 450 | + FoundationDB provides a mechanism for generating an automatically increasing integer id, called a [versionstamp](https://apple.github.io/foundationdb/data-modeling.html#versionstamps). |
| 451 | + Versionstamps are guaranteed to be unique and increasing along with the total order of the serializable transactions. |
| 452 | + They are not guaranteed to increment by exactly one. |
| 453 | + |
| 454 | + To use Versionstamps, your Schema must have a primary key field of type `Versionstamp`, with `autogenerate: false`. |
| 455 | + |
| 456 | + ```elixir |
| 457 | + @primary_key {:id, EctoFoundationDB.Versionstamp, autogenerate: false} |
| 458 | + ``` |
| 459 | + |
| 460 | + The easiest way to insert such records is by using `Repo.async_insert_all/3`. This function follows a different |
| 461 | + approach than the other `Repo.async_*` functions: the future must be awaited *outside* of the transaction. |
| 462 | + |
| 463 | + ```elixir |
| 464 | + alias MyApp.{Repo, Event} |
| 465 | + future = Repo.transactional(tenant, fn -> |
| 466 | + Repo.async_insert_all(Event, [%Event{data: "event_a"}, %Event{data: "event_b"}]) |
| 467 | + end) |
| 468 | + [event_a, event_b] = Repo.await(future) |
| 469 | + ``` |
| 470 | + |
| 471 | + The purpose of the async/await pattern here is for the FDB client to discover the committed versionstamp, and to apply |
| 472 | + it to the inserted structs after the transaction is committed. Since the computation of the versionstamp is a |
| 473 | + commit-time decision by the FoundationDB Server, the final value is unknowable by the client until after the transaction |
| 474 | + is committed. |
| 475 | + |
| 476 | + Inspecting the contents of the `:id` field will reveal the structure of the versionstamp: |
| 477 | + |
| 478 | + ```elixir |
| 479 | + event_a.id |
| 480 | + # => {:versionstamp, 1111, 2222, 0} |
| 481 | + ``` |
| 482 | + |
| 483 | + The Versionstamp can be converted to an integer using the `EctoFoundationDB.Versionstamp.to_integer/1` function. |
| 484 | + |
| 485 | + ```elixir |
| 486 | + alias EctoFoundationDB.Versionstamp |
| 487 | + Versionstamp.to_integer(event_a.id) |
| 488 | + # => 4771854286848 |
| 489 | + ``` |
| 490 | + |
| 491 | + Note: |
| 492 | + |
| 493 | + - The approach to inserting above is guaranteed to have zero conflicts with other keys in the database. |
| 494 | + - A group of records inserted in a single transaction **will** have versionatmps that are guaranteed to increment by 1. |
| 495 | + - Records inserted in different transactions **will not** have a predictable versionstamp distance from each other. That distance |
| 496 | + is very unlikely to ever be 1. |
| 497 | + |
| 396 498 | ## Watches |
| 397 499 | |
| 398 500 | [FoundationDB Watches](https://apple.github.io/foundationdb/developer-guide.html#watches) are |
| @@ -452,19 +554,19 @@ defmodule Ecto.Adapters.FoundationDB do | |
| 452 554 | # ... |
| 453 555 | |
| 454 556 | def init(tenant) do |
| 455 | - {alice, future} = MyRepo.transaction( |
| 557 | + {alice, future} = MyRepo.transactional(tenant, |
| 456 558 | fn -> |
| 457 559 | alice = MyRepo.get_by(User, name: "Alice") |
| 458 560 | {alice, MyRepo.watch(alice, label: :alice)} |
| 459 | - end, prefix: tenant) |
| 561 | + end) |
| 460 562 | assigns = [alice: alice] |
| 461 563 | {:ok, %{assigns: assigns, futures: [future]}} |
| 462 564 | end |
| 463 565 | |
| 464 566 | # ... |
| 465 567 | |
| 466 | - def handle_info({ref, :ready}, state=%{assigns: assigns, futures: futures}) when is_reference(ref) do |
| 467 | - {new_assigns, new_futures} = MyRepo.assign_ready(futures, [ref], watch?: true) |
| 568 | + def handle_info({ref, :ready}, state=%{assigns: assigns, futures: futures, tenant: tenant}) when is_reference(ref) do |
| 569 | + {new_assigns, new_futures} = MyRepo.assign_ready(futures, [ref], watch?: true, prefix: tenant) |
| 468 570 | {:noreply, %{ |
| 469 571 | state | |
| 470 572 | assigns: Map.merge(assigns, Enum.into(new_assigns, %{})), |
| @@ -481,6 +583,54 @@ defmodule Ecto.Adapters.FoundationDB do | |
| 481 583 | And `Repo.assign_ready/3` provides a conventional way to retrieve the result and register a new watch |
| 482 584 | in one go, so that updates to Alice are not missed. |
| 483 585 | |
| 586 | + For more on watches, see |
| 587 | + |
| 588 | + * [Sync Engine Part I - Single Object](watches.html) |
| 589 | + * [Schema Metadata](#module-schema-metadata) |
| 590 | + |
| 591 | + ## Schema Metadata |
| 592 | + |
| 593 | + In addition to [Default Indexes](#module-indexes-and-queries), EctoFDB provides another built-in Indexer called `EctoFoundationDB.Indexer.SchemaMetadata`. |
| 594 | + |
| 595 | + This is an optional Indexer that will keep track of various actions for a Schema: |
| 596 | + |
| 597 | + - `inserts`: Incremented for each insert or upsert |
| 598 | + - `deletes`: Incremented for each delete |
| 599 | + - `collection`: Incremented for each insert, update, or delete |
| 600 | + - `updates`: Incremented for each update (via `Repo.update/*`) |
| 601 | + - `changes`: Incremented for each insert, upsert, delete, or update |
| 602 | + |
| 603 | + These keys are useful for creating watches that will notify your application of those actions. For example, if you create a watch on the `inserts` key, |
| 604 | + your application will be notified when a new record is inserted, and you can react however you like. |
| 605 | + |
| 606 | + The counters are scoped to a particular tenant. If you desire a different scoping for the watches, then we recommend you split your tenant. |
| 607 | + |
| 608 | + To create this Indexer, you use [Migrations](#module-migrations): |
| 609 | + |
| 610 | + ```elixir |
| 611 | + defmodule MyApp.UserSchemaMetadata do |
| 612 | + @moduledoc false |
| 613 | + use EctoFoundationDB.Migration |
| 614 | + |
| 615 | + @impl true |
| 616 | + def change() do |
| 617 | + [create metadata(User)] |
| 618 | + end |
| 619 | + end |
| 620 | + ``` |
| 621 | + |
| 622 | + ```elixir |
| 623 | + def migrations() do |
| 624 | + [ |
| 625 | + # ... |
| 626 | + {2, MyApp.UserSchemaMetadata}, |
| 627 | + # ... |
| 628 | + ] |
| 629 | + end |
| 630 | + ``` |
| 631 | + |
| 632 | + For more on these watches, see [Sync Engine Part II - Collections](collection_syncing.html) |
| 633 | + |
| 484 634 | ## Upserts |
| 485 635 | |
| 486 636 | The FoundationDB Adapter supports the following upsert approaches. |
| @@ -539,12 +689,12 @@ defmodule Ecto.Adapters.FoundationDB do | |
| 539 689 | list is constructed and the transaction ends. |
| 540 690 | |
| 541 691 | ```elixir |
| 542 | - result = MyApp.Repo.transaction(fn -> |
| 692 | + result = MyApp.Repo.transactional(tenant, fn -> |
| 543 693 | [ |
| 544 694 | MyApp.Repo.get_by(User, name: "Alice"), |
| 545 695 | MyApp.Repo.get_by(User, name: "Bob") |
| 546 696 | ] |
| 547 | - end, prefix: tenant) |
| 697 | + end) |
| 548 698 | |
| 549 699 | [alice, bob] = result |
| 550 700 | ``` |
| @@ -554,14 +704,14 @@ defmodule Ecto.Adapters.FoundationDB do | |
| 554 704 | is constructed, and then we `await` on them. |
| 555 705 | |
| 556 706 | ```elixir |
| 557 | - result = MyApp.Repo.transaction(fn -> |
| 707 | + result = MyApp.Repo.transactional(tenant, fn -> |
| 558 708 | futures = [ |
| 559 709 | MyApp.Repo.async_get_by(User, name: "Alice"), |
| 560 710 | MyApp.Repo.async_get_by(User, name: "Bob") |
| 561 711 | ] |
| 562 712 | |
| 563 713 | MyApp.Repo.await(futures) |
| 564 | - end, prefix: tenant) |
| 714 | + end) |
| 565 715 | |
| 566 716 | [alice, bob] = result |
| 567 717 | ``` |
| @@ -579,14 +729,14 @@ defmodule Ecto.Adapters.FoundationDB do | |
| 579 729 | Consider this example, where we're retrieving all Users and all Posts in the same transaction. |
| 580 730 | |
| 581 731 | ```elixir |
| 582 | - MyApp.Repo.transaction(fn -> |
| 732 | + MyApp.Repo.transactional(tenant, fn -> |
| 583 733 | futures = [ |
| 584 734 | MyApp.Repo.async_all(User), |
| 585 735 | MyApp.Repo.async_all(Post) |
| 586 736 | ] |
| 587 737 | |
| 588 738 | MyApp.Repo.await(futures) |
| 589 | - end, prefix: tenant) |
| 739 | + end) |
| 590 740 | ``` |
| 591 741 | |
| 592 742 | Each query individually executed would be multiple round-trips to the database as pages of results are retrieved. |
| @@ -618,7 +768,7 @@ defmodule Ecto.Adapters.FoundationDB do | |
| 618 768 | |
| 619 769 | This happens automatically when you include the list of futures to your `Repo.await/1`. |
| 620 770 | |
| 621 | - ## Standard Options |
| 771 | + ## Standard options |
| 622 772 | |
| 623 773 | * `:cluster_file` - The path to the fdb.cluster file. The default is |
| 624 774 | `"/etc/foundationdb/fdb.cluster"`. |
| @@ -626,7 +776,7 @@ defmodule Ecto.Adapters.FoundationDB do | |
| 626 776 | behaviour. Required when using any indexes. Defaults to `nil`. When `nil`, |
| 627 777 | your Repo module is assumed to be the Migrator. |
| 628 778 | |
| 629 | - ## Advanced Options |
| 779 | + ## Advanced options |
| 630 780 | |
| 631 781 | * `:storage_id` - All tenants created by this adapter are prefixed with |
| 632 782 | this string. This allows multiple configurations to operate on the |
| @@ -638,7 +788,7 @@ defmodule Ecto.Adapters.FoundationDB do | |
| 638 788 | * `:migration_step` - The maximum number of keys to process in a single transaction |
| 639 789 | when running migrations. Defaults to `1000`. If you use a number that is |
| 640 790 | too large, the FDB transactions run by the Migrator will fail. |
| 641 | - * `:idx_cache` - When set to `:enabled`, the Ecto ets cache is used to store the |
| 791 | + * `:metadata_cache` - When set to `:enabled`, the Ecto ets cache is used to store the |
| 642 792 | available indexes per tenant. This speeds up all database operations. |
| 643 793 | Defaults to `:enabled`. |
| 644 794 | * `:max_single_value_size` - Number of Bytes above which an encoded struct will |
| @@ -669,7 +819,7 @@ defmodule Ecto.Adapters.FoundationDB do | |
| 669 819 | |
| 670 820 | Read more at [FDB Developer Guide | Loading Data](https://apple.github.io/foundationdb/developer-guide.html#loading-data). |
| 671 821 | |
| 672 | - ## Limitations and Caveats |
| 822 | + ## Limitations and caveats |
| 673 823 | |
| 674 824 | ### FoundationDB |
| 675 825 | |
| @@ -696,10 +846,11 @@ defmodule Ecto.Adapters.FoundationDB do | |
| 696 846 | |
| 697 847 | * Renaming a table |
| 698 848 | * Renaming a field |
| 849 | + * Deleting a field [GitHub #32](https://github.com/foundationdb-beam/ecto_foundationdb/issues/32) |
| 699 850 | * Dropping an index |
| 700 | - * Moving down in migration bersions (rollback) |
| 851 | + * Moving down in migration versions (rollback) |
| 701 852 | |
| 702 | - Finally, the Ecto Mix tasks are known to be unsupported: |
| 853 | + Finally, do not use the Ecto Mix tasks: |
| 703 854 | |
| 704 855 | ```elixir |
| 705 856 | # These commands are not supported. Do not use them with :ecto_foundationdb! |
| @@ -711,13 +862,15 @@ defmodule Ecto.Adapters.FoundationDB do | |
| 711 862 | # mix ecto.migrations |
| 712 863 | ``` |
| 713 864 | |
| 714 | - ### Key and Value Size |
| 865 | + ### Key and value size |
| 715 866 | |
| 716 867 | [FoundationDB has limits on key and value size](https://apple.github.io/foundationdb/known-limitations.html#large-keys-and-values) |
| 717 | - Please be aware of these limitations as you develop. EctoFoundationDB doesn't make |
| 718 | - any attempt to protect you from these errors. |
| 868 | + Please be aware of these limitations as you develop. EctoFDB does support splitting an individual struct |
| 869 | + across multiple keys in the database. This is automatic and doesn't require any additional input from the developer. The |
| 870 | + [options](#module-advanced-options) `:max_single_value_size` and `:max_value_size` allow you to tweak this feature. |
| 719 871 | |
| 720 | - ### Ecto Queries |
| 872 | + |
| 873 | + ### Ecto queries |
| 721 874 | |
| 722 875 | You'll find that most queries outside of those detailed in the documentation fail with |
| 723 876 | an `EctoFoundationDB.Exception.Unsupported` exception. The FoundationDB Layer concept |
| @@ -729,15 +882,11 @@ defmodule Ecto.Adapters.FoundationDB do | |
| 729 882 | |
| 730 883 | Refer to the integration tests for a collection of queries that is known to work. |
| 731 884 | |
| 732 | - ### Other Ecto Features |
| 885 | + ### Other Ecto features |
| 733 886 | |
| 734 887 | As you test the boundaries of EctoFoundationDB, you'll find that many other Ecto features just plain |
| 735 | - don't work with this adapter. Ecto has a lot of features, and this adapter is in the early stage. |
| 736 | - For any given unsupported feature, the underlying reason will be one of the following. |
| 737 | - |
| 738 | - 1. It makes sense for EctoFoundationDB, but it just hasn't been implemented yet. |
| 739 | - 2. The fundamental differences between FDB and SQL backends mean that there is |
| 740 | - no practical reason to implement it in EctoFoundationDB. |
| 888 | + don't work with this adapter. Ecto has a lot of features, and this adapter is in active development. |
| 889 | + Please let us know what you find! |
| 741 890 | """ |
| 742 891 | |
| 743 892 | @behaviour Ecto.Adapter |
| @@ -798,6 +947,39 @@ defmodule Ecto.Adapters.FoundationDB do | |
| 798 947 | @impl Ecto.Adapter |
| 799 948 | defmacro __before_compile__(_env) do |
| 800 949 | quote do |
| 950 | + def transactional(tenant, fun), do: Ecto.Adapters.FoundationDB.transactional(tenant, fun) |
| 951 | + |
| 952 | + def async_insert_all!(schema, list, opts \\ []) do |
| 953 | + repo = get_dynamic_repo() |
| 954 | + EctoAdapterAsync.async_insert_all!(__MODULE__, repo, schema, list, opts) |
| 955 | + end |
| 956 | + |
| 957 | + def async_all_range(queryable, id_s, id_e, opts \\ []) do |
| 958 | + async_query(fn -> |
| 959 | + repo = get_dynamic_repo() |
| 960 | + |
| 961 | + EctoAdapterQueryable.execute_all_range( |
| 962 | + __MODULE__, |
| 963 | + repo, |
| 964 | + queryable, |
| 965 | + id_s, |
| 966 | + id_e, |
| 967 | + Ecto.Repo.Supervisor.tuplet( |
| 968 | + repo, |
| 969 | + Keyword.merge( |
| 970 | + __MODULE__.default_options(:all), |
| 971 | + opts |
| 972 | + ) |
| 973 | + ) |
| 974 | + ) |
| 975 | + end) |
| 976 | + end |
| 977 | + |
| 978 | + def all_range(queryable, id_s, id_e, opts \\ []) do |
| 979 | + future = async_all_range(queryable, id_s, id_e, opts) |
| 980 | + await(future) |
| 981 | + end |
| 982 | + |
| 801 983 | def async_get(queryable, id, opts \\ []), |
| 802 984 | do: async_query(fn -> get(queryable, id, opts ++ [returning: {:future, :one}]) end) |
| @@ -1,15 +1,12 @@ | |
| 1 1 | defmodule Ecto.Adapters.FoundationDB.EctoAdapterAssigns do |
| 2 2 | @moduledoc false |
| 3 3 | alias EctoFoundationDB.Future |
| 4 | + alias EctoFoundationDB.Indexer.SchemaMetadata |
| 4 5 | alias EctoFoundationDB.Layer.Tx |
| 5 6 | |
| 6 7 | def assign_ready(_module, repo, futures, ready_refs, options) when is_list(ready_refs) do |
| 7 8 | Tx.transactional(options[:prefix], fn _tx -> |
| 8 | - {assign_futures_rev, futures} = |
| 9 | - Enum.reduce(ready_refs, {[], futures}, fn ready_ref, {acc, futures} -> |
| 10 | - {assign_future, futures} = repo.async_assign_ready(futures, ready_ref, options) |
| 11 | - {[assign_future | acc], futures} |
| 12 | - end) |
| 9 | + {assign_futures_rev, futures} = filter_ready(repo, futures, ready_refs, options) |
| 13 10 | |
| 14 11 | res = repo.await(Enum.reverse(assign_futures_rev)) |
| 15 12 | |
| @@ -20,6 +17,18 @@ defmodule Ecto.Adapters.FoundationDB.EctoAdapterAssigns do | |
| 20 17 | end) |
| 21 18 | end |
| 22 19 | |
| 20 | + defp filter_ready(repo, futures, ready_refs, options) do |
| 21 | + Enum.reduce(ready_refs, {[], futures}, fn ready_ref, {acc, futures} -> |
| 22 | + case async_assign_ready(__MODULE__, repo, futures, ready_ref, options) do |
| 23 | + {nil, futures} -> |
| 24 | + {acc, futures} |
| 25 | + |
| 26 | + {assign_future, futures} -> |
| 27 | + {[assign_future | acc], futures} |
| 28 | + end |
| 29 | + end) |
| 30 | + end |
| 31 | + |
| 23 32 | defp append_new_future(futures, nil), do: futures |
| 24 33 | defp append_new_future(futures, future), do: [future | futures] |
| 25 34 | |
| @@ -27,27 +36,41 @@ defmodule Ecto.Adapters.FoundationDB.EctoAdapterAssigns do | |
| 27 36 | when is_reference(ready_ref) do |
| 28 37 | case Future.find_ready(futures, ready_ref) do |
| 29 38 | {nil, futures} -> |
| 30 | - {[], futures} |
| 39 | + {nil, futures} |
| 31 40 | |
| 32 41 | {future, futures} -> |
| 33 | - {schema, id, watch_options} = Future.result(future) |
| 42 | + {schema, query, watch_options, new_watch_fn} = Future.result(future) |
| 34 43 | |
| 35 44 | if not Keyword.has_key?(watch_options, :label) do |
| 36 | - raise "To use Repo.assign_ready/3, you must have previously called Repo.watch(struct, label: mykey)." |
| 45 | + raise """ |
| 46 | + To use Repo.assign_ready/3, you must have previously created a watch with a label |
| 47 | + |
| 48 | + Examples: |
| 49 | + |
| 50 | + Repo.watch(struct, label: :mykey) |
| 51 | + SchemaMetadata.watch_collection(MySchema, label: :mykey) |
| 52 | + """ |
| 37 53 | end |
| 38 54 | |
| 39 | - async_get(repo, futures, schema, id, watch_options, options) |
| 55 | + case query do |
| 56 | + {:pk, pk} -> |
| 57 | + async_get(repo, futures, schema, pk, watch_options, options, new_watch_fn) |
| 58 | + |
| 59 | + {SchemaMetadata, name} |
| 60 | + when name in [:inserts, :deletes, :collection, :updates, :changes] -> |
| 61 | + async_all(repo, futures, schema, watch_options, options, new_watch_fn) |
| 62 | + end |
| 40 63 | end |
| 41 64 | end |
| 42 65 | |
| 43 | - defp async_get(repo, futures, schema, id, watch_options, options) do |
| 66 | + defp async_get(repo, futures, schema, id, watch_options, options, new_watch_fn) do |
| 44 67 | label = watch_options[:label] |
| 45 68 | |
| 46 69 | Tx.transactional(options[:prefix], fn _tx -> |
| 47 70 | assign_future = |
| 48 71 | repo.async_get(schema, id, options) |
| 49 72 | |> Future.apply(fn struct_or_nil -> |
| 50 | - new_future = maybe_new_watch(repo, struct_or_nil, watch_options, options) |
| 73 | + new_future = maybe_new_watch(struct_or_nil, watch_options, options, new_watch_fn) |
| 51 74 | |
| 52 75 | {[{label, struct_or_nil}], new_future} |
| 53 76 | end) |
| @@ -56,9 +79,25 @@ defmodule Ecto.Adapters.FoundationDB.EctoAdapterAssigns do | |
| 56 79 | end) |
| 57 80 | end |
| 58 81 | |
| 59 | - defp maybe_new_watch(repo, struct_or_nil, watch_options, options) do |
| 60 | - if not is_nil(struct_or_nil) and Keyword.get(options, :watch?, false) do |
| 61 | - repo.watch(struct_or_nil, Keyword.merge(options, watch_options)) |
| 82 | + defp async_all(repo, futures, schema, watch_options, options, new_watch_fn) do |
| 83 | + label = watch_options[:label] |
| 84 | + |
| 85 | + Tx.transactional(options[:prefix], fn _tx -> |
| 86 | + assign_future = |
| 87 | + repo.async_all(schema, options) |
| 88 | + |> Future.apply(fn result -> |
| 89 | + new_future = maybe_new_watch(result, watch_options, options, new_watch_fn) |
| 90 | + |
| 91 | + {[{label, result}], new_future} |
| 92 | + end) |
| 93 | + |
| 94 | + {assign_future, futures} |
| 95 | + end) |
| 96 | + end |
| 97 | + |
| 98 | + defp maybe_new_watch(result, watch_options, options, new_watch_fn) do |
| 99 | + if Keyword.get(options, :watch?, false) do |
| 100 | + new_watch_fn.(result, watch_options) |
| 62 101 | else |
| 63 102 | nil |
| 64 103 | end |
| @@ -1,6 +1,67 @@ | |
| 1 1 | defmodule Ecto.Adapters.FoundationDB.EctoAdapterAsync do |
| 2 2 | @moduledoc false |
| 3 | + alias Ecto.Adapters.FoundationDB |
| 4 | + alias EctoFoundationDB.Exception.Unsupported |
| 3 5 | alias EctoFoundationDB.Future |
| 6 | + alias EctoFoundationDB.Layer.Fields |
| 7 | + alias EctoFoundationDB.Layer.Tx |
| 8 | + alias EctoFoundationDB.Versionstamp |
| 9 | + import Ecto.Query |
| 10 | + |
| 11 | + def async_insert_all!(_module, repo, schema, list, opts) do |
| 12 | + {tx?, tenant} = Tx.in_tenant_tx?() |
| 13 | + |
| 14 | + if not tx?, |
| 15 | + do: raise(Unsupported, "`Repo.async_insert_all!` must be called within a transaction") |
| 16 | + |
| 17 | + pk_field = Fields.get_pk_field!(schema) |
| 18 | + |
| 19 | + tx = Tx.get() |
| 20 | + |
| 21 | + forced_no_conflict? = [] == Keyword.get(opts, :conflict_target) |
| 22 | + |
| 23 | + list = |
| 24 | + for x <- list do |
| 25 | + if not is_struct(x, schema) do |
| 26 | + raise Unsupported, """ |
| 27 | + `Repo.async_insert_all!` must be called with a list of Ecto.Schema structs |
| 28 | + """ |
| 29 | + end |
| 30 | + |
| 31 | + pk = Map.get(x, pk_field) |
| 32 | + |
| 33 | + x = |
| 34 | + if is_nil(pk) and |
| 35 | + schema.__schema__(:type, pk_field) == Versionstamp do |
| 36 | + Map.put(x, pk_field, Versionstamp.next(tx)) |
| 37 | + else |
| 38 | + x |
| 39 | + end |
| 40 | + |
| 41 | + pk = Map.get(x, pk_field) |
| 42 | + |
| 43 | + if not forced_no_conflict? and not Versionstamp.incomplete?(pk) do |
| 44 | + raise Unsupported, """ |
| 45 | + `Repo.async_insert_all!` is designed to be called with either |
| 46 | + |
| 47 | + 1. A list of Ecto.Schema structs with incomplete Versionstamp in the `:id` field |
| 48 | + 2. The option `conflict_target: []`. (Make sure you understand the implications of this option) |
| 49 | + """ |
| 50 | + end |
| 51 | + |
| 52 | + x |
| 53 | + end |
| 54 | + |
| 55 | + # The insert_all function does not return the structs, so instead we make sure individual calls to `insert!` are |
| 56 | + # non-blocking by enforcing Versionstamps or conflict_target == [] |
| 57 | + result = for x <- list, do: repo.insert!(x, opts) |
| 58 | + |
| 59 | + vs_future = Versionstamp.get(tx) |
| 60 | + |
| 61 | + Future.apply(vs_future, fn vs -> |
| 62 | + Enum.map(result, &resolve_versionstamp(tenant, &1, vs, pk_field)) |
| 63 | + end) |
| 64 | + end |
| 4 65 | |
| 5 66 | def async_query(_module, repo, fun) do |
| 6 67 | # Executes the repo function (e.g. get, get_by, all, etc). Caller must ensure |
| @@ -12,29 +73,48 @@ defmodule Ecto.Adapters.FoundationDB.EctoAdapterAsync do | |
| 12 73 | nil -> |
| 13 74 | raise "Pipelining failure" |
| 14 75 | |
| 15 | - future -> |
| 16 | - schema = Future.schema(future) |
| 17 | - |
| 18 | - Future.apply(future, fn {all_or_one, result} -> |
| 19 | - handle_all_or_one(repo, schema, all_or_one, result) |
| 76 | + {{source, schema}, future} -> |
| 77 | + Future.apply(future, fn {return_handler, result} -> |
| 78 | + invoke_return_handler(repo, source, schema, return_handler, result) |
| 20 79 | end) |
| 21 80 | end |
| 22 81 | after |
| 23 82 | Process.delete(Future.token()) |
| 24 83 | end |
| 25 84 | |
| 26 | - defp handle_all_or_one(repo, schema, all_or_one, result) do |
| 85 | + defp invoke_return_handler(repo, source, schema, return_handler, result) do |
| 27 86 | if is_nil(result), do: raise("Pipelining failure") |
| 28 87 | |
| 88 | + queryable = if is_nil(schema), do: source, else: schema |
| 89 | + |
| 29 90 | # Abuse a :noop option here to signal to the backend that we don't |
| 30 91 | # actually want to run a query. Instead, we just want the result to |
| 31 92 | # be transformed by Ecto's internal logic. |
| 32 | - case all_or_one do |
| 93 | + case return_handler do |
| 33 94 | :all -> |
| 34 | - repo.all(schema, noop: result) |
| 95 | + repo.all(queryable, noop: result) |
| 35 96 | |
| 36 97 | :one -> |
| 37 | - repo.one(schema, noop: result) |
| 98 | + repo.one(queryable, noop: result) |
| 99 | + |
| 100 | + :all_from_source -> |
| 101 | + {select_fields, data_result} = result |
| 102 | + query = from(_ in source, select: ^select_fields) |
| 103 | + repo.all(query, noop: data_result) |
| 38 104 | end |
| 39 105 | end |
| 106 | + |
| 107 | + defp resolve_versionstamp(tenant, x, vs, pk_field) do |
| 108 | + pk = Map.get(x, pk_field) |
| 109 | + |
| 110 | + x = |
| 111 | + if Versionstamp.incomplete?(pk) do |
| 112 | + pk = Versionstamp.resolve(Map.get(x, pk_field), vs) |
| 113 | + Map.put(x, pk_field, pk) |
| 114 | + else |
| 115 | + x |
| 116 | + end |
| 117 | + |
| 118 | + FoundationDB.usetenant(x, tenant) |
| 119 | + end |
| 40 120 | end |
Loading more files…