active_memory

defmodule ActiveMemory.Table do
  @moduledoc """

  Define your table attributes and options. 

  Example Table (without auto generated uuid):
  ```elixir
  defmodule Test.Support.People.Person do
    use ActiveMemory.Table,
      options: [index: [:last, :cylon?]]

    attributes do
      field :email
      field :first
      field :last
      field :hair_color
      field :age
      field :cylon?
    end
  end
  ```
  ### Auto Generated UUID
  A table can have an auto generated UUID. Specify the option `auto_generate_uuid: true` in the
  attributes as an option.

  Example Table with auto generated uuid:
  ```elixir
  defmodule Test.Support.People.Person do
    use ActiveMemory.Table,
      options: [index: [:last, :cylon?]]

    attributes auto_generate_uuid: true do
      field :email
      field :first
      field :last
      field :hair_color
      field :age
      field :cylon?
    end
  end
  ```

  ## Field types and Ecto Changesets
  Fields accept an optional [Ecto type](https://hexdocs.pm/ecto/Ecto.Schema.html#module-types-and-casting)
  as the second argument (defaulting to `:any` when omitted):

  ```elixir
  defmodule MyApp.Planet do
    use ActiveMemory.Table, type: :ets

    attributes auto_generate_uuid: true do
      field :name, :string
      field :gravity, :float
      field :moons, :integer, default: 0
    end
  end
  ```

  Types are not enforced on `write/1` — ETS and Mnesia store any term — they exist
  to power `Ecto.Changeset` casting and validation, which works directly on the
  table struct. `write/1` accepts the changeset itself, the way
  `c:Ecto.Repo.insert/2` does:

  ```elixir
  {:ok, planet} =
    %MyApp.Planet{}
    |> Ecto.Changeset.cast(params, [:name, :gravity, :moons])
    |> Ecto.Changeset.validate_required([:name])
    |> MyApp.Planet.Store.write()
  ```

  An invalid changeset returns `{:error, changeset}` with its `action` set to
  `:insert`, so a Phoenix form renders the errors without any extra work.

  The declared types are available as `__attributes__(:types)`.

  ## Using an Ecto schema instead of attributes
  A table can skip the `attributes` block entirely and define an Ecto
  `embedded_schema`. All table metadata is derived from the schema, and the module
  is a real Ecto schema, so every changeset function works out of the box:

  ```elixir
  defmodule MyApp.Comet do
    use ActiveMemory.Table, type: :ets

    use Ecto.Schema

    embedded_schema do
      field :name, :string
      field :orbit_years, :integer
    end
  end
  ```

  Notes for Ecto schema tables:
  - Autogenerated fields are honored: `write/1` fills any field the schema declares
    as autogenerated when it is still `nil`, leaving a value you set yourself alone.
    That covers the default `embedded_schema` primary key
    (`{:id, :binary_id, autogenerate: true}`), an explicit uuid key such as
    `@primary_key {:uuid, Ecto.UUID, autogenerate: true}`, and `timestamps()`.
  - An autogenerated **integer** primary key (`{:id, :id, autogenerate: true}`)
    cannot be generated in memory and raises when the table is created. Use
    `Ecto.UUID`/`:binary_id`, or `@primary_key false` and assign the key yourself.
  - With `@primary_key false` the first declared field is the table key.
  - Virtual fields are never stored; they reset to their defaults on read.
  - `timestamps()` are stamped on the first write that leaves them `nil`, but they
    are not refreshed on a later write — `write/1` is an upsert, not an update.
  - A table with a `ttl` must declare its own expiry field:
    `field :expires_at, :integer` (milliseconds since epoch, stamped on write).

  ## Options when creating tables
  `ActiveMemory.Table` support almost all of the same options as `:ets` and `:mneisia`. 
  Please be aware that the options are different for `:ets` and `:mneisia`. 
  Further reading can be found with [ETS docs](https://www.erlang.org/doc/man/ets.html) and [Mnesia docs](https://www.erlang.org/doc/man/mnesia.html).
  All options should be structured as a [Keyword list](https://hexdocs.pm/elixir/1.12/Keyword.html). 

  Example:
  ```elixir
  use ActiveMemory.Table,
    type: :ets,
    options: [compressed: true, read_concurrency: true, type: :protected]
  ```

  ### Record Expiry (`ttl`)
  Pass a `ttl` (time-to-live, in milliseconds) to give every record in the table a
  lifetime. Expiry is tracked in an `expires_at` field holding milliseconds since
  the epoch, stamped on each write as `now + ttl`.

  ```elixir
  use ActiveMemory.Table,
    type: :ets,
    ttl: :timer.hours(1)
  ```

  In an `attributes` block the field is added for you, appended last so it never
  becomes the table key.

  **An Ecto schema table must declare it explicitly**, because ActiveMemory does not
  add fields to a schema you wrote — the schema stays the single description of the
  record, exactly as Ecto treats it:

  ```elixir
  defmodule MyApp.Token do
    use ActiveMemory.Table, type: :ets, ttl: :timer.minutes(15)

    use Ecto.Schema

    @primary_key {:uuid, Ecto.UUID, autogenerate: true}
    embedded_schema do
      field :value, :string
      field :expires_at, :integer
    end
  end
  ```

  Declare it last, so it does not take the key position, and type it `:integer`. A
  `ttl` table without an `expires_at` field would silently never expire, so the
  table raises when it is created instead.

  Expiry is enforced in two complementary ways (see `ActiveMemory.Store` and
  `ActiveMemory.ActiveRepo`): reads never return an expired record, and the owning
  `Store`/`ActiveRepo` periodically sweeps expired records to reclaim memory. This
  makes a `ttl` table well suited to one time use tokens, 2FA codes, magic links
  and similar short-lived data.

  ### Mnesia Options
  #### Table Read and Write Access
  Mnesia tables can be set to `read_only` or `read_write`. The default is `read_write`.
  Read only tables updates cannot be performed.
  if you need to change the access use the following syntax: `[access_mode: :read_only]`

  #### Table Types
  Tables can be either a `:set`, `:ordered_set`, or a `:bag`. The default is `:set`
  if you need to change the type use the following syntax: `[type: :bag]`

  #### Disk Copies
  A list of nodes can be specified to maintain disk copies of the table. Nodes specified will recieve a replica of the table. Disk copy talbes still maintain a ram copy of the table as well.
  By default all tables are `ram_copies` and no `disc_copies` are specified.
  if you need to specify nodes use following syntax: `[disc_copies: [node1, node2, node3, ...]]`

  #### Disk Only Copies
  A list of nodes can be specified to maintain only disk copies. A disc only table replica is kept on disc only and unlike the other replica types, the contents of the replica do not reside in RAM. These replicas are considerably slower than replicas held in RAM.
  if you need to specify nodes use following syntax: `[disc_only_copies: [node1, node2, node3, ...]]`

  #### Ram Copies
  A list of nodes can be specified to maintain ram copies of the table. Nodes specified will recieve a replica of the table.
  By default all tables are set to ram_copies: `[ram_copies: [node()]]`
  if you need to specify nodes use following syntax: `[ram_copies: [node1, node2, node3, ...]]`

  #### Indexes
  If Indexes are desired specify an atom attribute list for which Mnesia is to build and maintain an extra index table. 
  The qlc query compiler may be able to optimize queries if there are indexes available.
  To specify Indexes use the following syntax: `[index: [:age, :hair_color, :cylon?]]`

  #### Table Load Order
  The load order priority is by default 0 (zero) but can be set to any integer. The tables with the highest load order priority are loaded first at startup.
  If you need to change the load order use the following syntax: `[load_order: 2]`

  #### Majority (quorum writes, and surviving a network partition)
  `[majority: true]` requires a **majority of a table's replicas to be reachable
  before any transactional write commits**. A write on the minority side of a network
  partition is aborted rather than accepted, which is the single most effective
  thing you can do about Mnesia's partition behavior. It defaults to `false`.

  ```elixir
  use ActiveMemory.Table,
    options: [
      majority: true,
      ram_copies: [:"node1@host", :"node2@host", :"node3@host"]
    ]
  ```

  ##### Why it matters
  With the default `majority: false`, each side of a partition keeps accepting writes
  against its own replicas. Mnesia does not merge conflicting histories, and it does
  not stop you from creating them. When the nodes reconnect and each has logged the
  other as down, Mnesia emits an
  `{inconsistent_database, running_partitioned_network, node}` system event — and the
  default handler **logs an error and carries on**, serving whichever replica a given
  node reads from.

  That is not carelessness: there is no correct automatic merge without knowing what
  the data means. But it does mean divergence is not loud, and recovery is operator
  work — choose an authoritative replica with `:mnesia.set_master_nodes/1,2` and
  restart the nodes that should resynchronise from it, or restore from a backup. It is
  the reason many teams give up on Mnesia.

  With `majority: true` the minority side refuses writes for that table, so there is
  much less to reconcile — the same trade CP systems make, and roughly what Mnesia's
  `pause_minority` strategy does at the node level.

  ##### What it costs
    - Writes on the minority side fail. Availability is traded for consistency.
    - It needs an odd number of replicas to be useful; with two replicas neither side
      of a split holds a majority, so writes stop on both.
    - It gates **updates**, not reads, and dirty operations bypass it entirely.
      ActiveMemory's Mnesia reads run inside a transaction but commit nothing, so
      they still succeed on the minority side and return that replica's contents,
      which may be behind the majority's.
    - It is per table, so a table can opt in without changing the rest.

  ##### If you need more than this
  Quorum writes reduce divergence; they do not make Mnesia a partition tolerant
  database. If your data genuinely cannot tolerate a partition, the options are to
  keep the system of record in a database and treat the ActiveMemory table as
  derived, or to reach for a consensus backed store such as
  [Khepri](https://hexdocs.pm/khepri) when the data model suits a leader and quorum.
  Khepri is not a drop-in for arbitrary Mnesia tables — it is a tree structured store
  built for strongly consistent state, and the RabbitMQ team adopted it for their
  metadata rather than as a general replacement.

  For a single node application none of this applies: the only replica is always a
  majority, so `majority: true` adds no availability constraint.

  ### ETS Options
  #### Table Access
  Access options are: `:public` `:protected` or `:private`. The default access is `:public` 
  if you need to change the access use the following syntax: `[access: :private]`

  #### Table Types
  Tables can be either a `:set`, `:ordered_set`, `:bag`, or a `:duplicate_bag`. The default is `:set`
  if you need to change the type use the following syntax: `[type: :bag]`

  #### Compression
  Compression can be used to help shrink the size of the memory the data consumes, however this does mean the access is slower.
  The default is `false` where no compression happens. 
  if you need to change the compression use the following syntax: `[Compression: true]`

  #### Read Concurrency
  From ETS documentation:
  Performance tuning. Defaults to `false`. When set to true, the table is optimized for concurrent read operations. When this option is enabled read operations become much cheaper; especially on systems with multiple physical processors. However, switching between read and write operations becomes more expensive.
  You typically want to enable this option when concurrent read operations are much more frequent than write operations, or when concurrent reads and writes comes in large read and write bursts (that is, many reads not interrupted by writes, and many writes not interrupted by reads).
  You typically do not want to enable this option when the common access pattern is a few read operations interleaved with a few write operations repeatedly. In this case, you would get a performance degradation by enabling this option.
  Option read_concurrency can be combined with option write_concurrency. You typically want to combine these when large concurrent read bursts and large concurrent write bursts are common.
  if you need to change the read_concurrency use the following syntax: `[read_concurrency: true]`

  #### Write Concurrency
  From ETS documentation:
  Performance tuning. Defaults to `false`, in which case an operation that mutates (writes to) the table obtains exclusive access, blocking any concurrent access of the same table until finished. If set to true, the table is optimized for concurrent write access. Different objects of the same table can be mutated (and read) by concurrent processes. This is achieved to some degree at the expense of memory consumption and the performance of sequential access and concurrent reading.
  The auto alternative for the write_concurrency option is similar to the true option but automatically adjusts the synchronization granularity during runtime depending on how the table is used. This is the recommended write_concurrency option when using Erlang/OTP 25 and above as it performs well in most scenarios.
  The write_concurrency option can be combined with the options read_concurrency and decentralized_counters. You typically want to combine write_concurrency with read_concurrency when large concurrent read bursts and large concurrent write bursts are common; for more information, see option read_concurrency. It is almost always a good idea to combine the write_concurrency option with the decentralized_counters option.
  Notice that this option does not change any guarantees about atomicity and isolation. Functions that makes such promises over many objects (like insert/2) gain less (or nothing) from this option.
  The memory consumption inflicted by both write_concurrency and read_concurrency is a constant overhead per table for set, bag and duplicate_bag when the true alternative for the write_concurrency option is not used. For all tables with the auto alternative and ordered_set tables with true alternative the memory overhead depends on the amount of actual detected concurrency during runtime. The memory overhead can be especially large when both write_concurrency and read_concurrency are combined.
  if you need to change the write_concurrency use the following syntax: `[write_concurrency: true]` or `[write_concurrency: :auto]`

  #### Decentralized Counters
  From ETS documentation:
  Performance tuning. Defaults to true for all tables with the write_concurrency option set to auto. For tables of type ordered_set the option also defaults to true when the write_concurrency option is set to true. The option defaults to false for all other configurations. This option has no effect if the write_concurrency option is set to false.
  When this option is set to true, the table is optimized for frequent concurrent calls to operations that modify the tables size and/or its memory consumption (e.g., insert/2 and delete/2). The drawback is that calls to info/1 and info/2 with size or memory as the second argument can get much slower when the decentralized_counters option is turned on.
  When this option is enabled the counters for the table size and memory consumption are distributed over several cache lines and the scheduling threads are mapped to one of those cache lines. The erl option +dcg can be used to control the number of cache lines that the counters are distributed over.
  if you need to change the decentralized_counters use the following syntax: `[decentralized_counters: true]`

  """
  alias ActiveMemory.Adapters.Helpers

  defmacro __using__(opts) do
    quote do
      import ActiveMemory.Table, only: [attributes: 1, attributes: 2]

      @before_compile ActiveMemory.Table

      Module.register_attribute(__MODULE__, :active_memory_fields, accumulate: true)
      Module.register_attribute(__MODULE__, :active_memory_query_fields, accumulate: true)
      Module.register_attribute(__MODULE__, :active_memory_field_sources, accumulate: true)
      Module.register_attribute(__MODULE__, :active_memory_types, accumulate: true)

      opts = unquote(Macro.expand(opts, __CALLER__))

      table_type = Keyword.get(opts, :type, :mnesia)

      table_options = Keyword.get(opts, :options, :defaults)

      Module.put_attribute(__MODULE__, :adapter, Helpers.set_adapter(table_type))

      Module.put_attribute(
        __MODULE__,
        :table_options,
        Helpers.build_options(table_options, table_type)
      )

      Module.put_attribute(__MODULE__, :ttl, Keyword.get(opts, :ttl, nil))
    end
  end

  defmacro attributes(opts \\ [], do: block) do
    define_attributes(opts, block)
  end

  defmacro field(name, type \\ :any, opts \\ []) do
    quote do
      ActiveMemory.Table.__field__(
        __MODULE__,
        unquote(name),
        unquote(type),
        unquote(opts)
      )
    end
  end

  @doc false
  def __after_compile__(%{module: _module}, _) do
    :ok
  end

  @doc false
  defmacro __before_compile__(env) do
    cond do
      Module.defines?(env.module, {:__attributes__, 1}) ->
        :ok

      Module.defines?(env.module, {:__schema__, 1}) ->
        define_ecto_schema_attributes(env.module)

      true ->
        raise ArgumentError,
              "#{inspect(env.module)} uses ActiveMemory.Table but defines no fields. " <>
                "Define them with an `attributes do ... end` block or with an Ecto schema " <>
                "(`embedded_schema do ... end`)."
    end
  end

  @doc false
  def __attributes__(fields, field_sources) do
    load =
      for name <- fields do
        if alias = field_sources[name] do
          {name, {:source, alias}}
        else
          name
        end
      end

    dump =
      for name <- fields do
        {name, field_sources[name] || name}
      end

    field_sources_quoted =
      for name <- fields do
        {[:field_source, name], field_sources[name] || name}
      end

    single_arg = [
      {[:dump], dump |> Map.new() |> Macro.escape()},
      {[:load], load |> Macro.escape()}
    ]

    catch_all = [
      {[:field_source, quote(do: _)], nil}
    ]

    [
      single_arg,
      field_sources_quoted,
      catch_all
    ]
  end

  @doc false
  # Ecto splits autogeneration in two: `:autogenerate_id` holds an `:id`/`:binary_id`
  # primary key, while `:autogenerate` holds every other autogenerating field (a
  # custom type such as `Ecto.UUID`, and `timestamps()`). Both are honored, as the
  # specs Ecto itself uses: `{fields, {module, function, args}}`.
  # ETS and Mnesia key a record on its first field, so that field is the primary
  # key regardless of what an Ecto schema declares. A declared key anywhere else
  # would silently read the wrong field, and a composite key cannot be expressed
  # at all, so both are rejected.
  def __primary_key__(module, declared, fields) do
    first = hd(fields)

    case declared do
      [] ->
        first

      [^first] ->
        first

      [elsewhere] ->
        raise ArgumentError,
              "#{inspect(module)} declares #{inspect(elsewhere)} as its primary key but " <>
                "#{inspect(first)} is the first field, and the first field is the table key. " <>
                "Declare the primary key first, or use `@primary_key false`."

      [_ | _] = composite ->
        raise ArgumentError,
              "#{inspect(module)} declares a composite primary key #{inspect(composite)}, " <>
                "which an in memory table cannot express. Use a single key field."
    end
  end

  @doc false
  def __autogenerate__(nil, autogenerate), do: autogenerate

  def __autogenerate__({field, _source, type}, autogenerate)
      when type in [:binary_id, Ecto.UUID] do
    [{[field], {Ecto.UUID, :autogenerate, []}} | autogenerate]
  end

  def __autogenerate__({field, _source, type}, _autogenerate) do
    raise ArgumentError,
          "cannot autogenerate the #{inspect(field)} primary key of type #{inspect(type)} " <>
            "in memory. Use `Ecto.UUID` or `:binary_id` for a generated key, or set " <>
            "`@primary_key false` and assign the value yourself."
  end

  @doc false
  def __field__(mod, name, type, opts) do
    {type, opts} = normalize_field_args(type, opts)

    validate_type!(name, type)

    define_field(mod, name, type, opts)
  end

  defp define_field(mod, name, type, opts) do
    put_struct_field(mod, name, Keyword.get(opts, :default))

    Module.put_attribute(mod, :active_memory_types, {name, type})

    Module.put_attribute(mod, :active_memory_query_fields, name)

    Module.put_attribute(mod, :active_memory_fields, name)
  end

  # `field :name, default: "x"` predates types: a keyword list in the type
  # position is the options list of an untyped field.
  defp normalize_field_args(opts, []) when is_list(opts), do: {:any, opts}

  defp normalize_field_args(type, opts), do: {type, opts}

  defp validate_type!(name, type) when not is_atom(type) and not is_tuple(type) do
    raise ArgumentError,
          "invalid type #{inspect(type)} for field #{inspect(name)}. " <>
            "Use an Ecto type such as :string, :integer, {:array, :string} or a custom type module."
  end

  defp validate_type!(_name, _type), do: :ok

  defp define_ecto_schema_attributes(module) do
    adapter = Module.get_attribute(module, :adapter)
    table_options = Module.get_attribute(module, :table_options)
    ttl = Module.get_attribute(module, :ttl)

    quote do
      def __attributes__(:adapter), do: unquote(adapter)

      def __attributes__(:auto_generate_uuid), do: false

      # Mirrors `auto_generate_uuid` for Ecto schema tables: every field the schema
      # declares as autogenerated is populated on write when it is still `nil`.
      def __attributes__(:autogenerate) do
        ActiveMemory.Table.__autogenerate__(
          __schema__(:autogenerate_id),
          __schema__(:autogenerate)
        )
      end

      def __attributes__(:match_head) do
        ActiveMemory.Adapters.Helpers.build_match_head(
          __attributes__(:query_map),
          __MODULE__,
          unquote(adapter)
        )
      end

      # The table key is the first stored field, so a schema whose declared primary
      # key sits elsewhere would make `get/1` read the wrong field.
      def __attributes__(:primary_key),
        do:
          ActiveMemory.Table.__primary_key__(
            __MODULE__,
            __schema__(:primary_key),
            __schema__(:fields)
          )

      def __attributes__(:query_fields), do: __schema__(:fields)

      def __attributes__(:query_map),
        do: ActiveMemory.Adapters.Helpers.build_query_map(__schema__(:fields))

      def __attributes__(:table_options), do: unquote(Macro.escape(table_options))

      def __attributes__(:ttl), do: unquote(ttl)

      def __attributes__(:types),
        do: Map.new(__schema__(:fields), fn field -> {field, __schema__(:type, field)} end)
    end
  end

  defp define_attributes(options, block) do
    prelude =
      quote do
        opts = unquote(options)

        @after_compile ActiveMemory.Table

        auto_generate_uuid = Keyword.get(opts, :auto_generate_uuid, false)

        Module.put_attribute(__MODULE__, :auto_generate_uuid, auto_generate_uuid)

        Module.register_attribute(__MODULE__, :active_memory_struct_fields, accumulate: true)

        if auto_generate_uuid do
          ActiveMemory.Table.__field__(
            __MODULE__,
            :uuid,
            Ecto.UUID,
            primary_key: true,
            autogenerate: true
          )
        end

        try do
          import ActiveMemory.Table
          unquote(block)
        after
          :ok
        end

        # When a `ttl` is configured the `expires_at` field is appended last so it
        # never displaces the table key (the first query field, or the uuid).
        if Module.get_attribute(__MODULE__, :ttl) do
          ActiveMemory.Table.__field__(__MODULE__, :expires_at, :integer, [])
        end
      end

    postlude =
      quote unquote: false do
        fields = @active_memory_fields |> Enum.reverse()
        active_memory_query_fields = @active_memory_query_fields |> Enum.reverse()
        field_sources = @active_memory_field_sources |> Enum.reverse()
        query_fields = Enum.map(active_memory_query_fields, & &1)
        query_map = Helpers.build_query_map(query_fields)
        types = Map.new(@active_memory_types)

        # A `uuid` attribute is generated on write, whether it came from
        # `auto_generate_uuid: true` or was declared by hand.
        autogenerate =
          if Enum.member?(fields, :uuid),
            do: [{[:uuid], {Ecto.UUID, :autogenerate, []}}],
            else: []

        defstruct Enum.reverse(@active_memory_struct_fields)

        def __attributes__(:adapter), do: unquote(Macro.escape(@adapter))

        def __attributes__(:auto_generate_uuid), do: unquote(Macro.escape(@auto_generate_uuid))

        def __attributes__(:autogenerate), do: unquote(Macro.escape(autogenerate))

        # The first attribute is the table key: `auto_generate_uuid: true` puts
        # `:uuid` there, otherwise it is the first field declared.
        def __attributes__(:primary_key), do: unquote(hd(query_fields))

        def __attributes__(:match_head),
          do:
            Helpers.build_match_head(
              unquote(query_map),
              unquote(__MODULE__),
              unquote(Macro.escape(@adapter))
            )

        def __attributes__(:query_fields), do: unquote(query_fields)

        def __attributes__(:query_map), do: unquote(query_map)

        def __attributes__(:table_options), do: unquote(Macro.escape(@table_options))

        def __attributes__(:ttl), do: unquote(Macro.escape(@ttl))

        def __attributes__(:types), do: unquote(Macro.escape(types))

        # The reflection `Ecto.Changeset.cast/4` looks up when given a struct,
        # so table structs can be cast and validated like any Ecto schema.
        @doc false
        def __changeset__, do: unquote(Macro.escape(types))

        for clauses <-
              ActiveMemory.Table.__attributes__(
                fields,
                field_sources
              ),
            {args, body} <- clauses do
          def __attributes__(unquote_splicing(args)), do: unquote(body)
        end
      end

    quote do
      unquote(prelude)
      unquote(postlude)
    end
  end

  defp put_struct_field(mod, name, assoc) do
    fields = Module.get_attribute(mod, :active_memory_struct_fields)

    if List.keyfind(fields, name, 0) do
      raise ArgumentError,
            "field/association #{inspect(name)} already exists on attributes, you must either remove the duplication or choose a different name"
    end

    Module.put_attribute(mod, :active_memory_struct_fields, {name, assoc})
  end
end