meshx_consul

defmodule MeshxConsul do
  @readme File.read!("docs/README.md") |> String.split("<!-- MDOC !-->") |> Enum.fetch!(1)

  @moduledoc """
  #{@readme}

  ## Configuration options
  #### Consul Agent
  `MeshxConsul` requires Consul Agent API endpoint address and ACL token to manage services and upstreams. Additionally environment variables for command starting proxy binary should be configured here.

  List of shell environment variables, command options and http request headers supported by Envoy Proxy: [consul.io](https://www.consul.io/commands/connect/envoy), Consul Connect Proxy: [consul.io](https://www.consul.io/commands/connect/proxy).
  * `:cli_env` - shell environment variables that will be passed with command starting sidecar service proxy binary. Variables are defined as tuple, first element being variable name and second variable value. Environment variables can be used as alternative to proxy command arguments. Environment variables are preferred over command arguments when passing secrets, eg. Consul ACL token. Example:
  ```elixir
  cli_env: [
    {"CONSUL_HTTP_ADDR", "unix:///run/consul/http.sock"},
    {"CONSUL_HTTP_TOKEN", ""}
  ]
  ```
  Default: `[]`.
  * `:uri` - `%URI{}` scheme which should be used when accessing Consul agent http(s) API endpoint. Default: `%URI{scheme: "http", host: ""}`.

  <br>
  `MeshxConsul` uses `:httpc.request/4` function when accessing Consul agent http endpoint; some `:httpc` configuration is required.
  * `:httpc_opts` - (required) option passed directly to `:httpc.set_options/1`. It specifies options `:httpc` will use for subsequent http(s) requests.
  Example:
  ```elixir
  httpc_opts: [
    ipfamily: :local,
    unix_socket: '/run/consul/http.sock'
  ]
  ```
  * `:httpc_headers` - `:httpc` http request headers used when running `:httpc.request/4`.
  Example:
  ```elixir
  httpc_headers: [{'X-Consul-Token', ''}]
  ```
  Default:  [].
  * `:httpc_request_http_options` - `:httpc` http request options, passed as 3rd argument of `:httpc.request/4`. Default:  [].
  * `:httpc_request_options` - `:httpc` options, passed as 4th argument of `:httpc.request/4`. Default:  [].

  #### Templates
  * `service_template` - default value for `template` argument of `start/4` function. Check `start/4` description below for details. Default:  [].
  * `upstream_template` - default value for `template` argument of `connect/3` function. Check `connect/4` description below for details. Default:  [].

  #### Proxy management
  * `proxy_stdout_fun` - 3-arity function invoked when binary proxy command will provide `stdout` output.
  First function argument is proxy service ID, second argument is output device as atom `in [:stdout, :stderr]`. Last third argument is message generated by proxy command. Example:
  ```elixir
  fn _service_id, _dev, msg -> IO.inspect(msg) end
  ```
  Default: function sending formatted args to `Logger.debug/2`.
  * `proxy_stderr_fun` - as above `proxy_stdout_fun`, invoked when proxy command generates `stderr` output.
  Default: function sending formatted args to `Logger.error/2`.
  * `proxy_down_fun` - 5-arity function invoked when binary proxy command dies. Function arguments are as follows:
  1. proxy service ID,
  2. pid of process which was running proxy command,
  3. OS pid of process which was running proxy command,
  4. reason command died,
  5. number of proxy command restarts so far.

  Example:
  ```elixir
  fn _service_id, _pid, _ospid, reason, _restarts -> IO.inspect(reason) end
  ```
  Default: function sending formatted args to `Logger.error/2`.
  * `max_proxy_restarts` - when binary proxy command dies it is automatically restarted; option specifies maximum number of allowed command restarts. Default: `5`.

  #### TCP port generation
  * `tcp_address` - option used to specify TCP address and associated ports range that will be used to automatically find new unused TCP port number when preparing mesh service or upstream endpoint with `start/4` or `connect/3`. It accepts keyword list: `[ip: ip, port_range: range]`. `ip` specifies network interface address which will be used by endpoint. `ip` should be defined as tuple and in most situations it should point at loopback interface: `{127, 0, 0, 1}`. TCP traffic passing here is unencrypted, it means that unauthorized users should never have access to this interface. It never should be a public interface, even in private networks. `port_range` specifies range in which available TCP ports will be allocated. Service ports are starting from lower range limit and are increasing, upstream ports are decreasing from upper range limit. Default: `[ip: {127, 0, 0, 1}, port_range: 1024..65535]`.

  ## Templates
  `MeshxConsul` is using customized Mustache template system [[wikipedia](https://en.wikipedia.org/wiki/Mustache_(template_system))] to render following items:
  * registration data when registering service or upstream with Consul agent,
  * binary proxy command when starting sidecar proxy,
  * service TTL health check worker ID.

  Original Mustache system implementation assumes that rendered templates are defined as strings.
  `MeshxConsul` prefers structured data (maps and lists) as templates, with individual template fields defined as strings and being Mustache rendered.

  To allow variables escaping in structured data, `$` notation is added to standard Mustache specification. For example (see `"int"` template key):
  ```elixir
  hash_params = %{"string_key" => "123abc", "int_key" => 123}
  template = %{"string" => "{{string_key}}", "int" => "{{$int_key$}}", "static" => "static_string"}
  ```
  will be rendered by `MeshxConsul` Mustache extended version to:
  ```elixir
  %{"string" => "123abc", "int" => 123, "static" => "static_string"}
  ```
  """

  @typedoc """
  Mesh endpoint address for user service providers and upstream clients.

  **Note:** UDS (Unix Domain Socket) support should be available in Consul 1.10?, see [pull #9981](https://github.com/hashicorp/consul/pull/9981).
  """
  @type address() ::
          {:tcp, ip :: :inet.ip_address(), port :: :inet.port_number()}
          | {:uds, path :: String.t()}

  @behaviour Meshx.ServiceMesh
  require Logger
  alias MeshxConsul.{App.C, Dummy, Proxy, Ttl}
  alias MeshxConsul.Service.{Mustache, Template, Reg, Ets, GenTcpPort}

  @doc """
  Consul configuration info for `service_id`.

  Function returns result of Consul API `GET` query at path `/agent/service/:service_id`.
  ```elixir
  iex(1)> MeshxConsul.start({"service1", "service1-mynode-myhost"})
  {:ok, "service1-mynode-myhost", {:tcp, {127, 0, 0, 1}, 1024}}
  iex(2)> MeshxConsul.info("service1-mynode-myhost")
  {:ok,
  %{
   "Address" => "",
   "ContentHash" => "aaaaaa0000000000",
   "Datacenter" => "my-dc",
   "EnableTagOverride" => false,
   "ID" => "service1-mynode-myhost",
   "Meta" => %{},
   "Port" => 0,
   "Service" => "service1",
   "Tags" => [],
   "Weights" => %{"Passing" => 1, "Warning" => 1}
  }}
  ```
  """
  @spec info(service_id :: String.t() | atom()) :: {:ok, info :: map()} | {:error, error :: term()} | term()
  def info(service_id), do: Reg.config(service_id)

  @doc """
  List services registered on current node.
  ```elixir
  iex(1)> MeshxConsul.start({"service1", "service1-mynode-myhost"})
  {:ok, "service1-mynode-myhost", {:tcp, {127, 0, 0, 1}, 1024}}
  iex(2)> MeshxConsul.list
  ["service1-mynode-myhost"]
  ```
  """
  @spec list() :: [String.t()]
  def list(), do: Ets.list()

  @doc """
  List upstreams registered with default proxy service.

  Default proxy service ID: "upstream-" concatenated with host name.
  """
  @spec list_upstream() :: [String.t()]
  def list_upstream() do
    {_name, id} = Template.default_proxy()
    list_upstream(id)
  end

  @doc """
  List upstreams registered with proxy `service_id`.

  ```elixir
  iex(1)> MeshxConsul.connect(["service1", :service2])
  {:ok,
  [
   ok: {:tcp, {127, 0, 0, 1}, 65535},
   ok: {:tcp, {127, 0, 0, 1}, 65534}
  ]}
  iex(2)> MeshxConsul.list
  ["upstream-h11"]
  iex(3)> MeshxConsul.list_upstream
  ["service1", "service2"]
  iex(4)> MeshxConsul.list_upstream("upstream-h11")
  ["service1", "service2"]
  iex(5)> MeshxConsul.list_upstream("not_existing")
  {:error, :service_not_owned}
  ```

  """
  @spec list_upstream(service_id :: String.t() | atom()) :: [String.t()]
  def list_upstream(service_id) do
    case Reg.get_upstreams(to_string(service_id)) do
      {:ok, upstreams, _proxy_name, _proxy_id, _proxy_conf} -> Enum.map(upstreams, fn u -> Map.fetch!(u, "DestinationName") end)
      e -> e
    end
  end

  @doc """
  Prepares mesh service endpoint when starting new user service provider.

  ## Basic use
  ```elixir
  iex(1)> MeshxConsul.start(:service1)
  {:ok, "service1-h11", {:tcp, {127, 0, 0, 1}, 1024}}
  ```
  If successful function returns tuple with registered service ID and mesh service endpoint `address()`. Service ID by default is service name concatenated with host name. User can start service providing both service name and service ID:
  ```elixir
  iex(1)> MeshxConsul.start({"service1", "service1-mynode-myhost"})
  {:ok, "service1-mynode-myhost", {:tcp, {127, 0, 0, 1}, 1024}}
  ```
  If service with same service ID was already registered by current node and service is healthy in Consul agent registry, function will return `{:ok, :already_started}`.
  If service with same service ID is registered with Consul agent but registration was not executed by current node function will return by default `{:error, :service_not_owned}`. User can force service re-registration with current node by setting `force_registration?` to `true`. If service was registered by current node but cannot be found in Consul agent registry function will return `{:error, :invalid_state}`.

  If `timeout` is set greater than `0`, `start/4` will wait `timeout` milliseconds for service to have "passing" state in Consul agent after registration. If service is not healthy and alive after `timeout` function will return `{:error, :service_alive_timeout}`. If `timeout` is to `0` this check will be skipped.

  ## Customization
  Consul agent documentation suggested reading:
  * [service registration options](https://www.consul.io/docs/discovery/services),
  * [health checks](https://www.consul.io/docs/discovery/checks),
  * [Connect commands](https://www.consul.io/commands/connect).

  If `params` function argument defines service as atom, string or `{name, id}` tuple following Mustache hash is created:
  ```elixir
  # 1. Generate new mesh service endpoint address:
  {:tcp, ip, port} = MeshxConsul.Service.GenTcpPort.new(:lo)
  # 2a. Build Mustache hash if name is given:
  %{"name" => name, "id" => name <> "-" <> hostname, "address" => ip, "port" => port}
  # 2b. Build Mustache hash if {name, id} is given:
  %{"name" => name, "id" => id, "address" => ip, "port" => port}
  # Example Mustache hash:
  %{"name" => "service1", "id" => "service1-my-hostname", "address" => "127.0.0.1", "port" => 1024}
  ```
  If `params` argument is defined by user as `map()`, keys `"address"` and `"port"` are used to inject automatically generated TCP port address similarly to code on snippet above. Automatic address injection can be cancelled by assigning both `"address"` and `"port"` keys some values, `"address"` must be not empty string and `"port"` any integer value, eg.: `%{"address" => "undefined", "port" => -999_999}`. If user provided values for both `"address"` and `"port"`, they will be fetched from input map and used to build function result `{:ok, service_id, address()}` tuple.

  If `template` is not defined as function argument, value of `config/config.exs` `:service_template` key will be used.
  If user `template` does not contain all required keys `[:registration, :ttl, :proxy]`, missing keys will be taken from following built-in defaults:
  ```elixir
    [
      registration: %{
        "ID" => "{{id}}",
        "Name" => "{{name}}",
        "Checks" => [
          %{
            "Name" => "TTL check",
            "CheckID" => "ttl:{{id}}",
            "TTL" => "10s"
          }
        ],
        "Connect" => %{
          "SidecarService" => %{
            "Proxy" => %{
              "LocalServiceAddress" => "{{address}}",
              "LocalServicePort" => "{{$port$}}"
            }
          }
        }
      },
      ttl: %{
        id: "ttl:{{id}}",
        status: "passing",
        ttl: 5_000
      },
      proxy: ["/bin/sh", "-c", "consul connect proxy -log-level err -sidecar-for {{id}}"]
    ]
  ```
  ### Example:
  ```elixir
  # start service using Envoy Proxy instead of default Consul Connect Proxy:
  iex(1)>  MeshxConsul.start("service1", proxy: ["/bin/sh", "-c", "consul connect envoy -sidecar-for {{id}} -- -l error"])
  {:ok, "service1-h11", {:tcp, {127, 0, 0, 1}, 1024}}
  ```
  """
  @spec start(
          params ::
            (name :: atom() | String.t())
            | {name :: atom() | String.t(), id :: atom() | String.t()}
            | map(),
          template ::
            [
              registration: map(),
              ttl: nil | %{id: String.t(), status: String.t(), ttl: pos_integer()},
              proxy: nil | [String.t()]
            ],
          force_registration? :: boolean(),
          timeout :: non_neg_integer()
        ) ::
          {:ok, service_id :: String.t(), addr :: address()}
          | {:ok, :already_started}
          | {:error, :invalid_state}
          | {:error, :service_not_owned}
          | {:error, :service_alive_timeout}
          | term()

  def start(params, template \\ C.service_template(), force_registration? \\ false, timeout \\ 5000)

  def start(name, template, force_registration?, timeout) when is_atom(name) or is_bitstring(name) do
    name = to_string(name)
    params = %{"name" => name, "id" => name <> "-" <> Template.default_nodename()}
    start(params, template, force_registration?, timeout)
  end

  def start({name, id}, template, force_registration?, timeout)
      when (is_atom(name) or is_bitstring(name)) and (is_atom(id) or is_bitstring(id)) do
    params = %{"name" => to_string(name), "id" => to_string(id)}
    start(params, template, force_registration?, timeout)
  end

  def start(params, template, force_registration?, timeout) do
    {params, address} = build_address(params, :lo)
    svc = Template.validate_service!(template)
    svc_reg = Keyword.fetch!(svc, :registration)
    ttl_check = Keyword.fetch!(svc, :ttl)
    proxy_cmd = Keyword.fetch!(svc, :proxy)

    with {:ok, reg} <- Mustache.render2map(svc_reg, params),
         {:ok, service_id} <- fetch_id(reg) do
      owned? = Ets.has_service?(service_id)
      passing? = Reg.passing?(service_id)

      cond do
        owned? and passing? ->
          {:ok, :already_started}

        owned? and !passing? ->
          {:error, :invalid_state}

        !owned? and passing? and !force_registration? ->
          {:error, :service_not_owned}

        true ->
          with {:ok, ttl} <- Mustache.ext_render2map(ttl_check, params, :atoms),
               {:ok, proxy} <- Mustache.ext_render(proxy_cmd, params),
               :ok <- Reg.register_service(reg, true),
               {:ok, _pid} <- Ttl.start(service_id, ttl),
               {:ok, _pid} <- Proxy.start(service_id, proxy),
               :ok <- wait_for_service(service_id, timeout),
               true <- Ets.insert(service_id) do
            {:ok, service_id, address}
          else
            err ->
              Ets.delete(service_id)
              Proxy.stop(service_id)
              Ttl.stop(service_id)
              Reg.deregister_proxy(service_id)
              Reg.deregister_service(service_id)
              log_start_err(params, template, address, force_registration?, err)
              err
          end
      end
    else
      err ->
        log_start_err(params, template, address, force_registration?, err)
        err
    end
  end

  @doc """
  Prepares mesh upstream endpoint for new user upstream client connection.

  ## Basic use
  Basic use of `connect/3` requires `upstream_params` argument to be a list of upstream names defined as strings or atoms:
  ```elixir
  iex(1)> MeshxConsul.connect(["service1", :service2])
  {:ok,
  [
   ok: {:tcp, {127, 0, 0, 1}, 65535},
   ok: {:tcp, {127, 0, 0, 1}, 65534}
  ]}
  ```
  Function returns list of tuples (keyword list), one tuple result per each `upstream_params` element, preserved ordering. Tuple elements are `{:ok, address()}` if upstream addition was successful or `{:error, reason}` if operation failed for given upstream. If upstream is already registered with `proxy`, function will return `{:ok, address()}` with mesh upstream endpoint `address()` fetched from Consul `proxy` sidecar registration.

  ## Customization
  Customization of `template` function argument requires understanding of [Consul upstream service configuration options](https://www.consul.io/docs/connect/registration/service-registration#upstream-configuration-reference).

  If `upstream_params` list element is defined as upstream name (atom or string), it is used to create following Mustache hash:
  ```elixir
  # 1. Generate new mesh upstream endpoint address:
  {:tcp, ip, port} = MeshxConsul.Service.GenTcpPort.new(:hi)
  # 2. Build Mustache hash:
  %{"name" => to_string(upstream_name), "address" => ip, "port" => port}
  # Example Mustache hash:
  %{"name" => "service1", "address" => "127.0.0.1", "port" => 65535}
  ```
  If `upstream_params` element is defined by user as `map()`, keys `"address"` and `"port"` are used to inject automatically generated TCP port address similarly to code on snippet above. Automatic address injection can be cancelled by assigning both `"address"` and `"port"` keys some values, `"address"` must be not empty string and `"port"` any integer value, eg.: `%{"address" => "undefined", "port" => -999_999}`. If user provided values for both `"address"` and `"port"`, they will be fetched from input map and used to build result `{:ok, address()}` tuple for given upstream.

  If Mustache upstream `template` is not defined as function argument, `:upstream_template` option value defined in `config/config.exs` will be used. If both are undefined (`nil`, empty map or empty list), following built-in upstream Mustache `template` will be used:
  ```elixir
  %{
    "DestinationName" => "{{name}}",
    "LocalBindAddress" => "{{address}}",
    "LocalBindPort" => "{{$port$}}"
  }
  ```
  Using Mustache hash from previous snippet, above template would register with `proxy` sidecar service following upstream:
  ```elixir
  {
    "DestinationType":"service",
    "DestinationName":"service1",
    "LocalBindAddress":"127.0.0.1",
    "LocalBindPort":65535,
    "MeshGateway":{}
  }
  ```
  **Note**: fields required by Consul in upstream registration `template`: `"DestinationName" (string)` and `"LocalBindPort" (int)`.

  Last `connect/3` function argument `proxy` specifies service registered with sidecar-proxy as a tuple `{proxy_service_name, proxy_service_id}`. Sidecar proxy service will be used as parent service for all upstreams in `upstream_params`. If proxy service name/id is not provided, it will be generated by concatenation of prefix `"upstream-"` with host name. If proxy service doesn't exist, new service will be started by running `MeshxConsul.start({proxy_service_name, proxy_service_id})`. If `start/4` fails, generated error will cascade to `connect/3` and upstreams will not be added to service mesh.

  Running `MeshxConsul.connect(["service1"])` on host `h11` should register following services with Consul agent:
  ```elixir
  {
  "upstream-h11": {
    "ID": "upstream-h11",
    "Service": "upstream-h11",
    "Tags": [],
    "Meta": {},
    "Port": 0,
    "Address": "",
    "Weights": {
      "Passing": 1,
      "Warning": 1
    },
    "EnableTagOverride": false,
    "Datacenter": "my-dc"
  },
  "upstream-h11-sidecar-proxy": {
    "Kind": "connect-proxy",
    "ID": "upstream-h11-sidecar-proxy",
    "Service": "upstream-h11-sidecar-proxy",
    "Tags": [],
    "Meta": {},
    "Port": 21001,
    "Address": "",
    "Weights": {
      "Passing": 1,
      "Warning": 1
    },
    "EnableTagOverride": false,
    "Proxy": {
      "DestinationServiceName": "upstream-h11",
      "DestinationServiceID": "upstream-h11",
      "LocalServiceAddress": "127.0.0.1",
      "LocalServicePort": 1024,
      "Upstreams": [
        {
          "DestinationType": "service",
          "DestinationName": "service1",
          "LocalBindAddress": "127.0.0.1",
          "LocalBindPort": 65535,
          "MeshGateway": {}
        }
      ],
      "MeshGateway": {},
      "Expose": {}
    },
    "Datacenter": "my-dc"
  }
  }
  ```
  """
  @spec connect(
          upstream_params :: [upstream :: atom() | String.t() | map()],
          template :: map(),
          proxy :: nil | {proxy_service_name :: String.t() | atom(), proxy_service_id :: String.t() | atom()}
        ) ::
          {:ok, []}
          | {:ok, [{:ok, addr :: address()} | {:error, err :: term()}]}
          | {:error, :invalid_state}
          | {:error, :service_not_owned}
          | term()
  def connect(
        upstream_params,
        template \\ C.upstream_template(),
        {proxy_service_name, proxy_service_id} = _proxy \\ Template.default_proxy()
      )
      when is_list(upstream_params) do
    proxy_service_name = to_string(proxy_service_name)
    proxy_service_id = to_string(proxy_service_id)

    case start({proxy_service_name, proxy_service_id}) do
      {:ok, :already_started} ->
        add_upstreams(upstream_params, template, proxy_service_id)

      {:ok, proxy_service_id, address} ->
        Dummy.start(proxy_service_id, address)
        add_upstreams(upstream_params, template, proxy_service_id)

      e ->
        e
    end
  end

  @doc """
  Stops service `service_id` started with `start/4`.
  Function reverses actions performed by `start/4`:
  * proxy binary command is terminated,
  * TTL health check worker is stopped,
  * service is deregistered with Consul agent.

  If service was not started by current node function returns: `{:error, :service_not_owned}`.

  ```elixir
  iex(1)> MeshxConsul.start(:service1)
  {:ok, "service1-h11", {:tcp, {127, 0, 0, 1}, 1024}}
  iex(2)> MeshxConsul.stop("service1-h11")
  :ok
  iex(3)> MeshxConsul.stop("service1-h11")
  {:error, :service_not_owned}
  ```
  """
  @spec stop(service_id :: String.t() | atom()) :: :ok | {:error, :service_not_owned}
  def stop(service_id) do
    service_id = to_string(service_id)

    if Ets.has_service?(service_id) do
      Ets.delete(service_id)
      Proxy.stop(service_id)
      Ttl.stop(service_id)
      Reg.deregister_proxy(service_id)
      Reg.deregister_service(service_id)
    else
      {:error, :service_not_owned}
    end
  end

  @doc """
  Disconnects mesh `upstreams` endpoints created earlier with `connect/3`.

  Function will deregister `upstreams` list from `proxy_service_id` parent sidecar proxy service:
  ```elixir
  iex(1)> MeshxConsul.connect(["service1", "service2"])
  {:ok,
  [
   ok: {:tcp, {127, 0, 0, 1}, 65535},
   ok: {:tcp, {127, 0, 0, 1}, 65534}
  ]}
  iex(2)> MeshxConsul.list_upstream
  ["service1", "service2"]
  iex(3)> MeshxConsul.disconnect(["service1", "service2", "service3"])
  {:ok, ["service2", "service1"]}
  iex(4)> MeshxConsul.list_upstream
  []
  ```
  Function returns list of disconnected upstreams.
  If `proxy_service_id` is not provided as function argument, default proxy ID will be used: "upstream-" concatenated with host name.

  Deregistering upstream from sidecar proxy service doesn't close established connections. Closing existing connections can be done by:
  * cold proxy restart: set `restart_proxy?` function argument to true,
  * hot proxy restart if supported by proxy, example script for Envoy: [[github](https://github.com/envoyproxy/envoy/blob/main/restarter/hot-restarter.py)].

  """
  @spec disconnect(
          upstreams :: [upstream :: atom() | String.t()],
          proxy_service_id :: nil | atom() | String.t(),
          restart_proxy? :: boolean()
        ) ::
          {:ok, []}
          | {:ok, [deleted_upstream_name :: String.t()]}
          | (err :: term())
  def disconnect(upstreams, proxy_service_id \\ nil, restart_proxy? \\ false) when is_list(upstreams) do
    proxy_service_id =
      if is_nil(proxy_service_id) do
        {_proxy_service_name, proxy_service_id} = Template.default_proxy()
        proxy_service_id
      else
        to_string(proxy_service_id)
      end

    with {:ok, current_upstreams, proxy_name, _proxy_id, proxy_conf} <- Reg.get_upstreams(proxy_service_id),
         {new_upstreams, deleted} <- del_upstreams(current_upstreams, upstreams),
         false <- Enum.empty?(deleted),
         :ok <- update_upstreams(proxy_conf, proxy_name, new_upstreams) do
      if restart_proxy?, do: Proxy.restart(proxy_service_id)
      {:ok, deleted}
    else
      true -> {:ok, []}
      err -> err
    end
  end

  defp add_upstreams(params, template, proxy_service_id) do
    case Reg.get_upstreams(proxy_service_id) do
      {:ok, curr_u, proxy_name, _proxy_id, proxy_conf} ->
        tpl = Template.validate_upstream!(template)
        curr_u_names = Enum.map(curr_u, fn u -> Map.fetch!(u, "DestinationName") end)

        {new_upstreams, addresses} = add_u(params, tpl, curr_u, curr_u_names)

        with false <- curr_u == new_upstreams,
             :ok <- update_upstreams(proxy_conf, proxy_name, new_upstreams) do
          {:ok, addresses}
        else
          true -> {:ok, addresses}
          e -> e
        end

      e ->
        e
    end
  end

  defp add_u(params, tpl, curr_u, curr_u_names, added \\ [])

  defp add_u([u | tail], tpl, curr_u, curr_u_names, added) when is_atom(u) or is_bitstring(u),
    do:
      add_u(
        [%{"name" => to_string(u)} | tail],
        tpl,
        curr_u,
        curr_u_names,
        added
      )

  defp add_u([u | tail], tpl, curr_u, curr_u_names, added) do
    tmp_u = maybe_inject_fake_address(u)

    case Mustache.render2map(tpl, tmp_u) do
      {:ok, tmp_render} ->
        name = Map.fetch!(tmp_render, "DestinationName")

        case Enum.find_index(curr_u_names, fn n -> n == name end) do
          nil ->
            {param, address} = build_address(u, :hi)

            case Mustache.render2map(tpl, param) do
              {:ok, render} ->
                name = Map.fetch!(render, "DestinationName")
                add_u(tail, tpl, curr_u ++ [render], curr_u_names ++ [name], added ++ [{:ok, address}])

              err ->
                add_u(tail, tpl, curr_u, curr_u_names, added ++ [{:error, err}])
            end

          index ->
            existing_u = Enum.at(curr_u, index)
            {:ok, ip} = Map.fetch!(existing_u, "LocalBindAddress") |> to_charlist() |> :inet.parse_address()
            port = Map.fetch!(existing_u, "LocalBindPort")
            add_u(tail, tpl, curr_u, curr_u_names, added ++ [{:ok, {:tcp, ip, port}}])
        end

      {:error, e} ->
        {:error, e}
    end
  end

  defp add_u([], _tpl, curr_u, _curr_u_names, added), do: {curr_u, added}

  defp del_upstreams(c_upstreams, upstreams, new_upstreams \\ [], deleted \\ [])

  defp del_upstreams([u | tail], upstreams, new_upstreams, deleted) do
    u_name = Map.fetch!(u, "DestinationName")

    if Enum.member?(upstreams, u_name),
      do: del_upstreams(tail, upstreams, new_upstreams, [u_name] ++ deleted),
      else: del_upstreams(tail, upstreams, [u] ++ new_upstreams, deleted)
  end

  defp del_upstreams([], _upstreams, new_upstreams, deleted), do: {new_upstreams, deleted}

  defp update_upstreams(proxy_conf, proxy_name, new_upstreams) do
    proxy_conf
    |> Map.drop(["ContentHash", "Datacenter", "Service"])
    |> Map.put("Name", proxy_name)
    |> put_in(["Proxy", "Upstreams"], new_upstreams)
    |> Reg.register_service(false)
  end

  defp fetch_id(service_reg) do
    service_id = Map.get(service_reg, "ID") || Map.get(service_reg, "Name")
    if is_nil(service_id), do: {:error, :missing_service_name}, else: {:ok, service_id}
  end

  defp log_start_err(params, template, address, force_registration?, err),
    do:
      Logger.error("""
      [#{__MODULE__}]: Problem starting service using registration params:
      #{inspect(params)}
      registration template:
      #{inspect(template)}
      address: #{inspect(address)},
      force registration: #{inspect(force_registration?)}
      Error:
      #{inspect(err)}
      """)

  defp build_address(params, tcp_range) do
    with {:ok, address} <- Map.fetch(params, "address"),
         {:ok, port} <- Map.fetch(params, "port"),
         true <- is_bitstring(address) and address != "" and is_integer(port) do
      {params, {:tcp, address, port}}
    else
      _ ->
        {:tcp, ip, port} = address = GenTcpPort.new(tcp_range)
        {Map.put(params, "address", :inet.ntoa(ip) |> to_string()) |> Map.put("port", port), address}
    end
  end

  defp maybe_inject_fake_address(u) when is_map(u) do
    with {:ok, address} <- Map.fetch(u, "address"),
         {:ok, port} <- Map.fetch(u, "port"),
         true <- is_bitstring(address) and address != "" and is_integer(port) do
      u
    else
      _ ->
        Map.put(u, "address", "") |> Map.put("port", -999_999)
    end
  end

  @healthy_retries 100
  defp wait_for_service(_service_id, 0), do: :ok

  defp wait_for_service(service_id, timeout, retry \\ 0) do
    if Reg.passing?(service_id) do
      :ok
    else
      if retry < @healthy_retries do
        Process.sleep(round(timeout / @healthy_retries))
        wait_for_service(service_id, timeout, retry + 1)
      else
        {:error, :service_alive_timeout}
      end
    end
  end
end