Current section

39 Versions

Jump to

Compare versions

24 files changed
+599 additions
-277 deletions
  @@ -0,0 +1,26 @@
1 + eDeliver Versions
2 + =================
3 +
4 + 1.1.1
5 +
6 + - added command line option `--mix-env=<env>`
7 + - automatic authorization of/on release store host
8 + - allow to build different apps in one project when `APP` env is used
9 + - use always explicit compilation for exrm compatibility
10 +
11 + __1.1.0__
12 +
13 + - allow incremental builds to decrease build time
14 + - support for executing ecto migrations
15 + - support exrm versions from `0.16.0` to > `1.0.0`
16 + - allow to link sys.config and vm.args in release
17 + - display command output when build command failed
18 + - keeps build repository size constant
19 + - return correct status code if mix command failed
20 + - deploy release non-interactively if also upgrade exists
21 + - install hex non-interactive
22 +
23 +
24 + __1.0.0__
25 +
26 + - initial version with elixir support
\ No newline at end of file
  @@ -8,29 +8,33 @@
8 8
9 9 The **[erlang releases](http://www.erlang.org/doc/design_principles/release_handling.html)** are **built** on a **remote host** that have a similar configuration to the deployment target systems and can then be **deployed to several production systems**.
10 10
11 - This is necessary because the **[release](http://www.erlang.org/doc/design_principles/release_handling.html) contains** the full **[erts (erlang runtime system)](http://erlang.org/doc/apps/erts/users_guide.html)**, all **[dependencies (erlang applications)](http://www.erlang.org/doc/design_principles/applications.html)**, native port drivers and **your own erlang application(s)** in a **standalone embedded node**.
11 + This is necessary because the **[release](http://www.erlang.org/doc/design_principles/release_handling.html) contains** the full **[erts (erlang runtime system)](http://erlang.org/doc/apps/erts/users_guide.html)**, all **[dependencies (erlang or elixir applications)](http://www.erlang.org/doc/design_principles/applications.html)**, **elixir runtime**, native port drivers and **your own erlang/elixir application(s)** in a **standalone embedded node**.
12 12
13 13 Examples:
14 14
15 15 **Build** an erlang/elixir release **and deploy** it on your **production hosts**:
16 16
17 - ./edeliver build release --branch=feature
18 - ./edeliver deploy release to production
19 - ./edeliver start production
17 + mix edeliver build release --branch=feature
18 + mix edeliver deploy release to production
19 + mix edeliver start production
20 20
21 21
22 22 Build a **live upgrade** from v1.0 to v2.0 for an erlang/elixir release and deploy it to your production hosts:
23 23
24 24 # build upgrade from tag v1.0 to v2.0
25 25
26 - ./edeliver build upgrade --from=v1.0 --to=v2.0
27 - ./edeliver deploy upgrade to production
26 + mix edeliver build upgrade --from=v1.0 --to=v2.0
27 + mix edeliver deploy upgrade to production
28 28
29 29 # or if you have the old release in your release store,
30 30 # you can build the upgrade with that old release instead of the old git revision/tag
31 31
32 - ./edeliver build upgrade --with=v1.0 --to=v2.0
33 - ./edeliver deploy upgrade to production
32 + mix edeliver build upgrade --with=v1.0 --to=v2.0
33 + mix edeliver deploy upgrade to production
34 +
35 + # run ecto migrations manually:
36 + mix edeliver migrate production
37 + # or automatically during upgrade when upgrade is built with --run-migrations
34 38
35 39 The deployed upgrade will be **available immediately, without restarting** your application. If the generated [upgrade instructions (relup)](http://www.erlang.org/doc/man/relup.html) for the hot code upgrade are not sufficient, you can modify these files before installing the upgrade by using the `edit relup` command.
36 40
  @@ -53,15 +57,25 @@ Edeliver tries to autodetect which system to use to compile the sources and buil
53 57
54 58 This can be overridden by the config variables `BUILD_CMD=rebar|mix` and `RELEASE_CMD=rebar|mix|relx`.
55 59
56 - If using [mix](http://elixir-lang.org/getting_started/mix/1.html), add it to you `mix.exs` config:
60 + __If using [mix](http://elixir-lang.org/getting_started/mix/1.html)__, add it as [hex package]({ :edeliver, "~> 1.0.0"},) to you `mix.exs` config:
57 61
58 62 defp deps do
59 - [{ :edeliver, github: "boldpoker/edeliver", compile: "mkdir -p ebin && cp src/edeliver.app.src ebin/edeliver.app" } ]
63 + [{:edeliver, ">= 1.1.1"}]
60 64 end
61 65
62 66 And run `mix do deps.get, deps.compile`. Edeliver is then available as __mix task__: `mix edeliver`.
63 67
64 - When using rebar, edeliver can be added as [rebar](https://github.com/basho/rebar) depencency. Just add it to your `rebar.config` (and ensure that a `./rebar` binary/link is in your project directory:
68 + For __additional edeliver commands__ `version`, `migrate` and `show migrations` __add edeliver as application for your relase__ in the `mix.exs` file:
69 +
70 + def application, do: [
71 + applications: [
72 + # ...
73 + :edeliver,
74 + ],
75 + ]
76 +
77 +
78 + __When using rebar__, edeliver can be added as [rebar](https://github.com/basho/rebar) depencency. Just add it to your `rebar.config` (and ensure that a `./rebar` binary/link is in your project directory:
65 79
66 80 {deps, [
67 81 % ...
  @@ -75,6 +89,8 @@ And link the `edeliver` binary to the root of your project directory:
75 89 ./rebar get-deps # when using rebar, or ...
76 90 ln -s ./deps/edeliver/bin/edeliver .
77 91
92 + Then use the linked binary `./edeliver` instead of the `mix edeliver` tasks from the examples.
93 +
78 94 ### Configuration
79 95
80 96 Create a `.deliver` directory in your project folder and add the `config` file:
  @@ -100,11 +116,13 @@ It uses ssh and scp to build and deploy the releases. Is is **recommended** that
100 116
101 117 Maybe it is required to __configure git on your build host__ (git user name / email) or to clone the repository initially at the `BUILD_AT` path. And of course you need to __install [erlang](http://www.erlang.org/) and [elixir](http://elixir-lang.org/)__ on the `BUILD_HOST`. If you use mix to build the releases, you should __install [hex](https://hex.pm)__ on the build hosts before the first build (otherwise mix asks interactively to install it). Run the build command with __`--verbose`__ if it fails the first time.
102 118
119 + To use __different configurations on different hosts__ for your erlang/elixir release, you can configure edeliver to link the `vm.args` and / or the `sys.config` files in the release package by setting the `LINK_VM_ARGS=/path/to/vm.args` and/or `LINK_SYS_CONFIG=/path/to/sys.config` variables in the edeliver config if you use [mix](http://elixir-lang.org/getting-started/mix-otp/introduction-to-mix.html) and [exrm](https://github.com/bitwalker/exrm) to build the releases.
120 +
103 121 There are **four kinds of commands**: **build commands** which compile the sources and build the erlang release **on the remote build system**, **deploy commands** which deliver the built releases to the **remote production systems**, **node commands** that **control** the nodes (e.g. starting/stopping) and **local commands**.
104 122
105 123 ### Build Commands
106 124
107 - The releases must be built on a system that is similar to the target system. E.g. if you want to deploy to a production system based on linux, the release must also be built on a linux system. Furthermore the [erlang runtime / OPT version](http://www.erlang.org/download.html) (e.g. OTP 17.5) of the remote build system is included into the release built and delivered to all production system. It is not required to install the otp runtime on the production systems.
125 + The releases must be built on a system that is similar to the target system. E.g. if you want to deploy to a production system based on linux, the release must also be built on a linux system. Furthermore the [erlang runtime / OPT version](http://www.erlang.org/download.html) (e.g. OTP 17.5) of the remote build system and the elixir runtime is included into the release built and delivered to all production system. It is not required to install the otp runtime nor elixir on the production systems.
108 126 For build commands the following **configuration** variables must be set:
109 127
110 128 - `APP`: the name of your release which should be built
  @@ -112,26 +130,28 @@ For build commands the following **configuration** variables must be set:
112 130 - `BUILD_USER`: the local user at build host
113 131 - `BUILD_AT`: the directory on build host where to build the release. must exist.
114 132
115 - The built release it then **copied to your local directory** `.deliver/releases` and can then be **delivered to your production servers** by using one of the **deploy commands**.
133 + The built release is then **copied to your local directory** `.deliver/releases` and can then be **delivered to your production servers** by using one of the **deploy commands**.
116 134
117 135 If compiling and generating the release build was successful, the release is **copied from the remote build host** to the **release store**. The default release store is the __local__ `.deliver` __directory__ but you can configure any destination with the `RELEASE_STORE=` environment variables, also __remote ssh destinations__ (in your server network) like `RELEASE_STORE=user@releases.acme.org:/releases/` or **amazon s3** locations like `s3://AWS_ACCESS_KEY_ID@AWS_SECRET_ACCESS_KEY:bucket`. The release is copied from the remote build host using the `RELEASE_DIR=` environment variable. If this is not set, the default directory is found by finding the subdirectory that contains the generated `RELEASES` file and has the `$APP` name in the path. e.g. if `$APP=myApp` and the `RELEASES` file is found at `rel/myApp/myApp/releases/RELEASE` the `rel/myApp/myApp` is copied to the release store.
118 136
137 + To __build releases__ and upgrades __faster__, you might adjust the `GIT_CLEAN_PATHS` variable in your config file e.g. to something like `="_build rel priv/generated"` which defaults to `.`. That value means, that everything from the last build is reset (beam files, release files, deps, generated assets etc.) before the next build is started to ensure that no conflicts with old (e.g. removed or renamed) files might arise. You can also use the command line option `--skip-git-clean` to skip this step completely and in addition with the `--skip-mix-clean` option for full __incremental builds__.
138 +
119 139 #### Build Initial Release
120 140
121 - ./edeliver build release [--revision=<git-revision>|--tag=<git-tag>] [--branch=<git-branch>]
141 + mix edeliver build release [--revision=<git-revision>|--tag=<git-tag>] [--branch=<git-branch>]
122 142
123 143 Builds an initial release that can be deployed to the production hosts. If you want to build a different tag or revision, use the `--revision=` or the `--tag` argument. If you want to build a different branch or the tag / revision is in a different branch, use the `--branch=` arguemtn.
124 144
125 145 #### Generate and Edit Upgrade Files (appup)
126 146
127 - ./edeliver build appups --from=<git-tag-or-revision>|--with=<release-version-from-store>
147 + mix edeliver build appups --from=<git-tag-or-revision>|--with=<release-version-from-store>
128 148 [--to=<git-tag-or-revision>] [--branch=<git-branch>]
129 149
130 150 Builds [release upgrade files (appup)](http://www.erlang.org/doc/man/appup.html) with instructions how to load the new code when an upgrade package is built. If your application __isn't used as dependency / library__ for other applications, you might __just__ want to __edit the final relup file__ as described later. The appup files are generated between two git revisions or tags or from an old revision / tag to the current master branch. Requires that the `--from=` parameter is passed at the command line which referes the the old git revision or tag to build the appup files from. If an **old release exists** already **in the release store**, it can be used by passing the old release number to the `--with=` argument. In that case the **building the old release** from the previous git revision **can be skipped**. The **generated appup files will be copied** to the `appup/OldVersion-NewVersion/*.appup` directory in your release store. You can **modify the generated** appup **files of your applications**, and delete all upgrade files of dependend apps or also of your apps, if the generated default upgrade script is sufficient. These files will be **incuded when the upgrade is built** with the `build upgrade` command and **overwrite the generated default appup files**.
131 151
132 152 #### Build an Upgrade Package for Live Updates of Running Nodes
133 153
134 - ./edeliver build upgrade --from=<git-tag-or-revision>|--with=<release-version-from-store>
154 + mix edeliver build upgrade --from=<git-tag-or-revision>|--with=<release-version-from-store>
135 155 [--to=<git-tag-or-revision>] [--branch=<git-branch>]
136 156
137 157 Builds a release upgrade package that can be deployed to production hosts with running nodes. The upgrade is generated between two git revisions or tags or from an old revision / tag to the current master branch. Requires that the `--from=` argument passed at the command line which referes the the old git revision or tag to build the upgrade from and an optional `--to=` argument, if the upgrade should not be created to the latest version. If an **old release exists** already **in the release store**, it can be used by passing the old release number to the `--with=` argument. In that case the **building the old release** from the previous git revision **can be skipped** (and build is faster). For the live upgrade process, you can **provide custom application upgrade files [(appup)](http://www.erlang.org/doc/man/appup.html)** as described in the previous section, or __modify the generated final upgrade instructions ([relup](http://www.erlang.org/doc/man/relup.html))__ as described in the next section (more convenient).
  @@ -139,7 +159,7 @@ Builds a release upgrade package that can be deployed to production hosts with r
139 159
140 160 #### Edit the final upgrade instructions (relup)
141 161
142 - ./edeliver edit relup [--version=<upgrade-version>]
162 + mix edeliver edit relup [--version=<upgrade-version>]
143 163
144 164 From the appup instructions of all included and updated applications, a __[relup](http://www.erlang.org/doc/man/relup.html)__ file is generated during the `build upgrade` command and included in the upgrade package. It contains the final upgrade instructions for the new release version. If there are __dependencies between applications__, it might be necessary to __modify this file__, e.g. changing the order of the applications or modules that are reloaded. Also if you don't want to reuse the appups for other releases, __it is much more convenient__ to modify this file instead of the appups which must be generated in a special step before.
145 165
  @@ -151,7 +171,7 @@ The reason for that is, that when the upgrade is build with rebar, rebar tries t
151 171
152 172 ### Deploy Commands
153 173
154 - ./edeliver deploy release|upgrade [[to] staging|production] [--version=<release-version>] [Options]
174 + mix edeliver deploy release|upgrade [[to] staging|production] [--version=<release-version>] [Options]
155 175
156 176 Deploy commands **deliver the builds** (that were created with a build command before) **to** your staging or **prodution hosts** and can perform the **live code upgrade**. The releases or upgrades to deliver are then available in your local directory `.deliver/releases`. To deploy releases the following **configuration** variables must be set:
  @@ -1,13 +1,15 @@
1 1 {<<"app">>,<<"edeliver">>}.
2 2 {<<"build_tools">>,[<<"mix">>]}.
3 3 {<<"description">>,
4 - <<"Build and Deploy Elixir Applications and perform Hot-Code Upgrades">>}.
4 + <<"Build and Deploy Elixir Applications and perform Hot-Code Upgrades and Schema Migrations">>}.
5 5 {<<"files">>,
6 - [<<"bin/deliver">>,<<"bin/edeliver">>,<<"mix.exs">>,
7 - <<"lib/mix/tasks/edeliver.ex">>,<<"libexec/app_config">>,<<"libexec/aws">>,
8 - <<"libexec/common">>,<<"libexec/core">>,<<"libexec/defaults">>,
9 - <<"libexec/deprecations">>,<<"libexec/erlang">>,<<"libexec/erlang-init">>,
10 - <<"libexec/generated">>,<<"libexec/init">>,<<"libexec/output">>,
6 + [<<"bin/deliver">>,<<"bin/edeliver">>,<<"CHANGELOG.md">>,
7 + <<"lib/edeliver.ex">>,<<"lib/exrm/plugins/link_sys_config.ex">>,
8 + <<"lib/exrm/plugins/link_vm_args.ex">>,<<"lib/mix/tasks/edeliver.ex">>,
9 + <<"libexec/app_config">>,<<"libexec/aws">>,<<"libexec/common">>,
10 + <<"libexec/core">>,<<"libexec/defaults">>,<<"libexec/deprecations">>,
11 + <<"libexec/erlang">>,<<"libexec/erlang-init">>,<<"libexec/generated">>,
12 + <<"libexec/init">>,<<"libexec/output">>,<<"mix.exs">>,
11 13 <<"src/edeliver.app.src">>,<<"strategies/README.md">>,
12 14 <<"strategies/erlang">>,<<"strategies/erlang-build-appup">>,
13 15 <<"strategies/erlang-build-release">>,<<"strategies/erlang-build-upgrade">>,
  @@ -18,10 +20,13 @@
18 20 <<"strategies/erlang-release-store-copy">>,
19 21 <<"strategies/erlang-show-releases">>,<<"strategies/erlang-show-relup">>,
20 22 <<"strategies/erlang-unpack-pack">>,<<"strategies/generated">>,
21 - <<"strategies/gh-pages">>,<<"strategies/nodejs">>,<<"strategies/ruby">>,
22 - <<"strategies/s3">>,<<"strategies/shared">>,<<"README.md">>]}.
23 + <<"strategies/gh-pages">>,<<"strategies/s3">>,<<"README.md">>]}.
23 24 {<<"licenses">>,[<<"MIT">>]}.
24 25 {<<"links">>,[{<<"GitHub">>,<<"https://github.com/boldpoker/edeliver">>}]}.
25 26 {<<"name">>,<<"edeliver">>}.
26 - {<<"requirements">>,[]}.
27 - {<<"version">>,<<"1.0.0">>}.
27 + {<<"requirements">>,
28 + [{<<"exrm">>,
29 + [{<<"app">>,<<"exrm">>},
30 + {<<"optional">>,false},
31 + {<<"requirement">>,<<">= 0.16.0">>}]}]}.
32 + {<<"version">>,<<"1.1.1">>}.
  @@ -0,0 +1,92 @@
1 + defmodule Edeliver do
2 +
3 +
4 + def run_command([command_name, application_name = [_|_] | arguments]) when is_atom(command_name) do
5 + application_name = to_string(application_name)
6 + application_name = String.to_atom(to_string(application_name))
7 + {^application_name, _description, application_version} = :application.which_applications |> List.keyfind application_name, 0
8 + application_version = to_string application_version
9 + apply __MODULE__, command_name, [application_name, application_version | arguments]
10 + end
11 +
12 + def release_version(application_name, _application_version \\ nil) do
13 + releases = :release_handler.which_releases
14 + application_name = Atom.to_char_list application_name
15 + case (for {name, version, _apps, status} <- releases, status == :current and name == application_name, do: to_string(version)) do
16 + [current_version] -> current_version
17 + _ ->
18 + case (for {name, version, _apps, status} <- releases, status == :permanent and name == application_name, do: to_string(version)) do
19 + [permanent_version] -> String.to_char_list(permanent_version)
20 + end
21 + end
22 + end
23 +
24 + def list_pending_migrations(application_name, application_version) do
25 + repository = ecto_repository!
26 + versions = Ecto.Migrator.migrated_versions(repository)
27 + pending_migrations = migrations_for(migrations_dir(application_name, application_version))
28 + |> Enum.filter(fn {version, _name, _file} -> not (version in versions) end)
29 + |> Enum.reverse
30 + |> Enum.map(fn {version, name, _file} -> {version, name} end)
31 + pending_migrations |> Enum.each(fn {version, name} ->
32 + warning "pending: #{name} (#{version})"
33 + end)
34 + end
35 +
36 + def migrate(application_name, application_version, direction, migration_version \\ :all) when is_atom(direction) do
37 + options = if migration_version == :all, do: [all: true], else: [to: to_string(migration_version)]
38 + Ecto.Migrator.run(ecto_repository!, migrations_dir(application_name, application_version), direction, options)
39 + end
40 +
41 + def migrations_dir(application_name, application_version) do
42 + # use priv dir from installed version
43 + lib_dir = :code.priv_dir(application_name) |> to_string |> Path.dirname |> Path.dirname
44 + application_with_version = "#{Atom.to_string(application_name)}-#{application_version}"
45 + Path.join([lib_dir, application_with_version, "priv", "repo", "migrations"])
46 + end
47 +
48 + def ecto_repository! do
49 + case System.get_env "ECTO_REPOSITORY" do
50 + ecto_repository = <<_,_::binary>> ->
51 + ecto_repository_module = List.to_atom ecto_repository
52 + unless :erlang.module_loaded(ecto_repository_module) && (ecto_repository_module.module_info(:exports) |> Dict.get(:__repo__, nil)) do
53 + error! "Module '#{ecto_repository_module}' is not an ecto repository.\n Please set the correct repository module in the edeliver config as ECTO_REPOSITORY env\n or remove that value to use autodetection of that module."
54 + else
55 + ecto_repository_module
56 + end
57 + _ ->
58 + case Enum.filter(:erlang.loaded |> Enum.reverse, fn(module) -> :erlang.module_loaded(module) && (module.module_info(:exports) |> Dict.get(:__repo__, nil)) end) do
59 + [ecto_repository_module] -> ecto_repository_module
60 + [] -> error! "No ecto repository module found.\n Please specify the repository in the edeliver config as ECTO_REPOSITORY env."
61 + modules =[_|_] -> error! "Found several ecto repository modules (#{inspect modules}).\n Please specify the repository to use in the edeliver config as ECTO_REPOSITORY env."
62 + end
63 + end
64 + end
65 +
66 +
67 + # taken from https://github.com/elixir-lang/ecto/blob/master/lib/ecto/migrator.ex#L183
68 + defp migrations_for(directory) do
69 + query = Path.join(directory, "*")
70 + for entry <- Path.wildcard(query),
71 + info = extract_migration_info(entry),
72 + do: info
73 + end
74 +
75 + defp extract_migration_info(file) do
76 + base = Path.basename(file)
77 + ext = Path.extname(base)
78 + case Integer.parse(Path.rootname(base)) do
79 + {integer, "_" <> name} when ext == ".exs" -> {integer, name, file}
80 + _ -> nil
81 + end
82 + end
83 +
84 + # defp info(message), do: IO.puts "==> #{IO.ANSI.green}#{message}#{IO.ANSI.reset}"
85 + defp warning(message), do: IO.puts "==> #{IO.ANSI.yellow}#{message}#{IO.ANSI.reset}"
86 + defp error!(message) do
87 + IO.puts "==> #{IO.ANSI.red}#{message}#{IO.ANSI.reset}"
88 + throw "error"
89 + end
90 +
91 +
92 + end
\ No newline at end of file
  @@ -0,0 +1,24 @@
1 + defmodule ReleaseManager.Plugin.LinkSysConfig do
2 + use ReleaseManager.Plugin
3 + alias ReleaseManager.Utils
4 +
5 +
6 + def before_release(_), do: nil
7 +
8 + def after_release(%Config{env: :prod, version: version, name: name}) do
9 + case System.get_env "LINK_SYS_CONFIG" do
10 + sys_config_link_destination = <<_,_::binary>> ->
11 + debug "Linking sys.config file"
12 + sysconfig_path = Utils.rel_dest_path(Path.join([name, "releases", version, "sys.config"]))
13 + if sysconfig_path |> File.exists?, do: sysconfig_path |> File.rm
14 + File.ln_s(sys_config_link_destination, sysconfig_path)
15 + _ -> nil
16 + end
17 + end
18 + def after_release(_), do: nil
19 +
20 + def after_cleanup(_), do: nil
21 +
22 + def after_package(_), do: nil
23 +
24 + end
Loading more files…