task_graph

#+TITLE: Yet another Erlang Make implementation

[[https://travis-ci.org/k32/task_graph.svg?branch=master]]

* Description

Yet another library rethinking good old /make/, implemented in pure
Erlang. This library aims to do one thing right: it runs tasks in
parallel in topological order. Hence it doesn't do globing,
templating, it doesn't provide any DSLs, and generally doesn't care
about the nature of tasks it executes.

* Features
** Dynamic tasks

Tasks may spawn other tasks or discover their own dependencies
dynamically

** Resources

=task_graph= further develops the concept of =make -j N= flag.
Instead of simply limiting the total number of parallel tasks,
=task_graph= allows to tag tasks with a list of so called
=resources=. These tags don't bear any meaning for =task_graph=
itself, so users are free to associate them with any application
specific entities, such as =poolboy= workers, network connections, RAM
or CPU demanding tasks.

The concept of resources may seem too abstract, but it allows to
offload resource allocation to third-party apps, e.g. =poolboy=

** Guards

=task_graph= allows to define custom checks for whether the task
should be executed or not. See =task_runner:guard/3= callback

** Written with build flow optimization in mind

Sooner or later everyone wants to optimize their builds. =task_graph=
comes with a few helpful tools:

*** Dependency graph visualization

** Painstakingly tested

Random task graph topologies are tested using [[http://proper.softlab.ntua.gr/][proper]]. Most of the code
in this repo is tests.

* Examples

Basic usage:

#+BEGIN_SRC erlang
-include_lib("task_graph/include/task_graph.hrl").

main() ->
    %% Define tasks:
    Tasks = [#tg_task{ id = I
                     , execute = fun(MyId, MyData) ->
                                     io:format("I'm ~p, doing ~p~n", [MyId, MyData])
                                 end,
                     , data = something
                     }
             || I <- lists:seq(1, 4)],
    %% Define dependnecies ({A, B} reads as "B depends on A"):
    Edges = [{1, 2}, {1, 3}, {2, 3}, {2, 4}],
    %% Execute graph:
    {ok, _} = task_graph:run_graph(my_graph, {Tasks, Edges}).
#+END_SRC

Got lost in the dependencies? It's possible to visualize the graph:

#+BEGIN_SRC erlang
    Opts = #{event_handlers =>
                 [{task_graph_draw_deps, #{ filename => "dependencies.dot"
                                          %% You can optionally customize vertices styles:
                                          , style => fun(#task{}) ->
                                                         "color=green shape=oval"
                                                     end
                                          }}]},
    {ok, _} = task_graph:run_graph(my_graph, Opts, {Tasks, Edges}).
#+END_SRC

Render the resulting file using graphviz:

#+BEGIN_SRC bash
dot -Tpng -O dependencies.dot
#+END_SRC

How to emulate =make -j N= flag using =resources=:

#+BEGIN_SRC erlang
%% Define tasks:
Tasks = [#tg_task{ id = I
                 , execute = fun(MyId, MyData) ->
                                 io:format("I'm ~p, doing ~p~n", [MyId, MyData])
                             end,
                 , data = something
                 %% Demand `jobs' resource:
                 , resources = [jobs]
                 }
         || I <- lists:seq(1, 4)],
%% Specify resource capacity:
Settings = #{resources => #{jobs => 3}},
{ok, _} = task_graph:run_graph(my_graph, Settings, {Tasks, Edges}).
#+END_SRC

:hidden:
Design goals:
 + Focus on doing one thing
 + Give users powerful abstactions instead of numerous features
 + But reserve all the abstactions for the user; internal
   implementation should be lean and fast
 + Features that are not =proper=-tested don't exist
:END: