nixpkgs/doc/languages-frameworks/beam.xml

<section xmlns="http://docbook.org/ns/docbook"
         xmlns:xlink="http://www.w3.org/1999/xlink"
         xml:id="sec-beam">

  <title>Beam Languages (Erlang, Elixir &amp; LFE)</title>
  <section xml:id="beam-introduction">
    <title>Introduction</title>
    <para>
      In this document and related Nix expressions we use the term
      <emphasis>Beam</emphasis> to describe the environment. BEAM is
      the name of the Erlang Virtial Machine and, as far as we know,
      from a packaging perspective all languages that run on BEAM are
      interchangable. The things that do change, like the build
      system, are transperant to the users of the package. So we make
      no distinction.
    </para>
  </section>
  <section xml:id="beam-structure">
    <title>Structure</title>
    <para>
      All Beam-related things are available via top-level
      <literal>beam</literal> attribute, which includes:
    </para>
    <itemizedlist>
      <listitem>
        <para>
          <literal>interpreters</literal>: contains a set of compilers running
          on Beam, including multiple Erlang/OTP versions
          (<literal>beam.interpreters.erlangR19</literal>, etc), Elixir
          (<literal>beam.interpreters.elixir</literal>) and LFE
          (<literal>beam.interpreters.lfe</literal>).
        </para>
      </listitem>
      <listitem>
        <para>
          <literal>packages</literal>: contains a set of package sets, each
          compiled with a specific Erlang/OTP version, e.g.
          <literal>beam.packages.erlangR19</literal>.
        </para>
      </listitem>
    </itemizedlist>
    <para>
      Default Erlang compiler is defined by
      <literal>beam.interpreters.erlang</literal> and aliased as
      <literal>erlang</literal>. Default package set with Beam packages is
      defined by <literal>beam.packages.erlang</literal> and aliased at
      top-level as <literal>beamPackages</literal>.
    </para>
    <para>
      If you want to create a package set built with a custom Erlang version,
      use lambda <literal>beam.packagesWith</literal>, which accepts an
      Erlang/OTP derivative and produces a package set similar to
      <literal>beam.packages.erlang</literal>.
    </para>
    <para>
      Many Erlang/OTP distributions available in
      <literal>beam.interpreters</literal> have their versions with ODBC and/or
      Java enabled. For example there's
      <literal>beam.interpreters.erlangR19_odbc_javac</literal> which
      corresponds to <literal>beam.interpreters.erlangR19</literal>.
    </para>
    <para>
      We also provide <literal>beam.packages.erlang.callPackage</literal>, which
      simplifies writing Beam package definitions, by injecting all packages from
      <literal>beam.packages.erlang</literal> into top-level context.
    </para>
  </section>
  <section xml:id="build-tools">
  <title>Build Tools</title>
  <section xml:id="build-tools-rebar3">
    <title>Rebar3</title>
    <para>
      By default Rebar3 wants to manage it's own dependencies. This is perfectly
      acceptable in the normal non-Nix setup. In the Nix world it is not. To
      support this we have created two versions of rebar3,
      <literal>rebar3</literal> and <literal>rebar3-open</literal>. The
      <literal>rebar3</literal> version has been patched to remove the ability
      to download anything from it. If you are not running it a nix-shell or a
      nix-build then its probably not going to work for you.
      <literal>rebar3-open</literal> is the normal, un-modified rebar3. It
      should work exactly as would any other version of rebar3. Any Erlang
      package should rely on <literal>rebar3</literal> and thats really what you
      should be using too. See <literal>buildRebar3</literal> below.
    </para>
  </section>
  <section xml:id="build-tools-other">
    <title>Mix &amp; Erlang.mk</title>
    <para>
      Both Mix and Erlang.mk work exactly as you would expect. There
      is a bootstrap process that needs to be run for both of
      them. However, that is supported by the
      <literal>buildMix</literal> and <literal>buildErlangMk</literal> derivations.
    </para>
  </section>
</section>

<section xml:id="how-to-install-beam-packages">
  <title>How to install Beam packages</title>
  <para>
    Beam packages are not registered in the top level simply because they are
    not relevant to the vast majority of Nix users. They are installable using
    the <literal>beam.packages.erlang</literal> attribute set and aliased as
    <literal>beamPackages</literal>. This attribute points at packages built by
    the default Erlang/OTP version in Nixpkgs as defined by
    <literal>beam.interpreters.erlang</literal>.

    You can list the avialable packages in the
    <literal>beamPackages</literal> with the following command:
  </para>

  <programlisting>
$ nix-env -f &quot;&lt;nixpkgs&gt;&quot; -qaP -A beamPackages
beamPackages.esqlite    esqlite-0.2.1
beamPackages.goldrush   goldrush-0.1.7
beamPackages.ibrowse    ibrowse-4.2.2
beamPackages.jiffy      jiffy-0.14.5
beamPackages.lager      lager-3.0.2
beamPackages.meck       meck-0.8.3
beamPackages.rebar3-pc  pc-1.1.0
  </programlisting>
  <para>
    To install any of those packages into your profile, refer to them by
    their attribute path (first column):
  </para>
  <programlisting>
$ nix-env -f &quot;&lt;nixpkgs&gt;&quot; -iA beamPackages.ibrowse
  </programlisting>
  <para>
    The attribute path of any Beam packages corresponds to the name
    of that particular package in Hex or its OTP Application/Release name.
  </para>
</section>
<section xml:id="packaging-beam-applications">
  <title>Packaging Beam Applications</title>
  <section  xml:id="packaging-erlang-applications">
    <title>Erlang Applications</title>
    <section xml:id="rebar3-packages">
      <title>Rebar3 Packages</title>
      <para>
        There is a Nix function called <literal>buildRebar3</literal> (defined
        in <literal>beam.packages.erlang.buildRebar3</literal> and aliased at
        top-level). We use this function to make a derivation that understands
        how to build the rebar3 project. For example, the expression we use to
        build the <link
        xlink:href="https://github.com/erlang-nix/hex2nix">hex2nix</link>
        project follows.
      </para>
      <programlisting>
        {stdenv, fetchFromGitHub, buildRebar3, ibrowse, jsx, erlware_commons }:

          buildRebar3 rec {
            name = "hex2nix";
            version = "0.0.1";

            src = fetchFromGitHub {
              owner = "ericbmerritt";
              repo = "hex2nix";
              rev = "${version}";
              sha256 = "1w7xjidz1l5yjmhlplfx7kphmnpvqm67w99hd2m7kdixwdxq0zqg";
            };

          beamDeps = [ ibrowse jsx erlware_commons ];
        }
      </programlisting>
      <para>
        This derivation is callable with
        <literal>beam.packages.erlang.callPackage</literal> (see above). If you
        want to call this package using normal <literal>callPackage</literal>,
        you should refer to dependency packages via
        <literal>beamPackages</literal>, e.g.
        <literal>beamPackages.ibrowse</literal>.
      </para>
      <para>
        The only visible difference between this derivation and
        something like <literal>stdenv.mkDerivation</literal> is that we
        have added <literal>beamDeps</literal> to the derivation. If
        you add your Beam dependencies here they will be correctly
        handled by the system.
      </para>
      <para>
        If your package needs to compile native code via Rebar's port
        compilation mechenism. You should add <literal>compilePort =
        true;</literal> to the derivation.
      </para>
    </section>
    <section xml:id="erlang-mk-packages">
      <title>Erlang.mk Packages</title>
      <para>
        Erlang.mk functions almost identically to Rebar. The only real
        difference is that <literal>buildErlangMk</literal> is called
        instead of <literal>buildRebar3</literal>
      </para>
      <programlisting>
        { buildErlangMk, fetchHex,  cowlib, ranch }:
          buildErlangMk {
            name = "cowboy";
            version = "1.0.4";
            src = fetchHex {
              pkg = "cowboy";
              version = "1.0.4";
              sha256 =
                "6a0edee96885fae3a8dd0ac1f333538a42e807db638a9453064ccfdaa6b9fdac";
            };
            beamDeps  = [ cowlib ranch ];

            meta = {
              description = ''Small, fast, modular HTTP server written in
                            Erlang.'';
              license = stdenv.lib.licenses.isc;
              homepage = "https://github.com/ninenines/cowboy";
            };
        }
      </programlisting>
    </section>
    <section xml:id="mix-packages">
      <title>Mix Packages</title>
      <para>
        Mix functions almost identically to Rebar. The only real
        difference is that <literal>buildMix</literal> is called
        instead of <literal>buildRebar3</literal>
      </para>
      <programlisting>
        { buildMix, fetchHex, plug, absinthe }:
        buildMix {
          name = "absinthe_plug";
          version = "1.0.0";
          src = fetchHex {
            pkg = "absinthe_plug";
            version = "1.0.0";
            sha256 =
              "08459823fe1fd4f0325a8bf0c937a4520583a5a26d73b193040ab30a1dfc0b33";
          };
          beamDeps  = [ plug absinthe ];

          meta = {
            description = ''A plug for Absinthe, an experimental GraphQL
                            toolkit'';
            license = stdenv.lib.licenses.bsd3;
            homepage = "https://github.com/CargoSense/absinthe_plug";
         };
       }
      </programlisting>
      <para>
        Alternatively one can use <literal>buildHex</literal> as a shortcut for the above:
      </para>
      <programlisting>
        { buildHex, buildMix, plug, absinthe }:
        buildHex {
          name = "absinthe_plug";
          version = "1.0.0";
          sha256 =
              "08459823fe1fd4f0325a8bf0c937a4520583a5a26d73b193040ab30a1dfc0b33";
          builder = buildMix;
          beamDeps  = [ plug absinthe];

          meta = {
            description = ''A plug for Absinthe, an experimental GraphQL
                            toolkit'';
            license = stdenv.lib.licenses.bsd3;
            homepage = "https://github.com/CargoSense/absinthe_plug";
         };
       }
      </programlisting>
    </section>
  </section>
</section>
<section xml:id="how-to-develop">
  <title>How to develop</title>
  <section xml:id="accessing-an-environment">
    <title>Accessing an Environment</title>
    <para>
      Often, all you want to do is be able to access a valid
      environment that contains a specific package and its
      dependencies. we can do that with the <literal>env</literal>
      part of a derivation. For example, lets say we want to access an
      erlang repl with ibrowse loaded up. We could do the following.
    </para>
    <programlisting>
      ~/w/nixpkgs ❯❯❯ nix-shell -A beamPackages.ibrowse.env --run "erl"
      Erlang/OTP 18 [erts-7.0] [source] [64-bit] [smp:4:4] [async-threads:10] [hipe] [kernel-poll:false]

      Eshell V7.0  (abort with ^G)
      1> m(ibrowse).
      Module: ibrowse
      MD5: 3b3e0137d0cbb28070146978a3392945
      Compiled: January 10 2016, 23:34
      Object file: /nix/store/g1rlf65rdgjs4abbyj4grp37ry7ywivj-ibrowse-4.2.2/lib/erlang/lib/ibrowse-4.2.2/ebin/ibrowse.beam
      Compiler options:  [{outdir,"/tmp/nix-build-ibrowse-4.2.2.drv-0/hex-source-ibrowse-4.2.2/_build/default/lib/ibrowse/ebin"},
      debug_info,debug_info,nowarn_shadow_vars,
      warn_unused_import,warn_unused_vars,warnings_as_errors,
      {i,"/tmp/nix-build-ibrowse-4.2.2.drv-0/hex-source-ibrowse-4.2.2/_build/default/lib/ibrowse/include"}]
      Exports:
      add_config/1                  send_req_direct/7
      all_trace_off/0               set_dest/3
      code_change/3                 set_max_attempts/3
      get_config_value/1            set_max_pipeline_size/3
      get_config_value/2            set_max_sessions/3
      get_metrics/0                 show_dest_status/0
      get_metrics/2                 show_dest_status/1
      handle_call/3                 show_dest_status/2
      handle_cast/2                 spawn_link_worker_process/1
      handle_info/2                 spawn_link_worker_process/2
      init/1                        spawn_worker_process/1
      module_info/0                 spawn_worker_process/2
      module_info/1                 start/0
      rescan_config/0               start_link/0
      rescan_config/1               stop/0
      send_req/3                    stop_worker_process/1
      send_req/4                    stream_close/1
      send_req/5                    stream_next/1
      send_req/6                    terminate/2
      send_req_direct/4             trace_off/0
      send_req_direct/5             trace_off/2
      send_req_direct/6             trace_on/0
      trace_on/2
      ok
      2>
    </programlisting>
    <para>
      Notice the <literal>-A beamPackages.ibrowse.env</literal>.That
      is the key to this functionality.
    </para>
  </section>
  <section xml:id="creating-a-shell">
    <title>Creating a Shell</title>
    <para>
      Getting access to an environment often isn't enough to do real
      development. Many times we need to create a
      <literal>shell.nix</literal> file and do our development inside
      of the environment specified by that file. This file looks a lot
      like the packaging described above. The main difference is that
      <literal>src</literal> points to project root and we call the
      package directly.
    </para>
    <programlisting>
{ pkgs ? import &quot;&lt;nixpkgs&quot;&gt; {} }:

with pkgs;

let

  f = { buildRebar3, ibrowse, jsx, erlware_commons }:
      buildRebar3 {
        name = "hex2nix";
        version = "0.1.0";
        src = ./.;
        beamDeps = [ ibrowse jsx erlware_commons ];
      };
  drv = beamPackages.callPackage f {};

in
 drv
    </programlisting>
    <section xml:id="building-in-a-shell">
    <title>Building in a shell (for mix projects)</title>
    <para>
      We can leverage the support of the Derivation, regardless of
      which build Derivation is called by calling the commands themselves.
    </para>
    <programlisting>
# =============================================================================
# Variables
# =============================================================================

NIX_TEMPLATES := "$(CURDIR)/nix-templates"

TARGET := "$(PREFIX)"

PROJECT_NAME := thorndyke

NIXPKGS=../nixpkgs
NIX_PATH=nixpkgs=$(NIXPKGS)
NIX_SHELL=nix-shell -I "$(NIX_PATH)" --pure
# =============================================================================
# Rules
# =============================================================================
.PHONY= all test clean repl shell build test analyze configure install \
        test-nix-install publish plt analyze

all: build

guard-%:
        @ if [ "${${*}}" == "" ]; then \
                echo "Environment variable $* not set"; \
                exit 1; \
        fi

clean:
        rm -rf _build
        rm -rf .cache

repl:
        $(NIX_SHELL) --run "iex -pa './_build/prod/lib/*/ebin'"

shell:
        $(NIX_SHELL)

configure:
        $(NIX_SHELL) --command 'eval "$$configurePhase"'

build: configure
        $(NIX_SHELL) --command 'eval "$$buildPhase"'

install:
        $(NIX_SHELL) --command 'eval "$$installPhase"'

test:
        $(NIX_SHELL) --command 'mix test --no-start --no-deps-check'

plt:
        $(NIX_SHELL) --run "mix dialyzer.plt --no-deps-check"

analyze: build plt
        $(NIX_SHELL) --run "mix dialyzer --no-compile"

    </programlisting>
    <para>
      If you add the <literal>shell.nix</literal> as described and
      user rebar as follows things should simply work. Aside from the
      <literal>test</literal>, <literal>plt</literal>, and
      <literal>analyze</literal> the talks work just fine for all of
      the build Derivations.
    </para>
  </section>
</section>
</section>
<section xml:id="generating-packages-from-hex-with-hex2nix">
  <title>Generating Packages from Hex with Hex2Nix</title>
  <para>
    Updating the Hex packages requires the use of the
    <literal>hex2nix</literal> tool. Given the path to the Erlang
    modules (usually
    <literal>pkgs/development/erlang-modules</literal>). It will
    dump a file called
    <literal>hex-packages.nix</literal>. That file will contain all
    the packages that use a recognized build system in Hex. However,
    it can't know whether or not all those packages are buildable.
    </para>
    <para>
      To make life easier for our users, it makes good sense to go
      ahead and attempt to build all those packages and remove the
      ones that don't build. To do that, simply run the command (in
      the root of your <literal>nixpkgs</literal> repository). that follows.
    </para>
    <programlisting>
$ nix-build -A beamPackages
    </programlisting>
    <para>
      That will build every package in
      <literal>beamPackages</literal>. Then you can go through and
      manually remove the ones that fail. Hopefully, someone will
      improve <literal>hex2nix</literal> in the future to automate
      that.
    </para>
</section>
</section>