[bug#66801,0/5] build Erlang packages with dependencies

Introduction
════════════

  • The objective is to introduce a `mix-build-system' for building
    Elixir packages.
  • It is first necessary to have `rebar-build-system' to build Erlang
    packages.
  • We start by showing a build procedure using a package named `lager'
    that operates without the need for network access.
  • We generalize the build procedure to any Erlang package by using
    [Checkout Dependencies].
  • To align this build procedure with Guix, we ensure that for any
    Erlang Guix package, all its dependencies, including the transitive
    ones, can be identified.
  • We show that the source code for each dependency is retrievable from
    the store.
  • We modify `rebar.scm' and `rebar-build-system.scm' to build
    procedure into Guix.
  • The packages defined in `erlang-xyz.scm' serve as the test cases for
    this implementation.
  • The changes to the Guix source code are minimal. While the primary
    goal is to validate the approach, it may be necessary to resolve
    stylistic issues or potential bugs.


[Checkout Dependencies]
<https://rebar3.org/docs/configuration/dependencies/#checkout-dependencies>


Bash
════

Rebar3 compiles and tests Lager without network access
──────────────────────────────────────────────────────

  There is a directory and an environment such that `rebar3 compile' and
   `rebar3 test' compile and test the [lager package] without network
   access.


[lager package] <https://hex.pm/packages/lager>

Sources are fetched and unpacked
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

  ┌────
  │ #! /usr/bin/env bash
  │ set -euo pipefail
  │ IFS=$'\n\t'
  │ 
  │ tmpdir=$(mktemp -d)
  │ cd $tmpdir
  │ wget -O source.tar https://repo.hex.pm/tarballs/lager-3.9.2.tar
  │ tar xf source.tar
  │ tar xf contents.tar.gz
  └────


Dependencies are fetched
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

  ┌────
  │ rebar3 compile
  │ rebar3 eunit
  └────


Checkouts are built
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

  #+begin_src bash mkdir _checkouts mv _build/default/lib/goldrush
  _checkouts rm -rf _build #+end_srcè


The directory is built
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

  ┌────
  │ .
  │ ├── _checkouts
  │ │   └── goldrush
  │ │       ├── ebin
  │ │       │   ├── …
  │ │       │   └── gr_sup.beam
  │ │       ├── hex_metadata.config
  │ │       ├── LICENSE
  │ │       ├── priv
  │ │       │   └── edoc.css
  │ │       ├── README.org
  │ │       ├── rebar.config
  │ │       ├── rebar.lock
  │ │       └── src
  │ │           ├── …
  │ │           └── gr_sup.erl
  │ ├── CHECKSUM
  │ ├── contents.tar.gz
  │ ├── include
  │ │   └── lager.hrl
  │ ├── LICENSE
  │ ├── metadata.config
  │ ├── priv
  │ │   └── edoc.css
  │ ├── README.md
  │ ├── rebar.config
  │ ├── rebar.config.script
  │ ├── rebar.lock
  │ ├── source.tar
  │ ├── src
  │ │   ├── …
  │ │   └── lager_util.erl
  │ └── VERSION
  │ 
  │ 9 directories, 73 files
  └────


The environment is defined
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

  ┌────
  │ (specifications->manifest '("coreutils" "rebar3"))
  └────


Lager is compiled and tested without network access
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

  ┌────
  │ guix shell -C -m manifest.scm -- rebar3 compile
  │ guix shell -C -m manifest.scm -- rebar3 eunit
  └────


Erlang can find and use the compiled module
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

  ┌────
  │ ERL_LIBS=_build/default/lib erl -noshell -eval 'io:format("~p~n", [code:which(lager)]).' -s init stop
  └────

  ┌────
  │ "_build/default/lib/lager/ebin/lager.beam"
  └────


Discussion
══════════

  Several observations can be made, and alternative approaches can be
  proposed, as listed below. Meanwhile, with the implementation
  described above, it becomes feasible to build Erlang packages with
  Guix, which is an essential step towards building Elixir packages.


Wasted Computations
───────────────────

  This build procedure needs to re-build of all Erlang dependencies for
  an Erlang package, even if they have been previously built. This is
  due to the `rebar3' build system's refusal to operate without network
  access unless the dependencies are checked out. If checked out, it
  insists on rebuilding them from sources.


Expensive computations
──────────────────────

  For a given Erlang dependency, it is necessary to retreive all its
  transitive dependencies which might be expensive to compute. See:
  [all-transitive-inputs].


[all-transitive-inputs] <./guix/build-system/rebar.scm::(define
(all-transitive-inputs>


Wasted Space
────────────

  The source code for all Erlang dependencies of a specific Erlang
  package must be available on the build machine, whereas it might be
  assumed that the build artifacts alone would suffice.


Alternatives
────────────

  Liliana Marie Pinker has suggested several alternatives listed below.


Store sources along the build artifacts
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

  From [issue/66801]:
        keep the sources in lib/erlang/lib/lib-name/src so that it
        gets symlinked by the phase we have.  Though at that point
        we can surely go with a less surprising install directory.


[issue/66801] <https://issues.guix.gnu.org/issue/66801>


Write the sources to a special source output
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

  From [issue/66801]:
        You could alternatively also write the sources to a
        special source output if and only if one such output is
        requested via the outputs field.  Again, I'd hazard a
        guess that this would be a very standard output for
        anything that needs to go into mix-build-system and thus
        not worth the split after all.


[issue/66801] <https://issues.guix.gnu.org/issue/66801>


Patch Rebar3
╌╌╌╌╌╌╌╌╌╌╌╌

  From [issue/66801]:
        you could patch mix to only look for compiled stuff and
        not sources.  That would tackle the issue at the root
        instead of trying to work around it, with the caveat being
        that we would need to maintain this patch ourselves if
        upstream doesn't accept it.


[issue/66801] <https://issues.guix.gnu.org/issue/66801>


Pierre-Henry Fröhring (5):
  guix: build-system: rebar: build Erlang packages with dependencies.
  gnu: Add erlang-goldrush.
  gnu: Add erlang-lager.
  gnu: Add erlang-unicode-util-compat.
  gnu: Add erlang-idna.

 gnu/packages/erlang-xyz.scm       | 111 +++++++++++++++
 guix/build-system/rebar.scm       | 223 ++++++++++++++++++++++--------
 guix/build/rebar-build-system.scm |  43 +++---
 3 files changed, 300 insertions(+), 77 deletions(-)
 create mode 100644 gnu/packages/erlang-xyz.scm


base-commit: 4dfbc536689b07e56aead3dd864b8af54613d091
--
2.41.0