From patchwork Tue Mar 21 20:57:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: "\\(" X-Patchwork-Id: 48549 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id E409F16F87; Tue, 21 Mar 2023 20:59:24 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-1.8 required=5.0 tests=DKIM_INVALID,DKIM_SIGNED, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 1873C16F63 for ; Tue, 21 Mar 2023 20:59:21 +0000 (GMT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pej4S-0003ft-DM; Tue, 21 Mar 2023 16:59:04 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pej4Q-0003fj-De for guix-patches@gnu.org; Tue, 21 Mar 2023 16:59:02 -0400 Received: from debbugs.gnu.org ([209.51.188.43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pej4Q-0004KD-5h for guix-patches@gnu.org; Tue, 21 Mar 2023 16:59:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1pej4Q-00027P-1X for guix-patches@gnu.org; Tue, 21 Mar 2023 16:59:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#62356] [PATCH guix-artwork] website: posts: Add Dissecting Guix, Part 3: G-Expressions. Resent-From: "(" Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Tue, 21 Mar 2023 20:59:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 62356 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 62356@debbugs.gnu.org Cc: "\(" X-Debbugs-Original-To: guix-patches@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.16794322958065 (code B ref -1); Tue, 21 Mar 2023 20:59:01 +0000 Received: (at submit) by debbugs.gnu.org; 21 Mar 2023 20:58:15 +0000 Received: from localhost ([127.0.0.1]:60945 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pej3d-00025v-CW for submit@debbugs.gnu.org; Tue, 21 Mar 2023 16:58:15 -0400 Received: from lists.gnu.org ([209.51.188.17]:40190) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pej3a-00025h-LA for submit@debbugs.gnu.org; Tue, 21 Mar 2023 16:58:12 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pej3W-0003Tt-AX for guix-patches@gnu.org; Tue, 21 Mar 2023 16:58:06 -0400 Received: from knopi.disroot.org ([178.21.23.139]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pej3S-0003xn-4s for guix-patches@gnu.org; Tue, 21 Mar 2023 16:58:06 -0400 Received: from localhost (localhost [127.0.0.1]) by disroot.org (Postfix) with ESMTP id A3EC34449A; Tue, 21 Mar 2023 21:57:57 +0100 (CET) X-Virus-Scanned: SPAM Filter at disroot.org Received: from knopi.disroot.org ([127.0.0.1]) by localhost (disroot.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id SEjwCLDnXdUe; Tue, 21 Mar 2023 21:57:54 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=disroot.org; s=mail; t=1679432274; bh=M6B4xqzv1U/Stc18snRaRbZ+2r8zNRV1j1ryBpM/H+M=; h=From:To:Cc:Subject:Date; b=krcH7vEPWn05C486SmbwjLxhkbW/R5uFu20qkUU9grAy+w9RXUTELdf07Vf94kQhe BSdHvjbP83XsHbldmKXrpz1ZLDtms2QqsW5fKVO+yWbswXqTnZkAnBwQFX7dJ19FFJ tayVhyxrUqeDk1mwiHoU8+XbOByGTCLciyWC101UP1WF8MnNYweDfYPXjywoSLu8BZ OtndfODfkPRcRdgUYN2p/Mw83GNgH6AmbQSTklM+ovKagpZRbpjDzZow1ujhd3XdMf DBCFiFGfmPpNZc9haMTh4uN4LxaWSYN1k3sjqRN002E5GgRDH5plY4XrNfy0ZXPnms 8LjtIFeLhQfrA== Date: Tue, 21 Mar 2023 20:57:49 +0000 Message-Id: <20230321205749.4974-1-paren@disroot.org> MIME-Version: 1.0 Received-SPF: pass client-ip=178.21.23.139; envelope-from=paren@disroot.org; helo=knopi.disroot.org X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: "\(" X-ACL-Warn: , "\( via Guix-patches" X-Patchwork-Original-From: "\( via Guix-patches" via From: "\\(" Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches * website/posts/dissecting-guix-3-gexps.md: New blog post. --- Heya Guix, Here's the third post in the Dissecting Guix series; this one aims to demystify g-expressions ;) -- ( website/posts/dissecting-guix-3-gexps.md | 673 +++++++++++++++++++++++ 1 file changed, 673 insertions(+) create mode 100644 website/posts/dissecting-guix-3-gexps.md base-commit: a81137f5c5eedc0c327e345285d40e84c68ed10b diff --git a/website/posts/dissecting-guix-3-gexps.md b/website/posts/dissecting-guix-3-gexps.md new file mode 100644 index 0000000..32f5d51 --- /dev/null +++ b/website/posts/dissecting-guix-3-gexps.md @@ -0,0 +1,673 @@ +title: Dissecting Guix, Part 3: G-Expressions +date: TBC +author: ( +tags: Dissecting Guix, Functional package management, Programming interfaces, Scheme API +--- +Welcome back to [Dissecting Guix](https://guix.gnu.org/en/blog/tags/dissecting-guix)! +Last time, we discussed [monads](https://guix.gnu.org/en/blog/2023/dissecting-guix-part-2-the-store-monad), +the functional programming idiom used by Guix to thread a store connection +through a series of store-related operations. + +Today, we'll be talking about a concept rather more specific to Guix: +_g-expressions_. Being an implementation of the Scheme language, Guile is built +around [_s-expressions_](https://en.wikipedia.org/wiki/S-expression), which can +represent, as the saying goes, _code as data_, thanks to the simple structure of +Scheme forms. + +As Guix's package recipes are written in Scheme, it naturally needs some way to +represent code that is to be run only when the package is built. Additionally, +there needs to be some way to reference dependencies and retrieve output paths; +otherwise, you wouldn't be able to, for instance, create a phase to install a +file in the output directory. + +So, how do we implement this "deferred" code? Well, initially Guix used plain +old s-expressions for this purpose. + +# Once Upon a Time + +Let's say we want to create a store item that's just a symlink to the +`bin/irssi` file of the `irssi` package. How would we do that with an +s-expression? Well, the s-expression itself, which we call the _builder_, is +fairly simple: + +```scheme +(define sexp-builder + `(let* ((out (assoc-ref %outputs "out")) + (irssi (assoc-ref %build-inputs "irssi")) + (bin/irssi (string-append irssi "/bin/irssi"))) + (symlink bin/irssi out))) +``` + +If you aren't familliar with the "quoting" syntax used to create s-expressions, +I strongly recommend that you read the excellent Scheme Primer; specifically, +section 7, [_Lists and +"cons"_](https://spritely.institute/static/papers/scheme-primer.html#scheme-lists-and-cons) +and section 11, [_On the extensibility of Scheme (and Lisps in +general)_](https://spritely.institute/static/papers/scheme-primer.html#scheme-extensibility) + +The `%outputs` and `%build-inputs` variables are bound within builder scripts to +_association lists_, which are lists of pairs that act like key/value stores, +for instance: + +```scheme +'(("foo" . "bar") + ("floob" . "blarb") + ("fvoolag" . "bvarlag")) +``` + +To retrieve values from association lists, which are often referred to as +_alists_, we use the `assoc-ref` procedure: + +```scheme +(assoc-ref '(("boing" . "bouncy") + ("floing" . "flouncy")) + "boing") +⇒ "bouncy" +``` + +`%outputs`, as the name might suggest, maps derivation output names to the paths +of their respective store items, the default output being `out`, and +`%build-inputs` maps inputs labels to their store items. + +The builder is the easy part; we now need to turn it into a derivation and tell +it what `"irssi"` actually refers to. For this, we use the +`build-expression->derivation` procedure from `(guix derivations)`: + +```scheme +(use-modules (guix derivations) + (guix packages) + (guix store) + (gnu packages guile) + (gnu packages irc)) + +(with-store store + (let ((guile-3.0-drv (package-derivation store guile-3.0)) + (irssi-drv (package-derivation store irssi))) + (build-expression->derivation store "irssi-symlink" sexp-builder + #:guile-for-build guile-3.0-drv + #:inputs `(("irssi" ,irssi-drv))))) +⇒ # /gnu/store/…-irssi-symlink …> +``` + +There are several things to note here: + +- The inputs _must_ all be derivations, so we need to first convert the packages + using `package-derivation`. +- We need to explicitly set `#:guile-for-build`; there's no default value. +- The `build-expression->derivation` and `package-derivation` procedures are + _not_ monadic, so we need to explicitly pass them the store connection. + +The shortcomings of using s-expressions in this way are numerous: we have to +convert everything to a derivation before using it, and _inputs are not an +inherent aspect of the builder_. G-expressions were designed to overcome these +issues. + +# Premortem Examination + +A gexp is fundumentally a record of type ``, which is, naturally, defined +in `(guix gexp)`. The two most important fields of this record type, out of a +total of five, are `proc` and `references`; the former is a procedure that +returns the equivalent sexp, the latter a list containing everything from the +"outside world" that's used by the gexp. + +When we want to turn the gexp into something that we can actually run as code, +we combine these two fields by first building any gexp inputs that can become +derivations (leaving alone those that cannot, such as and then passing the built +`references` as the arguments of `proc`. + +Here's an example gexp that is essentially equivalent to our `sexp-builder`: + +```scheme +(use-modules (guix gexp)) + +(define gexp-builder + #~(symlink #$(file-append irssi "/bin/irssi") + #$output)) +``` + +`gexp-builder` is far more concise than `sexp-builder`; let's examine the syntax +and the `` object we've created. To make a gexp, we use the `#~` syntax, +equivalent to the `gexp` macro, rather than the `quasiquote` backtick used to +create sexps. + +When we want to embed values from outside as references, we use `#$`, or +`ungexp`, which is, in appearance if not function, equivalent to `unquote` +(`,`). `ungexp` can accept any of four reference types: + +- Sexps (strings, lists, etc), which will be embedded literally. +- Other gexps, embedded literally. +- Expressions returning any sort of object that can be lowered into a + derivation, such as ``, embedding that object's `out` store item; if + the expression is specifically a symbol bound to a buildable object, you can + optionally follow it with a colon and an alternative output name, so + `package:lib` is permitted, but `(get-package):lib` isn't. +- The symbol `output`, embedding an output path. Like symbols bound to + buildable objects, this can be followed by a colon and the output name that + should be used rather than the default `out`. + +All these reference types will be represented by `` records in the +`references` field, except for the last kind, which will become `` +records. To give an example of each type of reference (with the return value +output formatted for easier reading): + +```scheme +(use-modules (gnu packages glib)) + +#~(list #$"foobar" ;s-expression + #$#~(string-append "foo" "bar") ;g-expression + #$(file-append irssi "/bin/irssi") ;buildable object (expression) + #$glib:bin ;buildable object (symbol) + #$output:out) ;output +⇒ # + #:out> + # "/bin/irssi">:out> + #:bin> + #) …> +``` + +Note the use of `file-append` in both the previous example and `gexp-builder`; +this procedure produces a `` object that builds its first argument +and is embedded as the concatenation of the first argument's output path and the +second argument, which should be a string. For instance, +`(file-append irssi "/bin/irssi")` builds `irssi` and expands to +`/gnu/store/…-irssi/bin/irssi`, rather than the `/gnu/store/…-irssi` that the +package alone would be embedded as. + +So, now that we have a gexp, how do we turn it into a derivation? This process +is known as _lowering_; it entails the use of the aptly-named `lower-gexp` +monadic procedure to combine `proc` and `references` and produce a +`` record, which acts as a sort of intermediate representation +between gexps and derivations. We can piece apart this lowered form to get a +sense of what the final derivation's builder script would look like: + +```scheme +(define lowered-gexp-builder + (with-store store + (run-with-store store + (lower-gexp gexp-builder)))) + +(lowered-gexp-sexp lowered-gexp-builder) +⇒ (symlink + "/gnu/store/…-irssi-1.4.3/bin/irssi" + ((@ (guile) getenv) "out")) +``` + +And there you have it: a s-expression compiled from a g-expression, ready to be +written into a builder script file in the store. So, how exactly do you turn +this into said derivation? + +Well, it turns out that there isn't an interface for turning lowered gexps into +derivations, only one for turning regular gexps into derivations that first uses +`lower-gexp`, then implements the aforementioned conversion internally, rather +than outsourcing it to some other procedure, so that's what we'll use. + +Unsurprisingly, that procedure is called `gexp->derivation`, and unlike its sexp +equivalent, it's monadic. (`build-expression->derivation` and other deprecated +procedures were in Guix since before the monads system existed.) + +```scheme +(with-store store + (run-with-store store + (gexp->derivation "irssi-symlink" gexp-builder))) +⇒ # /gnu/store/…-irssi-symlink …> +``` + +Finally, we have a gexp-based equivalent to the derivation we earlier created +with `build-expression->derivation`! Here's the code we used for the sexp +version in full: + +```scheme +(define sexp-builder + `(let* ((out (assoc-ref %outputs "out")) + (irssi (assoc-ref %build-inputs "irssi")) + (bin/irssi (string-append irssi "/bin/irssi"))) + (symlink bin/irssi out))) + +(with-store store + (let ((guile-3.0-drv (package-derivation store guile-3.0)) + (irssi-drv (package-derivation store irssi))) + (build-expression->derivation store "irssi-symlink" sexp-builder + #:guile-for-build guile-3.0-drv + #:inputs `(("irssi" ,irssi-drv))))) +``` + +And here's the gexp equivalent: + +```scheme +(define gexp-builder + #~(symlink #$(file-append irssi "/bin/irssi") + #$output)) + +(with-store store + (run-with-store store + (gexp->derivation "irssi-symlink" gexp-builder))) +``` + +That's a lot of complexity abstracted away! For more complex packages and +services, especially, gexps are a lifesaver; you can refer to the output paths +of inputs just as easily as you would a string constant. You do, however, have +to watch out for situations where `ungexp-native`, written as `#+`, would be +preferable over regular `ungexp`, and that's something we'll discuss later. + +A brief digression before we continue: if you'd like to look inside a `` +record, but you'd rather not build anything, you can use the +`gexp->approximate-sexp` procedure, which replaces all references with dummy +values: + +```scheme +(gexp->approximate-sexp gexp-builder) +⇒ (symlink (*approximate*) (*approximate*)) +``` + +# The Lowerable-Object Hardware Shop + +We've seen two examples already of records we can turn into derivations, which +are generally referred to as _lowerable objects_ or _file-like objects_: + +- ``, a Guix package. +- ``, which wraps another lowerable object and appends a string to + the embedded output path when ungexped. + +There are many more available to us. Recall from the previous post, +[_The Store Monad_](https://guix.gnu.org/en/blog/2023/dissecting-guix-part-2-the-store-monad), +that Guix provides the two monadic procedures `text-file` and `interned-file`, +which can be used, respectively, to put arbitrary text or files from the +filesystem in the store, returning the path to the created item. + +This doesn't work so well with gexps, though; you'd have to wrap each ungexped +use of either of them with `(with-store store (run-with-store store …))`, which +would be quite tedious. Thankfully, `(guix gexp)` provides the `plain-file` and +`local-file` procedures, which return equivalent lowerable objects. This code +example builds a directory containing symlinks to files greeting the world: + +```scheme +(use-modules (guix monads) + (ice-9 ftw) + (ice-9 textual-ports)) + +(define (build-derivation monadic-drv) + (with-store store + (run-with-store store + (mlet* %store-monad ((drv monadic-drv)) + (mbegin %store-monad + ;; BUILT-DERIVATIONS is the monadic version of BUILD-DERIVATIONS. + (built-derivations (list drv)) + (return (derivation-output-path + (assoc-ref (derivation-outputs drv) "out")))))))) + +(define world-greeting-output + (build-derivation + (gexp->derivation "world-greeting" + #~(begin + (mkdir #$output) + (symlink #$(plain-file "hi-world" + "Hi, world!") + (string-append #$output "/hi")) + (symlink #$(plain-file "hello-world" + "Hello, world!") + (string-append #$output "/hello")) + (symlink #$(plain-file "greetings-world" + "Greetings, world!") + (string-append #$output "/greetings")))))) + +;; We turn the list into multiple values using (APPLY VALUES …). +(apply values + (map (lambda (file-path) + (let* ((path (string-append world-greeting-output "/" file-path)) + (contents (call-with-input-file path get-string-all))) + (list path contents))) + ;; SCANDIR from (ICE-9 FTW) returns the list of all files in a + ;; directory (including ``.'' and ``..'', so we remove them with the + ;; second argument, SELECT?, which specifies a predicate). + (scandir world-greeting-output + (lambda (path) + (not (or (string=? path ".") + (string=? path ".."))))))) +⇒ ("/gnu/store/…-world-greeting/greetings" "Greetings, world!") +⇒ ("/gnu/store/…-world-greeting/hello" "Hello, world!") +⇒ ("/gnu/store/…-world-greeting/hi" "Hi, world!") +``` + +Note that we define a procedure for building the output; we will need to build +more derivations in a very similar fashion later, so it helps to have this to +reuse instead of copying the code in `world-greeting-output`. + +There are many other useful lowerable objects available as part of the gexp +library. These include `computed-file`, which accepts a gexp that builds +the output file, `program-file`, which creates an executable Scheme script in +the store using a gexp, and `mixed-text-file`, which allows you to, well, mix +text and lowerable objects; it creates a file from the concatenation of a +sequence of strings and file-likes. The +[G-Expressions](https://guix.gnu.org/manual/en/html_node/G_002dExpressions.html) +manual page has more details. + +So, you may be wondering, at this point: there's so many lowerable objects +included with the gexps library, surely there must be a way to define more? +Naturally, there is; this is Scheme, after all! We simply need to acquaint +ourselves with the `define-gexp-compiler` macro. + +The most basic usage of `define-gexp-compiler` essentially creates a procedure +that takes as arguments a record to lower, the host system, and the target +system, and returns a derivation or store item as a monadic value in +`%store-monad`. + +Let's try implementing a lowerable object representing a file that greets the +world. First, we'll define the record type: + +```scheme +(use-modules (srfi srfi-9)) + +(define-record-type + (greeting-file greeting) + greeting? + (greeting greeting-file-greeting)) +``` + +Now we use `define-gexp-compiler` like so; note how we can use `lower-object` +to compile down any sort of lowerable object into the equivalent store item or +derivation; essentially, `lower-object` is just the procedure for applying the +right gexp compiler to an object: + +```scheme +(use-modules (ice-9 i18n)) + +(define-gexp-compiler (greeting-file-compiler + (greeting-file ) + system target) + (lower-object + (let ((greeting (greeting-file-greeting greeting-file))) + (plain-file (string-append greeting "-greeting") + (string-append (string-locale-titlecase greeting) ", world!"))))) +``` + +Let's try it out now. Here's how we could rewrite our greetings directory +example from before using ``: + +```scheme +(define world-greeting-2-output + (build-derivation + (gexp->derivation "world-greeting-2" + #~(begin + (mkdir #$output) + (symlink #$(greeting-file "hi") + (string-append #$output "/hi")) + (symlink #$(greeting-file "hello") + (string-append #$output "/hello")) + (symlink #$(greeting-file "greetings") + (string-append #$output "/greetings")))))) + +(apply values + (map (lambda (file-path) + (let* ((path (string-append world-greeting-2-output + "/" file-path)) + (contents (call-with-input-file path get-string-all))) + (list path contents))) + (scandir world-greeting-2-output + (lambda (path) + (not (or (string=? path ".") + (string=? path ".."))))))) +⇒ ("/gnu/store/…-world-greeting-2/greetings" "Greetings, world!") +⇒ ("/gnu/store/…-world-greeting-2/hello" "Hello, world!") +⇒ ("/gnu/store/…-world-greeting-2/hi" "Hi, world!") +``` + +Now, this is probably not worth a whole new gexp compiler. How about something +a bit more complex? Sharp-eyed readers who are trying all this in the REPL may +have noticed the following output when they used `define-gexp-compiler` +(formatted for ease of reading): + +```scheme +⇒ #< + type: #> + lower: # + expand: #> +``` + +Now, the purpose of `type` and `lower` is self-explanatory, but what's this +`expand` procedure here? Well, if you recall `file-append`, you may realise +that the text produced by a gexp compiler for embedding into a gexp doesn't +necessarily have to be the exact output path of the produced derivation. + +There turns out to be another way to write a `define-gexp-compiler` form that +allows you to specify _both_ the lowering procedure, which produces the +derivation or store item, and the expanding procedure, which produces the text. + +Let's make another record; this one will let us build a store item containing a +`bin` directory with multiple scripts inside, and expand to the full path to +that script. + +```scheme +(define-record-type + (script-directory scripts) + script-directory? + (scripts script-directory-scripts)) +``` + +Here's how we define both a compiler and expander for our new record: + +```scheme +(define-gexp-compiler script-directory-compiler + compiler => (lambda (obj system target) + (gexp->derivation "script-directory" + #~(let ((bindir (string-append #$output "/bin"))) + (mkdir #$output) + (mkdir bindir) + (for-each + (lambda (pair) + (let* ((name (car pair)) + (path (cdr pair)) + (bin-path (string-append bindir "/" name))) + (symlink path bin-path))) + #$(script-directory-scripts obj))))) + expander => (lambda (obj drv output) + (string-append output "/bin/" (script-directory-name obj)))) +``` + +Let's try this out now: + +```scheme +(use-modules (gnu packages vim)) + +(define script-directory-output + (build-derivation + (lower-object + (script-directory + #~'(("irc" . #$(file-append irssi "/bin/irssi")) + ("editor" . #$(file-append neovim "/bin/nvim"))))))) + +(scandir (string-append script-directory-output "/bin")) +⇒ ("." ".." "editor" "irc") +``` + +Who knows why you'd want to do this, but it certainly works! We've looked at +why we need gexps, how they work, and how to extend them, and we've now only got +two more advanced features to cover: cross-build support, and modules. + +# Importing External Modules + +Let's try using one of the helpful procedures from the `(guix build utils)` +module in a gexp. + +```scheme +(define silly-directory-output + (build-derivation + (gexp->derivation "silly-directory" + #~(begin + (use-modules (guix build utils)) + (mkdir-p (string-append #$output "/what/a/silly/directory")))))) +``` + +Looks fine, right? We've even got a `use-modules` in th-- + +```Scheme +ERROR: + 1. &store-protocol-error: + message: "build of `/gnu/store/…-silly-directory.drv' failed" + status: 100 +``` + +OUTRAGEOUS. Fortunately, there's an explanation to be found in the Guix build +log directory, `/var/log/guix/drvs`; locate the file using the first two +characters of the store hash as the subdirectory, and the rest as the file name, +and remember to use `zcat` or `zless`, as the logs are gzipped: + +```scheme +Backtrace: + 9 (primitive-load "/gnu/store/…") +In ice-9/eval.scm: + 721:20 8 (primitive-eval (begin (use-modules (guix build #)) (?))) +In ice-9/psyntax.scm: + 1230:36 7 (expand-top-sequence ((begin (use-modules (guix ?)) #)) ?) + 1090:25 6 (parse _ (("placeholder" placeholder)) ((top) #(# # ?)) ?) + 1222:19 5 (parse _ (("placeholder" placeholder)) ((top) #(# # ?)) ?) + 259:10 4 (parse _ (("placeholder" placeholder)) (()) _ c&e (eval) ?) +In ice-9/boot-9.scm: + 3927:20 3 (process-use-modules _) + 222:17 2 (map1 (((guix build utils)))) + 3928:31 1 (_ ((guix build utils))) + 3329:6 0 (resolve-interface (guix build utils) #:select _ #:hide ?) + +ice-9/boot-9.scm:3329:6: In procedure resolve-interface: +no code for module (guix build utils) +``` + +It turns out `use-modules` can't actually find `(guix build utils)` at all. +There's no typo; it's just that to ensure the build is isolated, Guix builds +`module-import` and `module-importe-compiled` directories, and sets the +_Guile module path_ within the build environment to contain said directories, +along with those containing the Guile standard library modules. + +So, what to do? Turns out one of the fields in `` is `modules`, which, +funnily enough, contains the names of the modules which will be used to build +the aforementioned directories. To add to this field, we use the +`with-imported-modules` macro. (`gexp->derivation` _does_ provide a `modules` +parameter, but `with-imported-modules` lets you add the required modules +directly to the gexp value, rather than later on.) + +```scheme +(define silly-directory-output + (build-derivation + (gexp->derivation "silly-directory" + (with-imported-modules '((guix build utils)) + #~(begin + (use-modules (guix build utils)) + (mkdir-p (string-append #$output "/what/a/silly/directory"))))))) + +silly-directory-output +⇒ "/gnu/store/…-silly-directory" +``` + +It works, yay. It's worth noting that while passing just the list of modules to +`with-imported-modules` works in this case, this is only because +`(guix build utils)` has no dependencies on other Guix modules. Were we to try +adding, say, `(guix build emacs-build-system)`, we'd need to use the +`source-module-closure` procedure to add its dependencies to the list: + +```scheme +(source-module-closure '((guix build emacs-build-system))) +⇒ ((guix build emacs-build-system) + (guix build gnu-build-system) + (guix build utils) + (guix build gremlin) + (guix elf) + (guix build emacs-utils)) +``` + +Here's another scenario: what if we want to use a module not from Guix or Guile +but a third-party library? In this example, we'll use [guile-json +](https://github.com/aconchillo/guile-json), a library for converting between +S-expressions and [JavaScript Object Notation](https://json.org). + +We can't just `with-imported-modules` its modules, since it's not part of Guix, +so `` provides another field for this purpose: `extensions`. Each of +these extensions is a lowerable object that produces a Guile package directory; +so usually a package. Let's try it out. + +```scheme +(use-modules (gnu packages guile)) + +(define helpful-guide-output + (build-derivation + (gexp->derivation "json-file" + (with-extensions (list guile-json-4) + #~(begin + (use-modules (json)) + (mkdir #$output) + (call-with-output-file (string-append #$output "/helpful-guide.json") + (lambda (port) + (scm->json '((truth . "Guix is the best!") + (lies . "Guix isn't the best!")) + port)))))))) + +(call-with-input-file + (string-append helpful-guide-output "/helpful-guide.json") + get-string-all) +⇒ "{\"truth\":\"Guix is the best!\",\"lies\":\"Guix isn't the best!\"}" +``` + +Amen to that, `helpful-guide.json`. Before we continue on to cross-compilation, +there's one last feature of `with-imported-modules` you should note. We can +add modules to a gexp by name, but we can also create entirely new ones with +lowerable objects, like this pattern, which is used in several places in the +Guix source code: + +```scheme +(with-imported-modules `(((guix config) => ,(make-config.scm)) + …) + …) +``` + +In case you're wondering, `make-config.scm` is found in `(guix self)` and +returns a lowerable object that compiles to a version of the `(guix config)` +module, which contains constants usually substituted into the source code at +compile time. + +# Native Ungexp + +There is another piece of syntax we can use with gexps, and it's called +`ungexp-native`. This helps us distinguish between native inputs and regular +host-built inputs in cross-compilation situations. We'll cover +cross-compilation in detail at a later date, but the gist of it is that it +allows you to compile a derivation for one architecture X, the target, using a +machine of architecture Y, the host, and Guix has excellent support for it. + +If we cross-compile a gexp G that _non-natively_ ungexps L1, a lowerable object, +from architecture Y to architecture X, both G and L1 will be compiled for +architecture X. However, if G _natively_ ungexps L1, G will be compiled for X +and L1 for Y. + +Essentially, we use `ungexp-native` in situations where there would be no +difference between compiling on different architectures (for instance, if `L1` +were a `plain-file`), or where using L1 built for X would actually _break_ G +(for instance, if `L1` corresponds to a compiled executable that needs to be run +during the build; the executable would fail to run on Y if it was built for X.) + +The `ungexp-native` macro naturally has a corresponding reader syntax, `#+`, and +there's also `ungexp-native-splicing`, which is written as `#+@`. These two +pieces of syntax are used in the same way as their regular counterparts. + +# Conclusion + +Mastering gexps is essential to understanding Guix's inner workings, so the aim +of this blog post is to be as thorough as possible. However, if you still find +yourself with questions, please don't hesitate to stop by at the IRC channel +`#guix:libera.chat` and mailing list `help-guix@gnu.org`; we'll be glad to +assist you! + +#### About GNU Guix + +[GNU Guix](https://guix.gnu.org) is a transactional package manager and +an advanced distribution of the GNU system that [respects user +freedom](https://www.gnu.org/distros/free-system-distribution-guidelines.html). +Guix can be used on top of any system running the Hurd or the Linux +kernel, or it can be used as a standalone operating system distribution +for i686, x86_64, ARMv7, AArch64 and POWER9 machines. + +In addition to standard package management features, Guix supports +transactional upgrades and roll-backs, unprivileged package management, +per-user profiles, and garbage collection. When used as a standalone +GNU/Linux distribution, Guix offers a declarative, stateless approach to +operating system configuration management. Guix is highly customizable +and hackable through [Guile](https://www.gnu.org/software/guile) +programming interfaces and extensions to the +[Scheme](http://schemers.org) language.