diff mbox series

[bug#52555,RFC,v2,2/5] publish: Add ERIS URN to narinfo

Message ID 20220125192201.7582-3-pukkamustard@posteo.net
State New
Headers show
Series Decentralized substitute distribution with ERIS | expand

Checks

Context Check Description
cbaines/comparison success View comparision
cbaines/git branch success View Git branch
cbaines/applying patch success View Laminar job
cbaines/issue success View issue

Commit Message

pukkamustard Jan. 25, 2022, 7:21 p.m. UTC
* guix/scripts/publish.scm: (bake-narinfo+nar): Compute ERIS URN of compressed nars.
(narinfo-string): Add #:eris-urn parameter and honor it.
* guix/scripts/narinfo.scm: (<narinfo>)[eris-format,eris-urn]: New fields.
(narinfo-maker): Handle ERIS URN and ERIS format.
* configure.ac: (HAVE_GUILE_ERIS): New conditional.
* gnu/packages/package-management.scm: (guix)[native-inputs]: Add guile-eris.
---
 configure.ac                        |  5 +++++
 gnu/packages/package-management.scm |  1 +
 guix/narinfo.scm                    | 14 ++++++++++----
 guix/scripts/publish.scm            | 24 ++++++++++++++++++++----
 4 files changed, 36 insertions(+), 8 deletions(-)

Comments

M Jan. 29, 2022, 9:09 p.m. UTC | #1
pukkamustard schreef op di 25-01-2022 om 19:21 [+0000]:
> +dnl Check for Guile-eris.
> +GUILE_MODULE_AVAILABLE([have_guile_eris], [(eris)])
> +AM_CONDITIONAL([HAVE_GUILE_ERIS],
> +  [test "x$have_guile_eris" = "xyes"])

This could to be documented in (guix)Requirements.

Greetings,
Maxime.
M Jan. 29, 2022, 9:15 p.m. UTC | #2
pukkamustard schreef op di 25-01-2022 om 19:21 [+0000]:
> +  (define (eris-encode-nar compressions)
> +    (and (member %eris-zstd-compression compressions)
> +         (let* ((nar (nar-cache-file cache item
> +                                     #:compression %eris-zstd-compression))
> +                (stat (stat nar #f)))
> +           (and stat
> +                (call-with-input-file nar
> +                  (cut eris-encode->string <>
> +                       #:block-size %eris-block-size-large))))))

Why are exceptions turned into #f (in (stat nar #f))?
Should this be done for all I/O errors, including, say, EOVERFLOW,
ENOMEM or ENAMETOOLONG, or only for ENOENT?

Is a race condition possible here?  If so, maybe consider doing
something like

  (catch 'system-error
    (lambda () (call-with-input-file ...))
    (lambda exception
      (and it-is-a-ENOENT
           (apply throw exception))))

to avoid it?

Greetings,
Maxime.
pukkamustard Feb. 2, 2022, 10:16 a.m. UTC | #3
Maxime Devos <maximedevos@telenet.be> writes:

> pukkamustard schreef op di 25-01-2022 om 19:21 [+0000]:
>> +  (define (eris-encode-nar compressions)
>> +    (and (member %eris-zstd-compression compressions)
>> +         (let* ((nar (nar-cache-file cache item
>> +                                     #:compression %eris-zstd-compression))
>> +                (stat (stat nar #f)))
>> +           (and stat
>> +                (call-with-input-file nar
>> +                  (cut eris-encode->string <>
>> +                       #:block-size %eris-block-size-large))))))
>
> Why are exceptions turned into #f (in (stat nar #f))?
> Should this be done for all I/O errors, including, say, EOVERFLOW,
> ENOMEM or ENAMETOOLONG, or only for ENOENT?
>
> Is a race condition possible here?  If so, maybe consider doing
> something like
>
>   (catch 'system-error
>     (lambda () (call-with-input-file ...))
>     (lambda exception
>       (and it-is-a-ENOENT
>            (apply throw exception))))
>
> to avoid it?

A valid question. But (stat nar #f) is not something I introduced. It is
already in guix/scripts/publish.scm like that.

To me turning all exceptions to #f makes sense. Here we only want to
know if the file is readable. As the NAR is baked in the background by
another thread the case where the compressed NAR does exist yet will
happen. In that case we don't worry, we just don't publish the
`FileSize` and `ERIS` fields in the narinfo.

-pukkamustard
diff mbox series

Patch

diff --git a/configure.ac b/configure.ac
index 341cff8fbd..72396be8aa 100644
--- a/configure.ac
+++ b/configure.ac
@@ -170,6 +170,11 @@  GUILE_MODULE_AVAILABLE([have_guile_avahi], [(avahi)])
 AM_CONDITIONAL([HAVE_GUILE_AVAHI],
   [test "x$have_guile_avahi" = "xyes"])
 
+dnl Check for Guile-eris.
+GUILE_MODULE_AVAILABLE([have_guile_eris], [(eris)])
+AM_CONDITIONAL([HAVE_GUILE_ERIS],
+  [test "x$have_guile_eris" = "xyes"])
+
 dnl Guile-newt is used by the graphical installer.
 GUILE_MODULE_AVAILABLE([have_guile_newt], [(newt)])
 
diff --git a/gnu/packages/package-management.scm b/gnu/packages/package-management.scm
index 05795824b5..a9094b8b7f 100644
--- a/gnu/packages/package-management.scm
+++ b/gnu/packages/package-management.scm
@@ -404,6 +404,7 @@  (define code
                        ("guile-zstd" ,guile-zstd)
                        ("guile-ssh" ,guile-ssh)
                        ("guile-git" ,guile-git)
+                       ("guile-eris" ,guile-eris)
 
                        ;; XXX: Keep the development inputs here even though
                        ;; they're unnecessary, just so that 'guix environment
diff --git a/guix/narinfo.scm b/guix/narinfo.scm
index 4fc550aa6c..a6a5d3b84b 100644
--- a/guix/narinfo.scm
+++ b/guix/narinfo.scm
@@ -45,6 +45,8 @@  (define-module (guix narinfo)
             narinfo-file-sizes
             narinfo-hash
             narinfo-size
+            narinfo-eris-format
+            narinfo-eris-urn
             narinfo-references
             narinfo-deriver
             narinfo-system
@@ -68,8 +70,8 @@  (define-module (guix narinfo)
 
 (define-record-type <narinfo>
   (%make-narinfo path uri-base uris compressions file-sizes file-hashes
-                 nar-hash nar-size references deriver system
-                 signature contents)
+                 nar-hash nar-size eris-format eris-urn references deriver
+                 system signature contents)
   narinfo?
   (path         narinfo-path)
   (uri-base     narinfo-uri-base)        ;URI of the cache it originates from
@@ -79,6 +81,8 @@  (define-record-type <narinfo>
   (file-hashes  narinfo-file-hashes)
   (nar-hash     narinfo-hash)
   (nar-size     narinfo-size)
+  (eris-format  narinfo-eris-format)
+  (eris-urn     narinfo-eris-urn)
   (references   narinfo-references)
   (deriver      narinfo-deriver)
   (system       narinfo-system)
@@ -135,7 +139,7 @@  (define (narinfo-maker str cache-url)
   "Return a narinfo constructor for narinfos originating from CACHE-URL.  STR
 must contain the original contents of a narinfo file."
   (lambda (path urls compressions file-hashes file-sizes
-                nar-hash nar-size references deriver system
+                nar-hash nar-size eris-format eris-urn references deriver system
                 signature)
     "Return a new <narinfo> object."
     (define len (length urls))
@@ -157,6 +161,8 @@  (define len (length urls))
                      ((lst ...) (map string->number lst)))
                    nar-hash
                    (and=> nar-size string->number)
+                   eris-format
+                   (if eris-urn (string->uri eris-urn) #f)
                    (string-tokenize references)
                    (match deriver
                      ((or #f "") #f)
@@ -184,7 +190,7 @@  (define* (read-narinfo port #:optional url
                    (narinfo-maker str url)
                    '("StorePath" "URL" "Compression"
                      "FileHash" "FileSize" "NarHash" "NarSize"
-                     "References" "Deriver" "System"
+                     "ERISFormat" "ERIS" "References" "Deriver" "System"
                      "Signature")
                    '("URL" "Compression" "FileSize" "FileHash"))))
 
diff --git a/guix/scripts/publish.scm b/guix/scripts/publish.scm
index 6e2b4368da..9c83f5183d 100644
--- a/guix/scripts/publish.scm
+++ b/guix/scripts/publish.scm
@@ -58,6 +58,7 @@  (define-module (guix scripts publish)
   #:use-module (guix workers)
   #:use-module (guix store)
   #:use-module ((guix serialization) #:select (write-file))
+  #:use-module (eris)
   #:use-module (zlib)
   #:autoload   (lzlib) (call-with-lzip-output-port
                         make-lzip-output-port)
@@ -146,6 +147,9 @@  (define %default-gzip-compression
   ;; Since we compress on the fly, default to fast compression.
   (compression 'gzip 3))
 
+(define %eris-zstd-compression
+  (compression 'zstd 19))
+
 (define (default-compression type)
   (compression type 3))
 
@@ -324,7 +328,8 @@  (define* (store-item->recutils store-item
 
 (define* (narinfo-string store store-path
                          #:key (compressions (list %no-compression))
-                         (nar-path "nar") (file-sizes '()))
+                         (nar-path "nar") (file-sizes '())
+                         eris-urn)
   "Generate a narinfo key/value string for STORE-PATH; an exception is raised
 if STORE-PATH is invalid.  Produce a URL that corresponds to COMPRESSION.  The
 narinfo is signed with KEY.  NAR-PATH specifies the prefix for nar URLs.
@@ -347,7 +352,7 @@  (define* (narinfo-string store store-path
 StorePath: ~a
 ~{~a~}\
 NarHash: sha256:~a
-NarSize: ~d
+NarSize: ~d~@[~%ERISFormat: application/x-nix-archive+zstd-19~%ERIS: ~a~]
 References: ~a~%"
                              store-path
                              (map (lambda (compression)
@@ -359,7 +364,7 @@  (define* (narinfo-string store store-path
                                                             #:compression
                                                             compression)))
                                   compressions)
-                             hash size references))
+                             hash size eris-urn references))
          ;; Do not render a "Deriver" line if we are rendering info for a
          ;; derivation.  Also do not render a "System" line that would be
          ;; expensive to compute and is currently unused.
@@ -632,6 +637,16 @@  (define (compressed-nar-size compression)
       (and stat
            (cons compression (stat:size stat)))))
 
+  (define (eris-encode-nar compressions)
+    (and (member %eris-zstd-compression compressions)
+         (let* ((nar (nar-cache-file cache item
+                                     #:compression %eris-zstd-compression))
+                (stat (stat nar #f)))
+           (and stat
+                (call-with-input-file nar
+                  (cut eris-encode->string <>
+                       #:block-size %eris-block-size-large))))))
+
   (let ((compression (actual-compressions item compressions)))
 
     (for-each (cut compress-nar cache item <>) compressions)
@@ -650,7 +665,8 @@  (define (compressed-nar-size compression)
                  (display (narinfo-string store item
                                           #:nar-path nar-path
                                           #:compressions compressions
-                                          #:file-sizes sizes)
+                                          #:file-sizes sizes
+                                          #:eris-urn (eris-encode-nar compression))
                           port)))
 
              ;; Make the cached narinfo world-readable, contrary to what