From patchwork Thu Dec 28 09:40:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: pukkamustard X-Patchwork-Id: 58129 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 1FF0127BBEA; Thu, 28 Dec 2023 09:41:38 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.7 required=5.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,SPF_HELO_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 2DCF127BBE2 for ; Thu, 28 Dec 2023 09:41:36 +0000 (GMT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rImt2-0007eA-De; Thu, 28 Dec 2023 04:41:08 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rImt1-0007d9-6O for guix-patches@gnu.org; Thu, 28 Dec 2023 04:41:07 -0500 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rImt0-000834-T1; Thu, 28 Dec 2023 04:41:06 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1rImsx-0005lO-Hs; Thu, 28 Dec 2023 04:41:03 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#52555] [PATCH v4 4/7] publish: Add ERIS URN to narinfo. Resent-From: pukkamustard Original-Sender: "Debbugs-submit" Resent-CC: guix@cbaines.net, dev@jpoiret.xyz, ludo@gnu.org, othacehe@gnu.org, rekado@elephly.net, zimon.toutoune@gmail.com, me@tobias.gr, guix-patches@gnu.org Resent-Date: Thu, 28 Dec 2023 09:41:03 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 52555 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 52555@debbugs.gnu.org Cc: pukkamustard , ludo@gnu.org, maximedevos@telenet.be, Christopher Baines , Josselin Poiret , Ludovic =?utf-8?q?Court=C3=A8s?= , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice X-Debbugs-Original-Xcc: Christopher Baines , Josselin Poiret , Ludovic =?utf-8?q?Court=C3=A8s?= , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice Received: via spool by 52555-submit@debbugs.gnu.org id=B52555.170375644022058 (code B ref 52555); Thu, 28 Dec 2023 09:41:03 +0000 Received: (at 52555) by debbugs.gnu.org; 28 Dec 2023 09:40:40 +0000 Received: from localhost ([127.0.0.1]:38522 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rImsZ-0005ja-5k for submit@debbugs.gnu.org; Thu, 28 Dec 2023 04:40:40 -0500 Received: from mout02.posteo.de ([185.67.36.66]:35927) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rImsU-0005iQ-VW for 52555@debbugs.gnu.org; Thu, 28 Dec 2023 04:40:36 -0500 Received: from submission (posteo.de [185.67.36.169]) by mout02.posteo.de (Postfix) with ESMTPS id 2E175240105 for <52555@debbugs.gnu.org>; Thu, 28 Dec 2023 10:40:30 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017; t=1703756430; bh=D1gabgpydjsdJrKn5ayoFzKFjF17cQX7Sn4fJkp+bHs=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version: Content-Transfer-Encoding:From; b=fRVwKOzvGNfuAZTgVCe16rym1DbWmhdUhT1wMBo19eEOHq/YBv2HUtYoIPAY8Ag9S srbP3pWjMdCwOxaFBco1jZPUsNMEt+ewE7/07LqhpvdRg75pqG23/sclD3+5KMtBfc 7t/qdApgVyZwOmR3MCom6fQhsz+AdxT34b2iQSfEwuR4RSnk68jvlrNkf2d/0ogOkd 2VMuQMvWdcTl+5pSCPLB9WP8nkAmLWr9nxqnLGcKv8ZVp6avlX2Qe2wbjR1MNFUYiV 35zbj6ObC17W/5xNjubZBGwAY4dchvhPwHA9bfvoetjY4A+e6iu+c+GlzaLMPrzAyc 9ZU8Ey2hEvN0Q== Received: from customer (localhost [127.0.0.1]) by submission (posteo.de) with ESMTPSA id 4T13Qj2LFSz6tyf; Thu, 28 Dec 2023 10:40:29 +0100 (CET) From: pukkamustard Date: Thu, 28 Dec 2023 09:40:07 +0000 Message-ID: <000b47eb98b4f22a24f246cc12bb405c65efdf57.1703316055.git.pukkamustard@posteo.net> In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches * guix/scripts/publish.scm (bake-narinfo+nar): Encode store item using ERIS. (show-help, %options): Add '--eris'. (guix-publish): Honor '--eris'. * gnu/packages/package-management.scm (guix): Add guile-eris to native-inputs. * guix/eris.scm: New file. * Makefile.am (MODULES): Add new file. --- Makefile.am | 1 + configure.ac | 5 ++ gnu/packages/package-management.scm | 1 + guix/eris.scm | 73 +++++++++++++++++++++++++++++ guix/scripts/publish.scm | 51 ++++++++++++++------ 5 files changed, 117 insertions(+), 14 deletions(-) create mode 100644 guix/eris.scm diff --git a/Makefile.am b/Makefile.am index b64dcaa77c..86da4560e4 100644 --- a/Makefile.am +++ b/Makefile.am @@ -136,6 +136,7 @@ MODULES = \ guix/least-authority.scm \ guix/read-print.scm \ guix/ipfs.scm \ + guix/eris.scm \ guix/platform.scm \ guix/platforms/arm.scm \ guix/platforms/avr.scm \ diff --git a/configure.ac b/configure.ac index ecbd596a34..7e25099c4c 100644 --- a/configure.ac +++ b/configure.ac @@ -173,6 +173,11 @@ GUILE_MODULE_AVAILABLE([have_guile_avahi], [(avahi)]) AM_CONDITIONAL([HAVE_GUILE_AVAHI], [test "x$have_guile_avahi" = "xyes"]) +dnl Check for Guile-ERIS. +GUILE_MODULE_AVAILABLE([have_guile_eris], [(eris)]) +AM_CONDITIONAL([HAVE_GUILE_ERIS], + [test "x$have_guile_eris" = "xyes"]) + dnl Guile-newt is used by the graphical installer. GUILE_MODULE_AVAILABLE([have_guile_newt], [(newt)]) diff --git a/gnu/packages/package-management.scm b/gnu/packages/package-management.scm index 97ea41df66..9c0afb70dd 100644 --- a/gnu/packages/package-management.scm +++ b/gnu/packages/package-management.scm @@ -478,6 +478,7 @@ (define-public guix ("guile-zstd" ,guile-zstd) ("guile-ssh" ,guile-ssh) ("guile-git" ,guile-git) + ("guile-eris" ,guile-eris) ;; XXX: Keep the development inputs here even though ;; they're unnecessary, just so that 'guix environment diff --git a/guix/eris.scm b/guix/eris.scm new file mode 100644 index 0000000000..d98a9a62bd --- /dev/null +++ b/guix/eris.scm @@ -0,0 +1,73 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright © 2022 pukkamustard +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see . + +(define-module (guix eris) + + #:use-module (eris) + #:use-module (eris fs) + #:use-module (eris sqlite) + #:use-module (eris read-capability) + + #:use-module (web uri) + #:use-module (ice-9 match) + #:use-module (srfi srfi-171) + + #:export (%eris-store-url + eris-encode-store-item)) + +(define %eris-store-url + (make-parameter + (getenv "ERIS_STORE_URL") + (lambda (val) + (cond + ((uri? val) val) + ((string? val) (string->uri val)) + (else #f))))) + +(define %guix-eris-convergence-secret + (make-parameter %null-convergence-secret)) + +(define (guix-eris-block-reducer) + "Returns an ERIS block reducer." + (if (uri? (%eris-store-url)) + (match (uri-scheme (%eris-store-url)) + + ;; Store blocks in an SQLite database (see + ;; https://eris.codeberg.page/eer/sqlite.xml) + ('sqlite + (eris-sqlite-block-reducer (uri-path (%eris-store-url)))) + + ;; TODO + ;; ('coap+unix #f) + ;; ('coap+tcp #f) + + (_ (error "Don't know how to handle ERIS store URL " + (uri->string (%eris-store-url))))) + + ;; If no ERIS store URL is provided we just compute the ERIS URN without + ;; storing the blocks anywhere. As dummy block-reducer we use `rcount` from + ;; SRFI-171 that counts the number of blocks. + rcount)) + +(define* (eris-encode-store-item item) + "Encodes the store item ITEM using ERIS and returns the read capability as +string." + (eris-read-capability->string + (eris-fs-encode item + #:convergence-secret (%guix-eris-convergence-secret) + #:block-reducer (guix-eris-block-reducer)))) diff --git a/guix/scripts/publish.scm b/guix/scripts/publish.scm index 4457be1fce..2e7138f3c7 100644 --- a/guix/scripts/publish.scm +++ b/guix/scripts/publish.scm @@ -53,6 +53,7 @@ (define-module (guix scripts publish) #:use-module (guix workers) #:use-module (guix store) #:use-module ((guix serialization) #:select (write-file)) + #:use-module (guix eris) #:use-module (zlib) #:autoload (lzlib) (call-with-lzip-output-port make-lzip-output-port) @@ -96,6 +97,8 @@ (define (show-help) (display (G_ " --cache-bypass-threshold=SIZE serve store items below SIZE even when not cached")) + (display (G_ " + --eris[=STORE] encode items using ERIS and publish blocks to STORE")) (display (G_ " --workers=N use N workers to bake items")) (display (G_ " @@ -218,6 +221,9 @@ (define %options (lambda (opt name arg result) (alist-cons 'workers (string->number* arg) result))) + (option '("eris") #f #t + (lambda (opt name arg result) + (alist-cons 'eris (or arg #t) result))) (option '("ttl") #t #f (lambda (opt name arg result) (let ((duration (string->duration arg))) @@ -319,7 +325,8 @@ (define* (store-item->recutils store-item (define* (narinfo-string store store-path #:key (compressions (list %no-compression)) - (nar-path "nar") (file-sizes '())) + (nar-path "nar") (file-sizes '()) + eris-urn) "Generate a narinfo key/value string for STORE-PATH; an exception is raised if STORE-PATH is invalid. Produce a URL that corresponds to COMPRESSION. The narinfo is signed with KEY. NAR-PATH specifies the prefix for nar URLs. @@ -341,10 +348,10 @@ (define* (narinfo-string store store-path "\ StorePath: ~a NarHash: sha256:~a -NarSize: ~d +NarSize: ~d~@[~%ERIS: ~a~] References: ~a~%" store-path - hash size references)) + hash size eris-urn references)) ;; Do not render a "Deriver" line if we are rendering info for a ;; derivation. Also do not render a "System" line that would be ;; expensive to compute and is currently unused. @@ -530,7 +537,7 @@ (define (bypass-cache? store item) (define* (render-narinfo/cached store request hash #:key ttl (compressions (list %no-compression)) (nar-path "nar") negative-ttl - cache pool) + cache pool eris?) "Respond to the narinfo request for REQUEST. If the narinfo is available in CACHE, then send it; otherwise, return 404 and \"bake\" that nar and narinfo requested using POOL." @@ -575,7 +582,8 @@ (define* (render-narinfo/cached store request hash (bake-narinfo+nar cache item #:ttl ttl #:compressions compressions - #:nar-path nar-path))) + #:nar-path nar-path + #:eris? eris?))) (when ttl (single-baker 'cache-cleanup @@ -636,7 +644,8 @@ (define (compress-nar cache item compression) (define* (bake-narinfo+nar cache item #:key ttl (compressions (list %no-compression)) - (nar-path "/nar")) + (nar-path "/nar") + (eris? #f)) "Write the narinfo and nar for ITEM to CACHE." (define (compressed-nar-size compression) (let* ((nar (nar-cache-file cache item #:compression compression)) @@ -644,7 +653,10 @@ (define* (bake-narinfo+nar cache item (and stat (cons compression (stat:size stat))))) - (let ((compression (actual-compressions item compressions))) + (let ((compression (actual-compressions item compressions)) + (eris-urn (if eris? + (eris-encode-store-item item) + #f))) (for-each (cut compress-nar cache item <>) compressions) @@ -662,7 +674,8 @@ (define* (bake-narinfo+nar cache item (display (narinfo-string store item #:nar-path nar-path #:compressions compressions - #:file-sizes sizes) + #:file-sizes sizes + #:eris-urn eris-urn) port))) ;; Make the cached narinfo world-readable, contrary to what @@ -1060,7 +1073,8 @@ (define* (make-request-handler store cache pool narinfo-ttl narinfo-negative-ttl (nar-path "nar") - (compressions (list %no-compression))) + (compressions (list %no-compression)) + (eris? #f)) (define compression-type? string->compression-type) @@ -1092,7 +1106,8 @@ (define* (make-request-handler store #:ttl narinfo-ttl #:negative-ttl narinfo-negative-ttl #:nar-path nar-path - #:compressions compressions) + #:compressions compressions + #:eris? eris?) (render-narinfo store request hash #:ttl narinfo-ttl #:negative-ttl narinfo-negative-ttl @@ -1162,7 +1177,7 @@ (define* (run-publish-server socket store advertise? port (compressions (list %no-compression)) (nar-path "nar") narinfo-ttl narinfo-negative-ttl - cache pool) + cache pool eris?) (when advertise? (let ((name (service-name))) ;; XXX: Use a callback from Guile-Avahi here, as Avahi can pick a @@ -1178,7 +1193,8 @@ (define* (run-publish-server socket store #:nar-path nar-path #:narinfo-ttl narinfo-ttl #:narinfo-negative-ttl narinfo-negative-ttl - #:compressions compressions) + #:compressions compressions + #:eris? eris?) concurrent-http-server `(#:socket ,socket))) @@ -1262,6 +1278,7 @@ (define-command (guix-publish . args) (repl-port (assoc-ref opts 'repl)) (cache (assoc-ref opts 'cache)) (workers (assoc-ref opts 'workers)) + (eris? (assoc-ref opts 'eris)) ;; Read the key right away so that (1) we fail early on if we can't ;; access them, and (2) we can then drop privileges. @@ -1281,7 +1298,8 @@ (define-command (guix-publish . args) (%private-key private-key) (cache-bypass-threshold (or (assoc-ref opts 'cache-bypass-threshold) - (cache-bypass-threshold)))) + (cache-bypass-threshold))) + (%eris-store-url (assoc-ref opts 'eris))) (if (eq? style 'systemd) (info (G_ "publishing (started via socket activation)~%")) (info (G_ "publishing ~a on ~a, port ~d~%") @@ -1289,6 +1307,10 @@ (define-command (guix-publish . args) (inet-ntop (sockaddr:fam address) (sockaddr:addr address)) (sockaddr:port address))) + (when (string? (assoc-ref opts 'eris)) + (info (G_ "publishing ERIS blocks to ~a~%") + (assoc-ref opts 'eris))) + (for-each (lambda (compression) (info (G_ "using '~a' compression method, level ~a~%") (compression-type compression) @@ -1312,7 +1334,8 @@ (define-command (guix-publish . args) #:nar-path nar-path #:compressions compressions #:narinfo-negative-ttl negative-ttl - #:narinfo-ttl ttl)))))) + #:narinfo-ttl ttl + #:eris? eris?)))))) ;;; Local Variables: ;;; eval: (put 'single-baker 'scheme-indent-function 1)