From patchwork Fri Dec 28 23:12:05 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Ludovic_Court=C3=A8s?= X-Patchwork-Id: 567 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 6F800169D1; Fri, 28 Dec 2018 23:36:50 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_HI, URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mira.cbaines.net (Postfix) with ESMTP id 1EE471699E for ; Fri, 28 Dec 2018 23:36:50 +0000 (GMT) Received: from localhost ([127.0.0.1]:33820 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gd1gP-00063O-J9 for patchwork@mira.cbaines.net; Fri, 28 Dec 2018 18:36:49 -0500 Received: from eggs.gnu.org ([208.118.235.92]:38014) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gd1gC-0005r2-Nd for guix-patches@gnu.org; Fri, 28 Dec 2018 18:36:41 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gd1KP-00074Z-IV for guix-patches@gnu.org; Fri, 28 Dec 2018 18:14:09 -0500 Received: from debbugs.gnu.org ([208.118.235.43]:41571) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gd1KM-0006yO-Ft for guix-patches@gnu.org; Fri, 28 Dec 2018 18:14:04 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1gd1KM-0001Lf-29 for guix-patches@gnu.org; Fri, 28 Dec 2018 18:14:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#33899] [PATCH 0/5] Distributing substitutes over IPFS Resent-From: Ludovic =?utf-8?q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 28 Dec 2018 23:14:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 33899 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 33899@debbugs.gnu.org X-Debbugs-Original-To: guix-patches@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.15460387815093 (code B ref -1); Fri, 28 Dec 2018 23:14:01 +0000 Received: (at submit) by debbugs.gnu.org; 28 Dec 2018 23:13:01 +0000 Received: from localhost ([127.0.0.1]:40703 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gd1JM-0001Jx-Do for submit@debbugs.gnu.org; Fri, 28 Dec 2018 18:13:01 -0500 Received: from eggsout.gnu.org ([209.51.188.92]:39980) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gd1JK-0001Jh-T1 for submit@debbugs.gnu.org; Fri, 28 Dec 2018 18:12:59 -0500 Received: from lists.gnu.org ([208.118.235.17]:58459) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gd1JF-0005rK-EO for submit@debbugs.gnu.org; Fri, 28 Dec 2018 18:12:53 -0500 Received: from eggs.gnu.org ([208.118.235.92]:33661) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gd1J9-0000oR-VG for guix-patches@gnu.org; Fri, 28 Dec 2018 18:12:53 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gd1J3-0005bq-Kh for guix-patches@gnu.org; Fri, 28 Dec 2018 18:12:47 -0500 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:50661) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gd1Ib-00052a-Gm; Fri, 28 Dec 2018 18:12:13 -0500 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=51292 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1gd1Ib-0006mA-74; Fri, 28 Dec 2018 18:12:13 -0500 From: Ludovic =?utf-8?q?Court=C3=A8s?= Date: Sat, 29 Dec 2018 00:12:05 +0100 Message-Id: <20181228231205.8068-1-ludo@gnu.org> X-Mailer: git-send-email 2.20.1 MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 208.118.235.43 X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Hector Sanjuan , Pierre Neidhardt Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches Hello Guix! Here is a first draft adding support to distribute and retrieve substitutes over IPFS. This builds on discussions at the R-B Summit with Héctor Sanjuan of IPFS, lewo of Nix, Pierre Neidhardt, and also on the work Florian Paul Schmidt posted on guix-devel last month. The IPFS daemon exposes an HTTP API and the (guix ipfs) module provides bindings to a subset of that API. This module also implements a custom “directory” format to store directory trees in IPFS (IPFS already provides “UnixFS” and “tar” but they store too many or too few file attributes.) ‘guix publish’ and ‘guix substitute’ use (guix ipfs) to store and retrieve store items. Complete directory trees are stored in IPFS “as is”, rather than as compressed archives (nars). This allows for deduplication in IPFS. ‘guix publish’ adds a new “IPFS” field in narinfos and ‘guix substitute’ can then query those objects over IPFS. So the idea is that you still get narinfos over HTTP(S), and then you have the option of downloading substitutes over IPFS. I’ve pushed these patches in ‘wip-ipfs-substitutes’. This is rough on the edges and probably buggy, but the adventurous among us might want to give it a spin. :-) Thanks, Ludo’. Ludovic Courtès (5): Add (guix json). tests: 'file=?' now recurses on directories. Add (guix ipfs). publish: Add IPFS support. DRAFT substitute: Add IPFS support. Makefile.am | 3 + doc/guix.texi | 33 +++++ guix/ipfs.scm | 250 ++++++++++++++++++++++++++++++++++++ guix/json.scm | 63 +++++++++ guix/scripts/publish.scm | 67 +++++++--- guix/scripts/substitute.scm | 106 ++++++++------- guix/swh.scm | 35 +---- guix/tests.scm | 26 +++- tests/ipfs.scm | 55 ++++++++ 9 files changed, 535 insertions(+), 103 deletions(-) create mode 100644 guix/ipfs.scm create mode 100644 guix/json.scm create mode 100644 tests/ipfs.scm