From patchwork Mon Nov 19 16:13:25 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Ludovic_Court=C3=A8s?= X-Patchwork-Id: 229 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 44341167ED; Mon, 19 Nov 2018 16:14:10 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_HI, URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0 Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) by mira.cbaines.net (Postfix) with ESMTPS id DE75C167E8 for ; Mon, 19 Nov 2018 16:14:09 +0000 (GMT) Received: from localhost ([::1]:57487 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gOmBd-0006Aa-1O for patchwork@mira.cbaines.net; Mon, 19 Nov 2018 11:14:09 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:50382) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gOmBX-00068V-S8 for guix-patches@gnu.org; Mon, 19 Nov 2018 11:14:04 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gOmBW-0001zE-MZ for guix-patches@gnu.org; Mon, 19 Nov 2018 11:14:03 -0500 Received: from debbugs.gnu.org ([208.118.235.43]:56850) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gOmBW-0001yH-1F for guix-patches@gnu.org; Mon, 19 Nov 2018 11:14:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1gOmBV-0005VZ-Ok for guix-patches@gnu.org; Mon, 19 Nov 2018 11:14:01 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#33432] [PATCH 0/2] Download Git checkouts from Software Heritage as a last resort Resent-From: Ludovic =?utf-8?q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 19 Nov 2018 16:14:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 33432 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 33432@debbugs.gnu.org X-Debbugs-Original-To: guix-patches@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.154264403421147 (code B ref -1); Mon, 19 Nov 2018 16:14:01 +0000 Received: (at submit) by debbugs.gnu.org; 19 Nov 2018 16:13:54 +0000 Received: from localhost ([127.0.0.1]:32875 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gOmBO-0005V0-0x for submit@debbugs.gnu.org; Mon, 19 Nov 2018 11:13:54 -0500 Received: from eggs.gnu.org ([208.118.235.92]:40691) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gOmBM-0005Uk-It for submit@debbugs.gnu.org; Mon, 19 Nov 2018 11:13:52 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gOmBF-0001j4-HB for submit@debbugs.gnu.org; Mon, 19 Nov 2018 11:13:47 -0500 Received: from lists.gnu.org ([2001:4830:134:3::11]:59857) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gOmBF-0001ix-De for submit@debbugs.gnu.org; Mon, 19 Nov 2018 11:13:45 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:50296) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gOmBE-0005ym-Dv for guix-patches@gnu.org; Mon, 19 Nov 2018 11:13:45 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gOmBD-0001hG-IC for guix-patches@gnu.org; Mon, 19 Nov 2018 11:13:44 -0500 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:53847) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gOmB9-0001da-KI; Mon, 19 Nov 2018 11:13:39 -0500 Received: from [2001:660:6102:320:e120:2c8f:8909:cdfe] (port=60742 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1gOmB9-00065r-BM; Mon, 19 Nov 2018 11:13:39 -0500 From: Ludovic =?utf-8?q?Court=C3=A8s?= Date: Mon, 19 Nov 2018 17:13:25 +0100 Message-Id: <20181119161325.7801-1-ludo@gnu.org> X-Mailer: git-send-email 2.19.1 MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 208.118.235.43 X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches Hello Guix! This patch series adds the Software Heritage (SWH) client library initially discussed at: https://lists.gnu.org/archive/html/guix-devel/2018-11/msg00285.html Furthermore, it uses it in (guix git-download) to download code from SWH when it is unavailable upstream and on our servers. This bit relies on the “vault” API of SWH, which allows you to fetch a checkout as a tarball. Not all revisions are readily available as tarballs, understandably, so the vault API has a mechanism that allows you to request the “cooking” of a specific checkout. Cooking is asynchronous and can take some time. https://docs.softwareheritage.org/devel/swh-vault/api.html When downloading over SWH, the ‘swh-download’ procedure first resolves the tag (if it’s a tag), then tries to download the corresponding tarball from the vault. If the vault doesn’t have it yet, it sends a cooking request and waits for it to complete by periodically checking the cooking status. In the future, we should provide a “lister” and “loader” so that SWH can regularly obtain a list of Guix packages with their source URL and commit/tag: https://forge.softwareheritage.org/T1352 The SWH team is also considering pre-cooking all VCS tags such that every time we refer to a tag, we can be sure its contents are already available in the vault: https://forge.softwareheritage.org/T1350 Feedback welcome! Ludo’. Ludovic Courtès (2): Add (guix swh). git-download: Download from Software Heritage as a last resort. Makefile.am | 1 + guix/git-download.scm | 64 +++-- guix/swh.scm | 551 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 596 insertions(+), 20 deletions(-) create mode 100644 guix/swh.scm