From patchwork Thu Aug 29 23:16:53 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Ludovic_Court=C3=A8s?= X-Patchwork-Id: 15158 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 450461735D; Fri, 30 Aug 2019 00:18:15 +0100 (BST) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTP id F06D41735A for ; Fri, 30 Aug 2019 00:18:14 +0100 (BST) Received: from localhost ([::1]:55320 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1i3TgE-0000NM-FC for patchwork@mira.cbaines.net; Thu, 29 Aug 2019 19:18:14 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:56548) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1i3Tg3-0000Mz-D8 for guix-patches@gnu.org; Thu, 29 Aug 2019 19:18:04 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1i3Tg2-0004m4-5q for guix-patches@gnu.org; Thu, 29 Aug 2019 19:18:03 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:44198) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1i3Tg2-0004lq-2v for guix-patches@gnu.org; Thu, 29 Aug 2019 19:18:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1i3Tg1-0006Kl-TQ for guix-patches@gnu.org; Thu, 29 Aug 2019 19:18:01 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#37224] [PATCH 0/4] Add 'archival' checker for 'guix lint' Resent-From: Ludovic =?utf-8?q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 29 Aug 2019 23:18:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 37224 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 37224@debbugs.gnu.org X-Debbugs-Original-To: guix-patches@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.156712062924231 (code B ref -1); Thu, 29 Aug 2019 23:18:01 +0000 Received: (at submit) by debbugs.gnu.org; 29 Aug 2019 23:17:09 +0000 Received: from localhost ([127.0.0.1]:53019 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1i3Tf9-0006Ih-UQ for submit@debbugs.gnu.org; Thu, 29 Aug 2019 19:17:08 -0400 Received: from lists.gnu.org ([209.51.188.17]:37341) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1i3Tf7-0006IX-CN for submit@debbugs.gnu.org; Thu, 29 Aug 2019 19:17:06 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:54804) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1i3Tf6-0000BI-2O for guix-patches@gnu.org; Thu, 29 Aug 2019 19:17:05 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:48138) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1i3Tf5-00023y-HG; Thu, 29 Aug 2019 19:17:03 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=36110 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1i3Tf4-0005Os-Sc; Thu, 29 Aug 2019 19:17:03 -0400 From: Ludovic =?utf-8?q?Court=C3=A8s?= Date: Fri, 30 Aug 2019 01:16:53 +0200 Message-Id: <20190829231653.7607-1-ludo@gnu.org> X-Mailer: git-send-email 2.23.0 MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.51.188.43 X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches Hello Guix! This patch series adds an ‘archival’ checker for ‘guix lint’, documented like this: Checks whether the package’s source code is archived at Software Heritage (https://www.softwareheritage.org). When the source code that is not archived comes from a version-control system (VCS)—e.g., it’s obtained with ‘git-fetch’, send Software Heritage a “save” request so that it eventually archives it. This ensures that the source will remain available in the long term, and that Guix can fall back to Software Heritage should the source code disappear from its original host. The status of recent “save” requests can be viewed on-line (https://archive.softwareheritage.org/save/#requests). When source code is a tarball obtained with ‘url-fetch’, simply print a message when it is not archived. As of this writing Software Heritage does not allow requests to save arbitrary tarballs; we are working on ways to ensure that non-VCS source code is also archived. Software Heritage limits the request rate per IP address (https://archive.softwareheritage.org/api/#rate-limiting). When the limit is reached, ‘guix lint’ prints a message and the ‘archival’ checker stops doing anything until that limit has been reset. Currently, only 25% of our packages are not fetched with ‘url-fetch’. For the remaining 75%, this checker can only report whether the tarball is missing (and apart from ftp.gnu.org and a few other exceptions, it usually _is_ missing) and cannot actually save it. Anyway, it’s a first step in that direction. Feedback welcome! The second step will be to write a “lister” for Software Heritage that grabs the list of source code URLs from . That could would run at SWH and it could potentially grab the tarballs, not just the VCS checkouts. Here’s are examples: https://forge.softwareheritage.org/source/swh-lister/browse/master/swh/lister/packagist/lister.py https://forge.softwareheritage.org/source/swh-lister/browse/master/swh/lister/gnu/lister.py It should be quite easy for a Pythonista to write something similar for our ‘packages.json’. Any takers? :-) Ludo’. Ludovic Courtès (4): tests: 'with-http-server' accepts multiple responses. swh: Add hooks for rate limiting handling. swh: Make 'commit-id?' public. lint: Add 'archival' checker. doc/guix.texi | 25 ++++++ guix/lint.scm | 96 +++++++++++++++++++++- guix/swh.scm | 88 ++++++++++++++++----- guix/tests/http.scm | 39 +++++---- tests/derivations.scm | 12 +-- tests/lint.scm | 179 ++++++++++++++++++++++++++++++++---------- tests/swh.scm | 41 +++++++++- 7 files changed, 395 insertions(+), 85 deletions(-)