From patchwork Fri May 31 14:48:28 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Ludovic_Court=C3=A8s?= X-Patchwork-Id: 14176 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id ADE2C17057; Fri, 31 May 2019 15:49:14 +0100 (BST) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTP id C276F16FED for ; Fri, 31 May 2019 15:49:12 +0100 (BST) Received: from localhost ([127.0.0.1]:44753 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hWiqD-0000gs-Lh for patchwork@mira.cbaines.net; Fri, 31 May 2019 10:49:09 -0400 Received: from eggs.gnu.org ([209.51.188.92]:49527) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hWiq9-0000ga-6o for guix-patches@gnu.org; Fri, 31 May 2019 10:49:06 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hWiq6-0006Bh-Kl for guix-patches@gnu.org; Fri, 31 May 2019 10:49:04 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:50614) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1hWiq6-0006BY-Hm for guix-patches@gnu.org; Fri, 31 May 2019 10:49:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1hWiq6-0003O1-DW for guix-patches@gnu.org; Fri, 31 May 2019 10:49:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#36029] [PATCH 0/2] 'publish' and 'substitute' support several compression methods Resent-From: Ludovic =?utf-8?q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 31 May 2019 14:49:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 36029 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 36029@debbugs.gnu.org X-Debbugs-Original-To: guix-patches@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.155931413012981 (code B ref -1); Fri, 31 May 2019 14:49:02 +0000 Received: (at submit) by debbugs.gnu.org; 31 May 2019 14:48:50 +0000 Received: from localhost ([127.0.0.1]:35925 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hWipu-0003NJ-6a for submit@debbugs.gnu.org; Fri, 31 May 2019 10:48:50 -0400 Received: from eggs.gnu.org ([209.51.188.92]:35082) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hWipr-0003Mp-LH for submit@debbugs.gnu.org; Fri, 31 May 2019 10:48:49 -0400 Received: from lists.gnu.org ([209.51.188.17]:48936) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hWipm-0005tV-Ek for submit@debbugs.gnu.org; Fri, 31 May 2019 10:48:42 -0400 Received: from eggs.gnu.org ([209.51.188.92]:49454) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hWipl-0000gP-5l for guix-patches@gnu.org; Fri, 31 May 2019 10:48:42 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:46531) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hWipj-0005pt-2N; Fri, 31 May 2019 10:48:39 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=53130 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1hWipg-0004Wo-ME; Fri, 31 May 2019 10:48:38 -0400 From: Ludovic =?utf-8?q?Court=C3=A8s?= Date: Fri, 31 May 2019 16:48:28 +0200 Message-Id: <20190531144828.9585-1-ludo@gnu.org> X-Mailer: git-send-email 2.21.0 MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.51.188.43 X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Pierre Neidhardt Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches Hello Guix! This is a followup to . One idea we discussed there was to allow clients to pass an ‘X-Guix-Accepted-Encoding’ header in HTTP requests, and the server would return an lzip narinfo or a gzip narinfo depending on that. However, I thought that this was not very flexible, and that we were bound to mess up with caching. This patch implements a different solution: ‘guix publish’ can be passed multiple ‘-C’ options, in which case it compresses substitutes with all these compression methods. The corresponding narinfo looks like this: --8<---------------cut here---------------start------------->8--- StorePath: /gnu/store/9czlz7ss3187l2vi1hvrlkwlgrggdg5p-inkscape-0.92.4 URL: nar/gzip/9czlz7ss3187l2vi1hvrlkwlgrggdg5p-inkscape-0.92.4 Compression: gzip FileSize: 40308611 URL: nar/lzip/9czlz7ss3187l2vi1hvrlkwlgrggdg5p-inkscape-0.92.4 Compression: lzip FileSize: 19867767 NarHash: sha256:1jv4nkq68a7zwqhi9inrnh340a4jxcpb91wq7d25hgw0nk8isbbk NarSize: 136499024 References: … --8<---------------cut here---------------end--------------->8--- IOW, it’s like before, except that there are multiple URL/Compression/FileSize fields instead of just one of each. The trick is that old clients take the first occurrence of each of these fields and ignore subsequent occurrences. In the example above, they’d just take gzip and ignore the rest. The new ‘guix substitute’ (second patch) “sees” all these fields and is able to choose the most appropriate compression method (i.e., the best one among those it supports.) This adds a bit of complexity that is useless beyond the transitioning period from gzip to lzip, but I think that’s OK; plus there might be an lzip to super-lzip transition in the future, who knows. Thoughts? When we deploy that, we’ll obviously more use storage and more CPU on the build farm, but that seems unavoidable. OTOH, we’ll progressively end up sending less data over the wire (and paying less for the CDN!), given that lzip compresses better. Ludo’. Ludovic Courtès (2): publish: '--compression' can be repeated. substitute: Select the best compression methods. doc/guix.texi | 5 + guix/scripts/challenge.scm | 4 +- guix/scripts/publish.scm | 204 ++++++++++++++++++++++-------------- guix/scripts/substitute.scm | 141 ++++++++++++++++++------- guix/scripts/weather.scm | 5 +- tests/publish.scm | 89 ++++++++++++++-- tests/substitute.scm | 51 ++++++++- 7 files changed, 370 insertions(+), 129 deletions(-)