From patchwork Fri May 24 13:42:32 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Ludovic_Court=C3=A8s?= X-Patchwork-Id: 14059 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 21CA417040; Fri, 24 May 2019 14:43:08 +0100 (BST) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTP id 950EB1700D for ; Fri, 24 May 2019 14:43:07 +0100 (BST) Received: from localhost ([127.0.0.1]:54877 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hUATT-0008Pd-6s for patchwork@mira.cbaines.net; Fri, 24 May 2019 09:43:07 -0400 Received: from eggs.gnu.org ([209.51.188.92]:36880) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hUATQ-0008Oj-08 for guix-patches@gnu.org; Fri, 24 May 2019 09:43:05 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hUATO-0008Mi-OX for guix-patches@gnu.org; Fri, 24 May 2019 09:43:03 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:60990) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1hUATO-0008Mc-LV for guix-patches@gnu.org; Fri, 24 May 2019 09:43:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1hUATO-0002PR-CS for guix-patches@gnu.org; Fri, 24 May 2019 09:43:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#35880] [PATCH 1/7] lzlib: Add 'make-lzip-input-port/compressed'. References: <20190524133159.22568-1-ludo@gnu.org> In-Reply-To: <20190524133159.22568-1-ludo@gnu.org> Resent-From: Ludovic =?utf-8?q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 24 May 2019 13:43:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 35880 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 35880@debbugs.gnu.org Received: via spool by 35880-submit@debbugs.gnu.org id=B35880.15587053769220 (code B ref 35880); Fri, 24 May 2019 13:43:02 +0000 Received: (at 35880) by debbugs.gnu.org; 24 May 2019 13:42:56 +0000 Received: from localhost ([127.0.0.1]:46295 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hUATH-0002OT-Ou for submit@debbugs.gnu.org; Fri, 24 May 2019 09:42:56 -0400 Received: from eggs.gnu.org ([209.51.188.92]:50652) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hUATE-0002Nn-JF for 35880@debbugs.gnu.org; Fri, 24 May 2019 09:42:54 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:38746) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hUAT8-0008C6-Oc; Fri, 24 May 2019 09:42:46 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=36484 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1hUAT8-0007iN-AZ; Fri, 24 May 2019 09:42:46 -0400 From: Ludovic =?utf-8?q?Court=C3=A8s?= Date: Fri, 24 May 2019 15:42:32 +0200 Message-Id: <20190524134238.22802-1-ludo@gnu.org> X-Mailer: git-send-email 2.21.0 MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.51.188.43 X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Pierre Neidhardt Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches * guix/lzlib.scm (lzwrite!, make-lzip-input-port/compressed): New procedures. * tests/lzlib.scm ("make-lzip-input-port/compressed"): New test. * guix/tests.scm (%seed): Export. --- guix/lzlib.scm | 62 +++++++++++++++++++++++++++++++++++++++++++++++++ guix/tests.scm | 1 + tests/lzlib.scm | 10 ++++++++ 3 files changed, 73 insertions(+) diff --git a/guix/lzlib.scm b/guix/lzlib.scm index a6dac46049..48927c6262 100644 --- a/guix/lzlib.scm +++ b/guix/lzlib.scm @@ -1,5 +1,6 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2019 Pierre Neidhardt +;;; Copyright © 2019 Ludovic Courtès ;;; ;;; This file is part of GNU Guix. ;;; @@ -23,9 +24,11 @@ #:use-module (ice-9 match) #:use-module (system foreign) #:use-module (guix config) + #:use-module (srfi srfi-11) #:export (lzlib-available? make-lzip-input-port make-lzip-output-port + make-lzip-input-port/compressed call-with-lzip-input-port call-with-lzip-output-port %default-member-length-limit @@ -515,6 +518,23 @@ the end-of-stream has been reached." (loop rd))) read)) +(define (lzwrite! encoder source source-offset source-count + target target-offset target-count) + "Write up to SOURCE-COUNT bytes from SOURCE to ENCODER, and read up to +TARGET-COUNT bytes into TARGET at TARGET-OFFSET. Return two values: the +number of bytes read from SOURCE, and the number of bytes written to TARGET." + (define read + (if (< 0 (lz-compress-write-size encoder)) + (match (lz-compress-write encoder source source-offset source-count) + (0 (lz-compress-finish encoder) 0) + (n n)) + 0)) + + (let loop () + (match (lz-compress-read encoder target target-offset target-count) + (0 (loop)) + (written (values read written))))) + (define* (lzwrite encoder bv lz-port #:optional (start 0) (count (bytevector-length bv))) "Write up to COUNT bytes from BV at offset START into LZ-PORT. Return @@ -597,6 +617,48 @@ port is closed." (lz-compress-close encoder) (close-port port)))) +(define* (make-lzip-input-port/compressed port + #:key + (level %default-compression-level)) + "Return an input port that compresses data read from PORT, with the given LEVEL. +PORT is automatically closed when the resulting port is closed." + (define encoder (apply lz-compress-open + (car (assoc-ref %compression-levels level)))) + + (define input-buffer (make-bytevector 8192)) + (define input-len 0) + (define input-offset 0) + + (define input-eof? #f) + + (define (read! bv start count) + (cond + (input-eof? + (lz-compress-read encoder bv start count)) + ((= input-offset input-len) + (match (get-bytevector-n! port input-buffer 0 + (bytevector-length input-buffer)) + ((? eof-object?) + (set! input-eof? #t) + (lz-compress-finish encoder)) + (count + (set! input-offset 0) + (set! input-len count))) + (read! bv start count)) + (else + (let-values (((read written) + (lzwrite! encoder + input-buffer input-offset + (- input-len input-offset) + bv start count))) + (set! input-offset (+ input-offset read)) + written)))) + + (make-custom-binary-input-port "lzip-input/compressed" + read! #f #f + (lambda () + (close-port port)))) + (define* (call-with-lzip-input-port port proc) "Call PROC with a port that wraps PORT and decompresses data read from it. PORT is closed upon completion." diff --git a/guix/tests.scm b/guix/tests.scm index 35ebf8464d..66d60e964e 100644 --- a/guix/tests.scm +++ b/guix/tests.scm @@ -33,6 +33,7 @@ #:use-module (web uri) #:export (open-connection-for-tests with-external-store + %seed random-text random-bytevector file=? diff --git a/tests/lzlib.scm b/tests/lzlib.scm index cf53a9417d..543622bb45 100644 --- a/tests/lzlib.scm +++ b/tests/lzlib.scm @@ -108,4 +108,14 @@ (test-assert* "Bytevector of size relative to Lzip internal buffers (1MiB+1)" (compress-and-decompress (random-bytevector (1+ (* 1024 1024))))) +(test-assert "make-lzip-input-port/compressed" + (let* ((len (pk 'len (+ 10 (random 4000 %seed)))) + (data (random-bytevector len)) + (compressed (make-lzip-input-port/compressed + (open-bytevector-input-port data))) + (result (call-with-lzip-input-port compressed + get-bytevector-all))) + (pk (bytevector-length result) (bytevector-length data)) + (bytevector=? result data))) + (test-end) From patchwork Fri May 24 13:42:33 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?q?Ludovic_Court=C3=A8s?= X-Patchwork-Id: 14058 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 1FDD917040; Fri, 24 May 2019 14:43:07 +0100 (BST) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00 autolearn=ham autolearn_force=no version=3.4.0 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTP id B4A381700D for ; Fri, 24 May 2019 14:43:06 +0100 (BST) Received: from localhost ([127.0.0.1]:54873 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hUATS-0008Oz-BI for patchwork@mira.cbaines.net; Fri, 24 May 2019 09:43:06 -0400 Received: from eggs.gnu.org ([209.51.188.92]:36869) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hUATP-0008Oh-6x for guix-patches@gnu.org; Fri, 24 May 2019 09:43:04 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hUATO-0008MG-3A for guix-patches@gnu.org; Fri, 24 May 2019 09:43:03 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:60989) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1hUATN-0008MB-W3 for guix-patches@gnu.org; Fri, 24 May 2019 09:43:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1hUATN-0002PJ-T0 for guix-patches@gnu.org; Fri, 24 May 2019 09:43:01 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#35880] [PATCH 2/7] utils: Test 'compressed-port' and 'decompressed-port' for both gzip and xz. Resent-From: Ludovic =?utf-8?q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 24 May 2019 13:43:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 35880 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 35880@debbugs.gnu.org Received: via spool by 35880-submit@debbugs.gnu.org id=B35880.15587053759208 (code B ref 35880); Fri, 24 May 2019 13:43:01 +0000 Received: (at 35880) by debbugs.gnu.org; 24 May 2019 13:42:55 +0000 Received: from localhost ([127.0.0.1]:46293 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hUATH-0002OQ-EB for submit@debbugs.gnu.org; Fri, 24 May 2019 09:42:55 -0400 Received: from eggs.gnu.org ([209.51.188.92]:50660) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hUATE-0002Nr-Sz for 35880@debbugs.gnu.org; Fri, 24 May 2019 09:42:54 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:38747) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hUAT9-0008Cl-Nn; Fri, 24 May 2019 09:42:47 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=36484 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1hUAT9-0007iN-8Z; Fri, 24 May 2019 09:42:47 -0400 From: Ludovic =?utf-8?q?Court=C3=A8s?= Date: Fri, 24 May 2019 15:42:33 +0200 Message-Id: <20190524134238.22802-2-ludo@gnu.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190524134238.22802-1-ludo@gnu.org> References: <20190524134238.22802-1-ludo@gnu.org> MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.51.188.43 X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Pierre Neidhardt Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches * tests/utils.scm (test-compression/decompression): New procedure. : Call it for both 'xz and 'gzip. --- tests/utils.scm | 61 +++++++++++++++++++++++++++++++------------------ 1 file changed, 39 insertions(+), 22 deletions(-) diff --git a/tests/utils.scm b/tests/utils.scm index 3015b21b23..7d55107fda 100644 --- a/tests/utils.scm +++ b/tests/utils.scm @@ -174,30 +174,47 @@ (any (compose (negate zero?) cdr waitpid) pids)))) -(test-assert "compressed-port, decompressed-port, non-file" - (let ((data (call-with-input-file (search-path %load-path "guix.scm") - get-bytevector-all))) - (let*-values (((compressed pids1) - (compressed-port 'xz (open-bytevector-input-port data))) - ((decompressed pids2) - (decompressed-port 'xz compressed))) - (and (every (compose zero? cdr waitpid) - (append pids1 pids2)) - (equal? (get-bytevector-all decompressed) data))))) +(define (test-compression/decompression method run?) + "Test METHOD, a symbol such as 'gzip. Call RUN? to determine whether to +skip these tests." + (unless (run?) (test-skip 1)) + (test-assert (format #f "compressed-port, decompressed-port, non-file [~a]" + method) + (let ((data (call-with-input-file (search-path %load-path "guix.scm") + get-bytevector-all))) + (let*-values (((compressed pids1) + (compressed-port method (open-bytevector-input-port data))) + ((decompressed pids2) + (decompressed-port method compressed))) + (and (every (compose zero? cdr waitpid) + (pk 'pids method (append pids1 pids2))) + (let ((result (get-bytevector-all decompressed))) + (pk 'len method + (if (bytevector? result) + (bytevector-length result) + result) + (bytevector-length data)) + (equal? result data)))))) -(false-if-exception (delete-file temp-file)) -(test-assert "compressed-output-port + decompressed-port" - (let* ((file (search-path %load-path "guix/derivations.scm")) - (data (call-with-input-file file get-bytevector-all)) - (port (open-file temp-file "w0b"))) - (call-with-compressed-output-port 'xz port - (lambda (compressed) - (put-bytevector compressed data))) - (close-port port) + (false-if-exception (delete-file temp-file)) + (unless (run?) (test-skip 1)) + (test-assert (format #f "compressed-output-port + decompressed-port [~a]" + method) + (let* ((file (search-path %load-path "guix/derivations.scm")) + (data (call-with-input-file file get-bytevector-all)) + (port (open-file temp-file "w0b"))) + (call-with-compressed-output-port method port + (lambda (compressed) + (put-bytevector compressed data))) + (close-port port) - (bytevector=? data - (call-with-decompressed-port 'xz (open-file temp-file "r0b") - get-bytevector-all)))) + (bytevector=? data + (call-with-decompressed-port method (open-file temp-file "r0b") + get-bytevector-all))))) + +(for-each test-compression/decompression + '(gzip xz) + (list (const #t) (const #f))) ;; This is actually in (guix store). (test-equal "store-path-package-name" From patchwork Fri May 24 13:42:34 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Ludovic_Court=C3=A8s?= X-Patchwork-Id: 14060 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 4E21517040; Fri, 24 May 2019 14:43:15 +0100 (BST) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTP id 2E5681700D for ; Fri, 24 May 2019 14:43:14 +0100 (BST) Received: from localhost ([127.0.0.1]:54881 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hUATZ-0008Up-RZ for patchwork@mira.cbaines.net; Fri, 24 May 2019 09:43:13 -0400 Received: from eggs.gnu.org ([209.51.188.92]:36887) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hUATQ-0008Oo-Iy for guix-patches@gnu.org; Fri, 24 May 2019 09:43:05 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hUATP-0008NJ-Dm for guix-patches@gnu.org; Fri, 24 May 2019 09:43:04 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:60993) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1hUATP-0008N7-Af for guix-patches@gnu.org; Fri, 24 May 2019 09:43:03 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1hUATP-0002Pb-7k for guix-patches@gnu.org; Fri, 24 May 2019 09:43:03 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#35880] [PATCH 3/7] utils: Support compression and decompression with lzip. Resent-From: Ludovic =?utf-8?q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 24 May 2019 13:43:03 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 35880 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 35880@debbugs.gnu.org Received: via spool by 35880-submit@debbugs.gnu.org id=B35880.15587053769233 (code B ref 35880); Fri, 24 May 2019 13:43:03 +0000 Received: (at 35880) by debbugs.gnu.org; 24 May 2019 13:42:56 +0000 Received: from localhost ([127.0.0.1]:46298 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hUATI-0002Of-89 for submit@debbugs.gnu.org; Fri, 24 May 2019 09:42:56 -0400 Received: from eggs.gnu.org ([209.51.188.92]:50666) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hUATG-0002Nv-4F for 35880@debbugs.gnu.org; Fri, 24 May 2019 09:42:55 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:38748) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hUATA-0008Db-V2; Fri, 24 May 2019 09:42:49 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=36484 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1hUATA-0007iN-7s; Fri, 24 May 2019 09:42:48 -0400 From: Ludovic =?utf-8?q?Court=C3=A8s?= Date: Fri, 24 May 2019 15:42:34 +0200 Message-Id: <20190524134238.22802-3-ludo@gnu.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190524134238.22802-1-ludo@gnu.org> References: <20190524134238.22802-1-ludo@gnu.org> MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.51.188.43 X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Pierre Neidhardt Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches * guix/utils.scm (lzip-port): New procedure. (decompressed-port, compressed-port, compressed-output-port): Add 'lzip case. * tests/utils.scm : Call 'test-compression/decompression' for 'lzip as well. --- guix/utils.scm | 27 ++++++++++++++++++++++----- tests/utils.scm | 5 +++-- 2 files changed, 25 insertions(+), 7 deletions(-) diff --git a/guix/utils.scm b/guix/utils.scm index ed1a418cca..709cdf9353 100644 --- a/guix/utils.scm +++ b/guix/utils.scm @@ -1,5 +1,5 @@ ;;; GNU Guix --- Functional package management for GNU -;;; Copyright © 2012, 2013, 2014, 2015, 2016, 2017, 2018 Ludovic Courtès +;;; Copyright © 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019 Ludovic Courtès ;;; Copyright © 2013, 2014, 2015 Mark H Weaver ;;; Copyright © 2014 Eric Bavier ;;; Copyright © 2014 Ian Denhardt @@ -169,6 +169,17 @@ buffered data is lost." (close-port out) (loop in (cons child pids))))))))) +(define (lzip-port proc port . args) + "Return the lzip port produced by calling PROC (a symbol) on PORT and ARGS. +Raise an error if lzlib support is missing." + (let* ((lzlib (false-if-exception (resolve-interface '(guix lzlib)))) + (supported? (and lzlib + ((module-ref lzlib 'lzlib-available?))))) + (if supported? + (let ((make-port (module-ref lzlib proc))) + (values (make-port port) '())) + (error "lzip compression not supported" lzlib)))) + (define (decompressed-port compression input) "Return an input port where INPUT is decompressed according to COMPRESSION, a symbol such as 'xz." @@ -177,17 +188,21 @@ a symbol such as 'xz." ('bzip2 (filtered-port `(,%bzip2 "-dc") input)) ('xz (filtered-port `(,%xz "-dc") input)) ('gzip (filtered-port `(,%gzip "-dc") input)) - (else (error "unsupported compression scheme" compression)))) + ('lzip (values (lzip-port 'make-lzip-input-port input) + '())) + (_ (error "unsupported compression scheme" compression)))) (define (compressed-port compression input) - "Return an input port where INPUT is decompressed according to COMPRESSION, + "Return an input port where INPUT is compressed according to COMPRESSION, a symbol such as 'xz." (match compression ((or #f 'none) (values input '())) ('bzip2 (filtered-port `(,%bzip2 "-c") input)) ('xz (filtered-port `(,%xz "-c") input)) ('gzip (filtered-port `(,%gzip "-c") input)) - (else (error "unsupported compression scheme" compression)))) + ('lzip (values (lzip-port 'make-lzip-input-port/compressed input) + '())) + (_ (error "unsupported compression scheme" compression)))) (define (call-with-decompressed-port compression port proc) "Call PROC with a wrapper around PORT, a file port, that decompresses data @@ -244,7 +259,9 @@ program--e.g., '(\"--fast\")." ('bzip2 (filtered-output-port `(,%bzip2 "-c" ,@options) output)) ('xz (filtered-output-port `(,%xz "-c" ,@options) output)) ('gzip (filtered-output-port `(,%gzip "-c" ,@options) output)) - (else (error "unsupported compression scheme" compression)))) + ('lzip (values (lzip-port 'make-lzip-output-port output) + '())) + (_ (error "unsupported compression scheme" compression)))) (define* (call-with-compressed-output-port compression port proc #:key (options '())) diff --git a/tests/utils.scm b/tests/utils.scm index 7d55107fda..7c8f7c09d0 100644 --- a/tests/utils.scm +++ b/tests/utils.scm @@ -23,6 +23,7 @@ #:use-module (guix utils) #:use-module ((guix store) #:select (%store-prefix store-path-package-name)) #:use-module ((guix search-paths) #:select (string-tokenize*)) + #:use-module ((guix lzlib) #:select (lzlib-available?)) #:use-module (srfi srfi-1) #:use-module (srfi srfi-11) #:use-module (srfi srfi-64) @@ -213,8 +214,8 @@ skip these tests." get-bytevector-all))))) (for-each test-compression/decompression - '(gzip xz) - (list (const #t) (const #f))) + '(gzip xz lzip) + (list (const #t) (const #f) lzlib-available?)) ;; This is actually in (guix store). (test-equal "store-path-package-name" From patchwork Fri May 24 13:42:35 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Ludovic_Court=C3=A8s?= X-Patchwork-Id: 14064 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id A7AC717041; Fri, 24 May 2019 14:44:18 +0100 (BST) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-0.2 required=5.0 tests=BAYES_00,URIBL_BLOCKED, URIBL_SBL,URIBL_SBL_A autolearn=no autolearn_force=no version=3.4.0 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTP id 340291700D for ; Fri, 24 May 2019 14:44:15 +0100 (BST) Received: from localhost ([127.0.0.1]:54904 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hUAUY-0000G2-RD for patchwork@mira.cbaines.net; Fri, 24 May 2019 09:44:14 -0400 Received: from eggs.gnu.org ([209.51.188.92]:37116) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hUAUO-000097-4n for guix-patches@gnu.org; Fri, 24 May 2019 09:44:06 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hUAUM-0000X3-9D for guix-patches@gnu.org; Fri, 24 May 2019 09:44:04 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:61000) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1hUAUM-0000Wv-6G for guix-patches@gnu.org; Fri, 24 May 2019 09:44:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1hUAUM-0002Rn-1k for guix-patches@gnu.org; Fri, 24 May 2019 09:44:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#35880] [PATCH 4/7] publish: Add support for lzip. Resent-From: Ludovic =?utf-8?q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 24 May 2019 13:44:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 35880 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 35880@debbugs.gnu.org Received: via spool by 35880-submit@debbugs.gnu.org id=B35880.15587053849297 (code B ref 35880); Fri, 24 May 2019 13:44:02 +0000 Received: (at 35880) by debbugs.gnu.org; 24 May 2019 13:43:04 +0000 Received: from localhost ([127.0.0.1]:46304 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hUATO-0002PZ-U8 for submit@debbugs.gnu.org; Fri, 24 May 2019 09:43:03 -0400 Received: from eggs.gnu.org ([209.51.188.92]:50670) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hUATH-0002Nz-64 for 35880@debbugs.gnu.org; Fri, 24 May 2019 09:42:55 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:38750) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hUATC-0008EC-1C; Fri, 24 May 2019 09:42:50 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=36484 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1hUATB-0007iN-Ea; Fri, 24 May 2019 09:42:49 -0400 From: Ludovic =?utf-8?q?Court=C3=A8s?= Date: Fri, 24 May 2019 15:42:35 +0200 Message-Id: <20190524134238.22802-4-ludo@gnu.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190524134238.22802-1-ludo@gnu.org> References: <20190524134238.22802-1-ludo@gnu.org> MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.51.188.43 X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Pierre Neidhardt Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches * guix/scripts/publish.scm (show-help, %options): Support '-C METHOD' and '-C METHOD:LEVEL'. (default-compression): New procedure. (bake-narinfo+nar): Add lzip. (nar-response-port): Likewise. (string->compression-type): New procedure. (make-request-handler): Generalize /nar/gzip handler to handle /nar/lzip as well. * tests/publish.scm ("/nar/lzip/*"): New test. ("/*.narinfo with lzip compression"): New test. * doc/guix.texi (Invoking guix publish): Document it. (Requirements): Mention lzlib. --- .dir-locals.el | 2 + doc/guix.texi | 25 +++++++++--- guix/scripts/publish.scm | 84 +++++++++++++++++++++++++++++----------- tests/publish.scm | 36 +++++++++++++++++ 4 files changed, 119 insertions(+), 28 deletions(-) diff --git a/.dir-locals.el b/.dir-locals.el index 550e06ef09..f1196fd781 100644 --- a/.dir-locals.el +++ b/.dir-locals.el @@ -53,6 +53,8 @@ (eval . (put 'call-with-decompressed-port 'scheme-indent-function 2)) (eval . (put 'call-with-gzip-input-port 'scheme-indent-function 1)) (eval . (put 'call-with-gzip-output-port 'scheme-indent-function 1)) + (eval . (put 'call-with-lzip-input-port 'scheme-indent-function 1)) + (eval . (put 'call-with-lzip-output-port 'scheme-indent-function 1)) (eval . (put 'signature-case 'scheme-indent-function 1)) (eval . (put 'emacs-batch-eval 'scheme-indent-function 0)) (eval . (put 'emacs-batch-edit-file 'scheme-indent-function 1)) diff --git a/doc/guix.texi b/doc/guix.texi index f176bb0163..b0de5632e7 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -757,6 +757,11 @@ Support for build offloading (@pxref{Daemon Offload Setup}) and @uref{https://github.com/artyom-poptsov/guile-ssh, Guile-SSH}, version 0.10.2 or later. +@item +When @url{https://www.nongnu.org/lzip/lzlib.html, lzlib} is available, lzlib +substitutes can be used and @command{guix publish} can compress substitutes +with lzlib. + @item When @url{http://www.bzip.org, libbz2} is available, @command{guix-daemon} can use it to compress build logs. @@ -9656,12 +9661,20 @@ accept connections from any interface. Change privileges to @var{user} as soon as possible---i.e., once the server socket is open and the signing key has been read. -@item --compression[=@var{level}] -@itemx -C [@var{level}] -Compress data using the given @var{level}. When @var{level} is zero, -disable compression. The range 1 to 9 corresponds to different gzip -compression levels: 1 is the fastest, and 9 is the best (CPU-intensive). -The default is 3. +@item --compression[=@var{method}[:@var{level}]] +@itemx -C [@var{method}[:@var{level}]] +Compress data using the given @var{method} and @var{level}. @var{method} is +one of @code{lzip} and @code{gzip}; when @var{method} is omitted, @code{gzip} +is used. + +When @var{level} is zero, disable compression. The range 1 to 9 corresponds +to different compression levels: 1 is the fastest, and 9 is the best +(CPU-intensive). The default is 3. + +Usually, @code{lzip} compresses noticeably better than @code{gzip} for a small +increase in CPU usage; see +@uref{https://nongnu.org/lzip/lzip_benchmark.html,benchmarks on the lzip Web +page}. Unless @option{--cache} is used, compression occurs on the fly and the compressed streams are not diff --git a/guix/scripts/publish.scm b/guix/scripts/publish.scm index a236f3e45c..9e74d789ce 100644 --- a/guix/scripts/publish.scm +++ b/guix/scripts/publish.scm @@ -1,6 +1,6 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2015 David Thompson -;;; Copyright © 2015, 2016, 2017, 2018 Ludovic Courtès +;;; Copyright © 2015, 2016, 2017, 2018, 2019 Ludovic Courtès ;;; ;;; This file is part of GNU Guix. ;;; @@ -51,6 +51,7 @@ #:use-module (guix store) #:use-module ((guix serialization) #:select (write-file)) #:use-module (guix zlib) + #:autoload (guix lzlib) (lzlib-available?) #:use-module (guix cache) #:use-module (guix ui) #:use-module (guix scripts) @@ -74,8 +75,8 @@ Publish ~a over HTTP.\n") %store-directory) (display (G_ " -u, --user=USER change privileges to USER as soon as possible")) (display (G_ " - -C, --compression[=LEVEL] - compress archives at LEVEL")) + -C, --compression[=METHOD:LEVEL] + compress archives with METHOD at LEVEL")) (display (G_ " -c, --cache=DIRECTORY cache published items to DIRECTORY")) (display (G_ " @@ -121,6 +122,9 @@ Publish ~a over HTTP.\n") %store-directory) ;; Since we compress on the fly, default to fast compression. (compression 'gzip 3)) +(define (default-compression type) + (compression type 3)) + (define (actual-compression item requested) "Return the actual compression used for ITEM, which may be %NO-COMPRESSION if ITEM is already compressed." @@ -153,18 +157,28 @@ if ITEM is already compressed." name))))) (option '(#\C "compression") #f #t (lambda (opt name arg result) - (match (if arg (string->number* arg) 3) - (0 - (alist-cons 'compression %no-compression result)) - (level - (if (zlib-available?) - (alist-cons 'compression - (compression 'gzip level) - result) - (begin - (warning (G_ "zlib support is missing; \ -compression disabled~%")) - result)))))) + (let* ((colon (string-index arg #\:)) + (type (cond + (colon (string-take arg colon)) + ((string->number arg) "gzip") + (else arg))) + (level (if colon + (string->number* + (string-drop arg (+ 1 colon))) + (or (string->number arg) 3)))) + (match level + (0 + (alist-cons 'compression %no-compression result)) + (level + (match (string->compression-type type) + ((? symbol? type) + (alist-cons 'compression + (compression type level) + result)) + (_ + (warning (G_ "~a: unsupported compression type~%") + type) + result))))))) (option '(#\c "cache") #t #f (lambda (opt name arg result) (alist-cons 'cache arg result))) @@ -483,6 +497,13 @@ requested using POOL." #:level (compression-level compression) #:buffer-size (* 128 1024)) (rename-file (string-append nar ".tmp") nar)) + ('lzip + ;; Note: the file port gets closed along with the lzip port. + (call-with-lzip-output-port (open-output-file (string-append nar ".tmp")) + (lambda (port) + (write-file item port)) + #:level (compression-level compression)) + (rename-file (string-append nar ".tmp") nar)) ('none ;; Cache nars even when compression is disabled so that we can ;; guarantee the TTL (see .) @@ -687,6 +708,9 @@ example: \"/foo/bar\" yields '(\"foo\" \"bar\")." (make-gzip-output-port (response-port response) #:level level #:buffer-size (* 64 1024))) + (($ 'lzip level) + (make-lzip-output-port (response-port response) + #:level level)) (($ 'none) (response-port response)) (#f @@ -761,12 +785,23 @@ blocking." http-write (@@ (web server http) http-close)) +(define (string->compression-type string) + "Return a symbol denoting the compression method expressed by STRING; return +#f if STRING doesn't match any supported method." + (match string + ("gzip" (and (zlib-available?) 'gzip)) + ("lzip" (and (lzlib-available?) 'lzip)) + (_ #f))) + (define* (make-request-handler store #:key cache pool narinfo-ttl (nar-path "nar") (compression %no-compression)) + (define compression-type? + string->compression-type) + (define nar-path? (let ((expected (split-and-decode-uri-path nar-path))) (cut equal? expected <>))) @@ -815,13 +850,18 @@ blocking." ;; is restarted with different compression parameters. ;; /nar/gzip/ - ((components ... "gzip" store-item) - (if (and (nar-path? components) (zlib-available?)) - (let ((compression (match compression - (($ 'gzip) - compression) - (_ - %default-gzip-compression)))) + ((components ... (? compression-type? type) store-item) + (if (nar-path? components) + (let* ((compression-type (string->compression-type type)) + (compression (match compression + (($ type) + (if (eq? type compression-type) + compression + (default-compression + compression-type))) + (_ + (default-compression + compression-type))))) (if cache (render-nar/cached store cache request store-item #:ttl narinfo-ttl diff --git a/tests/publish.scm b/tests/publish.scm index 097ac036e0..10bc859695 100644 --- a/tests/publish.scm +++ b/tests/publish.scm @@ -36,6 +36,7 @@ #:use-module (gcrypt pk-crypto) #:use-module ((guix pki) #:select (%public-key-file %private-key-file)) #:use-module (guix zlib) + #:use-module (guix lzlib) #:use-module (web uri) #:use-module (web client) #:use-module (web response) @@ -229,6 +230,19 @@ FileSize: ~a~%" (string-append "/nar/gzip/" (basename %item)))))) (get-bytevector-n nar (bytevector-length %gzip-magic-bytes)))) +(unless (lzlib-available?) + (test-skip 1)) +(test-equal "/nar/lzip/*" + "bar" + (call-with-temporary-output-file + (lambda (temp port) + (let ((nar (http-get-port + (publish-uri + (string-append "/nar/lzip/" (basename %item)))))) + (call-with-lzip-input-port nar + (cut restore-file <> temp))) + (call-with-input-file temp read-string)))) + (unless (zlib-available?) (test-skip 1)) (test-equal "/*.narinfo with compression" @@ -251,6 +265,28 @@ FileSize: ~a~%" (_ #f))) (recutils->alist body))))) +(unless (lzlib-available?) + (test-skip 1)) +(test-equal "/*.narinfo with lzip compression" + `(("StorePath" . ,%item) + ("URL" . ,(string-append "nar/lzip/" (basename %item))) + ("Compression" . "lzip")) + (let ((thread (with-separate-output-ports + (call-with-new-thread + (lambda () + (guix-publish "--port=6790" "-Clzip")))))) + (wait-until-ready 6790) + (let* ((url (string-append "http://localhost:6790/" + (store-path-hash-part %item) ".narinfo")) + (body (http-get-port url))) + (filter (lambda (item) + (match item + (("Compression" . _) #t) + (("StorePath" . _) #t) + (("URL" . _) #t) + (_ #f))) + (recutils->alist body))))) + (unless (zlib-available?) (test-skip 1)) (test-equal "/*.narinfo for a compressed file" From patchwork Fri May 24 13:42:36 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?q?Ludovic_Court=C3=A8s?= X-Patchwork-Id: 14061 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id B765C17040; Fri, 24 May 2019 14:44:07 +0100 (BST) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTP id 4E7B81700D for ; Fri, 24 May 2019 14:44:07 +0100 (BST) Received: from localhost ([127.0.0.1]:54895 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hUAUQ-00009m-UK for patchwork@mira.cbaines.net; Fri, 24 May 2019 09:44:06 -0400 Received: from eggs.gnu.org ([209.51.188.92]:37112) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hUAUN-000096-T7 for guix-patches@gnu.org; Fri, 24 May 2019 09:44:04 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hUAUM-0000XR-Rq for guix-patches@gnu.org; Fri, 24 May 2019 09:44:03 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:32768) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1hUAUM-0000XM-Of for guix-patches@gnu.org; Fri, 24 May 2019 09:44:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1hUAUM-0002Rv-HT for guix-patches@gnu.org; Fri, 24 May 2019 09:44:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#35880] [PATCH 5/7] self: Add dependency on lzlib. Resent-From: Ludovic =?utf-8?q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 24 May 2019 13:44:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 35880 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 35880@debbugs.gnu.org Received: via spool by 35880-submit@debbugs.gnu.org id=B35880.15587053849305 (code B ref 35880); Fri, 24 May 2019 13:44:02 +0000 Received: (at 35880) by debbugs.gnu.org; 24 May 2019 13:43:04 +0000 Received: from localhost ([127.0.0.1]:46307 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hUATP-0002Pq-SV for submit@debbugs.gnu.org; Fri, 24 May 2019 09:43:04 -0400 Received: from eggs.gnu.org ([209.51.188.92]:50678) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hUATI-0002O1-5B for 35880@debbugs.gnu.org; Fri, 24 May 2019 09:42:56 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:38751) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hUATD-0008Ep-0C; Fri, 24 May 2019 09:42:51 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=36484 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1hUATC-0007iN-Hd; Fri, 24 May 2019 09:42:50 -0400 From: Ludovic =?utf-8?q?Court=C3=A8s?= Date: Fri, 24 May 2019 15:42:36 +0200 Message-Id: <20190524134238.22802-5-ludo@gnu.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190524134238.22802-1-ludo@gnu.org> References: <20190524134238.22802-1-ludo@gnu.org> MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.51.188.43 X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Pierre Neidhardt Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches * guix/self.scm (compiled-guix): Pass #:lzlib to 'make-config.scm'. (make-config.scm): Add #:lzlib and honor it. (specification->package): Add "lzlib". --- guix/self.scm | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/guix/self.scm b/guix/self.scm index 6d7569ec19..17d289e486 100644 --- a/guix/self.scm +++ b/guix/self.scm @@ -57,6 +57,7 @@ ("guile-gcrypt" (ref '(gnu packages gnupg) 'guile-gcrypt)) ("gnutls" (ref '(gnu packages tls) 'gnutls)) ("zlib" (ref '(gnu packages compression) 'zlib)) + ("lzlib" (ref '(gnu packages compression) 'lzlib)) ("gzip" (ref '(gnu packages compression) 'gzip)) ("bzip2" (ref '(gnu packages compression) 'bzip2)) ("xz" (ref '(gnu packages compression) 'xz)) @@ -646,6 +647,7 @@ Info manual." (guile-version (effective-version)) (guile-for-build (default-guile)) (zlib (specification->package "zlib")) + (lzlib (specification->package "lzlib")) (gzip (specification->package "gzip")) (bzip2 (specification->package "bzip2")) (xz (specification->package "xz")) @@ -800,6 +802,7 @@ Info manual." #:extra-modules `(((guix config) => ,(make-config.scm #:zlib zlib + #:lzlib lzlib #:gzip gzip #:bzip2 bzip2 #:xz xz @@ -897,7 +900,7 @@ Info manual." (variables rest ...)))))) (variables %localstatedir %storedir %sysconfdir))) -(define* (make-config.scm #:key zlib gzip xz bzip2 +(define* (make-config.scm #:key zlib lzlib gzip xz bzip2 (package-name "GNU Guix") (package-version "0") (bug-report-address "bug-guix@gnu.org") @@ -919,7 +922,7 @@ Info manual." %store-database-directory %config-directory %libz - ;; TODO: %liblz + %liblz %gzip %bzip2 %xz)) @@ -966,7 +969,11 @@ Info manual." (define %libz #+(and zlib - (file-append zlib "/lib/libz")))) + (file-append zlib "/lib/libz"))) + + (define %liblz + #+(and lzlib + (file-append lzlib "/lib/liblz")))) ;; Guile 2.0 *requires* the 'define-module' to be at the ;; top-level or the 'toplevel-ref' in the resulting .go file are From patchwork Fri May 24 13:42:37 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?q?Ludovic_Court=C3=A8s?= X-Patchwork-Id: 14063 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id E1C7017040; Fri, 24 May 2019 14:44:14 +0100 (BST) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00 autolearn=ham autolearn_force=no version=3.4.0 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTP id 93F5F1700D for ; Fri, 24 May 2019 14:44:14 +0100 (BST) Received: from localhost ([127.0.0.1]:54901 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hUAUY-0000Fd-6a for patchwork@mira.cbaines.net; Fri, 24 May 2019 09:44:14 -0400 Received: from eggs.gnu.org ([209.51.188.92]:37118) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hUAUO-000098-6h for guix-patches@gnu.org; Fri, 24 May 2019 09:44:05 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hUAUN-0000Xn-CH for guix-patches@gnu.org; Fri, 24 May 2019 09:44:04 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:32769) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1hUAUN-0000Xg-9F for guix-patches@gnu.org; Fri, 24 May 2019 09:44:03 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1hUAUN-0002S2-5i for guix-patches@gnu.org; Fri, 24 May 2019 09:44:03 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#35880] [PATCH 6/7] gnu: guix: Add dependency on lzlib. Resent-From: Ludovic =?utf-8?q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 24 May 2019 13:44:03 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 35880 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 35880@debbugs.gnu.org Received: via spool by 35880-submit@debbugs.gnu.org id=B35880.15587053849312 (code B ref 35880); Fri, 24 May 2019 13:44:03 +0000 Received: (at 35880) by debbugs.gnu.org; 24 May 2019 13:43:04 +0000 Received: from localhost ([127.0.0.1]:46309 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hUATQ-0002Q2-Fq for submit@debbugs.gnu.org; Fri, 24 May 2019 09:43:04 -0400 Received: from eggs.gnu.org ([209.51.188.92]:50681) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hUATJ-0002O2-4F for 35880@debbugs.gnu.org; Fri, 24 May 2019 09:42:57 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:38752) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hUATD-0008FM-Vd; Fri, 24 May 2019 09:42:52 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=36484 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1hUATD-0007iN-Gc; Fri, 24 May 2019 09:42:51 -0400 From: Ludovic =?utf-8?q?Court=C3=A8s?= Date: Fri, 24 May 2019 15:42:37 +0200 Message-Id: <20190524134238.22802-6-ludo@gnu.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190524134238.22802-1-ludo@gnu.org> References: <20190524134238.22802-1-ludo@gnu.org> MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.51.188.43 X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Pierre Neidhardt Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches * gnu/packages/package-management.scm (guix)[inputs]: Add LZLIB. --- gnu/packages/package-management.scm | 1 + 1 file changed, 1 insertion(+) diff --git a/gnu/packages/package-management.scm b/gnu/packages/package-management.scm index a356a6dab7..985fa801b1 100644 --- a/gnu/packages/package-management.scm +++ b/gnu/packages/package-management.scm @@ -272,6 +272,7 @@ `(("bzip2" ,bzip2) ("gzip" ,gzip) ("zlib" ,zlib) ;for 'guix publish' + ("lzlib" ,lzlib) ;for 'guix publish' and 'guix substitute' ("sqlite" ,sqlite) ("libgcrypt" ,libgcrypt) From patchwork Fri May 24 13:42:38 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?q?Ludovic_Court=C3=A8s?= X-Patchwork-Id: 14062 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 5150317040; Fri, 24 May 2019 14:44:08 +0100 (BST) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00 autolearn=ham autolearn_force=no version=3.4.0 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTP id D5D2C1700D for ; Fri, 24 May 2019 14:44:07 +0100 (BST) Received: from localhost ([127.0.0.1]:54897 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hUAUR-00009z-GG for patchwork@mira.cbaines.net; Fri, 24 May 2019 09:44:07 -0400 Received: from eggs.gnu.org ([209.51.188.92]:37132) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hUAUO-00009A-VA for guix-patches@gnu.org; Fri, 24 May 2019 09:44:05 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hUAUN-0000YJ-UD for guix-patches@gnu.org; Fri, 24 May 2019 09:44:04 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:32770) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1hUAUN-0000Y8-Qx for guix-patches@gnu.org; Fri, 24 May 2019 09:44:03 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1hUAUN-0002SA-MK for guix-patches@gnu.org; Fri, 24 May 2019 09:44:03 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#35880] [PATCH 7/7] lzlib: 'lzread!' never returns more than it was asked for. Resent-From: Ludovic =?utf-8?q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 24 May 2019 13:44:03 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 35880 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 35880@debbugs.gnu.org Received: via spool by 35880-submit@debbugs.gnu.org id=B35880.15587053859318 (code B ref 35880); Fri, 24 May 2019 13:44:03 +0000 Received: (at 35880) by debbugs.gnu.org; 24 May 2019 13:43:05 +0000 Received: from localhost ([127.0.0.1]:46311 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hUATQ-0002Q9-PA for submit@debbugs.gnu.org; Fri, 24 May 2019 09:43:05 -0400 Received: from eggs.gnu.org ([209.51.188.92]:50684) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hUATK-0002O4-3Y for 35880@debbugs.gnu.org; Fri, 24 May 2019 09:42:58 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:38753) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hUATE-0008Fs-UO; Fri, 24 May 2019 09:42:52 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=36484 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1hUATE-0007iN-Fs; Fri, 24 May 2019 09:42:52 -0400 From: Ludovic =?utf-8?q?Court=C3=A8s?= Date: Fri, 24 May 2019 15:42:38 +0200 Message-Id: <20190524134238.22802-7-ludo@gnu.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190524134238.22802-1-ludo@gnu.org> References: <20190524134238.22802-1-ludo@gnu.org> MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.51.188.43 X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Pierre Neidhardt Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches Fixes a bug whereby 'lzread!' could return more than COUNT. * guix/lzlib.scm (lzread!): Rewrite in a semi-functional style. --- guix/lzlib.scm | 39 +++++++++++++++++++-------------------- 1 file changed, 19 insertions(+), 20 deletions(-) diff --git a/guix/lzlib.scm b/guix/lzlib.scm index 48927c6262..0e384f6cb8 100644 --- a/guix/lzlib.scm +++ b/guix/lzlib.scm @@ -494,29 +494,28 @@ perhaps not yet read." ;; High level functions. -(define* (lzread! decoder file-port bv + +(define* (lzread! decoder port bv #:optional (start 0) (count (bytevector-length bv))) - "Read up to COUNT bytes from FILE-PORT into BV at offset START. Return the + "Read up to COUNT bytes from PORT into BV at offset START. Return the number of uncompressed bytes actually read; it is zero if COUNT is zero or if the end-of-stream has been reached." - ;; WARNING: Because we don't alternate between lz-reads and lz-writes, we can't - ;; process more than lz-decompress-write-size from the file-port. - (when (> count (lz-decompress-write-size decoder)) - (set! count (lz-decompress-write-size decoder))) - (let ((file-bv (get-bytevector-n file-port count))) - (unless (eof-object? file-bv) - (lz-decompress-write decoder file-bv 0 (bytevector-length file-bv)))) - (let ((read 0)) - (let loop ((rd 0)) - (if (< start (bytevector-length bv)) - (begin - (set! rd (lz-decompress-read decoder bv start (- (bytevector-length bv) start))) - (set! start (+ start rd)) - (set! read (+ read rd))) - (set! rd 0)) - (unless (= rd 0) - (loop rd))) - read)) + (define (feed-decoder! decoder) + ;; Feed DECODER with data read from PORT. + (match (get-bytevector-n port (lz-decompress-write-size decoder)) + ((? eof-object? eof) eof) + (bv (lz-decompress-write decoder bv)))) + + (let loop ((read 0) + (start start)) + (cond ((< read count) + (match (lz-decompress-read decoder bv start (- count read)) + (0 (if (eof-object? (feed-decoder! decoder)) + read + (loop read start))) + (n (loop (+ read n) (+ start n))))) + (else + read)))) (define (lzwrite! encoder source source-offset source-count target target-offset target-count)