From patchwork Mon May 6 21:18:38 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Ludovic_Court=C3=A8s?= X-Patchwork-Id: 13922 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 99E6816FB6; Mon, 6 May 2019 22:19:21 +0100 (BST) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTP id 342F616FB2 for ; Mon, 6 May 2019 22:19:18 +0100 (BST) Received: from localhost ([127.0.0.1]:34097 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hNl13-0007N7-Ag for patchwork@mira.cbaines.net; Mon, 06 May 2019 17:19:17 -0400 Received: from eggs.gnu.org ([209.51.188.92]:33457) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hNl0q-0007Hi-PK for guix-patches@gnu.org; Mon, 06 May 2019 17:19:06 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hNl0o-0002ds-Vp for guix-patches@gnu.org; Mon, 06 May 2019 17:19:04 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:44069) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1hNl0o-0002df-Rm for guix-patches@gnu.org; Mon, 06 May 2019 17:19:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1hNl0o-0002Rj-GG for guix-patches@gnu.org; Mon, 06 May 2019 17:19:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#34807] [PATCH 1/2] Add (guix lzlib). Resent-From: Ludovic =?utf-8?q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 06 May 2019 21:19:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 34807 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Pierre Neidhardt Received: via spool by 34807-submit@debbugs.gnu.org id=B34807.15571775399392 (code B ref 34807); Mon, 06 May 2019 21:19:01 +0000 Received: (at 34807) by debbugs.gnu.org; 6 May 2019 21:18:59 +0000 Received: from localhost ([127.0.0.1]:57613 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hNl0f-0002RL-Ec for submit@debbugs.gnu.org; Mon, 06 May 2019 17:18:59 -0400 Received: from eggs.gnu.org ([209.51.188.92]:47217) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hNl0a-0002R3-Ft for 34807@debbugs.gnu.org; Mon, 06 May 2019 17:18:52 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:46978) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hNl0U-0002Ur-Ax; Mon, 06 May 2019 17:18:42 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=58962 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1hNl0T-0006n4-Bu; Mon, 06 May 2019 17:18:41 -0400 From: Ludovic =?utf-8?q?Court=C3=A8s?= References: <20190310180209.11578-1-mail@ambrevar.xyz> <8736ne3855.fsf@gnu.org> <87pnp2f7gr.fsf@ambrevar.xyz> <8736lxdxn6.fsf@ambrevar.xyz> <87lfzm7fdz.fsf@gnu.org> <878svm5xic.fsf@ambrevar.xyz> <87ef5e0vvv.fsf@gnu.org> <87o94h527a.fsf@ambrevar.xyz> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 17 =?utf-8?q?Flor=C3=A9al?= an 227 de la =?utf-8?q?R?= =?utf-8?q?=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Mon, 06 May 2019 23:18:38 +0200 In-Reply-To: <87o94h527a.fsf@ambrevar.xyz> (Pierre Neidhardt's message of "Sat, 04 May 2019 23:39:37 +0200") Message-ID: <87a7fzffip.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.2 (gnu/linux) MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.51.188.43 X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: 34807@debbugs.gnu.org Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches Hi Pierre, Pierre Neidhardt skribis: > It's definitely the ideal route. Something like guile-compress or > guile-archive, with a high-level abstraction for a collection of > bindings including zlib and lzlib for now. > > Sadly I don't have the time for it at the moment. Unless you do (:p) I > suggest we add a TODO item and keep it for later. Sounds good! Below are the Autoconf-related changes I made. Committed! We’ll take care of (guix self) when (guix lzlib) is actually used by other parts of the code. > Regarding guix publish and the farms, what shall we do? I think we should arrange for the client part, ‘guix substitute’, to be ready to lzip-decode as soon as it talks to an lzip-capable server. Then we should add support in ‘guix publish’. At some later point, we’d deploy it on the build farms. For this migration to be incremental, we need (1) clients to be able to transparently switch to lzip when it’s available, and (2) servers to be able to produce both lzip archives (for new clients) and gzip archives (for old clients) during the transition period. That’s a bit of work in ‘guix publish’. It’ll be extra CPU and storage usage on the build farm since during the transition period it’d have to produce and store both gzip and lzip archives for each store item. I don’t really see any way around that, though. A difficulty is that narinfos currently include a fixed compression scheme: --8<---------------cut here---------------start------------->8--- $ wget -q -O - https://ci.guix.info/nrkm1683p1cqnkcmhlmhiig9q9qd7xqh.narinfo | head -3 StorePath: /gnu/store/nrkm1683p1cqnkcmhlmhiig9q9qd7xqh-sed-4.5 URL: nar/gzip/nrkm1683p1cqnkcmhlmhiig9q9qd7xqh-sed-4.5 Compression: gzip --8<---------------cut here---------------end--------------->8--- So, depending on the client, ‘guix publish’ should return either a narinfo-for-gzip or a narinfo-for-lzip. To make it possible, new clients could send an extra HTTP header, say ‘X-Guix-Compression’, that would specify their preferred compression method(s). ‘guix publish’ would take that into account when replying. How does that sound? Thanks, Ludo’. diff --git a/configure.ac b/configure.ac index ee7aa25742..3918550a79 100644 --- a/configure.ac +++ b/configure.ac @@ -251,14 +251,13 @@ AC_MSG_RESULT([$LIBZ]) AC_SUBST([LIBZ]) dnl Library name of lzlib suitable for 'dynamic-link'. -GUIX_LIBLZ_LIBDIR([liblz_libdir]) -if test "x$liblz_libdir" = "x"; then +GUIX_LIBLZ_FILE_NAME([LIBLZ]) +if test "x$LIBLZ" = "x"; then LIBLZ="liblz" else - LIBLZ="$liblz_libdir/liblz" + # Strip the .so or .so.1 extension since that's what 'dynamic-link' expects. + LIBLZ="`echo $LIBLZ | sed -es'/\.so\(\.[[0-9.]]\+\)\?//g'`" fi -AC_MSG_CHECKING([for lzlib's shared library name]) -AC_MSG_RESULT([$LIBLZ]) AC_SUBST([LIBLZ]) dnl Check for Guile-SSH, for the (guix ssh) module. diff --git a/guix/config.scm.in b/guix/config.scm.in index cd7036ca7e..0ada0f3c38 100644 --- a/guix/config.scm.in +++ b/guix/config.scm.in @@ -34,10 +34,10 @@ %system %libz + %liblz %gzip %bzip2 - %xz - %liblz)) + %xz)) ;;; Commentary: ;;; @@ -91,6 +91,9 @@ (define %libz "@LIBZ@") +(define %liblz + "@LIBLZ@") + (define %gzip "@GZIP@") @@ -100,9 +103,4 @@ (define %xz "@XZ@") -(define %liblz - ;; TODO: Set this dynamically. - ;; "@LIBLZ@" - "/gnu/store/8db7vivi8p9mpkbphb8xy8gh2bkwc4iz-lzlib-1.11/lib/liblz") - ;;; config.scm ends here diff --git a/guix/self.scm b/guix/self.scm index 983f3514d3..74ea65240c 100644 --- a/guix/self.scm +++ b/guix/self.scm @@ -925,6 +925,7 @@ Info manual." %store-database-directory %config-directory %libz + ;; TODO: %liblz %gzip %bzip2 %xz)) @@ -971,8 +972,7 @@ Info manual." (define %libz #+(and zlib - (file-append zlib "/lib/libz"))) - (define %liblz #f)) + (file-append zlib "/lib/libz")))) ;; Guile 2.0 *requires* the 'define-module' to be at the ;; top-level or the 'toplevel-ref' in the resulting .go file are diff --git a/m4/guix.m4 b/m4/guix.m4 index 78cc3777f8..d0c5ec0f08 100644 --- a/m4/guix.m4 +++ b/m4/guix.m4 @@ -1,5 +1,5 @@ dnl GNU Guix --- Functional package management for GNU -dnl Copyright © 2012, 2013, 2014, 2015, 2016, 2018 Ludovic Courtès +dnl Copyright © 2012, 2013, 2014, 2015, 2016, 2018, 2019 Ludovic Courtès dnl Copyright © 2014 Mark H Weaver dnl Copyright © 2017 Efraim Flashner dnl @@ -312,20 +312,18 @@ AC_DEFUN([GUIX_LIBZ_LIBDIR], [ $1="$guix_cv_libz_libdir" ]) -dnl GUIX_LIBLZ_LIBDIR VAR +dnl GUIX_LIBLZ_FILE_NAME VAR dnl -dnl Attempt to determine liblz's LIBDIR; store the result in VAR. -AC_DEFUN([GUIX_LIBLZ_LIBDIR], [ +dnl Attempt to determine liblz's absolute file name; store the result in VAR. +AC_DEFUN([GUIX_LIBLZ_FILE_NAME], [ AC_REQUIRE([PKG_PROG_PKG_CONFIG]) - AC_CACHE_CHECK([lzlib's library directory], + AC_CACHE_CHECK([lzlib's file name], [guix_cv_liblz_libdir], - [guix_cv_liblz_libdir="`$PKG_CONFIG lzlib --variable=libdir 2> /dev/null`"]) - dnl TODO: lzlib has no pkg-config so we need the following trick to find its directory. - dnl old_LIBS="$LIBS" - dnl LIBS="-llz" - dnl AC_LINK_IFELSE([LZ_decompress_open();], - dnl [guix_cv_libz_libdir="`ldd conftest$EXEEXT | grep liblz | sed '-es/.*=> \([^ ]*\).*$/\1/g'`"]) - dnl LIBS="$old_LIBS" + [old_LIBS="$LIBS" + LIBS="-llz" + AC_LINK_IFELSE([AC_LANG_SOURCE([int main () { return LZ_decompress_open(); }])], + [guix_cv_liblz_libdir="`ldd conftest$EXEEXT | grep liblz | sed '-es/.*=> \(.*\) .*$/\1/g'`"]) + LIBS="$old_LIBS"]) $1="$guix_cv_liblz_libdir" ]) diff --git a/tests/lzlib.scm b/tests/lzlib.scm index a6631ce91c..cf53a9417d 100644 --- a/tests/lzlib.scm +++ b/tests/lzlib.scm @@ -26,8 +26,11 @@ ;; Test the (guix lzlib) module. -(unless (lzlib-available?) - (exit 77)) +(define-syntax-rule (test-assert* description exp) + (begin + (unless (lzlib-available?) + (test-skip 1)) + (test-assert description exp))) (test-begin "lzlib") @@ -68,41 +71,41 @@ (port-closed? parent) (bytevector=? received data))))))))))) -(test-assert "null bytevector" +(test-assert* "null bytevector" (compress-and-decompress (make-bytevector (+ (random 100000) (* 20 1024))))) -(test-assert "random bytevector" +(test-assert* "random bytevector" (compress-and-decompress (random-bytevector (+ (random 100000) (* 20 1024))))) -(test-assert "small bytevector" +(test-assert* "small bytevector" (compress-and-decompress (random-bytevector 127))) -(test-assert "1 bytevector" +(test-assert* "1 bytevector" (compress-and-decompress (random-bytevector 1))) -(test-assert "Bytevector of size relative to Lzip internal buffers (2 * dictionary)" +(test-assert* "Bytevector of size relative to Lzip internal buffers (2 * dictionary)" (compress-and-decompress (random-bytevector (* 2 (car (car (assoc-ref (@@ (guix lzlib) %compression-levels) (@@ (guix lzlib) %default-compression-level)))))))) -(test-assert "Bytevector of size relative to Lzip internal buffers (64KiB)" +(test-assert* "Bytevector of size relative to Lzip internal buffers (64KiB)" (compress-and-decompress (random-bytevector (* 64 1024)))) -(test-assert "Bytevector of size relative to Lzip internal buffers (64KiB-1)" +(test-assert* "Bytevector of size relative to Lzip internal buffers (64KiB-1)" (compress-and-decompress (random-bytevector (1- (* 64 1024))))) -(test-assert "Bytevector of size relative to Lzip internal buffers (64KiB+1)" +(test-assert* "Bytevector of size relative to Lzip internal buffers (64KiB+1)" (compress-and-decompress (random-bytevector (1+ (* 64 1024))))) -(test-assert "Bytevector of size relative to Lzip internal buffers (1MiB)" +(test-assert* "Bytevector of size relative to Lzip internal buffers (1MiB)" (compress-and-decompress (random-bytevector (* 1024 1024)))) -(test-assert "Bytevector of size relative to Lzip internal buffers (1MiB-1)" +(test-assert* "Bytevector of size relative to Lzip internal buffers (1MiB-1)" (compress-and-decompress (random-bytevector (1- (* 1024 1024))))) -(test-assert "Bytevector of size relative to Lzip internal buffers (1MiB+1)" +(test-assert* "Bytevector of size relative to Lzip internal buffers (1MiB+1)" (compress-and-decompress (random-bytevector (1+ (* 1024 1024))))) (test-end)