From patchwork Sun Dec 8 11:26:31 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?q?Ludovic_Court=C3=A8s?= X-Patchwork-Id: 16412 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id F145117869; Sun, 8 Dec 2019 11:27:15 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00 autolearn=unavailable autolearn_force=no version=3.4.0 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTP id 7E90D177CB for ; Sun, 8 Dec 2019 11:27:15 +0000 (GMT) Received: from localhost ([::1]:57726 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iduiZ-0004kp-2e for patchwork@mira.cbaines.net; Sun, 08 Dec 2019 06:27:15 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:34231) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iduiN-0004c9-K8 for guix-patches@gnu.org; Sun, 08 Dec 2019 06:27:05 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1iduiL-0004n9-Ro for guix-patches@gnu.org; Sun, 08 Dec 2019 06:27:03 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:44611) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1iduiL-0004m8-NW for guix-patches@gnu.org; Sun, 08 Dec 2019 06:27:01 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1iduiL-0007sq-Is for guix-patches@gnu.org; Sun, 08 Dec 2019 06:27:01 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#38518] [PATCH 1/7] serialization: Add 'fold-archive'. References: <20191207214230.25653-1-ludo@gnu.org> In-Reply-To: <20191207214230.25653-1-ludo@gnu.org> Resent-From: Ludovic =?utf-8?q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sun, 08 Dec 2019 11:27:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 38518 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 38518@debbugs.gnu.org Cc: Ludovic =?utf-8?q?Court=C3=A8s?= Received: via spool by 38518-submit@debbugs.gnu.org id=B38518.157580441930262 (code B ref 38518); Sun, 08 Dec 2019 11:27:01 +0000 Received: (at 38518) by debbugs.gnu.org; 8 Dec 2019 11:26:59 +0000 Received: from localhost ([127.0.0.1]:50576 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1iduiI-0007s1-GP for submit@debbugs.gnu.org; Sun, 08 Dec 2019 06:26:58 -0500 Received: from eggs.gnu.org ([209.51.188.92]:52645) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1iduiF-0007rk-F1 for 38518@debbugs.gnu.org; Sun, 08 Dec 2019 06:26:55 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:43961) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1iduiA-0003vI-6V; Sun, 08 Dec 2019 06:26:50 -0500 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=55006 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1idui8-0003cW-Nm; Sun, 08 Dec 2019 06:26:49 -0500 From: Ludovic =?utf-8?q?Court=C3=A8s?= Date: Sun, 8 Dec 2019 12:26:31 +0100 Message-Id: <20191208112637.5534-1-ludo@gnu.org> X-Mailer: git-send-email 2.24.0 MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.51.188.43 X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches * guix/serialization.scm (read-contents): Remove. (read-file-type, fold-archive): New procedures. (restore-file): Rewrite in terms of 'fold-archive'. * tests/nar.scm ("write-file-tree + fold-archive") ("write-file-tree + fold-archive, flat file"): New tests. --- guix/serialization.scm | 134 ++++++++++++++++++++++++----------------- tests/nar.scm | 74 +++++++++++++++++++++++ 2 files changed, 153 insertions(+), 55 deletions(-) diff --git a/guix/serialization.scm b/guix/serialization.scm index e14b7d1b9f..cf263d321e 100644 --- a/guix/serialization.scm +++ b/guix/serialization.scm @@ -48,6 +48,7 @@ write-file write-file-tree + fold-archive restore-file)) ;;; Comment: @@ -226,38 +227,25 @@ substitute invalid byte sequences with question marks. This is a (dump input output size)) (write-padding size output)) -(define (read-contents in out) - "Read the contents of a file from the Nar at IN, write it to OUT, and return -the size in bytes." - (define executable? - (match (read-string in) - ("contents" - #f) - ("executable" - (match (list (read-string in) (read-string in)) - (("" "contents") #t) - (x (raise - (condition (&message - (message "unexpected executable file marker")) - (&nar-read-error (port in) - (file #f) - (token x)))))) - #t) - (x - (raise - (condition (&message (message "unsupported nar file type")) - (&nar-read-error (port in) (file #f) (token x))))))) - - (let ((size (read-long-long in))) - ;; Note: `sendfile' cannot be used here because of port buffering on IN. - (dump in out size) - - (when executable? - (chmod out #o755)) - (let ((m (modulo size 8))) - (unless (zero? m) - (get-bytevector-n* in (- 8 m)))) - size)) +(define (read-file-type port) + "Read the file type tag from PORT, and return either 'regular or +'executable." + (match (read-string port) + ("contents" + 'regular) + ("executable" + (match (list (read-string port) (read-string port)) + (("" "contents") 'executable) + (x (raise + (condition (&message + (message "unexpected executable file marker")) + (&nar-read-error (port port) + (file #f) + (token x))))))) + (x + (raise + (condition (&message (message "unsupported nar file type")) + (&nar-read-error (port port) (file #f) (token x))))))) (define %archive-version-1 ;; Magic cookie for Nix archives. @@ -383,9 +371,14 @@ which case you can use 'identity'." (define port-conversion-strategy (fluid->parameter %default-port-conversion-strategy)) -(define (restore-file port file) - "Read a file (possibly a directory structure) in Nar format from PORT. -Restore it as FILE." +(define (fold-archive proc seed port file) + "Read a file (possibly a directory structure) in Nar format from PORT. Call +PROC on each file or directory read from PORT using: + + (PROC FILE TYPE CONTENTS RESULT) + +using SEED as the first RESULT. TYPE is a symbol like 'regular, and CONTENTS +depends on TYPE." (parameterize ((currently-restored-file file) ;; Error out if we can convert file names to the current @@ -401,7 +394,8 @@ Restore it as FILE." (token signature) (file #f)))))) - (let restore ((file file)) + (let read ((file file) + (result seed)) (define (read-eof-marker) (match (read-string port) (")" #t) @@ -414,40 +408,49 @@ Restore it as FILE." (match (list (read-string port) (read-string port) (read-string port)) (("(" "type" "regular") - (call-with-output-file file (cut read-contents port <>)) - (read-eof-marker)) + (let* ((type (read-file-type port)) + (size (read-long-long port)) + + ;; The caller must read exactly SIZE bytes from PORT. + (result (proc file type `(,port . ,size) result))) + (let ((m (modulo size 8))) + (unless (zero? m) + (get-bytevector-n* port (- 8 m)))) + (read-eof-marker) + result)) (("(" "type" "symlink") (match (list (read-string port) (read-string port)) (("target" target) - (symlink target file) - (read-eof-marker)) + (let ((result (proc file 'symlink target result))) + (read-eof-marker) + result)) (x (raise (condition (&message (message "invalid symlink tokens")) (&nar-read-error (port port) (file file) (token x))))))) (("(" "type" "directory") (let ((dir file)) - (mkdir dir) - (let loop ((prefix (read-string port))) + (let loop ((prefix (read-string port)) + (result (proc file 'directory #f result))) (match prefix ("entry" (match (list (read-string port) (read-string port) (read-string port) (read-string port)) (("(" "name" file "node") - (restore (string-append dir "/" file)) - (match (read-string port) - (")" #t) - (x - (raise - (condition - (&message - (message "unexpected directory entry termination")) - (&nar-read-error (port port) - (file file) - (token x)))))) - (loop (read-string port))))) - (")" #t) ; done with DIR + (let ((result (read (string-append dir "/" file) result))) + (match (read-string port) + (")" #f) + (x + (raise + (condition + (&message + (message "unexpected directory entry termination")) + (&nar-read-error (port port) + (file file) + (token x)))))) + (loop (read-string port) result))))) + (")" result) ;done with DIR (x (raise (condition @@ -459,6 +462,27 @@ Restore it as FILE." (&message (message "unsupported nar entry type")) (&nar-read-error (port port) (file file) (token x))))))))) +(define (restore-file port file) + "Read a file (possibly a directory structure) in Nar format from PORT. +Restore it as FILE." + (fold-archive (lambda (file type content result) + (match type + ('directory + (mkdir file)) + ('symlink + (symlink content file)) + ((or 'regular 'executable) + (match content + ((input . size) + (call-with-output-file file + (lambda (output) + (dump input output size) + (when (eq? type 'executable) + (chmod output #o755))))))))) + #t + port + file)) + ;;; Local Variables: ;;; eval: (put 'call-with-binary-input-file 'scheme-indent-function 1) ;;; End: diff --git a/tests/nar.scm b/tests/nar.scm index bfc71c69a8..aeff3d3330 100644 --- a/tests/nar.scm +++ b/tests/nar.scm @@ -214,6 +214,80 @@ (lambda () (false-if-exception (rm-rf %test-dir)))))) +(test-equal "write-file-tree + fold-archive" + '(("R" directory #f) + ("R/dir" directory #f) + ("R/dir/exe" executable "1234") + ("R/foo" regular "abcdefg") + ("R/lnk" symlink "foo")) + + (let () + (define-values (port get-bytevector) + (open-bytevector-output-port)) + (write-file-tree "root" port + #:file-type+size + (match-lambda + ("root" + (values 'directory 0)) + ("root/foo" + (values 'regular 7)) + ("root/lnk" + (values 'symlink 0)) + ("root/dir" + (values 'directory 0)) + ("root/dir/exe" + (values 'executable 4))) + #:file-port + (match-lambda + ("root/foo" (open-input-string "abcdefg")) + ("root/dir/exe" (open-input-string "1234"))) + #:symlink-target + (match-lambda + ("root/lnk" "foo")) + #:directory-entries + (match-lambda + ("root" '("foo" "dir" "lnk")) + ("root/dir" '("exe")))) + (close-port port) + + (reverse + (fold-archive (lambda (file type contents result) + (let ((contents (if (memq type '(regular executable)) + (utf8->string + (get-bytevector-n (car contents) + (cdr contents))) + contents))) + (cons `(,file ,type ,contents) + result))) + '() + (open-bytevector-input-port (get-bytevector)) + "R")))) + +(test-equal "write-file-tree + fold-archive, flat file" + '(("R" regular "abcdefg")) + + (let () + (define-values (port get-bytevector) + (open-bytevector-output-port)) + (write-file-tree "root" port + #:file-type+size + (match-lambda + ("root" (values 'regular 7))) + #:file-port + (match-lambda + ("root" (open-input-string "abcdefg")))) + (close-port port) + + (reverse + (fold-archive (lambda (file type contents result) + (let ((contents (utf8->string + (get-bytevector-n (car contents) + (cdr contents))))) + (cons `(,file ,type ,contents) result))) + '() + (open-bytevector-input-port (get-bytevector)) + "R")))) + (test-assert "write-file supports non-file output ports" (let ((input (string-append (dirname (search-path %load-path "guix.scm")) "/guix"))