From patchwork Mon Aug 15 15:36:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Roman Scherer X-Patchwork-Id: 41691 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 4E6AA27BBEA; Mon, 15 Aug 2022 16:52:28 +0100 (BST) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.7 required=5.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,SPF_HELO_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 7E6B727BBE9 for ; Mon, 15 Aug 2022 16:52:27 +0100 (BST) Received: from localhost ([::1]:34806 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1oNcOA-0005mk-KN for patchwork@mira.cbaines.net; Mon, 15 Aug 2022 11:52:26 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:53914) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1oNcKu-0004VO-KM for guix-patches@gnu.org; Mon, 15 Aug 2022 11:49:06 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:54153) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1oNcKs-0000T5-IT for guix-patches@gnu.org; Mon, 15 Aug 2022 11:49:03 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1oNcKs-0007Zt-3v for guix-patches@gnu.org; Mon, 15 Aug 2022 11:49:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#56604] [PATCH 0/8] Update Clojure to 1.11.1. Resent-From: Roman Scherer Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 15 Aug 2022 15:49:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 56604 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Ludovic =?utf-8?q?Court=C3=A8s?= Cc: 56604-done@debbugs.gnu.org Received: via spool by 56604-done@debbugs.gnu.org id=D56604.166057853629118 (code D ref 56604); Mon, 15 Aug 2022 15:49:02 +0000 Received: (at 56604-done) by debbugs.gnu.org; 15 Aug 2022 15:48:56 +0000 Received: from localhost ([127.0.0.1]:43902 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1oNcKm-0007ZZ-3E for submit@debbugs.gnu.org; Mon, 15 Aug 2022 11:48:56 -0400 Received: from mail-wr1-f44.google.com ([209.85.221.44]:43677) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1oNcKk-0007ZL-3B for 56604-done@debbugs.gnu.org; Mon, 15 Aug 2022 11:48:54 -0400 Received: by mail-wr1-f44.google.com with SMTP id n4so9492622wrp.10 for <56604-done@debbugs.gnu.org>; Mon, 15 Aug 2022 08:48:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=burningswell-com.20210112.gappssmtp.com; s=20210112; h=mime-version:message-id:in-reply-to:date:subject:cc:to:from :user-agent:references:from:to:cc; bh=h74UlrG8AkaArG59KK2tF2L3UmfXS9gip0vpFTWMv9M=; b=bc+PumEiYr21O8rVaC0cqxLZLqEUKBfLw59nb70YncCgqmBnkx+tod0IKD55dhk5m3 5dzmx6O0QUShwjNzZCf8YoPp4LxQHovVzFjXDWxph39Enay3sr16WT+Mu+BXL/aAgN1V baoFPGb4MnLVrZvXSG9iaTpO+U3rsAJtAQknuMJH7PMqcTRYUy8QMjsqPvDPOEPOdiq+ 4MC4a20q9pkdzBOWIJUPG86h6vLqDh1gURUYXJViMYNEuiBm6EinZBPlvgb62gTqEuDz ziZwbVrJqlqUsmYyDMMsy/lMMuHZ8PW/QlPm4bDMVf9lGIrkwb3tDgOOTy5LtEa4YM2c oEww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=mime-version:message-id:in-reply-to:date:subject:cc:to:from :user-agent:references:x-gm-message-state:from:to:cc; bh=h74UlrG8AkaArG59KK2tF2L3UmfXS9gip0vpFTWMv9M=; b=2fuErp9wYmhUu7kl38RopRjHUmnATalt8oXJoLxzTzIEElELAViaVdqXGzmO4LVazU 99B7almINzTf2XHKq31ED46tUJturMtLLTcnuyaNAk8EyZN2rxh+FVax/QwnIrLLowZH AnqmqqjELjGIiEFPEGZ9f9as5WQdABSVrGDmRRaD3DJgwml/DE86S/J9YtAfpr2vmuPE 73+PG9TirOadcLAo9lfnGXufJpJjdJAQ7Nz+fuNYOEmc0XfswgjtFR60eOBJbUSGFDpN +HIQ84w/ZCf2ZC1ufNV5+6pl0L3RN4p0RQH2/ugDPg/7awYkRyh0kXAoPiI5/G6/vFJs 6ung== X-Gm-Message-State: ACgBeo3hTv3xrdixGuIh1GLQ/U1V3CdZeD9YYx3a51hGhksXDK227mXZ 9K2dB5jsVf3pgKlPngDrJeX1EcJysXp4hNfI X-Google-Smtp-Source: AA6agR6v0Isbc0TtIhesvCDrSXdfqhJIneDBlj92OQeqwLDrPDvIFHwVnW+Wfldlum7CnXxqx9A4bQ== X-Received: by 2002:a05:6000:18af:b0:222:c48d:9064 with SMTP id b15-20020a05600018af00b00222c48d9064mr9433891wri.18.1660578527780; Mon, 15 Aug 2022 08:48:47 -0700 (PDT) Received: from precision (tmo-119-192.customers.d1-online.com. [80.187.119.192]) by smtp.gmail.com with ESMTPSA id j3-20020a5d6043000000b0021d6dad334bsm7745313wrt.4.2022.08.15.08.48.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Aug 2022 08:48:47 -0700 (PDT) References: <87fsisg5mu.fsf@gnu.org> User-agent: mu4e 1.8.7; emacs 28.1 From: Roman Scherer Date: Mon, 15 Aug 2022 15:36:32 +0000 In-reply-to: <87fsisg5mu.fsf@gnu.org> Message-ID: <87zgg55wyq.fsf@burningswell.com> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches Hi Ludo, here's the promised patch to follow up with the code duplication I introduced in my previous patch. I tested this by compiling Clojure which uses the Ant build system, and by compiling clojure-tools-deps-alpha which uses the Clojure build system. Both of the build systems now call the repack-jar function. Could you have a look at it please? A related question: When I run the following commands after modifying the build systems they run for quite some time, because they were compiling a ton (the jdk, jetty) of things. ./pre-inst-env guix build clojure ./pre-inst-env guix build clojure-tools I guess this is expected, since a change in a build system might affect all packages being built with it. But I was wondering if there is a way to force only building the packages specified on the command line. Does such a thing exists? I was wondering what is the most efficient way to quickly iterate on changes to a build system, without recompiling the whole world for that build system. How would you do that? Thanks, Roman. Ludovic Courtès writes: > Hi, > > r0man skribis: > >> This phase makes sure the timestamp of compiled class files is set to a later >> point in time than the timestamp of the corresponding Clojure source files. If >> the timestamps of the class and source files are the same, the Clojure >> compiler will compile the sources again which can lead to issues. This problem >> has been discussed here [1]. The suggested solution was to keep/adjust the >> timestamps of the class files. > > Sounds reasonable. It’s a bummer though that the whole phase is pasted > from ant-build-system.scm, the only difference being the timestamps > (1980 instead of 1970). > > I added a TODO comment in clojure-build-system.scm when applying the > patch. Could you follow up with a patch to factorize that? > >> Btw, I was a bit surprised that in Guix Clojure packages are AOT compiled. The >> general wisdom in the Clojure community seems to be to avoid AOT compilation >> when distributing libraries, and only AOT compiling Uberjars for final >> deployment. Due to issues like I mentioned in clojure-instaparse. >> >> Are we sure that AOT compiling all Clojure source files by default is a good >> idea, instead of just compiling user declared namespaces which Leiningen and >> friends are doing? WDYT? > > Not much, but as you might have seen in ./etc/teams.scm, the project is > finally being structured as teams. There’s an opportunity for you to > start a Clojure team and to take the lead! :-) > > As a first step, I’d recommend getting in touch with people who have > worked on ‘clojure-build-system’ and packaged things in the past. > >> gnu: clojure-tools-cli: Update to 1.0.206. >> gnu: clojure-tools-gitlibs: Update to 2.4.181. >> gnu: clojure-tools-deps-alpha: Update to 0.14.1212. >> gnu: clojure-tools: Update to 1.11.1.1149. >> gnu: clojure: Update to 1.11.1. >> gnu: clojure-algo-generic: Fix test failing under AOT in Clojure 1.11.1. >> gnu: clojure-core-match: Update to 1.0.0. >> gnu: clojure-instaparse: Update to 1.4.12 (disabled AOT). > > I adjusted all the commit logs to follow our conventions; please > consider doing this next time: > > https://guix.gnu.org/manual/devel/en/html_node/Submitting-Patches.html > > The instaparse patch missed the hash update so I did that too. > > Thanks! > > Ludo’. From 756bfd3458ded38e1041ebb255c6b6ffe737732d Mon Sep 17 00:00:00 2001 From: Roman Scherer Date: Mon, 15 Aug 2022 15:29:25 +0000 Subject: [PATCH] build-system: Add repack-jar and use it in Ant & Clojure build systems * guix/build/ant-build-system.scm: Add repack-jar and use it in strip-jar-timestamps * guix/build/clojure-build-system.scm: Use repack-jar in reset-class-timestamps --- guix/build/ant-build-system.scm | 26 ++++++++------ guix/build/clojure-build-system.scm | 55 +++++------------------------ 2 files changed, 24 insertions(+), 57 deletions(-) diff --git a/guix/build/ant-build-system.scm b/guix/build/ant-build-system.scm index fae1b47ec5..63bdee4651 100644 --- a/guix/build/ant-build-system.scm +++ b/guix/build/ant-build-system.scm @@ -195,10 +195,9 @@ (define (generate-index jar) outputs) #t) -(define* (strip-jar-timestamps #:key outputs - #:allow-other-keys) - "Unpack all jar archives, reset the timestamp of all contained files, and -repack them. This is necessary to ensure that archives are reproducible." +(define-public (repack-jar outputs repack-fn) + "Unpack all jar archives, invoke repack-fn for each JAR with the directory +it has been unpacked to, and pack them again." (define (repack-archive jar) (format #t "repacking ~a\n" jar) (let* ((dir (mkdtemp! "jar-contents.XXXXXX")) @@ -206,13 +205,7 @@ (define (repack-archive jar) (with-directory-excursion dir (invoke "jar" "xf" jar)) (delete-file jar) - ;; XXX: copied from (gnu build install) - (for-each (lambda (file) - (let ((s (lstat file))) - (unless (eq? (stat:type s) 'symlink) - (utime file 0 0 0 0)))) - (find-files dir #:directories? #t)) - + (repack-fn dir) ;; The jar tool will always set the timestamp on the manifest file ;; and the containing directory to the current time, even when we ;; reuse an existing manifest file. To avoid this we use "zip" @@ -237,6 +230,17 @@ (define (repack-archive jar) outputs) #t) +(define* (strip-jar-timestamps #:key outputs + #:allow-other-keys) + "Unpack all jar archives, reset the timestamp of all contained files, and +repack them. This is necessary to ensure that archives are reproducible." + (repack-jar outputs (lambda (dir) + (for-each (lambda (file) + (let ((s (lstat file))) + (unless (eq? (stat:type s) 'symlink) + (utime file 0 0 0 0)))) + (find-files dir #:directories? #t))))) + (define* (check #:key target (make-flags '()) (tests? (not target)) (test-target "check") #:allow-other-keys) diff --git a/guix/build/clojure-build-system.scm b/guix/build/clojure-build-system.scm index cacbefb386..b82ebc30fe 100644 --- a/guix/build/clojure-build-system.scm +++ b/guix/build/clojure-build-system.scm @@ -19,7 +19,7 @@ (define-module (guix build clojure-build-system) #:use-module ((guix build ant-build-system) #:select ((%standard-phases . %standard-phases@ant) - ant-build)) + ant-build repack-jar)) #:use-module (guix build clojure-utils) #:use-module (guix build java-utils) #:use-module (guix build syscalls) @@ -112,54 +112,17 @@ (define* (check #:key jar-names))) #t) -(define (regular-jar-file? file stat) - "Predicate returning true if FILE is ending on '.jar' -and STAT indicates it is a regular file." - (and (string-suffix? ".jar" file) - (eq? 'regular (stat:type stat)))) - -;; XXX: The only difference compared to 'strip-jar-timestamps' in -;; ant-build-system.scm is the date. TODO: Adjust and factorize. (define* (reset-class-timestamps #:key outputs #:allow-other-keys) "Unpack all jar archives, reset the timestamp of all contained class files, and repack them. This is necessary to ensure that archives are reproducible." - (define (repack-archive jar) - (format #t "resetting class timestamps and repacking ~a\n" jar) - - ;; Note: .class files need to be strictly newer than source files, - ;; otherwise the Clojure compiler will recompile sources. - (let* ((early-1980 315619200) ; 1980-01-02 UTC - (dir (mkdtemp! "jar-contents.XXXXXX")) - (manifest (string-append dir "/META-INF/MANIFEST.MF"))) - (with-directory-excursion dir - (invoke "jar" "xf" jar)) - (delete-file jar) - (for-each (lambda (file) - (let ((s (lstat file))) - (unless (eq? (stat:type s) 'symlink) - (when (string-match "^(.*)\\.class$" file) - (utime file early-1980 early-1980))))) - (find-files dir #:directories? #t)) - ;; The jar tool will always set the timestamp on the manifest file - ;; and the containing directory to the current time, even when we - ;; reuse an existing manifest file. To avoid this we use "zip" - ;; instead of "jar". It is important that the manifest appears - ;; first. - (with-directory-excursion dir - (let* ((files (find-files "." ".*" #:directories? #t)) - ;; To ensure that the reference scanner can detect all - ;; store references in the jars we disable compression - ;; with the "-0" option. - (command (if (file-exists? manifest) - `("zip" "-0" "-X" ,jar ,manifest ,@files) - `("zip" "-0" "-X" ,jar ,@files)))) - (apply invoke command))) - (utime jar 0 0))) - (for-each (match-lambda - ((output . directory) - (for-each repack-archive - (find-files directory regular-jar-file?)))) - outputs)) + (repack-jar outputs (lambda (dir) + (for-each (lambda (file) + (let ((s (lstat file)) + (early-1980 315619200)) ; 1980-01-02 UTC + (unless (eq? (stat:type s) 'symlink) + (when (string-match "^(.*)\\.class$" file) + (utime file early-1980 early-1980))))) + (find-files dir #:directories? #t))))) (define-with-docs install "Standard 'install' phase for clojure-build-system." -- 2.37.1