diff mbox series

[bug#59078] lint: Split the derivation lint checker by system.

Message ID 20221106135532.5724-1-mail@cbaines.net
State New
Headers show
Series [bug#59078] lint: Split the derivation lint checker by system. | expand

Checks

Context Check Description
cbaines/comparison success View comparision
cbaines/git-branch success View Git branch
cbaines/applying patch success
cbaines/issue success View issue

Commit Message

Christopher Baines Nov. 6, 2022, 1:55 p.m. UTC
Currently, if you attempt to run the derivation checker on all packages, the
Guile process will run out of memory. I think a contributing factor to this is
that the checker forces an inefficient order when you want to generate
derivations for all the supported systems of each package, constantly
switching system then package.

This problem also impacts the Guix Data Service, since it tries to run the
derivation checker for all packages.

The changes in this commit to split the derivation lint checker in to several,
one for each system, means that you can now treat each system separately,
which should be better for caching purposes.

If it's desirable to keep some notion of checking all supported systems for a
single package, I think lint checker groups could be added, so that you could
ask for the "derivation" checker, and this would run all the derivation
checkers.

* guix/lint.scm (check-derivation): Adapt to make-check-derivation-for-system.
(%derivation-checkers): New variable.
(%local-checkers): Include all %derivation-checkers.
* doc/guix.texi (Invoking guix lint): Update.
---
 doc/guix.texi |   4 +-
 guix/lint.scm | 139 +++++++++++++++++++++++++++++---------------------
 2 files changed, 83 insertions(+), 60 deletions(-)

Comments

Christopher Baines Nov. 7, 2022, 5:37 p.m. UTC | #1
Christopher Baines <mail@cbaines.net> writes:

> This problem also impacts the Guix Data Service, since it tries to run the
> derivation checker for all packages.

This patch has now been processed by qa.guix.gnu.org. Looking at the
logs for the Guix Data Service processing the base and target revision,
and the change is more significant than I'd imagined:

Base:

  inferior heap after cleanup: 1739.0 MiB used (5160.0 MiB heap)
  debug: Finished getting formatting lint warnings, took 349 seconds
  debug: Finished fetching inferior lint warnings, took 3782 seconds

Target:

  inferior heap after cleanup: 1152.0 MiB used (1778.0 MiB heap)
  debug: Finished getting derivation/aarch64-linux lint warnings, took 334 seconds
  debug: Finished fetching inferior lint warnings, took 3285 seconds


So with the changes, it's a little faster, but the main difference is
that the heap ~3GiB smaller, so ~34% of what it was previously.

I did notice that this also subtly differs from how the linter behaved
previously, since some packages define support for systems not defined
through the platform module.

  https://data.qa.guix.gnu.org/compare?base_commit=a60dc46c2bb5de196858594b72b00d5f86ca7e98&target_commit=4e152714f55337015991e62e51e8dea15e889b9f

Personally, I think this change is still a good one. Maybe we can add a
separate linter to go round and check that packages don't declare
support for systems that aren't in the platform module.

Unless anyone objects, I'll like to push this sooner rather than later,
as I think the excessive heap size in the inferior process is not ideal.

Thanks,

Chris
diff mbox series

Patch

diff --git a/doc/guix.texi b/doc/guix.texi
index 7806b21a0f..8d4989a60c 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -14422,9 +14422,9 @@  Parse the @code{source} URL to determine if a tarball from GitHub is
 autogenerated or if it is a release tarball.  Unfortunately GitHub's
 autogenerated tarballs are sometimes regenerated.
 
-@item derivation
+@item derivation/SYSTEM
 Check that the derivation of the given packages can be successfully
-computed for all the supported systems (@pxref{Derivations}).
+computed for the specified system (@pxref{Derivations}).
 
 @item profile-collisions
 Check whether installing the given packages in a profile would lead to
diff --git a/guix/lint.scm b/guix/lint.scm
index 8e3976171f..f692856f42 100644
--- a/guix/lint.scm
+++ b/guix/lint.scm
@@ -52,6 +52,7 @@  (define-module (guix lint)
   #:use-module (guix memoization)
   #:use-module (guix profiles)
   #:use-module (guix monads)
+  #:use-module (guix platform)
   #:use-module (guix scripts)
   #:use-module ((guix ui) #:select (texi->plain-text fill-paragraph))
   #:use-module (guix gnu-maintenance)
@@ -98,7 +99,6 @@  (define-module (guix lint)
             check-patch-file-names
             check-patch-headers
             check-synopsis-style
-            check-derivation
             check-home-page
             check-name
             check-source
@@ -116,6 +116,8 @@  (define-module (guix lint)
             check-haskell-stackage
             check-tests-true
 
+            make-check-derivation-for-system
+
             lint-warning
             lint-warning?
             lint-warning-package
@@ -1369,56 +1371,6 @@  (define (check-phases-deltas deltas)
     (append-map check-phases-delta deltas))
   (find-phase-deltas package check-phases-deltas))
 
-(define* (check-derivation package #:key store)
-  "Emit a warning if we fail to compile PACKAGE to a derivation."
-  (define (try store system)
-    (guard (c ((store-protocol-error? c)
-               (make-warning package
-                             (G_ "failed to create ~a derivation: ~a")
-                             (list system
-                                   (store-protocol-error-message c))))
-              ((exception-with-kind-and-args? c)
-               (make-warning package
-                             (G_ "failed to create ~a derivation: ~s")
-                             (list system
-                                   (cons (exception-kind c)
-                                         (exception-args c)))))
-              ((message-condition? c)
-               (make-warning package
-                             (G_ "failed to create ~a derivation: ~a")
-                             (list system
-                                   (condition-message c))))
-              ((formatted-message? c)
-               (let ((str (apply format #f
-                                 (formatted-message-string c)
-                                 (formatted-message-arguments c))))
-                 (make-warning package
-                               (G_ "failed to create ~a derivation: ~a")
-                               (list system str))))
-              (else
-               (make-warning package
-                             (G_ "failed to create ~a derivation: ~a")
-                             (list system c))))
-      (parameterize ((%graft? #f))
-        (package-derivation store package system #:graft? #f)
-
-        ;; If there's a replacement, make sure we can compute its
-        ;; derivation.
-        (match (package-replacement package)
-          (#f #t)
-          (replacement
-           (package-derivation store replacement system
-                               #:graft? #f))))))
-
-  (define (check-with-store store)
-    (filter lint-warning?
-            (map (cut try store <>) (package-supported-systems package))))
-
-  ;; For backwards compatability, don't rely on store being set
-  (or (and=> store check-with-store)
-      (with-store store
-        (check-with-store store))))
-
 (define* (check-profile-collisions package #:key store)
   "Check for collisions that would occur when installing PACKAGE as a result
 of the propagated inputs it pulls in."
@@ -1843,13 +1795,88 @@  (define (check-formatting package)
                                        (G_ "source file not found"))))))))
         '())))
 
+(define (make-check-derivation-for-system system)
+  (define (try package proc)
+    (guard (c ((store-protocol-error? c)
+               (make-warning package
+                             (G_ "failed to create ~a derivation: ~a")
+                             (list system
+                                   (store-protocol-error-message c))))
+              ((exception-with-kind-and-args? c)
+               (make-warning package
+                             (G_ "failed to create ~a derivation: ~s")
+                             (list system
+                                   (cons (exception-kind c)
+                                         (exception-args c)))))
+              ((message-condition? c)
+               (make-warning package
+                             (G_ "failed to create ~a derivation: ~a")
+                             (list system
+                                   (condition-message c))))
+              ((formatted-message? c)
+               (let ((str (apply format #f
+                                 (formatted-message-string c)
+                                 (formatted-message-arguments c))))
+                 (make-warning package
+                               (G_ "failed to create ~a derivation: ~a")
+                               (list system str))))
+              (else
+               (make-warning package
+                             (G_ "failed to create ~a derivation: ~a")
+                             (list system c))))
+      (proc)))
+
+
+
+  (lambda* (package #:key store)
+    "Emit a warning if we fail to compile PACKAGE to a derivation."
+
+    (define (check-with-store store)
+      (if (member system (package-supported-systems package))
+          (filter
+           lint-warning?
+           (map (cut try package <>)
+                (list
+                 (lambda ()
+                   (parameterize ((%graft? #f))
+                     (package-derivation store package system #:graft? #f)))
+                 (lambda ()
+                   ;; If there's a replacement, make sure we can compute its
+                   ;; derivation.
+                   (match (package-replacement package)
+                     (#f #t)
+                     (replacement
+                      (parameterize ((%graft? #f))
+                        (package-derivation store replacement system
+                                            #:graft? #f))))))))
+          '()))
+
+    ;; For backwards compatability, don't rely on store being set
+    (or (and=> store check-with-store)
+        (with-store store
+          (check-with-store store)))))
+
 
 ;;;
 ;;; List of checkers.
 ;;;
 
+(define %derivation-checkers
+  (map (lambda (system)
+         (lint-checker
+          (name (string->symbol
+                 (simple-format #f "derivation/~A" system)))
+          (description
+           (simple-format
+            #f
+            "Report failure to compile a package to a derivation for ~A"
+            system))
+          (check (make-check-derivation-for-system system))
+          (requires-store? #t)))
+       (systems)))
+
 (define %local-checkers
-  (list
+  (cons*
    (lint-checker
      (name        'name)
      (description "Validate package names")
@@ -1901,11 +1928,6 @@  (define %local-checkers
      (name        'source-unstable-tarball)
      (description "Check for autogenerated tarballs")
      (check       check-source-unstable-tarball))
-   (lint-checker
-     (name            'derivation)
-     (description     "Report failure to compile a package to a derivation")
-     (check           check-derivation)
-     (requires-store? #t))
    (lint-checker
      (name            'profile-collisions)
      (description     "Report collisions that would occur due to propagated inputs")
@@ -1922,7 +1944,8 @@  (define %local-checkers
    (lint-checker
      (name        'formatting)
      (description "Look for formatting issues in the source")
-     (check       check-formatting))))
+     (check       check-formatting))
+   %derivation-checkers))
 
 (define %network-dependent-checkers
   (list