diff mbox series

[bug#44590,WIP,CORE-UPDATES] build/python: Check for cythonized files.

Message ID 20201112101630.1302-1-efraim@flashner.co.il
State Work in progress
Headers show
Series [bug#44590,WIP,CORE-UPDATES] build/python: Check for cythonized files. | expand

Checks

Context Check Description
cbaines/applying patch fail View Laminar job

Commit Message

Efraim Flashner Nov. 12, 2020, 10:16 a.m. UTC 
* guix/build/python-build-system.scm (ensure-no-cythonized-files): New procedure.
(%standard-phases): Add it.
---

This patch only checks if a file exists and suggests it may be
cythonized. It doesn't check the first line for the string "generated by
Cython" in that file or in any file. It also doesn't error when it
thinks it finds a file. It does provide a good visual queue that the
source needs more scrutiny before being added.

---
 guix/build/python-build-system.scm | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

Comments

Efraim Flashner Dec. 9, 2020, 7:28 a.m. UTC | #1 
Patch pushed
Simon South Dec. 13, 2020, 4:53 p.m. UTC | #2 
Efraim,

I'm seeing an issue I think is related to this change. Trying to build
glib in core-updates today fails for me with "Unbound variable: warning"
in python-build-system.scm:

    starting phase `ensure-no-cythonized-files'
    Backtrace:
               9 (primitive-load "/gnu/store/ycv4j8dqk44g6d72m2ykvlwb9wc?")
    In ice-9/eval.scm:
       191:35  8 (_ _)
    In guix/build/gnu-build-system.scm:
        886:2  7 (gnu-build #:source _ #:outputs _ #:inputs _ #:phases . #)
    In ice-9/boot-9.scm:
      1736:10  6 (with-exception-handler _ _ #:unwind? _ # _)
    In srfi/srfi-1.scm:
        634:9  5 (for-each #<procedure 7ffff4f662e0 at guix/build/gnu-b?> ?)
    In guix/build/gnu-build-system.scm:
       895:33  4 (_ _)
    In guix/build/python-build-system.scm:
        272:2  3 (ensure-no-cythonized-files . _)
    In srfi/srfi-1.scm:
        634:9  2 (for-each #<procedure 7ffff5ad3978 at guix/build/pytho?> ?)
    In guix/build/python-build-system.scm:
       277:10  1 (_ _)
    In ice-9/boot-9.scm:
      1669:16  0 (raise-exception _ #:continuable? _)

    ice-9/boot-9.scm:1669:16: In procedure raise-exception:
    Unbound variable: warning
    builder for `/gnu/store/cgmgrddwjvryxmj4nn3naiwy42yi3wzn-meson-for-build-0.55.1.drv' failed with exit code 1
    build of /gnu/store/cgmgrddwjvryxmj4nn3naiwy42yi3wzn-meson-for-build-0.55.1.drv failed
    View build log at '/var/log/guix/drvs/cg/mgrddwjvryxmj4nn3naiwy42yi3wzn-meson-for-build-0.55.1.drv.bz2'.
    cannot build derivation `/gnu/store/a00wr5wj4cd0b98ixxynyxgbw2nyv8kg-glib-2.62.6.drv': 1 dependencies couldn't be built
    guix build: error: build of `/gnu/store/a00wr5wj4cd0b98ixxynyxgbw2nyv8kg-glib-2.62.6.drv' failed

And in fact building python-build-system.scm shows

    [100%] GUILEC   guix/build/python-build-system.go
    WARNING: (guix build python-build-system): imported module (guix build utils) overrides core binding `delete'
    guix/build/python-build-system.scm:277:10: warning: possibly unbound variable `warning'
    guix/build/python-build-system.scm:277:19: warning: possibly unbound variable `G_'

Adding use-module directives for (guix diagnostics) and (guix i18n) to
the module doesn't seem to fix the problem, presumably (if I understand
correctly) because these modules are not then also imported on the build
side of things.

Are you aware of this issue?
Efraim Flashner Dec. 13, 2020, 7:42 p.m. UTC | #3 
On Sun, Dec 13, 2020 at 11:53:31AM -0500, Simon South wrote:
> Efraim,
> 
> I'm seeing an issue I think is related to this change. Trying to build
> glib in core-updates today fails for me with "Unbound variable: warning"
> in python-build-system.scm:
> 
>     starting phase `ensure-no-cythonized-files'
>     Backtrace:
>                9 (primitive-load "/gnu/store/ycv4j8dqk44g6d72m2ykvlwb9wc?")
>     In ice-9/eval.scm:
>        191:35  8 (_ _)
>     In guix/build/gnu-build-system.scm:
>         886:2  7 (gnu-build #:source _ #:outputs _ #:inputs _ #:phases . #)
>     In ice-9/boot-9.scm:
>       1736:10  6 (with-exception-handler _ _ #:unwind? _ # _)
>     In srfi/srfi-1.scm:
>         634:9  5 (for-each #<procedure 7ffff4f662e0 at guix/build/gnu-b?> ?)
>     In guix/build/gnu-build-system.scm:
>        895:33  4 (_ _)
>     In guix/build/python-build-system.scm:
>         272:2  3 (ensure-no-cythonized-files . _)
>     In srfi/srfi-1.scm:
>         634:9  2 (for-each #<procedure 7ffff5ad3978 at guix/build/pytho?> ?)
>     In guix/build/python-build-system.scm:
>        277:10  1 (_ _)
>     In ice-9/boot-9.scm:
>       1669:16  0 (raise-exception _ #:continuable? _)
> 
>     ice-9/boot-9.scm:1669:16: In procedure raise-exception:
>     Unbound variable: warning
>     builder for `/gnu/store/cgmgrddwjvryxmj4nn3naiwy42yi3wzn-meson-for-build-0.55.1.drv' failed with exit code 1
>     build of /gnu/store/cgmgrddwjvryxmj4nn3naiwy42yi3wzn-meson-for-build-0.55.1.drv failed
>     View build log at '/var/log/guix/drvs/cg/mgrddwjvryxmj4nn3naiwy42yi3wzn-meson-for-build-0.55.1.drv.bz2'.
>     cannot build derivation `/gnu/store/a00wr5wj4cd0b98ixxynyxgbw2nyv8kg-glib-2.62.6.drv': 1 dependencies couldn't be built
>     guix build: error: build of `/gnu/store/a00wr5wj4cd0b98ixxynyxgbw2nyv8kg-glib-2.62.6.drv' failed
> 
> And in fact building python-build-system.scm shows
> 
>     [100%] GUILEC   guix/build/python-build-system.go
>     WARNING: (guix build python-build-system): imported module (guix build utils) overrides core binding `delete'
>     guix/build/python-build-system.scm:277:10: warning: possibly unbound variable `warning'
>     guix/build/python-build-system.scm:277:19: warning: possibly unbound variable `G_'
> 
> Adding use-module directives for (guix diagnostics) and (guix i18n) to
> the module doesn't seem to fix the problem, presumably (if I understand
> correctly) because these modules are not then also imported on the build
> side of things.
> 
> Are you aware of this issue?
> 

I tested it mostly against master. I'll see if I can figure out how I
broke it.
Efraim Flashner Dec. 13, 2020, 8:23 p.m. UTC | #4 
On Sun, Dec 13, 2020 at 11:53:31AM -0500, Simon South wrote:
> Efraim,
> 
> I'm seeing an issue I think is related to this change. Trying to build
> glib in core-updates today fails for me with "Unbound variable: warning"
> in python-build-system.scm:
> 
>     starting phase `ensure-no-cythonized-files'
>     Backtrace:
>                9 (primitive-load "/gnu/store/ycv4j8dqk44g6d72m2ykvlwb9wc?")
>     In ice-9/eval.scm:
>        191:35  8 (_ _)
>     In guix/build/gnu-build-system.scm:
>         886:2  7 (gnu-build #:source _ #:outputs _ #:inputs _ #:phases . #)
>     In ice-9/boot-9.scm:
>       1736:10  6 (with-exception-handler _ _ #:unwind? _ # _)
>     In srfi/srfi-1.scm:
>         634:9  5 (for-each #<procedure 7ffff4f662e0 at guix/build/gnu-b?> ?)
>     In guix/build/gnu-build-system.scm:
>        895:33  4 (_ _)
>     In guix/build/python-build-system.scm:
>         272:2  3 (ensure-no-cythonized-files . _)
>     In srfi/srfi-1.scm:
>         634:9  2 (for-each #<procedure 7ffff5ad3978 at guix/build/pytho?> ?)
>     In guix/build/python-build-system.scm:
>        277:10  1 (_ _)
>     In ice-9/boot-9.scm:
>       1669:16  0 (raise-exception _ #:continuable? _)
> 
>     ice-9/boot-9.scm:1669:16: In procedure raise-exception:
>     Unbound variable: warning
>     builder for `/gnu/store/cgmgrddwjvryxmj4nn3naiwy42yi3wzn-meson-for-build-0.55.1.drv' failed with exit code 1
>     build of /gnu/store/cgmgrddwjvryxmj4nn3naiwy42yi3wzn-meson-for-build-0.55.1.drv failed
>     View build log at '/var/log/guix/drvs/cg/mgrddwjvryxmj4nn3naiwy42yi3wzn-meson-for-build-0.55.1.drv.bz2'.
>     cannot build derivation `/gnu/store/a00wr5wj4cd0b98ixxynyxgbw2nyv8kg-glib-2.62.6.drv': 1 dependencies couldn't be built
>     guix build: error: build of `/gnu/store/a00wr5wj4cd0b98ixxynyxgbw2nyv8kg-glib-2.62.6.drv' failed
> 
> And in fact building python-build-system.scm shows
> 
>     [100%] GUILEC   guix/build/python-build-system.go
>     WARNING: (guix build python-build-system): imported module (guix build utils) overrides core binding `delete'
>     guix/build/python-build-system.scm:277:10: warning: possibly unbound variable `warning'
>     guix/build/python-build-system.scm:277:19: warning: possibly unbound variable `G_'
> 
> Adding use-module directives for (guix diagnostics) and (guix i18n) to
> the module doesn't seem to fix the problem, presumably (if I understand
> correctly) because these modules are not then also imported on the build
> side of things.
> 
> Are you aware of this issue?
> 

I fixed it. I changed the code to output using format and (ice-9 format)
and I tested it building meson-for-build and python-setuptools.

I suppose I should point out now that meson-for-build has some possibly
cythonized files :)
diff mbox series

Patch

diff --git a/guix/build/python-build-system.scm b/guix/build/python-build-system.scm
index 4a07496d22..bc34e22df8 100644
--- a/guix/build/python-build-system.scm
+++ b/guix/build/python-build-system.scm
@@ -8,6 +8,7 @@ 
 ;;; Copyright © 2018 Arun Isaac <arunisaac@systemreboot.net>
 ;;; Copyright © 2019, 2020 Maxim Cournoyer <maxim.cournoyer@gmail.com>
 ;;; Copyright © 2020 Jakub Kądziołka <kuba@kadziolka.net>
+;;; Copyright © 2020 Efraim Flashner <efraim@flashner.co.il>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -262,6 +263,18 @@  installed with setuptools."
   (setenv "PYTHONHASHSEED" "0")
   #t)
 
+(define* (ensure-no-cythonized-files #:rest _)
+  "Check the source code for @code{.c} files which may have been pre-generated
+by Cython."
+  (for-each
+    (lambda (file)
+      (let ((generated-file
+              (string-append (string-drop-right file 3) "c")))
+        (when (file-exists? generated-file)
+          (warning (G_ "Possible Cythonized file found: ~a~%") generated-file))))
+    (find-files "." "\\.pyx$"))
+  #t)
+
 (define %standard-phases
   ;; The build phase only builds C extensions and copies the Python sources,
   ;; while the install phase copies then byte-compiles the sources to the
@@ -271,6 +284,8 @@  installed with setuptools."
     (add-after 'unpack 'ensure-no-mtimes-pre-1980 ensure-no-mtimes-pre-1980)
     (add-after 'ensure-no-mtimes-pre-1980 'enable-bytecode-determinism
       enable-bytecode-determinism)
+    (add-after 'enable-bytecode-determinism 'ensure-no-cythonized-files
+      ensure-no-cythonized-files)
     (delete 'bootstrap)
     (delete 'configure)                 ;not needed
     (replace 'build build)