@@ -166,6 +166,7 @@ MODULES = \
guix/build-system/maven.scm \
guix/build-system/node.scm \
guix/build-system/perl.scm \
+ guix/build-system/pyproject.scm \
guix/build-system/python.scm \
guix/build-system/renpy.scm \
guix/build-system/ocaml.scm \
@@ -222,6 +223,7 @@ MODULES = \
guix/build/minetest-build-system.scm \
guix/build/node-build-system.scm \
guix/build/perl-build-system.scm \
+ guix/build/pyproject-build-system.scm \
guix/build/python-build-system.scm \
guix/build/ocaml-build-system.scm \
guix/build/qt-build-system.scm \
@@ -786,12 +786,29 @@ for instance, the module python-dateutil is packaged under the names
starts with @code{py} (e.g.@: @code{pytz}), we keep it and prefix it as
described above.
+@quotation Note
+Currently there are two different build systems for Python packages in Guix:
+@var{python-build-system} and @var{pyproject-build-system}. For the
+longest time, Python packages were built from an informally specified
+@file{setup.py} file. That worked amazingly well, considering Python's
+success, but was difficult to build tooling around. As a result, a host
+of alternative build systems emerged and the community eventually settled on a
+@url{https://peps.python.org/pep-0517/, formal standard} for specifying build
+requirements. @var{pyproject-build-system} is Guix's implementation of this
+standard. It is considered ``experimental'' in that it does not yet support
+all the various PEP-517 @emph{build backends}, but you are encouraged to try
+it for new Python packages and report any problems. It will eventually be
+deprecated and merged into @var{python-build-system}.
+@end quotation
+
@subsubsection Specifying Dependencies
@cindex inputs, for Python packages
Dependency information for Python packages is usually available in the
package source tree, with varying degrees of accuracy: in the
-@file{setup.py} file, in @file{requirements.txt}, or in @file{tox.ini}.
+@file{pyproject.toml} file, the @file{setup.py} file, in
+@file{requirements.txt}, or in @file{tox.ini} (the latter mostly for
+test dependencies).
Your mission, when writing a recipe for a Python package, is to map
these dependencies to the appropriate type of ``input'' (@pxref{package
@@ -802,10 +819,12 @@ following check list to determine which dependency goes where.
@itemize
@item
-We currently package Python 2 with @code{setuptools} and @code{pip}
-installed like Python 3.4 has per default. Thus you don't need to
-specify either of these as an input. @command{guix lint} will warn you
-if you do.
+We currently package Python with @code{setuptools} and @code{pip}
+installed per default. This is about to change, and users are encouraged
+to use @code{python-toolchain} if they want a build environment for Python.
+
+@command{guix lint} will warn if @code{setuptools} or @code{pip} are
+added as native-inputs because they are generally not necessary.
@item
Python dependencies required at run time go into
@@ -814,9 +833,10 @@ Python dependencies required at run time go into
@file{requirements.txt} file.
@item
-Python packages required only at build time---e.g., those listed with
-the @code{setup_requires} keyword in @file{setup.py}---or only for
-testing---e.g., those in @code{tests_require}---go into
+Python packages required only at build time---e.g., those listed under
+@code{build-system.requires} in @file{pyproject.toml} or with the
+@code{setup_requires} keyword in @file{setup.py}---or dependencies only
+for testing---e.g., those in @code{tests_require} or @file{tox.ini}---go into
@code{native-inputs}. The rationale is that (1) they do not need to be
propagated because they are not needed at run time, and (2) in a
cross-compilation context, it's the ``native'' input that we'd want.
@@ -9311,7 +9311,36 @@ instead of the default @code{"out"} output. This is useful for packages that
include a Python package as only a part of the software, and thus want to
combine the phases of @code{python-build-system} with another build system.
Python bindings are a common usecase.
+@end defvr
+
+@defvr {Scheme Variable} pyproject-build-system
+This is a variable exported by @code{guix build-system pyproject}. It
+is based on of @var{python-build-system}, and adds support for
+@file{pyproject.toml} and @url{https://peps.python.org/pep-0517/, PEP 517}.
+It also supports a variety of build backends and test frameworks.
+
+The API is slightly different from @var{python-build-system}:
+@enumerate
+@item
+@code{#:use-setuptools?} and @code{#:test-target} is removed.
+@item
+@code{#:build-backend} is added. It defaults to @code{#false} and will try
+to guess the appropriate backend based on @file{pyproject.toml}.
+@item
+@code{#:test-backend} is added. It defaults to @code{#false} and will guess
+an appropriate test backend based on what is available in package inputs.
+@item
+@code{#:test-flags} is added. The default is @code{#false}, and varies based
+on the detected @code{#:test-backend}.
+@end enumerate
+
+It is considered ``experimental'' in that the implementation details are
+not set in stone yet, however users are encouraged to try it for new
+Python projects (even those using @file{setup.py}). The API is subject to
+change, but any breaking changes in the Guix channel will be dealt with.
+Eventually this build system will be deprecated and merged back into
+@var{python-build-system}, probably some time in 2024.
@end defvr
@defvr {Scheme Variable} perl-build-system
new file mode 100644
@@ -0,0 +1,98 @@
+# -*- coding: utf-8 -*-
+
+# This version adds a small change to accomodate missing python-setuptools.
+# Original patch by Lars-Dominik Braun in wip-python-pep517, commit
+# 720dbe22d431262938be29dd9a9ddb78c44a99b3.
+# --- sanity-check.py 2022-06-12 14:40:06.814337702 +0200
+# +++ sanity-check.py 2022-10-16 23:21:38.990651568 +0200
+# @@ -19,9 +19,13 @@
+# from __future__ import print_function # Python 2 support.
+# import importlib
+# -import pkg_resources
+# import sys
+# import traceback
+# +try:
+# + import pkg_resources
+# +except ImportError:
+# + print('Warning: Skipping, because python-setuptools are not available.')
+# + sys.exit(0)
+
+# TODO: Merge with sanity-check.py in the next core-updates cycle.
+
+from __future__ import print_function # Python 2 support.
+import importlib
+import sys
+import traceback
+try:
+ import pkg_resources
+except ImportError:
+ print('Warning: Skipping, because python-setuptools are not available.')
+ sys.exit(0)
+
+try:
+ from importlib.machinery import PathFinder
+except ImportError:
+ PathFinder = None
+
+ret = 0
+
+# Only check site-packages installed by this package, but not dependencies
+# (which pkg_resources.working_set would include). Path supplied via argv.
+ws = pkg_resources.find_distributions(sys.argv[1])
+
+for dist in ws:
+ print('validating', repr(dist.project_name), dist.location)
+ try:
+ print('...checking requirements: ', end='')
+ req = str(dist.as_requirement())
+ # dist.activate() is not enough to actually check requirements, we
+ # have to .require() it.
+ pkg_resources.require(req)
+ print('OK')
+ except Exception as e:
+ print('ERROR:', req, repr(e))
+ ret = 1
+ continue
+
+ # Try to load top level modules. This should not have any side-effects.
+ try:
+ metalines = dist.get_metadata_lines('top_level.txt')
+ except (KeyError, EnvironmentError):
+ # distutils (i.e. #:use-setuptools? #f) will not install any metadata.
+ # This file is also missing for packages built using a PEP 517 builder
+ # such as poetry.
+ print('WARNING: cannot determine top-level modules')
+ continue
+ for name in metalines:
+ # Only available on Python 3.
+ if PathFinder and PathFinder.find_spec(name) is None:
+ # Ignore unavailable modules, often C modules, which were not
+ # installed at the top-level. Cannot use ModuleNotFoundError,
+ # because it is raised by failed imports too.
+ continue
+ try:
+ print('...trying to load module', name, end=': ')
+ importlib.import_module(name)
+ print('OK')
+ except Exception:
+ print('ERROR:')
+ traceback.print_exc(file=sys.stdout)
+ ret = 1
+
+ # Try to load entry points of console scripts too, making sure they
+ # work. They should be removed if they don't. Other groups may not be
+ # safe, as they can depend on optional packages.
+ for group, v in dist.get_entry_map().items():
+ if group not in {'console_scripts', 'gui_scripts'}:
+ continue
+ for name, ep in v.items():
+ try:
+ print('...trying to load endpoint', group, name, end=': ')
+ ep.load()
+ print('OK')
+ except Exception:
+ print('ERROR:')
+ traceback.print_exc(file=sys.stdout)
+ ret = 1
+
+sys.exit(ret)
@@ -26,7 +26,7 @@
;;; Copyright © 2016, 2017 Nikita <nikita@n0.is>
;;; Copyright © 2016 Dylan Jeffers <sapientech@sapientech@openmailbox.org>
;;; Copyright © 2016 David Craven <david@craven.ch>
-;;; Copyright © 2016, 2017, 2018, 2019, 2020, 2021 Marius Bakke <marius@gnu.org>
+;;; Copyright © 2016-2022 Marius Bakke <marius@gnu.org>
;;; Copyright © 2016, 2017 Stefan Reichör <stefan@xsteve.at>
;;; Copyright © 2016, 2017 Alex Vong <alexvong1995@gmail.com>
;;; Copyright © 2016, 2017, 2018 Arun Isaac <arunisaac@systemreboot.net>
@@ -60,6 +60,7 @@
;;; Copyright © 2020, 2021 Greg Hogan <code@greghogan.com>
;;; Copyright © 2022 Philip McGrath <philip@philipmcgrath.com>
;;; Copyright © 2022 jgart <jgart@dismail.de>
+;;; Copyright © 2021 Lars-Dominik Braun <lars@6xq.net>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -87,6 +88,7 @@ (define-module (gnu packages python)
#:use-module (gnu packages hurd)
#:use-module (gnu packages libffi)
#:use-module (gnu packages pkg-config)
+ #:use-module (gnu packages python-build)
#:use-module (gnu packages readline)
#:use-module (gnu packages sqlite)
#:use-module (gnu packages tcl)
@@ -674,6 +676,54 @@ (define* (wrap-python3 python
(define-public python-wrapper (wrap-python3 python))
(define-public python-minimal-wrapper (wrap-python3 python-minimal))
+;; The Python used in pyproject-build-system.
+(define-public python-sans-pip
+ (hidden-package
+ (package/inherit python
+ (arguments
+ (substitute-keyword-arguments (package-arguments python)
+ ((#:configure-flags flags #~())
+ #~(append '("--with-ensurepip=no")
+ (delete "--with-ensurepip=install" #$flags))))))))
+
+(define-public python-sans-pip-wrapper
+ (wrap-python3 python-sans-pip))
+
+(define-public python-toolchain
+ (package
+ (name "python-toolchain")
+ (version (package-version python))
+ (source #f)
+ (build-system trivial-build-system)
+ (arguments
+ (list #:modules '((guix build union))
+ #:builder
+ #~(begin
+ (use-modules (ice-9 match)
+ (srfi srfi-1)
+ (guix build union))
+ (union-build #$output
+ (filter-map (match-lambda
+ ((_ . directory) directory))
+ %build-inputs)))))
+ (inputs
+ (list python-sans-pip-wrapper
+ python-pip
+ python-pypa-build
+ python-setuptools
+ python-wheel))
+ (native-search-paths
+ (package-native-search-paths python))
+ (search-paths
+ (package-search-paths python))
+ (license (package-license python))
+ (synopsis "Python toolchain")
+ (description
+ "Python toolchain including Python itself, setuptools and pip. Use this
+package if you need a minimal Python toolchain instead of just the
+interpreter.")
+ (home-page (package-home-page python))))
+
(define-public micropython
(package
(name "micropython")
new file mode 100644
@@ -0,0 +1,147 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2021 Lars-Dominik Braun <lars@6xq.net>
+;;; Copyright © 2022 Marius Bakke <marius@gnu.org>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix build-system pyproject)
+ #:use-module ((gnu packages) #:select (search-auxiliary-file))
+ #:use-module (guix gexp)
+ #:use-module (guix store)
+ #:use-module (guix utils)
+ #:use-module (guix memoization)
+ #:use-module (guix gexp)
+ #:use-module (guix monads)
+ #:use-module (guix packages)
+ #:use-module (guix derivations)
+ #:use-module (guix search-paths)
+ #:use-module (guix build-system)
+ #:use-module (guix build-system gnu)
+ #:use-module (guix build-system python)
+ #:use-module (ice-9 match)
+ #:use-module (srfi srfi-1)
+ #:use-module (srfi srfi-26)
+ #:export (%pyproject-build-system-modules
+ default-python
+ pyproject-build
+ pyproject-build-system))
+
+;; Commentary:
+;;
+;; Standard build procedure for Python packages using 'pyproject.toml'.
+;; This is implemented as an extension of 'python-build-system'.
+;;
+;; Code:
+
+(define %pyproject-build-system-modules
+ ;; Build-side modules imported by default.
+ `((guix build pyproject-build-system)
+ (guix build json)
+ ,@%python-build-system-modules))
+
+(define (default-python)
+ "Return the default Python package."
+ ;; Lazily resolve the binding to avoid a circular dependency.
+ (let ((python (resolve-interface '(gnu packages python))))
+ (module-ref python 'python-toolchain)))
+
+(define sanity-check.py
+ ;; TODO: Merge with sanity-check.py in the next rebuild cycle.
+ (search-auxiliary-file "python/sanity-check-next.py"))
+
+(define* (lower name
+ #:key source inputs native-inputs outputs system target
+ (python (default-python))
+ #:allow-other-keys
+ #:rest arguments)
+ "Return a bag for NAME."
+ (define private-keywords
+ '(#:target #:python #:inputs #:native-inputs))
+
+ (and (not target) ;XXX: no cross-compilation
+ (bag
+ (name name)
+ (system system)
+ (host-inputs `(,@(if source
+ `(("source" ,source))
+ '())
+ ,@inputs
+
+ ;; Keep the standard inputs of 'gnu-build-system'.
+ ,@(standard-packages)))
+ (build-inputs `(("python" ,python)
+ ("sanity-check.py" ,(local-file sanity-check.py))
+ ,@native-inputs))
+ (outputs (append outputs '(wheel)))
+ (build pyproject-build)
+ (arguments (strip-keyword-arguments private-keywords arguments)))))
+
+(define* (pyproject-build name inputs
+ #:key source
+ (tests? #t)
+ (configure-flags ''())
+ (build-backend #f)
+ (test-backend #f)
+ (test-flags #f)
+ (phases '%standard-phases)
+ (outputs '("out" "wheel"))
+ (search-paths '())
+ (system (%current-system))
+ (guile #f)
+ (imported-modules %pyproject-build-system-modules)
+ (modules '((guix build pyproject-build-system)
+ (guix build utils))))
+ "Build SOURCE using PYTHON, and with INPUTS."
+ (define build
+ (with-imported-modules imported-modules
+ #~(begin
+ (use-modules #$@(sexp->gexp modules))
+
+ #$(with-build-variables inputs outputs
+ #~(pyproject-build
+ #:name #$name
+ #:source #+source
+ #:configure-flags #$configure-flags
+ #:system #$system
+ #:build-backend #$build-backend
+ #:test-backend #$test-backend
+ #:test-flags #$test-flags
+ #:tests? #$tests?
+ #:phases #$(if (pair? phases)
+ (sexp->gexp phases)
+ phases)
+ #:outputs %outputs
+ #:search-paths '#$(sexp->gexp
+ (map search-path-specification->sexp
+ search-paths))
+ #:inputs %build-inputs)))))
+
+
+ (mlet %store-monad ((guile (package->derivation (or guile (default-guile))
+ system #:graft? #f)))
+ (gexp->derivation name build
+ #:system system
+ #:graft? #f ;consistent with 'gnu-build'
+ #:target #f
+ #:guile-for-build guile)))
+
+(define pyproject-build-system
+ (build-system
+ (name 'pyproject)
+ (description "The PEP517-compliant Python build system")
+ (lower lower)))
+
+;;; pyproject.scm ends here
new file mode 100644
@@ -0,0 +1,367 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2021 Lars-Dominik Braun <lars@6xq.net>
+;;; Copyright © 2022 Marius Bakke <marius@gnu.org>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix build pyproject-build-system)
+ #:use-module ((guix build python-build-system) #:prefix python:)
+ #:use-module (guix build utils)
+ #:use-module (guix build json)
+ #:use-module (ice-9 match)
+ #:use-module (ice-9 ftw)
+ #:use-module (ice-9 format)
+ #:use-module (ice-9 rdelim)
+ #:use-module (ice-9 regex)
+ #:use-module (srfi srfi-1)
+ #:use-module (srfi srfi-26)
+ #:use-module (srfi srfi-34)
+ #:use-module (srfi srfi-35)
+ #:export (%standard-phases
+ add-installed-pythonpath
+ site-packages
+ python-version
+ pyproject-build))
+
+;;; Commentary:
+;;;
+;;; PEP 517-compatible build system for Python packages.
+;;;
+;;; PEP 517 mandates the use of a TOML file called pyproject.toml at the
+;;; project root, describing build and runtime dependencies, as well as the
+;;; build system, which can be different from setuptools. This module uses
+;;; that file to extract the build system used and call its wheel-building
+;;; entry point build_wheel (see 'build). setuptools’ wheel builder is
+;;; used as a fallback if either no pyproject.toml exists or it does not
+;;; declare a build-system. It supports config_settings through the
+;;; standard #:configure-flags argument.
+;;;
+;;; This wheel, which is just a ZIP file with a file structure defined
+;;; by PEP 427 (https://www.python.org/dev/peps/pep-0427/), is then unpacked
+;;; and its contents are moved to the appropriate locations in 'install.
+;;;
+;;; Then entry points, as defined by the PyPa Entry Point Specification
+;;; (https://packaging.python.org/specifications/entry-points/) are read
+;;; from a file called entry_points.txt in the package’s site-packages
+;;; subdirectory and scripts are written to bin/. These are not part of a
+;;; wheel and expected to be created by the installing utility.
+;;; TODO: Add support for PEP-621 entry points.
+;;;
+;;; Caveats:
+;;; - There is no support for in-tree build backends.
+;;;
+;;; Code:
+;;;
+
+;; Re-export these variables from python-build-system as many packages
+;; rely on these.
+(define python-version python:python-version)
+(define site-packages python:site-packages)
+(define add-installed-pythonpath python:add-installed-pythonpath)
+
+;; Base error type.
+(define-condition-type &python-build-error &error
+ python-build-error?)
+
+;; Raised when 'check cannot find a valid test system in the inputs.
+(define-condition-type &test-system-not-found &python-build-error
+ test-system-not-found?)
+
+;; Raised when multiple wheels are created by 'build.
+(define-condition-type &cannot-extract-multiple-wheels &python-build-error
+ cannot-extract-multiple-wheels?)
+
+;; Raised, when no wheel has been built by the build system.
+(define-condition-type &no-wheels-built &python-build-error
+ no-wheels-built?)
+
+(define* (build #:key outputs build-backend configure-flags #:allow-other-keys)
+ "Build a given Python package."
+
+ (define (pyproject.toml->build-backend file)
+ "Look up the build backend in a pyproject.toml file."
+ (call-with-input-file file
+ (lambda (in)
+ (let loop ((line (read-line in 'concat)))
+ (if (eof-object? line)
+ #f
+ (let ((m (string-match "build-backend = [\"'](.+)[\"']" line)))
+ (if m (match:substring m 1)
+ (loop (read-line in 'concat)))))))))
+
+ (let* ((wheel-output (assoc-ref outputs "wheel"))
+ (wheel-dir (if wheel-output wheel-output "dist"))
+ ;; There is no easy way to get data from Guile into Python via
+ ;; s-expressions, but we have JSON serialization already, which Python
+ ;; also supports out-of-the-box.
+ (config-settings (call-with-output-string (cut write-json configure-flags <>)))
+ ;; python-setuptools’ default backend supports setup.py *and*
+ ;; pyproject.toml. Allow overriding this automatic detection via
+ ;; build-backend.
+ (auto-build-backend (if (file-exists? "pyproject.toml")
+ (pyproject.toml->build-backend "pyproject.toml")
+ #f))
+ ;; Use build system detection here and not in importer, because a) we
+ ;; have alot of legacy packages and b) the importer cannot update arbitrary
+ ;; fields in case a package switches its build system.
+ (use-build-backend (or
+ build-backend
+ auto-build-backend
+ "setuptools.build_meta")))
+ (format #t "Using '~a' to build wheels, auto-detected '~a', override '~a'.~%"
+ use-build-backend auto-build-backend build-backend)
+ (mkdir-p wheel-dir)
+ ;; Call the PEP 517 build function, which drops a .whl into wheel-dir.
+ (invoke "python" "-c" "import sys, importlib, json
+config_settings = json.loads (sys.argv[3])
+builder = importlib.import_module(sys.argv[1])
+builder.build_wheel(sys.argv[2], config_settings=config_settings)"
+ use-build-backend wheel-dir config-settings)))
+
+(define* (check #:key inputs outputs tests? test-backend test-flags #:allow-other-keys)
+ "Run the test suite of a given Python package."
+ (if tests?
+ ;; Unfortunately with PEP 517 there is no common method to specify test
+ ;; systems. Guess test system based on inputs instead.
+ (let* ((pytest (which "pytest"))
+ (nosetests (which "nosetests"))
+ (nose2 (which "nose2"))
+ (have-setup-py (file-exists? "setup.py"))
+ (use-test-backend
+ (or
+ test-backend
+ ;; Prefer pytest
+ (if pytest 'pytest #f)
+ (if nosetests 'nose #f)
+ (if nose2 'nose2 #f)
+ ;; But fall back to setup.py, which should work for most
+ ;; packages. XXX: would be nice not to depend on setup.py here? fails
+ ;; more often than not to find any tests at all. Maybe we can run
+ ;; `python -m unittest`?
+ (if have-setup-py 'setup.py #f))))
+ (format #t "Using ~a~%" use-test-backend)
+ (match use-test-backend
+ ('pytest
+ (apply invoke (cons pytest (or test-flags '("-vv")))))
+ ('nose
+ (apply invoke (cons nosetests (or test-flags '("-v")))))
+ ('nose2
+ (apply invoke (cons nose2 (or test-flags '("-v" "--pretty-assert")))))
+ ('setup.py
+ (apply invoke (append '("python" "setup.py") (or test-flags '("test" "-v")))))
+ ;; The developer should explicitly disable tests in this case.
+ (else (raise (condition (&test-system-not-found))))))
+ (format #t "test suite not run~%")))
+
+(define* (install #:key inputs outputs (configure-flags '()) #:allow-other-keys)
+ "Install a wheel file according to PEP 427"
+ ;; See https://www.python.org/dev/peps/pep-0427/#installing-a-wheel-distribution-1-0-py32-none-any-whl
+ (let* ((site-dir (site-packages inputs outputs))
+ (python (assoc-ref inputs "python"))
+ (out (assoc-ref outputs "out")))
+ (define (extract file)
+ "Extract wheel (ZIP file) into site-packages directory"
+ ;; Use Python’s zipfile to avoid extra dependency
+ (invoke "python" "-m" "zipfile" "-e" file site-dir))
+
+ (define python-hashbang
+ (string-append "#!" python "/bin/python"))
+
+ (define* (merge-directories source destination #:optional (post-move #f))
+ "Move all files in SOURCE into DESTINATION, merging the two directories."
+ (format #t "Merging directory ~a into ~a~%" source destination)
+ (for-each
+ (lambda (file)
+ (format #t "~a/~a -> ~a/~a~%" source file destination file)
+ (mkdir-p destination)
+ (rename-file
+ (string-append source "/" file)
+ (string-append destination "/" file))
+ (when post-move
+ (post-move file)))
+ (scandir source (negate (cut member <> '("." "..")))))
+ (rmdir source))
+
+ (define (expand-data-directory directory)
+ "Move files from all .data subdirectories to their respective
+destinations."
+ ;; Python’s distutils.command.install defines this mapping from source to
+ ;; destination mapping.
+ (let ((source (string-append directory "/scripts"))
+ (destination (string-append out "/bin")))
+ (when (file-exists? source)
+ (merge-directories
+ source
+ destination
+ (lambda (f)
+ (let ((dest-path (string-append destination "/" f)))
+ (chmod dest-path #o755)
+ (substitute* dest-path (("#!python") python-hashbang)))))))
+ ;; data can create arbitrary directory structures. Most commonly
+ ;; it is used for share/.
+ (let ((source (string-append directory "/data"))
+ (destination out))
+ (when (file-exists? source)
+ (merge-directories source destination)))
+ (let* ((distribution (car (string-split (basename directory) #\-)))
+ (source (string-append directory "/headers"))
+ (destination (string-append out "/include/python" (python-version python) "/" distribution)))
+ (when (file-exists? source)
+ (merge-directories source destination))))
+
+ (define (list-directories base predicate)
+ ;; Cannot use find-files here, because it’s recursive.
+ (scandir
+ base
+ (lambda (name)
+ (let ((stat (lstat (string-append base "/" name))))
+ (and
+ (not (member name '("." "..")))
+ (eq? (stat:type stat) 'directory)
+ (predicate name stat))))))
+
+ (let* ((wheel-output (assoc-ref outputs "wheel"))
+ (wheel-dir (if wheel-output wheel-output "dist"))
+ (wheels (map (cut string-append wheel-dir "/" <>)
+ (scandir wheel-dir (cut string-suffix? ".whl" <>)))))
+ (cond
+ ((> (length wheels) 1) ; This code does not support multiple wheels
+ ; yet, because their outputs would have to be
+ ; merged properly.
+ (raise (condition (&cannot-extract-multiple-wheels))))
+ ((= (length wheels) 0)
+ (raise (condition (&no-wheels-built)))))
+ (for-each extract wheels))
+ (let ((datadirs (map
+ (cut string-append site-dir "/" <>)
+ (list-directories site-dir (file-name-predicate "\\.data$")))))
+ (for-each (lambda (directory)
+ (expand-data-directory directory)
+ (rmdir directory))
+ datadirs))))
+
+(define* (compile-bytecode #:key inputs outputs (configure-flags '()) #:allow-other-keys)
+ "Compile installed byte-code in site-packages."
+ (let* ((site-dir (site-packages inputs outputs))
+ (python (assoc-ref inputs "python"))
+ (major-minor (map string->number
+ (take (string-split (python-version python) #\.) 2)))
+ (<3.7? (match major-minor
+ ((major minor)
+ (or (< major 3) (and (= major 3) (< minor 7)))))))
+ (if <3.7?
+ ;; These versions don’t have the hash invalidation modes and do
+ ;; not produce reproducible bytecode files.
+ (format #t "Skipping bytecode compilation for Python version ~a < 3.7~%"
+ (python-version python))
+ (invoke "python" "-m" "compileall" "--invalidation-mode=unchecked-hash"
+ site-dir))))
+
+(define* (create-entrypoints #:key inputs outputs (configure-flags '()) #:allow-other-keys)
+ "Implement Entry Points Specification
+(https://packaging.python.org/specifications/entry-points/) by PyPa,
+which creates runnable scripts in bin/ from entry point specification
+file entry_points.txt. This is necessary, because wheels do not contain
+these binaries and installers are expected to create them."
+
+ (define (entry-points.txt->entry-points file)
+ "Specialized parser for Python configfile-like files, in particular
+entry_points.txt. Returns a list of console_script and gui_scripts
+entry points."
+ (call-with-input-file file
+ (lambda (in)
+ (let loop ((line (read-line in))
+ (inside #f)
+ (result '()))
+ (if (eof-object? line)
+ result
+ (let* ((group-match (string-match "^\\[(.+)\\]$" line))
+ (group-name (if group-match (match:substring group-match 1) #f))
+ (next-inside
+ (if (not group-name)
+ inside
+ (or
+ (string=? group-name "console_scripts")
+ (string=? group-name "gui_scripts"))))
+ (item-match (string-match "^([^ =]+)\\s*=\\s*([^:]+):(.+)$" line)))
+ (if (and inside item-match)
+ (loop (read-line in) next-inside (cons (list
+ (match:substring item-match 1)
+ (match:substring item-match 2)
+ (match:substring item-match 3))
+ result))
+ (loop (read-line in) next-inside result))))))))
+
+ (define (create-script path name module function)
+ "Create a Python script from an entry point’s NAME, MODULE and
+ FUNCTION and return write it to PATH/NAME."
+ (let ((interpreter (which "python"))
+ (file-path (string-append path "/" name)))
+ (format #t "Creating entry point for '~a.~a' at '~a'.~%" module function
+ file-path)
+ (call-with-output-file file-path
+ (lambda (port)
+ ;; Technically the script could also include search-paths,
+ ;; but having a generic 'wrap phases also handles manually
+ ;; written entry point scripts.
+ (format port "#!~a
+# Auto-generated entry point script.
+import sys
+import ~a as mod
+sys.exit (mod.~a ())~%" interpreter module function)))
+ (chmod file-path #o755)))
+
+ (let* ((site-dir (site-packages inputs outputs))
+ (out (assoc-ref outputs "out"))
+ (bin-dir (string-append out "/bin"))
+ (entry-point-files (find-files site-dir "^entry_points.txt$")))
+ (mkdir-p bin-dir)
+ (for-each
+ (lambda (f)
+ (for-each
+ (lambda (ep) (apply create-script (cons bin-dir ep)))
+ (entry-points.txt->entry-points f)))
+ entry-point-files)))
+
+(define* (set-SOURCE-DATE-EPOCH #:rest _)
+ "Set the 'SOURCE_DATE_EPOCH' environment variable. This is used by tools
+that incorporate timestamps as a way to tell them to use a fixed timestamp.
+See https://reproducible-builds.org/specs/source-date-epoch/."
+ ;; Use a post-1980 timestamp because the Zip format used in wheels do
+ ;; not support timestamps before 1980.
+ (setenv "SOURCE_DATE_EPOCH" "315619200"))
+
+(define %standard-phases
+ ;; The build phase only builds C extensions and copies the Python sources,
+ ;; while the install phase copies then byte-compiles the sources to the
+ ;; prefix directory. The check phase is moved after the installation phase
+ ;; to ease testing the built package.
+ (modify-phases python:%standard-phases
+ (replace 'set-SOURCE-DATE-EPOCH set-SOURCE-DATE-EPOCH)
+ (replace 'build build)
+ (replace 'install install)
+ (delete 'check)
+ ;; Must be before tests, so they can use installed packages’ entry points.
+ (add-before 'wrap 'create-entrypoints create-entrypoints)
+ (add-after 'wrap 'check check)
+ (add-before 'check 'compile-bytecode compile-bytecode)))
+
+(define* (pyproject-build #:key inputs (phases %standard-phases)
+ #:allow-other-keys #:rest args)
+ "Build the given Python package, applying all of PHASES in order."
+ (apply python:python-build #:inputs inputs #:phases phases args))
+
+;;; pyproject-build-system.scm ends here
From: Lars-Dominik Braun <lars@6xq.net> This is an experimental build system based on python-build-system that implements PEP 517-compliant builds. * doc/guix.texi (Build Systems): Add pyproject-build-system section. * doc/contributing.texi (Python Modules): Mention pyproject.toml and the PYTHON-TOOLCHAIN package, as well as differences to python-build-system. * guix/build-system/pyproject.scm, guix/build/pyproject-build-system.scm, gnu/packages/aux-files/python/sanity-check-next.py, gnu/packages/python-commencement.scm: New files. * Makefile.am (MODULES): Register the new build systems. * gnu/local.mk (GNU_SYSTEM_MODULES): Add python-commencement.scm. * gnu/packages/python.scm (python-sans-pip, python-sans-pip-wrapper): New variables. Co-authored-by: Marius Bakke <marius@gnu.org> --- Makefile.am | 2 + doc/contributing.texi | 36 +- doc/guix.texi | 29 ++ .../aux-files/python/sanity-check-next.py | 98 +++++ gnu/packages/python.scm | 52 ++- guix/build-system/pyproject.scm | 147 +++++++ guix/build/pyproject-build-system.scm | 367 ++++++++++++++++++ 7 files changed, 722 insertions(+), 9 deletions(-) create mode 100644 gnu/packages/aux-files/python/sanity-check-next.py create mode 100644 guix/build-system/pyproject.scm create mode 100644 guix/build/pyproject-build-system.scm