From patchwork Mon Oct 17 20:11:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Marius Bakke X-Patchwork-Id: 43483 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 4441D27BBEA; Mon, 17 Oct 2022 21:15:33 +0100 (BST) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-3.7 required=5.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS, URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 6BCF227BBE9 for ; Mon, 17 Oct 2022 21:15:29 +0100 (BST) Received: from localhost ([::1]:50394 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1okWWG-0000Fo-G5 for patchwork@mira.cbaines.net; Mon, 17 Oct 2022 16:15:28 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:48356) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1okWTz-0006xM-1v for guix-patches@gnu.org; Mon, 17 Oct 2022 16:13:07 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:51105) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1okWTy-0002Hw-Px for guix-patches@gnu.org; Mon, 17 Oct 2022 16:13:06 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1okWTy-0007PY-LB for guix-patches@gnu.org; Mon, 17 Oct 2022 16:13:06 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#58587] [PATCH 04/14] build-system: Add pyproject-build-system. Resent-From: Marius Bakke Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 17 Oct 2022 20:13:06 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 58587 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 58587@debbugs.gnu.org Cc: Lars-Dominik Braun Received: via spool by 58587-submit@debbugs.gnu.org id=B58587.166603755628302 (code B ref 58587); Mon, 17 Oct 2022 20:13:06 +0000 Received: (at 58587) by debbugs.gnu.org; 17 Oct 2022 20:12:36 +0000 Received: from localhost ([127.0.0.1]:50148 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1okWTR-0007ML-TD for submit@debbugs.gnu.org; Mon, 17 Oct 2022 16:12:35 -0400 Received: from eggs.gnu.org ([209.51.188.92]:39136) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1okWTA-0007Jw-8H for 58587@debbugs.gnu.org; Mon, 17 Oct 2022 16:12:24 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:51486) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1okWT5-0002Cb-2T; Mon, 17 Oct 2022 16:12:11 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:References:In-Reply-To:Date:Subject:To: From; bh=kvcBYIgshpDeQaAkYkS5TE6V0wR3OW8oHzvoMqXrvDw=; b=Tx9pnCZVvQMWAXS3Yaq3 7IwsNSkngQuUW//ZnNHSI8vSNxqE3zCbhI1OQnXLfwXDo7nA0x38JXGqlg7nO5FkJhlBb3dbbSW5t 3MJO5onyNXzL09kM3+gT1DMc78icLfmICia8OAOgIqPF5ybXaAZuWA1rVQphNna3dnGFONXHM9AQv YqtsHSlgWLRTnvZGE3HZtmAUL7znvTVp24DtCGnLuuXWFWIlHV5XngTRhoiH7Nk5k/69PjTemJvGX 7+kRtFks4590UpAqWbt1cET4Ly3zf90BKVFoqDoRfntfcCvDeuUO85zjZoOPMIYimmUJThBtoVjFz QtKCerrAspn6lA==; Received: from [84.211.203.64] (port=45530 helo=localhost) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1okWT4-0004zz-AT; Mon, 17 Oct 2022 16:12:10 -0400 From: Marius Bakke Date: Mon, 17 Oct 2022 22:11:51 +0200 Message-Id: <20221017201201.4808-4-marius@gnu.org> X-Mailer: git-send-email 2.38.0 In-Reply-To: <20221017201201.4808-1-marius@gnu.org> References: <20221017200624.4076-1-marius@gnu.org> <20221017201201.4808-1-marius@gnu.org> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches From: Lars-Dominik Braun This is an experimental build system based on python-build-system that implements PEP 517-compliant builds. * guix/build-system/pyproject.scm, guix/build/pyproject-build-system.scm, gnu/packages/aux-files/python/sanity-check-next.py, gnu/packages/python-commencement.scm: New files. * Makefile.am (MODULES): Register the new build systems. * gnu/local.mk (GNU_SYSTEM_MODULES): Add python-commencement.scm. * gnu/packages/python.scm (python-sans-pip, python-sans-pip-wrapper): New variables. Co-authored-by: Marius Bakke --- Makefile.am | 2 + gnu/local.mk | 1 + .../aux-files/python/sanity-check-next.py | 98 ++++ gnu/packages/python-commencement.scm | 64 +++ gnu/packages/python.scm | 13 + guix/build-system/pyproject.scm | 148 ++++++ guix/build/pyproject-build-system.scm | 460 ++++++++++++++++++ 7 files changed, 786 insertions(+) create mode 100644 gnu/packages/aux-files/python/sanity-check-next.py create mode 100644 gnu/packages/python-commencement.scm create mode 100644 guix/build-system/pyproject.scm create mode 100644 guix/build/pyproject-build-system.scm diff --git a/Makefile.am b/Makefile.am index 22dcc43f99..6ccb790c11 100644 --- a/Makefile.am +++ b/Makefile.am @@ -166,6 +166,7 @@ MODULES = \ guix/build-system/maven.scm \ guix/build-system/node.scm \ guix/build-system/perl.scm \ + guix/build-system/pyproject.scm \ guix/build-system/python.scm \ guix/build-system/renpy.scm \ guix/build-system/ocaml.scm \ @@ -222,6 +223,7 @@ MODULES = \ guix/build/minetest-build-system.scm \ guix/build/node-build-system.scm \ guix/build/perl-build-system.scm \ + guix/build/pyproject-build-system.scm \ guix/build/python-build-system.scm \ guix/build/ocaml-build-system.scm \ guix/build/qt-build-system.scm \ diff --git a/gnu/local.mk b/gnu/local.mk index bf598cec8b..b6e0d262be 100644 --- a/gnu/local.mk +++ b/gnu/local.mk @@ -511,6 +511,7 @@ GNU_SYSTEM_MODULES = \ %D%/packages/python-build.scm \ %D%/packages/python-check.scm \ %D%/packages/python-compression.scm \ + %D%/packages/python-commencement.scm \ %D%/packages/python-crypto.scm \ %D%/packages/python-science.scm \ %D%/packages/python-web.scm \ diff --git a/gnu/packages/aux-files/python/sanity-check-next.py b/gnu/packages/aux-files/python/sanity-check-next.py new file mode 100644 index 0000000000..891606f72b --- /dev/null +++ b/gnu/packages/aux-files/python/sanity-check-next.py @@ -0,0 +1,98 @@ +# -*- coding: utf-8 -*- + +# This version adds a small change to accomodate missing python-setuptools. +# Original patch by Lars-Dominik Braun in wip-python-pep517, commit +# 720dbe22d431262938be29dd9a9ddb78c44a99b3. +# --- sanity-check.py 2022-06-12 14:40:06.814337702 +0200 +# +++ sanity-check.py 2022-10-16 23:21:38.990651568 +0200 +# @@ -19,9 +19,13 @@ +# from __future__ import print_function # Python 2 support. +# import importlib +# -import pkg_resources +# import sys +# import traceback +# +try: +# + import pkg_resources +# +except ImportError: +# + print('Warning: Skipping, because python-setuptools are not available.') +# + sys.exit(0) + +# TODO: Merge with sanity-check.py in the next core-updates cycle. + +from __future__ import print_function # Python 2 support. +import importlib +import sys +import traceback +try: + import pkg_resources +except ImportError: + print('Warning: Skipping, because python-setuptools are not available.') + sys.exit(0) + +try: + from importlib.machinery import PathFinder +except ImportError: + PathFinder = None + +ret = 0 + +# Only check site-packages installed by this package, but not dependencies +# (which pkg_resources.working_set would include). Path supplied via argv. +ws = pkg_resources.find_distributions(sys.argv[1]) + +for dist in ws: + print('validating', repr(dist.project_name), dist.location) + try: + print('...checking requirements: ', end='') + req = str(dist.as_requirement()) + # dist.activate() is not enough to actually check requirements, we + # have to .require() it. + pkg_resources.require(req) + print('OK') + except Exception as e: + print('ERROR:', req, repr(e)) + ret = 1 + continue + + # Try to load top level modules. This should not have any side-effects. + try: + metalines = dist.get_metadata_lines('top_level.txt') + except (KeyError, EnvironmentError): + # distutils (i.e. #:use-setuptools? #f) will not install any metadata. + # This file is also missing for packages built using a PEP 517 builder + # such as poetry. + print('WARNING: cannot determine top-level modules') + continue + for name in metalines: + # Only available on Python 3. + if PathFinder and PathFinder.find_spec(name) is None: + # Ignore unavailable modules, often C modules, which were not + # installed at the top-level. Cannot use ModuleNotFoundError, + # because it is raised by failed imports too. + continue + try: + print('...trying to load module', name, end=': ') + importlib.import_module(name) + print('OK') + except Exception: + print('ERROR:') + traceback.print_exc(file=sys.stdout) + ret = 1 + + # Try to load entry points of console scripts too, making sure they + # work. They should be removed if they don't. Other groups may not be + # safe, as they can depend on optional packages. + for group, v in dist.get_entry_map().items(): + if group not in {'console_scripts', 'gui_scripts'}: + continue + for name, ep in v.items(): + try: + print('...trying to load endpoint', group, name, end=': ') + ep.load() + print('OK') + except Exception: + print('ERROR:') + traceback.print_exc(file=sys.stdout) + ret = 1 + +sys.exit(ret) diff --git a/gnu/packages/python-commencement.scm b/gnu/packages/python-commencement.scm new file mode 100644 index 0000000000..011ba2c38d --- /dev/null +++ b/gnu/packages/python-commencement.scm @@ -0,0 +1,64 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright © 2021 Lars-Dominik Braun +;;; Copyright © 2022 Marius Bakke +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see . + +(define-module (gnu packages python-commencement) + #:use-module ((guix licenses) #:prefix license:) + #:use-module (guix packages) + #:use-module (guix gexp) + #:use-module (guix build-system trivial) + #:use-module (gnu packages) + #:use-module (gnu packages python) + #:use-module (gnu packages python-build) + #:use-module (srfi srfi-1) + #:use-module (srfi srfi-26)) + +(define-public python-toolchain + (package + (name "python-toolchain") + (version (package-version python)) + (source #f) + (build-system trivial-build-system) + (arguments + (list #:modules '((guix build union)) + #:builder + #~(begin + (use-modules (ice-9 match) + (srfi srfi-1) + (guix build union)) + (union-build #$output + (filter-map (match-lambda + ((_ . directory) directory)) + %build-inputs))))) + (inputs + (list python-sans-pip-wrapper + python-pypa-build + python-pip + python-setuptools + python-wheel)) + (native-search-paths + (package-native-search-paths python)) + (search-paths + (package-search-paths python)) + (license (package-license python)) + (synopsis "Python toolchain") + (description + "Python toolchain including Python itself, setuptools and pip. Use this +package if you need a minimal Python toolchain instead of just the +interpreter.") + (home-page (package-home-page python)))) diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm index b54e393eb4..377fd1c282 100644 --- a/gnu/packages/python.scm +++ b/gnu/packages/python.scm @@ -677,6 +677,19 @@ (define* (wrap-python3 python (define-public python-wrapper (wrap-python3 python)) (define-public python-minimal-wrapper (wrap-python3 python-minimal)) +;; The Python used in pyproject-build-system. +(define-public python-sans-pip + (hidden-package + (package/inherit python + (arguments + (substitute-keyword-arguments (package-arguments python) + ((#:configure-flags flags #~()) + #~(append '("--with-ensurepip=no") + (delete "--with-ensurepip=install" #$flags)))))))) + +(define-public python-sans-pip-wrapper + (wrap-python3 python-sans-pip)) + (define-public micropython (package (name "micropython") diff --git a/guix/build-system/pyproject.scm b/guix/build-system/pyproject.scm new file mode 100644 index 0000000000..8e320529cc --- /dev/null +++ b/guix/build-system/pyproject.scm @@ -0,0 +1,148 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright © 2021 Lars-Dominik Braun +;;; Copyright © 2022 Marius Bakke +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see . + +(define-module (guix build-system pyproject) + #:use-module ((gnu packages) #:select (search-auxiliary-file)) + #:use-module (guix gexp) + #:use-module (guix store) + #:use-module (guix utils) + #:use-module (guix memoization) + #:use-module (guix gexp) + #:use-module (guix monads) + #:use-module (guix packages) + #:use-module (guix derivations) + #:use-module (guix search-paths) + #:use-module (guix build-system) + #:use-module (guix build-system gnu) + #:use-module (guix build-system python) + #:use-module (ice-9 match) + #:use-module (srfi srfi-1) + #:use-module (srfi srfi-26) + #:export (%pyproject-build-system-modules + default-python + pyproject-build + pyproject-build-system)) + +;; Commentary: +;; +;; Standard build procedure for Python packages using 'pyproject.toml'. +;; This is implemented as an extension of 'python-build-system'. +;; +;; Code: + +(define %pyproject-build-system-modules + ;; Build-side modules imported by default. + `((guix build pyproject-build-system) + (guix build json) + ,@%python-build-system-modules + ,@%gnu-build-system-modules)) + +(define (default-python) + "Return the default Python package." + ;; Lazily resolve the binding to avoid a circular dependency. + (let ((python (resolve-interface '(gnu packages python-commencement)))) + (module-ref python 'python-toolchain))) + +(define sanity-check.py + ;; TODO: Merge with sanity-check.py in the next rebuild cycle. + (search-auxiliary-file "python/sanity-check-next.py")) + +(define* (lower name + #:key source inputs native-inputs outputs system target + (python (default-python)) + #:allow-other-keys + #:rest arguments) + "Return a bag for NAME." + (define private-keywords + '(#:target #:python #:inputs #:native-inputs)) + + (and (not target) ;XXX: no cross-compilation + (bag + (name name) + (system system) + (host-inputs `(,@(if source + `(("source" ,source)) + '()) + ,@inputs + + ;; Keep the standard inputs of 'gnu-build-system'. + ,@(standard-packages))) + (build-inputs `(("python" ,python) + ("sanity-check.py" ,(local-file sanity-check.py)) + ,@native-inputs)) + (outputs (append outputs '(wheel))) + (build pyproject-build) + (arguments (strip-keyword-arguments private-keywords arguments))))) + +(define* (pyproject-build name inputs + #:key source + (tests? #t) + (configure-flags ''()) + (build-backend #f) + (test-backend #f) + (test-flags #f) + (phases '%standard-phases) + (outputs '("out" "wheel")) + (search-paths '()) + (system (%current-system)) + (guile #f) + (imported-modules %pyproject-build-system-modules) + (modules '((guix build pyproject-build-system) + (guix build utils)))) + "Build SOURCE using PYTHON, and with INPUTS." + (define build + (with-imported-modules imported-modules + #~(begin + (use-modules #$@(sexp->gexp modules)) + + #$(with-build-variables inputs outputs + #~(pyproject-build + #:name #$name + #:source #+source + #:configure-flags #$configure-flags + #:system #$system + #:build-backend #$build-backend + #:test-backend #$test-backend + #:test-flags #$test-flags + #:tests? #$tests? + #:phases #$(if (pair? phases) + (sexp->gexp phases) + phases) + #:outputs %outputs + #:search-paths '#$(sexp->gexp + (map search-path-specification->sexp + search-paths)) + #:inputs %build-inputs))))) + + + (mlet %store-monad ((guile (package->derivation (or guile (default-guile)) + system #:graft? #f))) + (gexp->derivation name build + #:system system + #:graft? #f ;consistent with 'gnu-build' + #:target #f + #:guile-for-build guile))) + +(define pyproject-build-system + (build-system + (name 'pyproject) + (description "The PEP517-compliant Python build system") + (lower lower))) + +;;; pyproject.scm ends here diff --git a/guix/build/pyproject-build-system.scm b/guix/build/pyproject-build-system.scm new file mode 100644 index 0000000000..141d4d849f --- /dev/null +++ b/guix/build/pyproject-build-system.scm @@ -0,0 +1,460 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright © 2013, 2015, 2016, 2018, 2019 Ludovic Courtès +;;; Copyright © 2013 Andreas Enge +;;; Copyright © 2013 Nikita Karetnikov +;;; Copyright © 2015, 2018 Mark H Weaver +;;; Copyright © 2016 Hartmut Goebel +;;; Copyright © 2018 Ricardo Wurmus +;;; Copyright © 2018 Arun Isaac +;;; Copyright © 2019, 2020, 2021 Maxim Cournoyer +;;; Copyright © 2020 Jakub Kądziołka +;;; Copyright © 2020 Efraim Flashner +;;; Copyright © 2021 Maxime Devos +;;; +;;; The above copyright is taken from python-build-system. Contributions +;;; made only to this module are listed below: +;;; Copyright © 2021 Lars-Dominik Braun +;;; Copyright © 2022 Marius Bakke +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see . + +(define-module (guix build pyproject-build-system) + #:use-module ((guix build gnu-build-system) #:prefix gnu:) + #:use-module (guix build utils) + #:use-module (guix build json) + #:use-module (ice-9 match) + #:use-module (ice-9 ftw) + #:use-module (ice-9 format) + #:use-module (ice-9 rdelim) + #:use-module (ice-9 regex) + #:use-module (srfi srfi-1) + #:use-module (srfi srfi-26) + #:use-module (srfi srfi-34) + #:use-module (srfi srfi-35) + #:export (%standard-phases + add-installed-pythonpath + site-packages + python-version + pyproject-build)) + +;; Commentary: +;; +;; PEP 517-compatible build system for Python packages. +;; +;; PEP 517 mandates the use of a TOML file called pyproject.toml at the +;; project root, describing build and runtime dependencies, as well as the +;; build system, which can be different from setuptools. This module uses +;; that file to extract the build system used and call its wheel-building +;; entry point build_wheel (see 'build). setuptools’ wheel builder is +;; used as a fallback if either no pyproject.toml exists or it does not +;; declare a build-system. It supports config_settings through the +;; standard #:configure-flags argument. +;; +;; This wheel, which is just a ZIP file with a file structure defined +;; by PEP 427 (https://www.python.org/dev/peps/pep-0427/), is then unpacked +;; and its contents are moved to the appropriate locations in 'install. +;; +;; Then entry points, as defined by the PyPa Entry Point Specification +;; (https://packaging.python.org/specifications/entry-points/) are read +;; from a file called entry_points.txt in the package’s site-packages +;; subdirectory and scripts are written to bin/. These are not part of a +;; wheel and expected to be created by the installing utility. +;; +;; Caveats: +;; - There is no support for in-tree build backends. + +;; Base error type. +(define-condition-type &python-build-error &error + python-build-error?) + +;; Raised when 'check cannot find a valid test system in the inputs. +(define-condition-type &test-system-not-found &python-build-error + test-system-not-found?) + +;; Raised when multiple wheels are created by 'build. +(define-condition-type &cannot-extract-multiple-wheels &python-build-error + cannot-extract-multiple-wheels?) + +;; Raised, when no wheel has been built by the build system. +(define-condition-type &no-wheels-built &python-build-error + no-wheels-built?) + +(define* (sanity-check #:key tests? inputs outputs #:allow-other-keys) + "Ensure packages depending on this package via setuptools work properly, +their advertised endpoints work and their top level modules are importable +without errors." + (let ((sanity-check.py (assoc-ref inputs "sanity-check.py"))) + ;; Make sure the working directory is empty (i.e. no Python modules in it) + (with-directory-excursion "/tmp" + (invoke "python" sanity-check.py (site-packages inputs outputs))))) + +(define* (build #:key outputs build-backend configure-flags #:allow-other-keys) + "Build a given Python package." + + (define (pyproject.toml->build-backend file) + "Look up the build backend in a pyproject.toml file." + (call-with-input-file file + (lambda (in) + (let loop ((line (read-line in 'concat))) + (if (eof-object? line) + #f + (let ((m (string-match "build-backend = [\"'](.+)[\"']" line))) + (if m (match:substring m 1) + (loop (read-line in 'concat))))))))) + + (let* ((wheel-output (assoc-ref outputs "wheel")) + (wheel-dir (if wheel-output wheel-output "dist")) + ;; There is no easy way to get data from Guile into Python via + ;; s-expressions, but we have JSON serialization already, which Python + ;; also supports out-of-the-box. + (config-settings (call-with-output-string (cut write-json configure-flags <>))) + ;; python-setuptools’ default backend supports setup.py *and* + ;; pyproject.toml. Allow overriding this automatic detection via + ;; build-backend. + (auto-build-backend (if (file-exists? "pyproject.toml") + (pyproject.toml->build-backend "pyproject.toml") + #f)) + ;; Use build system detection here and not in importer, because a) we + ;; have alot of legacy packages and b) the importer cannot update arbitrary + ;; fields in case a package switches its build system. + (use-build-backend (or + build-backend + auto-build-backend + "setuptools.build_meta"))) + (format #t "Using '~a' to build wheels, auto-detected '~a', override '~a'.~%" + use-build-backend auto-build-backend build-backend) + (mkdir-p wheel-dir) + ;; Call the PEP 517 build function, which drops a .whl into wheel-dir. + (invoke "python" "-c" "import sys, importlib, json +config_settings = json.loads (sys.argv[3]) +builder = importlib.import_module(sys.argv[1]) +builder.build_wheel(sys.argv[2], config_settings=config_settings)" + use-build-backend wheel-dir config-settings))) + +(define* (check #:key inputs outputs tests? test-backend test-flags #:allow-other-keys) + "Run the test suite of a given Python package." + (if tests? + ;; Unfortunately with PEP 517 there is no common method to specify test + ;; systems. Guess test system based on inputs instead. + (let* ((pytest (which "pytest")) + (nosetests (which "nosetests")) + (nose2 (which "nose2")) + (have-setup-py (file-exists? "setup.py")) + (use-test-backend + (or + test-backend + ;; Prefer pytest + (if pytest 'pytest #f) + (if nosetests 'nose #f) + (if nose2 'nose2 #f) + ;; But fall back to setup.py, which should work for most + ;; packages. XXX: would be nice not to depend on setup.py here? fails + ;; more often than not to find any tests at all. Maybe we can run + ;; `python -m unittest`? + (if have-setup-py 'setup.py #f)))) + (format #t "Using ~a~%" use-test-backend) + (match use-test-backend + ('pytest + (apply invoke (cons pytest (or test-flags '("-vv"))))) + ('nose + (apply invoke (cons nosetests (or test-flags '("-v"))))) + ('nose2 + (apply invoke (cons nose2 (or test-flags '("-v" "--pretty-assert"))))) + ('setup.py + (apply invoke (append '("python" "setup.py") (or test-flags '("test" "-v"))))) + ;; The developer should explicitly disable tests in this case. + (else (raise (condition (&test-system-not-found)))))) + (format #t "test suite not run~%"))) + +(define (python-version python) + (let* ((version (last (string-split python #\-))) + (components (string-split version #\.)) + (major+minor (take components 2))) + (string-join major+minor "."))) + +(define (python-output outputs) + "Return the path of the python output, if there is one, or fall-back to out." + (or (assoc-ref outputs "python") + (assoc-ref outputs "out"))) + +(define (site-packages inputs outputs) + "Return the path of the current output's Python site-package." + (let* ((out (python-output outputs)) + (python (assoc-ref inputs "python"))) + (string-append out "/lib/python" (python-version python) "/site-packages"))) + +(define (add-installed-pythonpath inputs outputs) + "Prepend the site-package of OUTPUT to GUIX_PYTHONPATH. This is useful when +running checks after installing the package." + (setenv "GUIX_PYTHONPATH" (string-append (site-packages inputs outputs) ":" + (getenv "GUIX_PYTHONPATH")))) + +(define* (add-install-to-pythonpath #:key inputs outputs #:allow-other-keys) + "A phase that just wraps the 'add-installed-pythonpath' procedure." + (add-installed-pythonpath inputs outputs)) + +(define* (add-install-to-path #:key outputs #:allow-other-keys) + "Adding Python scripts to PATH is also often useful in tests." + (setenv "PATH" (string-append (assoc-ref outputs "out") + "/bin:" + (getenv "PATH")))) + +(define* (install #:key inputs outputs (configure-flags '()) #:allow-other-keys) + "Install a wheel file according to PEP 427" + ;; See https://www.python.org/dev/peps/pep-0427/#installing-a-wheel-distribution-1-0-py32-none-any-whl + (let* ((site-dir (site-packages inputs outputs)) + (python (assoc-ref inputs "python")) + (out (assoc-ref outputs "out"))) + (define (extract file) + "Extract wheel (ZIP file) into site-packages directory" + ;; Use Python’s zipfile to avoid extra dependency + (invoke "python" "-m" "zipfile" "-e" file site-dir)) + + (define python-hashbang + (string-append "#!" python "/bin/python")) + + (define* (merge-directories source destination #:optional (post-move #f)) + "Move all files in SOURCE into DESTINATION, merging the two directories." + (format #t "Merging directory ~a into ~a~%" source destination) + (for-each + (lambda (file) + (format #t "~a/~a -> ~a/~a~%" source file destination file) + (mkdir-p destination) + (rename-file + (string-append source "/" file) + (string-append destination "/" file)) + (when post-move + (post-move file))) + (scandir source (negate (cut member <> '("." ".."))))) + (rmdir source)) + + (define (expand-data-directory directory) + "Move files from all .data subdirectories to their respective +destinations." + ;; Python’s distutils.command.install defines this mapping from source to + ;; destination mapping. + (let ((source (string-append directory "/scripts")) + (destination (string-append out "/bin"))) + (when (file-exists? source) + (merge-directories + source + destination + (lambda (f) + (let ((dest-path (string-append destination "/" f))) + (chmod dest-path #o755) + (substitute* dest-path (("#!python") python-hashbang))))))) + ;; data can create arbitrary directory structures. Most commonly + ;; it is used for share/. + (let ((source (string-append directory "/data")) + (destination out)) + (when (file-exists? source) + (merge-directories source destination))) + (let* ((distribution (car (string-split (basename directory) #\-))) + (source (string-append directory "/headers")) + (destination (string-append out "/include/python" (python-version python) "/" distribution))) + (when (file-exists? source) + (merge-directories source destination)))) + + (define (list-directories base predicate) + ;; Cannot use find-files here, because it’s recursive. + (scandir + base + (lambda (name) + (let ((stat (lstat (string-append base "/" name)))) + (and + (not (member name '("." ".."))) + (eq? (stat:type stat) 'directory) + (predicate name stat)))))) + + (let* ((wheel-output (assoc-ref outputs "wheel")) + (wheel-dir (if wheel-output wheel-output "dist")) + (wheels (map (cut string-append wheel-dir "/" <>) + (scandir wheel-dir (cut string-suffix? ".whl" <>))))) + (cond + ((> (length wheels) 1) ; This code does not support multiple wheels + ; yet, because their outputs would have to be + ; merged properly. + (raise (condition (&cannot-extract-multiple-wheels)))) + ((= (length wheels) 0) + (raise (condition (&no-wheels-built))))) + (for-each extract wheels)) + (let ((datadirs (map + (cut string-append site-dir "/" <>) + (list-directories site-dir (file-name-predicate "\\.data$"))))) + (for-each (lambda (directory) + (expand-data-directory directory) + (rmdir directory)) + datadirs)))) + +(define* (compile-bytecode #:key inputs outputs (configure-flags '()) #:allow-other-keys) + "Compile installed byte-code in site-packages." + (let ((site-dir (site-packages inputs outputs))) + (invoke "python" "-m" "compileall" "--invalidation-mode=unchecked-hash" + site-dir))) + +(define* (create-entrypoints #:key inputs outputs (configure-flags '()) #:allow-other-keys) + "Implement Entry Points Specification +(https://packaging.python.org/specifications/entry-points/) by PyPa, +which creates runnable scripts in bin/ from entry point specification +file entry_points.txt. This is necessary, because wheels do not contain +these binaries and installers are expected to create them." + + (define (entry-points.txt->entry-points file) + "Specialized parser for Python configfile-like files, in particular +entry_points.txt. Returns a list of console_script and gui_scripts +entry points." + (call-with-input-file file + (lambda (in) + (let loop ((line (read-line in)) + (inside #f) + (result '())) + (if (eof-object? line) + result + (let* ((group-match (string-match "^\\[(.+)\\]$" line)) + (group-name (if group-match (match:substring group-match 1) #f)) + (next-inside + (if (not group-name) + inside + (or + (string=? group-name "console_scripts") + (string=? group-name "gui_scripts")))) + (item-match (string-match "^([^ =]+)\\s*=\\s*([^:]+):(.+)$" line))) + (if (and inside item-match) + (loop (read-line in) next-inside (cons (list + (match:substring item-match 1) + (match:substring item-match 2) + (match:substring item-match 3)) + result)) + (loop (read-line in) next-inside result)))))))) + + (define (create-script path name module function) + "Create a Python script from an entry point’s NAME, MODULE and + FUNCTION and return write it to PATH/NAME." + (let ((interpreter (which "python")) + (file-path (string-append path "/" name))) + (format #t "Creating entry point for '~a.~a' at '~a'.~%" module function + file-path) + (call-with-output-file file-path + (lambda (port) + ;; Technically the script could also include search-paths, + ;; but having a generic 'wrap phases also handles manually + ;; written entry point scripts. + (format port "#!~a +# Auto-generated entry point script. +import sys +import ~a as mod +sys.exit (mod.~a ())~%" interpreter module function))) + (chmod file-path #o755))) + + (let* ((site-dir (site-packages inputs outputs)) + (out (assoc-ref outputs "out")) + (bin-dir (string-append out "/bin")) + (entry-point-files (find-files site-dir "^entry_points.txt$"))) + (mkdir-p bin-dir) + (for-each + (lambda (f) + (for-each + (lambda (ep) (apply create-script (cons bin-dir ep))) + (entry-points.txt->entry-points f))) + entry-point-files))) + +(define* (wrap #:key inputs outputs search-paths #:allow-other-keys) + (define (list-of-files dir) + (find-files dir (lambda (file stat) + (and (eq? 'regular (stat:type stat)) + (not (wrapped-program? file)))))) + + (define bindirs + (append-map (match-lambda + ((_ . dir) + (list (string-append dir "/bin") + (string-append dir "/sbin")))) + outputs)) + + ;; Do not require "bash" to be present in the package inputs + ;; even when there is nothing to wrap. + ;; Also, calculate (sh) only once to prevent some I/O. + (define %sh (delay (search-input-file inputs "bin/bash"))) + (define (sh) (force %sh)) + + (let* ((var `("GUIX_PYTHONPATH" prefix + ,(search-path-as-string->list + (or (getenv "GUIX_PYTHONPATH") ""))))) + (for-each (lambda (dir) + (let ((files (list-of-files dir))) + (for-each (cut wrap-program <> #:sh (sh) var) + files))) + bindirs))) + +(define* (set-SOURCE-DATE-EPOCH #:rest _) + "Set the 'SOURCE_DATE_EPOCH' environment variable. This is used by tools +that incorporate timestamps as a way to tell them to use a fixed timestamp. +See https://reproducible-builds.org/specs/source-date-epoch/." + ;; Use a post-1980 timestamp because the Zip format used in wheels do + ;; not support timestamps before 1980. + (setenv "SOURCE_DATE_EPOCH" "315619200")) + +(define* (enable-bytecode-determinism #:rest _) + "Improve determinism of pyc files." + ;; Use deterministic hashes for strings, bytes, and datetime objects. + (setenv "PYTHONHASHSEED" "0") + ;; Prevent Python from creating .pyc files when loading modules (such as + ;; when running a test suite). + (setenv "PYTHONDONTWRITEBYTECODE" "1")) + +(define* (ensure-no-cythonized-files #:rest _) + "Check the source code for @code{.c} files which may have been pre-generated +by Cython." + (for-each + (lambda (file) + (let ((generated-file + (string-append (string-drop-right file 3) "c"))) + (when (file-exists? generated-file) + (format #t "Possible Cythonized file found: ~a~%" generated-file)))) + (find-files "." "\\.pyx$"))) + +(define %standard-phases + ;; The build phase only builds C extensions and copies the Python sources, + ;; while the install phase copies then byte-compiles the sources to the + ;; prefix directory. The check phase is moved after the installation phase + ;; to ease testing the built package. + (modify-phases gnu:%standard-phases + (add-after 'unpack 'enable-bytecode-determinism + enable-bytecode-determinism) + (add-after 'enable-bytecode-determinism 'ensure-no-cythonized-files + ensure-no-cythonized-files) + (replace 'set-SOURCE-DATE-EPOCH set-SOURCE-DATE-EPOCH) + (delete 'bootstrap) + (delete 'configure) ;not needed + (replace 'build build) + (delete 'check) ;moved after the install phase + (replace 'install install) + (add-after 'install 'add-install-to-pythonpath add-install-to-pythonpath) + (add-after 'add-install-to-pythonpath 'add-install-to-path + add-install-to-path) + (add-after 'add-install-to-path 'wrap wrap) + ;; must be before tests, so they can use installed packages’ entry points. + (add-before 'wrap 'create-entrypoints create-entrypoints) + (add-after 'wrap 'check check) + (add-after 'check 'sanity-check sanity-check) + (add-before 'check 'compile-bytecode compile-bytecode))) + +(define* (pyproject-build #:key inputs (phases %standard-phases) + #:allow-other-keys #:rest args) + "Build the given Python package, applying all of PHASES in order." + (apply gnu:gnu-build #:inputs inputs #:phases phases args)) + +;;; pyproject-build-system.scm ends here