[bug#46848,PATCHES,core-updates] PEP 517 python-build-system

Hi,

I rebased my changes on top of the current core-updates HEAD.

Cheers,
Lars

Hi,

I’m working on version 3 of this patchset, removing the dependency
on python-pypa-build, which simplifies bootstrapping. A rough version
is available at

https://github.com/PromyLOPh/guix/commits/work-python-build-pep517

Cheers,
Lars

Hi everyone,

I pushed the 3rd version of my new PEP 517-based python-build-system
into wip-python-pep517. I removed the dependency on python-pypa-build
for building packages and instead implemented building with a single
python invokation. This simplifies bootstrapping Python. The 'install
phase also learned how to create scripts in bin/ from an entry points
specification.

Currently 600 of 2212 packages using python-build-system are failing to
build. That’s alot, but most failures so far were related to testsuites,
which were not enabled correctly previously because 'check was not
replaced manually. And some fail because upstream does not separate
sources and tests and a name clash occurs. This also happens when C
extensions are build and there does not seem to be a workaround.

This number does not include Python 2 packages, which will probably
break with this new build system. Since Python 2 is EOL anyway I suggest
to entirely remove Python 2 support before merging this changeset. After
merging we should upgrade the entire Python ecosystem, because alot of
packages are already years old.

Cheers,
Lars

PS: Maxim: This is your ping for a review (via #52269). Thanks :)

Lars-Dominik Braun <lars@6xq.net> skriver:

> I pushed the 3rd version of my new PEP 517-based python-build-system
> into wip-python-pep517. I removed the dependency on python-pypa-build
> for building packages and instead implemented building with a single
> python invokation. This simplifies bootstrapping Python. The 'install
> phase also learned how to create scripts in bin/ from an entry points
> specification.

This is great, thank you.

> Currently 600 of 2212 packages using python-build-system are failing to
> build. That’s alot, but most failures so far were related to testsuites,
> which were not enabled correctly previously because 'check was not
> replaced manually. And some fail because upstream does not separate
> sources and tests and a name clash occurs. This also happens when C
> extensions are build and there does not seem to be a workaround.

Can you elaborate on the name clash issue?

> This number does not include Python 2 packages, which will probably
> break with this new build system. Since Python 2 is EOL anyway I suggest
> to entirely remove Python 2 support before merging this changeset. After
> merging we should upgrade the entire Python ecosystem, because alot of
> packages are already years old.

Unfortunately we need Python 2 for some time still.  It is used to
bootstrap old versions of GHC, Node, and probably other things.

Do you think it is feasible to provide this as a new build system, say
pep517-build-system, during a transitional period?  Then we can a) start
using it right away for the increasing amount of packages that lack a
setup.py; and b) flesh out most bugs before eventually merging it back
(possibly piecemeal) into python-build-system.

I've had a cursory look and it looks good overall.  A few comments:

* Zipping a wheel just to unpack it afterwards is weird, but there seems
  to be no way around it.
* I also think trying "python setup.py test" is unnecessary.
* It would be nice to support the "no tests" case without having to add
  explicit #:tests? everywhere.  Perhaps along the lines of...

  (match use-test-backend
   ('pytest (apply invoke ...))
   [...]
   (_ (match (find-files "." "^test_.*\\.py$")
        (() (format #t "no tests found~%"))
        (_ (apply invoke `("python" "-m" "unittest" ,@test-flags))))))

WDYT?

Great work, and apologies for not chiming in earlier!

Hi Marius,

> Can you elaborate on the name clash issue?
the problem seems to be that most packages do not put the source code
into a subdirectory in the sdist. I.e. package foobar will have a foobar/
directory, instead of src/foobar. Thus Python tries to load the module
foobar.barbaz from that directory, instead of the store. Thus it’ll
also look there for C extensions and fail. I don’t know why it does
that, even when *not* using `python -m` – according to the documentation
it should not. If anyone has any pointers I’d like to have a look again.

> Unfortunately we need Python 2 for some time still.  It is used to
> bootstrap old versions of GHC, Node, and probably other things.
But as far as I see these specific examples do not use
python-build-system. Only if some package using `#:python python-2` is
required during bootstrapping, then we’d need a way to support Python
2, right?

> Do you think it is feasible to provide this as a new build system, say
> pep517-build-system, during a transitional period?  Then we can a) start
> using it right away for the increasing amount of packages that lack a
> setup.py; and b) flesh out most bugs before eventually merging it back
> (possibly piecemeal) into python-build-system.
Actually, I think that’s a good idea. I tried, but I cannot fix all
packages broken by this change on my own, so smoothing the transition
could help and – as you said – we could catch bugs early on.

I could prepare a patch, if there is interest in this path.

> * Zipping a wheel just to unpack it afterwards is weird, but there seems
>   to be no way around it.
Indeed, that’s how PEP 517 works, which always builds a wheel
(i.e. ZIP file).

> * I also think trying "python setup.py test" is unnecessary.
It still works quite often, although its usefulness will decrease in
the future I guess. Another problem I see is that this command will not
fail if there are no tests.

> * It would be nice to support the "no tests" case without having to add
>   explicit #:tests? everywhere.  Perhaps along the lines of...
My idea was to force packagers to make and explain this decision
explicitly. If you don’t run tests, you have to add `#:tests? #f`
and leave a comment why they are disabled. I do see this could become
a hassle with packages that simply don’t have any tests. But my hope
is that it increases package quality.
 
>         (_ (apply invoke `("python" "-m" "unittest" ,@test-flags))))))
If I remember correctly I tried this and it did not work for some
reason. I’ll have a look again.

Cheers,
Lars

Lars-Dominik Braun <lars@6xq.net> skriver:

> Hi Marius,
>
>> Can you elaborate on the name clash issue?
> the problem seems to be that most packages do not put the source code
> into a subdirectory in the sdist. I.e. package foobar will have a foobar/
> directory, instead of src/foobar. Thus Python tries to load the module
> foobar.barbaz from that directory, instead of the store. Thus it’ll
> also look there for C extensions and fail. I don’t know why it does
> that, even when *not* using `python -m` – according to the documentation
> it should not. If anyone has any pointers I’d like to have a look again.

Perhaps it could be worked around by building in a separate directory,
i.e. ./build/lib, like setuptools does?  And ensure that the original
source directory does not end up on {,GUIX_}PYTHONPATH.

(I don't really understand the problem, just throwing ideas around.)

>> Unfortunately we need Python 2 for some time still.  It is used to
>> bootstrap old versions of GHC, Node, and probably other things.
> But as far as I see these specific examples do not use
> python-build-system. Only if some package using `#:python python-2` is
> required during bootstrapping, then we’d need a way to support Python
> 2, right?

Right.  If we don't need any Python 2 modules it should be fine to
remove support from the build system.

>> Do you think it is feasible to provide this as a new build system, say
>> pep517-build-system, during a transitional period?  Then we can a) start
>> using it right away for the increasing amount of packages that lack a
>> setup.py; and b) flesh out most bugs before eventually merging it back
>> (possibly piecemeal) into python-build-system.
> Actually, I think that’s a good idea. I tried, but I cannot fix all
> packages broken by this change on my own, so smoothing the transition
> could help and – as you said – we could catch bugs early on.
>
> I could prepare a patch, if there is interest in this path.

It would lower the bar significantly for testing and contributing, so I
would appreciate it.  :-)

>> * I also think trying "python setup.py test" is unnecessary.
> It still works quite often, although its usefulness will decrease in
> the future I guess. Another problem I see is that this command will not
> fail if there are no tests.
>
>> * It would be nice to support the "no tests" case without having to add
>>   explicit #:tests? everywhere.  Perhaps along the lines of...
> My idea was to force packagers to make and explain this decision
> explicitly. If you don’t run tests, you have to add `#:tests? #f`
> and leave a comment why they are disabled. I do see this could become
> a hassle with packages that simply don’t have any tests. But my hope
> is that it increases package quality.

My main concern is end-user experience when they just want to get some
random library working on their local channel.  But it's arguably
something that can be solved on the importer level and not a strong
opinion.

Thinking about importers, perhaps they could start emitting git origins
when possible, as there is a trend to strip tests before uploading to
PyPI.

>>         (_ (apply invoke `("python" "-m" "unittest" ,@test-flags))))))
> If I remember correctly I tried this and it did not work for some
> reason. I’ll have a look again.

Eh, it should be (apply invoke "python" "-m" "unittest" test-flags), but
you probably got that.  :-P

Thanks!

Hi Lars-Dominik,

Lars-Dominik Braun <lars@6xq.net> writes:

When you judge the branch ready to merge, could you please send a subset
of the patches (at least the ones touching the python-build-system
directly) to this issue (marked as v2 -- git send-email -v2) so that
they can be more easily commented?

Thanks,

Maxim

Hi Maxim,

> When you judge the branch ready to merge, could you please send a subset
> of the patches (at least the ones touching the python-build-system
> directly) to this issue (marked as v2 -- git send-email -v2) so that
> they can be more easily commented?

as soon as time permits I can do that. Which route do we want to
take? Replace python-build-system entirely in one big merge or add a
new one and slowly migrate?

Cheers,
Lars

Hi Lars,

Lars-Dominik Braun <lars@6xq.net> writes:

> Hi Maxim,
>
>> When you judge the branch ready to merge, could you please send a subset
>> of the patches (at least the ones touching the python-build-system
>> directly) to this issue (marked as v2 -- git send-email -v2) so that
>> they can be more easily commented?
>
> as soon as time permits I can do that. Which route do we want to
> take? Replace python-build-system entirely in one big merge or add a
> new one and slowly migrate?

After the Berlin can be rebooted onto its new file system, I expect it
should allow us to keep the build farm much busier than in the past give
us the headroom to experiment with a PEP 517-focused world rebuilding
branch.

I'd aim to have the build system completely replaced; unless your
experience suggests it's going to take multiple weeks to fix iron out
the kinks?

Thanks,

Maxim

Hello!

With the pyproject build system now in master, can we close this issue?

Hi,

> With the pyproject build system now in master, can we close this issue?
yes, I think so.

Cheers,
Lars