[bug#53878,07/11] gnu: chez-scheme: Explicitly package bootstrap bootfiles.

This might seem a bit silly in isolation, but it makes the structure of
the upstream Chez Scheme package the same as for the Racket variant, it
sets things up for (one day, hopefully) actually being able to bootstrap
the upstream Chez Scheme bootfiles, and it may be useful for
cross-compilation and adding support for architectures without pre-built
bootfiles from upstream.

* gnu/packages/chez-and-racket-bootstrap.scm
(chez-scheme-bootstrap-bootfiles): New variable.
(chez-scheme)[native-inputs]: Add it.
[arguments]: Add new phase 'unpack-bootfiles'.
[version, source, home-page]: Derive from 'chez-scheme-bootstrap-bootfiles'.
---
 gnu/packages/chez-and-racket-bootstrap.scm | 57 ++++++++++++++++++++--
 1 file changed, 52 insertions(+), 5 deletions(-)
  

Am Sonntag, dem 13.02.2022 um 16:51 -0500 schrieb Philip McGrath:
> This might seem a bit silly in isolation, but it makes the structure
> of the upstream Chez Scheme package the same as for the Racket
> variant, it sets things up for (one day, hopefully) actually being
> able to bootstrap the upstream Chez Scheme bootfiles, and it may be
> useful for cross-compilation and adding support for architectures
> without pre-built bootfiles from upstream.
> 
> * gnu/packages/chez-and-racket-bootstrap.scm
> (chez-scheme-bootstrap-bootfiles): New variable.
> (chez-scheme)[native-inputs]: Add it.
> [arguments]: Add new phase 'unpack-bootfiles'.
> [version, source, home-page]: Derive from 'chez-scheme-bootstrap-
> bootfiles'.
> ---
While having chez-scheme-bootstrap-bootfiles (silly name) does make
some kind of sense, making chez-scheme inherit from it does not.  Given
that we don't have a chez-scheme bootstrap tower at hand, you should
probably make (chez-scheme-bootstrap) a procedure which takes chez-
scheme's origin as argument and returns the full package.


Also, while technically a violation of the DRY principle, you are
allowed to type out the homepage multiple times.
  

Hi,

On 2/14/22 09:54, Liliana Marie Prikler wrote:
> Am Sonntag, dem 13.02.2022 um 16:51 -0500 schrieb Philip McGrath:
>> This might seem a bit silly in isolation, but it makes the structure
>> of the upstream Chez Scheme package the same as for the Racket
>> variant, it sets things up for (one day, hopefully) actually being
>> able to bootstrap the upstream Chez Scheme bootfiles, and it may be
>> useful for cross-compilation and adding support for architectures
>> without pre-built bootfiles from upstream.
>>
>> * gnu/packages/chez-and-racket-bootstrap.scm
>> (chez-scheme-bootstrap-bootfiles): New variable.
>> (chez-scheme)[native-inputs]: Add it.
>> [arguments]: Add new phase 'unpack-bootfiles'.
>> [version, source, home-page]: Derive from 'chez-scheme-bootstrap-
>> bootfiles'.
>> ---
> While having chez-scheme-bootstrap-bootfiles (silly name) does make
> some kind of sense, making chez-scheme inherit from it does not.  Given
> that we don't have a chez-scheme bootstrap tower at hand, you should
> probably make (chez-scheme-bootstrap) a procedure which takes chez-
> scheme's origin as argument and returns the full package.
> 
Making a function is an interesting idea, but I'm not sure I'm quite 
picturing what you have in mind. I will see if I can figure out 
something that seems reasonable as I revise this series, if I don't hear 
from you before then.

One reason I like making the bootfiles a package is that a set of 
bootfiles includes artifacts in addition to the bootfiles themselves, 
such as generated C headers describing the layout of Scheme objects in 
memory, some of which are not included as part of an installed Chez 
Scheme. For example, imagine someone wants to run Chez Scheme on 
FreeBSD: upstream does not distribute BSD bootfiles, so they must be 
cross-compiled. Even though Guix doesn't have a C toolchain for FreeBSD 
(AFAIK), Guix could be used to reproducibly build the needed bootfiles 
and pack a "source" tarball to be used on a FreeBSD build machine.

Also, the process for building bootfiles is largely orthogonal to 
building the actual `scheme` executable, and it seems like eventually it 
may be useful to be able to override options separately.

There may be other ways to address these sorts of things, and it's true 
that a lot of this has more to do with my intuition of what might be 
work well in the future, rather than something that actually works right 
now.

Is there a technical reason to prefer either repeating the home page, 
license, etc. or writing e.g. `(package-license 
chez-scheme-bootstrap-bootfiles)` rather than using inheritance?

-Philip
  

Hi,

Am Mittwoch, dem 16.02.2022 um 16:13 -0500 schrieb Philip McGrath:
> Hi,
> 
> On 2/14/22 09:54, Liliana Marie Prikler wrote:
> > Am Sonntag, dem 13.02.2022 um 16:51 -0500 schrieb Philip McGrath:
> > > This might seem a bit silly in isolation, but it makes the
> > > structure of the upstream Chez Scheme package the same as for the
> > > Racket variant, it sets things up for (one day, hopefully)
> > > actually being able to bootstrap the upstream Chez Scheme
> > > bootfiles, and it may be useful for cross-compilation and adding
> > > support for architectures without pre-built bootfiles from
> > > upstream.
> > > 
> > > * gnu/packages/chez-and-racket-bootstrap.scm
> > > (chez-scheme-bootstrap-bootfiles): New variable.
> > > (chez-scheme)[native-inputs]: Add it.
> > > [arguments]: Add new phase 'unpack-bootfiles'.
> > > [version, source, home-page]: Derive from 'chez-scheme-bootstrap-
> > > bootfiles'.
> > > ---
> > While having chez-scheme-bootstrap-bootfiles (silly name) does make
> > some kind of sense, making chez-scheme inherit from it does not. 
> > Given that we don't have a chez-scheme bootstrap tower at hand, you
> > should probably make (chez-scheme-bootstrap) a procedure which
> > takes chez-scheme's origin as argument and returns the full
> > package.
> > 
> Making a function is an interesting idea, but I'm not sure I'm quite 
> picturing what you have in mind. I will see if I can figure out 
> something that seems reasonable as I revise this series, if I don't
> hear from you before then.
I was picturing something like

(define chez-bootfiles (chez ...)
  (package/inherit chez
    (inputs ...)
    (native-inputs ...)
    (build-system ...)
    (arguments ...)))

> One reason I like making the bootfiles a package is that a set of 
> bootfiles includes artifacts in addition to the bootfiles themselves,
> such as generated C headers describing the layout of Scheme objects
> in memory, some of which are not included as part of an installed
> Chez Scheme. For example, imagine someone wants to run Chez Scheme on
> FreeBSD: upstream does not distribute BSD bootfiles, so they must be 
> cross-compiled. Even though Guix doesn't have a C toolchain for
> FreeBSD (AFAIK), Guix could be used to reproducibly build the needed
> bootfiles and pack a "source" tarball to be used on a FreeBSD build
> machine.
> 
> Also, the process for building bootfiles is largely orthogonal to 
> building the actual `scheme` executable, and it seems like eventually
> it may be useful to be able to override options separately.
Again, I'm with you on making chez-bootfiles a package and your
rationale sounds reasonable, but I don't think it's correct to say that
chez inherits from them.  IIUC it is rather the other way around; the
bootfiles contain precompiled versions of Chez among other things.

> Is there a technical reason to prefer either repeating the home page,
> license, etc. or writing e.g. `(package-license 
> chez-scheme-bootstrap-bootfiles)` rather than using inheritance?
You should not write (package-license chez-scheme-bootstrap-files),
that's the point!  For one, that's exactly what inheritance would do
unless you specify the field (technical reason), but more importantly,
as a reader, using (package-license this-other-chez-thing) sends me on
a journey to track down this-other-chez-thing while determining the
license of chez!  That's just silly (social reason).

Cheers
  

Hi,

On 2/17/22 02:10, Liliana Marie Prikler wrote:
> Hi,
> 
> Am Mittwoch, dem 16.02.2022 um 16:13 -0500 schrieb Philip McGrath:
>> Hi,
>>
>> On 2/14/22 09:54, Liliana Marie Prikler wrote:
>>> Am Sonntag, dem 13.02.2022 um 16:51 -0500 schrieb Philip McGrath:
>>>> This might seem a bit silly in isolation, but it makes the
>>>> structure of the upstream Chez Scheme package the same as for the
>>>> Racket variant, it sets things up for (one day, hopefully)
>>>> actually being able to bootstrap the upstream Chez Scheme
>>>> bootfiles, and it may be useful for cross-compilation and adding
>>>> support for architectures without pre-built bootfiles from
>>>> upstream.
>>>>
>>>> * gnu/packages/chez-and-racket-bootstrap.scm
>>>> (chez-scheme-bootstrap-bootfiles): New variable.
>>>> (chez-scheme)[native-inputs]: Add it.
>>>> [arguments]: Add new phase 'unpack-bootfiles'.
>>>> [version, source, home-page]: Derive from 'chez-scheme-bootstrap-
>>>> bootfiles'.
>>>> ---
>>> While having chez-scheme-bootstrap-bootfiles (silly name) does make
>>> some kind of sense, making chez-scheme inherit from it does not.
>>> Given that we don't have a chez-scheme bootstrap tower at hand, you
>>> should probably make (chez-scheme-bootstrap) a procedure which
>>> takes chez-scheme's origin as argument and returns the full
>>> package.
>>>
>> Making a function is an interesting idea, but I'm not sure I'm quite
>> picturing what you have in mind. I will see if I can figure out
>> something that seems reasonable as I revise this series, if I don't
>> hear from you before then.
> I was picturing something like
> 
> (define chez-bootfiles (chez ...)
>    (package/inherit chez
>      (inputs ...)
>      (native-inputs ...)
>      (build-system ...)
>      (arguments ...)))
> 

Sorry, I still don't think I'm following. Would this rely on the 
`mative-inputs` being thunked to let the result of this function be an 
input to `chez-scheme`? What commonality is the function abstracting 
over, compared to having 'chez-scheme-for-racket-bootstrap-bootfiles' 
inherit from 'chez-scheme-bootstrap-bootfiles'?

(I'm using "-bootstrap-bootfiles" because there are also other kinds of 
bootfiles: applications can create their own bootfiles, e.g. 
"racket.boot", and using Chez as a cross-compiler also involves more 
bootfiles.)

>> Is there a technical reason to prefer either repeating the home page,
>> license, etc. or writing e.g. `(package-license
>> chez-scheme-bootstrap-bootfiles)` rather than using inheritance?
> You should not write (package-license chez-scheme-bootstrap-files),
> that's the point!  For one, that's exactly what inheritance would do
> unless you specify the field (technical reason), but more importantly,
> as a reader, using (package-license this-other-chez-thing) sends me on
> a journey to track down this-other-chez-thing while determining the
> license of chez!  That's just silly (social reason).

That makes some sense with respect to the license and home page, but 
what about 'package-source' and 'package-version'?

-Philip
  

Hi,

Am Donnerstag, dem 17.02.2022 um 03:06 -0500 schrieb Philip McGrath:
> Hi,
> 
> On 2/17/22 02:10, Liliana Marie Prikler wrote:
> > [...]
> > I was picturing something like
> > 
> > (define chez-bootfiles (chez ...)
> >    (package/inherit chez
> >      (inputs ...)
> >      (native-inputs ...)
> >      (build-system ...)
> >      (arguments ...)))
> > 
> 
> Sorry, I still don't think I'm following. Would this rely on the 
> `mative-inputs` being thunked to let the result of this function be
> an input to `chez-scheme`? 
Yes.

> What commonality is the function abstracting over, compared to having
> 'chez-scheme-for-racket-bootstrap-bootfiles' inherit from 'chez-
> scheme-bootstrap-bootfiles'?
At the moment version, source, home-page and license.  I don't really
think bootstrap files ought to be a part of chez' source, so if you
wanted to do this really cleanly, you'd have to drop them from chez and
add restrict chez-bootstrap to them, which would imply you'd have to
use (version (package-version chez-scheme)) explicitly – for now I
don't want to add too much burden to that patch and you can assume
source to be the same between the two.

> (I'm using "-bootstrap-bootfiles" because there are also other kinds
> of bootfiles: applications can create their own bootfiles, e.g. 
> "racket.boot", and using Chez as a cross-compiler also involves more 
> bootfiles.)
I have no idea how useful that feature is or how widely it is used, but
if those are just binary blobs needed to get chez scheme running, we
ought to treat them as that.
That makes some sense with respect to the license and home page, but 
> > 

> That makes some sense with respect to the license and home page, but 
> what about 'package-source' and 'package-version'?
Your patch currently makes them the same for both, so you tell me :P
See above for long-term plans.

Cheers
  

Hi,

On 2/17/22 03:19, Liliana Marie Prikler wrote:
> Hi,
> 
> Am Donnerstag, dem 17.02.2022 um 03:06 -0500 schrieb Philip McGrath:
>> Hi,
>>
>> On 2/17/22 02:10, Liliana Marie Prikler wrote:
>>> [...]
>>> I was picturing something like
>>>
>>> (define chez-bootfiles (chez ...)
>>>     (package/inherit chez
>>>       (inputs ...)
>>>       (native-inputs ...)
>>>       (build-system ...)
>>>       (arguments ...)))
>>>
>>
>> Sorry, I still don't think I'm following. Would this rely on the
>> `mative-inputs` being thunked to let the result of this function be
>> an input to `chez-scheme`?
> Yes.
> 
>> What commonality is the function abstracting over, compared to having
>> 'chez-scheme-for-racket-bootstrap-bootfiles' inherit from 'chez-
>> scheme-bootstrap-bootfiles'?
> At the moment version, source, home-page and license.  I don't really
> think bootstrap files ought to be a part of chez' source, so if you
> wanted to do this really cleanly, you'd have to drop them from chez and
> add restrict chez-bootstrap to them, which would imply you'd have to
> use (version (package-version chez-scheme)) explicitly – for now I
> don't want to add too much burden to that patch and you can assume
> source to be the same between the two.
> 

Ok, this helps, I think. I'll give it a try.

In <https://issues.guix.gnu.org/47153#9> you argued---and convinced 
me---that we should keep the pre-built bootfiles in the origin until we 
actually don't need them. With 'chez-scheme-for-racket', that's already 
true, and there aren't bootfiles in that origin anyway. When someday we 
can actually bootstrap upstream Chez Scheme, presumably we'll add a 
snippet to delete the pre-built bootfiles from the origin, at which 
point the it will still be the same origin for both.

-Philip
  

Hi,

Am Donnerstag, dem 17.02.2022 um 03:40 -0500 schrieb Philip McGrath:
> Hi,
> 
> On 2/17/22 03:19, Liliana Marie Prikler wrote:
> > Hi,
> > 
> > Am Donnerstag, dem 17.02.2022 um 03:06 -0500 schrieb Philip
> > McGrath:
> > > Hi,
> > > 
> > > On 2/17/22 02:10, Liliana Marie Prikler wrote:
> > > > [...]
> > > > I was picturing something like
> > > > 
> > > > (define chez-bootfiles (chez ...)
> > > >     (package/inherit chez
> > > >       (inputs ...)
> > > >       (native-inputs ...)
> > > >       (build-system ...)
> > > >       (arguments ...)))
> > > > 
> > > 
> > > Sorry, I still don't think I'm following. Would this rely on the
> > > `mative-inputs` being thunked to let the result of this function
> > > be
> > > an input to `chez-scheme`?
> > Yes.
> > 
> > > What commonality is the function abstracting over, compared to
> > > having
> > > 'chez-scheme-for-racket-bootstrap-bootfiles' inherit from 'chez-
> > > scheme-bootstrap-bootfiles'?
> > At the moment version, source, home-page and license.  I don't
> > really
> > think bootstrap files ought to be a part of chez' source, so if you
> > wanted to do this really cleanly, you'd have to drop them from chez
> > and
> > add restrict chez-bootstrap to them, which would imply you'd have
> > to
> > use (version (package-version chez-scheme)) explicitly – for now I
> > don't want to add too much burden to that patch and you can assume
> > source to be the same between the two.
> > 
> 
> Ok, this helps, I think. I'll give it a try.
> 
> In <https://issues.guix.gnu.org/47153#9> you argued---and convinced 
> me---that we should keep the pre-built bootfiles in the origin until
> we actually don't need them. With 'chez-scheme-for-racket', that's
> already true, and there aren't bootfiles in that origin anyway. When
> someday we can actually bootstrap upstream Chez Scheme, presumably
> we'll add a snippet to delete the pre-built bootfiles from the
> origin, at which point the it will still be the same origin for both.
This concerns schez-scheme-for-racket, not the current chez-scheme for
which we are packaging the bootfiles, am I right?  Given that we build
a set of bootfiles that would be enough to go forward, we no longer
need the original bootfiles beyond that point, i.e. the "bootstrap
bootfiles" would only be needed inside chez-bootstrap, but not chez
itself (which can instead use the native-input).  Am I missing
something here?
  

Hi,

Here is a v2!

Liliana, I tried using a function to generate the bootfile packages as you
suggested, but I didn't like the result---though it is very possible I still
didn't properly understand what you had in mind. I hope what I've done in this
version addresses the concerns you raised about inverted inheritance and such
anyway: if you'd like to compare the two, in the repository
at <https://gitlab.com/philip1/guix-patches> I've tagged this series as
`guix-issue-53878-v2` and the alternate as
`guix-issue-53878-v2-bootfiles-proc`. I'll also include the diff below.

Other than that, I hope I've addressed all your comments.

> diff --git a/gnu/packages/chez-and-racket-bootstrap.scm b/gnu/packages/chez-and-racket-bootstrap.scm
> index 3e90a15d94..cbdddb1e98 100644
> --- a/gnu/packages/chez-and-racket-bootstrap.scm
> +++ b/gnu/packages/chez-and-racket-bootstrap.scm
> @@ -720,21 +720,18 @@ (define* (stex-make #:optional (suffix ""))
>  and 32-bit PowerPC architectures.")
>      (license license:asl2.0)))
>  
> -(define (bootfiles-for-chez chez)
> -  (package/inherit chez
> -    (outputs '("out"))
> +(define-public chez-scheme-bootstrap-bootfiles
> +  (package
> +    (inherit chez-scheme)
> +    (name "chez-scheme-bootstrap-bootfiles")
>      (inputs '())
>      (native-inputs '())
> +    (outputs '("out"))
>      (build-system copy-build-system)
>      ;; TODO: cross compilation
>      (arguments
>       (list #:install-plan
> -           #~`(("boot/" "lib/chez-scheme-bootfiles"))))))
> -
> -(define-public chez-scheme-bootstrap-bootfiles
> -  (package
> -    (inherit (bootfiles-for-chez chez-scheme))
> -    (name "chez-scheme-bootstrap-bootfiles")
> +           #~`(("boot/" "lib/chez-scheme-bootfiles"))))
>      (supported-systems
>       ;; Upstream only distributes pre-built bootfiles for
>       ;; arm32le and t?(i3|a6)(le|nt|osx)
> @@ -822,28 +819,34 @@ (define-public chez-scheme-for-racket
>      (license license:asl2.0)))
>  
>  (define-public chez-scheme-for-racket-bootstrap-bootfiles
> -  (let ((chez (bootfiles-for-chez chez-scheme-for-racket)))
> -    (package
> -      (inherit (bootfiles-for-chez chez))
> -      (name "chez-scheme-for-racket-bootstrap-bootfiles")
> -      (native-inputs (list chez-nanopass-bootstrap racket-vm-bc))
> -      (arguments
> -       (substitute-keyword-arguments (package-arguments chez)
> -         ((#:phases those-phases #~%standard-phases)
> -          #~(modify-phases #$those-phases
> -              (add-after 'unpack 'chdir
> -                (lambda args
> -                  (chdir "racket/src/ChezScheme")))
> -              (add-after 'chdir 'unpack-nanopass+stex
> -                (lambda args
> -                  #$unpack-nanopass+stex))
> -              (add-before 'install 'build
> -                (lambda* (#:key native-inputs inputs #:allow-other-keys)
> -                  (invoke (search-input-file (or native-inputs inputs)
> -                                             "/opt/racket-vm/bin/racket")
> -                          "rktboot/main.rkt")))))))
> -      (synopsis "Chez Scheme bootfiles bootstrapped by Racket")
> -      (description "Chez Scheme is a self-hosting compiler: building it
> +  (package
> +    (inherit chez-scheme-bootstrap-bootfiles)
> +    (name "chez-scheme-for-racket-bootstrap-bootfiles")
> +    (version (package-version chez-scheme-for-racket))
> +    (source (package-source chez-scheme-for-racket))
> +    (native-inputs (list chez-nanopass-bootstrap racket-vm-bc))
> +    (arguments
> +     (substitute-keyword-arguments
> +         (package-arguments chez-scheme-bootstrap-bootfiles)
> +       ((#:phases those-phases #~%standard-phases)
> +        #~(modify-phases #$those-phases
> +            (add-after 'unpack 'chdir
> +              (lambda args
> +                (chdir "racket/src/ChezScheme")))
> +            (add-after 'chdir 'unpack-nanopass+stex
> +              (lambda args
> +                #$unpack-nanopass+stex))
> +            (add-before 'install 'build
> +              (lambda* (#:key native-inputs inputs #:allow-other-keys)
> +                (invoke (search-input-file (or native-inputs inputs)
> +                                           "/opt/racket-vm/bin/racket")
> +                        "rktboot/main.rkt")))))))
> +    (home-page "https://github.com/racket/ChezScheme")
> +    ;; ^ This is downstream of https://github.com/racket/racket,
> +    ;; but it's designed to be a friendly landing place for people
> +    ;; who want a ChezScheme-shaped repositroy.
> +    (synopsis "Chez Scheme bootfiles bootstrapped by Racket")
> +    (description "Chez Scheme is a self-hosting compiler: building it
>  requires ``bootfiles'' containing the Scheme-implemented portions compiled for
>  the current platform.  (Chez can then cross-compile bootfiles for all other
>  supported platforms.)
> @@ -857,7 +860,7 @@ (define-public chez-scheme-for-racket-bootstrap-bootfiles
>  
>  Note that the generated bootfiles are specific to Racket's fork of Chez
>  Scheme, and @code{cs-bootstrap} does not currently support building upstream
> -Chez Scheme."))))
> +Chez Scheme.")))
>  
>  ;;
>  ;; Chez's bootstrap dependencies:

 -Philip

Philip McGrath (15):
  gnu: chez-scheme: Move to (gnu packages chez-and-racket-bootstrap).
  gnu: chez-scheme: Use "lib/chez-scheme" for search path.
  gnu: chez-scheme: Use shared zlib and lz4.
  gnu: chez-and-racket-bootstrap: Add utilities for Chez machine types.
  gnu: chez-scheme: Use new package style.
  gnu: Add stex.
  gnu: Add chez-nanopass.
  gnu: chez-scheme: Explicitly package bootstrap bootfiles.
  gnu: Add racket-vm-cgc.
  gnu: Add racket-vm-bc.
  gnu: Add chez-scheme-for-racket.
  gnu: Add racket-vm-cs.
  gnu: chez-mit: Support chez-scheme-for-racket.
  gnu: chez-and-racket-bootstrap: Add 'chez-scheme-for-system'.
  gnu: racket: Update to 8.4.

 gnu/local.mk                                  |    5 +-
 gnu/packages/chez-and-racket-bootstrap.scm    | 1070 ++++++++++++
 gnu/packages/chez.scm                         |  628 +++----
 gnu/packages/emacs-xyz.scm                    |    4 +-
 gnu/packages/loko.scm                         |    4 +-
 .../racket-enable-scheme-backport.patch       |  465 +++++
 ...acket-gui-tethered-launcher-backport.patch |   26 +
 gnu/packages/racket.scm                       | 1553 +++++++++++------
 8 files changed, 2813 insertions(+), 942 deletions(-)
 create mode 100644 gnu/packages/chez-and-racket-bootstrap.scm
 create mode 100644 gnu/packages/patches/racket-enable-scheme-backport.patch
 create mode 100644 gnu/packages/patches/racket-gui-tethered-launcher-backport.patch