[bug#53878,RFC,0/9] Update Racket to 8.4. Adjust Chez Scheme packages.

Racket 8.4 hasn't been released yet (expect it in a matter of days), but I
wanted to solicit feedback on these changes early.

On the Racket side of things, discussions upstream and with other distribution
packagers have clarified what a ``minimal Racket'' package ought to consist
of, which in turn suggested a nice new boundary between the lowest-level
Racket VM packages, which are concerned with bootstrapping and platform
support, and a more uniform use of layered and tethered installations for both
the 'racket-minimal' and 'racket' packages. I've also started getting the
sources for the other main-distribution Racket packages directly from their
respective repositories, rather than extracting them from the bundled
tarballs. All in all, it's a few steps closer to the goal of a build system
for Racket packages.

For Chez Scheme, I set out to package Racket's variant of Chez Scheme---which
supports architectures not yet supported upstream, e.g. aarch64---and to share
more between the upstream and Racket packages. I also tweaked various things I
found along the way: hopefully some are clear improvements, though there are
some prime opportunities for bikeshedding about naming things.

Let me know what you think!

 -Philip

Philip McGrath (9):
  gnu: chez-scheme: Move to (gnu packages chez-and-racket-bootstrap).
  gnu: chez-scheme: Use "lib/chez-scheme" for search path.
  gnu: chez-scheme: Use shared zlib and lz4.
  gnu: chez-and-racket-bootstrap: Add utilities for Chez machine types.
  gnu: Add stex.
  gnu: Add chez-nanopass.
  gnu: chez-scheme: Explicitly package bootstrap bootfiles.
  gnu: Add chez-scheme-racket-variant.
  gnu: racket: Update to 8.3.900.

 gnu/local.mk                                  |    5 +-
 gnu/packages/chez-and-racket-bootstrap.scm    | 1108 ++++++++++++
 gnu/packages/chez.scm                         |  628 +++----
 gnu/packages/emacs-xyz.scm                    |    4 +-
 gnu/packages/loko.scm                         |    4 +-
 .../racket-enable-scheme-backport.patch       |  465 +++++
 ...acket-gui-tethered-launcher-backport.patch |   26 +
 gnu/packages/racket.scm                       | 1552 +++++++++++------
 8 files changed, 2850 insertions(+), 942 deletions(-)
 create mode 100644 gnu/packages/chez-and-racket-bootstrap.scm
 create mode 100644 gnu/packages/patches/racket-enable-scheme-backport.patch
 create mode 100644 gnu/packages/patches/racket-gui-tethered-launcher-backport.patch
  

Hi,

As promised, here is v3!

Changes since v2:

  * 05/15: Fix typo in commit message.

  * 09/15: Add `%racket-version` and `%racket-origin`.

  * 11/15: Add `%chez-scheme-for-racket-version`.

  * 15/15: Use `%racket-version` and `%racket-origin`. Also:

      * `racket-minimal`: The package to install explicitly is
        "racket-lib" ("base" is implicit).

      * `make-installation-layer.rkt`: Adjust workaround for
        <https://github.com/racket/racket/issues/4133>, because `racket`
        (but not `racket-minimal`) ends up actually creating the bogus
        untethered "bin" directory.

I've also put these patches up
at <https://gitlab.com/philip1/guix-patches/-/tags/guix-issue-53878-v3>

 -Philip

Philip McGrath (15):
  gnu: chez-scheme: Move to (gnu packages chez-and-racket-bootstrap).
  gnu: chez-scheme: Use "lib/chez-scheme" for search path.
  gnu: chez-scheme: Use shared zlib and lz4.
  gnu: chez-and-racket-bootstrap: Add utilities for Chez machine types.
  gnu: chez-scheme: Use new package style.
  gnu: Add stex.
  gnu: Add chez-nanopass.
  gnu: chez-scheme: Explicitly package bootstrap bootfiles.
  gnu: Add racket-vm-cgc.
  gnu: Add racket-vm-bc.
  gnu: Add chez-scheme-for-racket.
  gnu: Add racket-vm-cs.
  gnu: chez-mit: Support chez-scheme-for-racket.
  gnu: chez-and-racket-bootstrap: Add 'chez-scheme-for-system'.
  gnu: racket: Update to 8.4.

 gnu/local.mk                                  |    5 +-
 gnu/packages/chez-and-racket-bootstrap.scm    | 1074 ++++++++++++
 gnu/packages/chez.scm                         |  628 +++----
 gnu/packages/emacs-xyz.scm                    |    4 +-
 gnu/packages/loko.scm                         |    4 +-
 .../racket-enable-scheme-backport.patch       |  465 +++++
 ...acket-gui-tethered-launcher-backport.patch |   26 +
 gnu/packages/racket.scm                       | 1557 +++++++++++------
 8 files changed, 2821 insertions(+), 942 deletions(-)
 create mode 100644 gnu/packages/chez-and-racket-bootstrap.scm
 create mode 100644 gnu/packages/patches/racket-enable-scheme-backport.patch
 create mode 100644 gnu/packages/patches/racket-gui-tethered-launcher-backport.patch
  

(sorry, missed "reply all" ...)

----------  Forwarded Message  ----------

Subject: Re: [PATCH v3 14/15] gnu: chez-and-racket-bootstrap: Add 'chez-
scheme-for-system'.
Date: Saturday, February 19, 2022, 5:29:05 PM EST
From: Philip McGrath <philip@philipmcgrath.com>
To: Liliana Marie Prikler <liliana.prikler@gmail.com>

Hi,

On Saturday, February 19, 2022 3:00:37 PM EST you wrote:
> Am Samstag, dem 19.02.2022 um 01:42 -0500 schrieb Philip McGrath:
> > +(define* (chez-scheme-for-system #:optional
> > +                                 (system (or (%current-target-
> > system)
> > +                                             (%current-system))))
> > +  "Return 'chez-scheme' unless only 'chez-scheme-for-racket'
> > supports SYSTEM,
> > +including support for native threads."
> > +  (if (and (nix-system->chez-machine system)
> > +           (not (and=> (chez-upstream-features-for-system system)
> > +                       (cut memq 'threads <>))))
> > +      chez-scheme-for-racket
> > +      chez-scheme))
> 
> Given your previous explanation this series looks clean enough so far,
> but this looks like a bug.  You probably want
> (if (and (nix-system->chez-machine system)
>     (and=> (chez-upstream-features-for-system system)
>            (cut memq 'threads <>)))
>     chez-scheme
>     chez-scheme-for-racket)
> This variant would in particular use chez-scheme-for-racket if nix-
> system->chez-machine returns #f.
> 
> I can make that adjustment for you assuming I interpreted this
> correctly.

Thanks for catching this! I must have gotten myself confused while changing to 
chez-upstream-features-for-system from the chez-machine->upstream-restriction 
version.

-Philip

-----------------------------------------
  

(... on this one, too)

----------  Forwarded Message  ----------

Subject: Re: [PATCH v3 09/15] gnu: Add racket-vm-cgc.
Date: Saturday, February 19, 2022, 5:27:39 PM EST
From: Philip McGrath <philip@philipmcgrath.com>
To: Liliana Marie Prikler <liliana.prikler@gmail.com>

Hi,

On Saturday, February 19, 2022 3:46:47 PM EST you wrote:
> Am Samstag, dem 19.02.2022 um 01:42 -0500 schrieb Philip McGrath:
> > * gnu/packages/patches/racket-enable-scheme-backport.patch: New
> > patch.
> > * gnu/local.mk (dist_patch_DATA): Add it.
> > * gnu/packages/chez-and-racket-bootstrap.scm (unbundle-chez-
> > submodules,
> > %racket-version, %racket-origin, racket-vm-cgc): New variables.
> > (chez-scheme)[source]<snippet>: Use 'unbundle-chez-submodules'.
> 
> Something weird happened to me just now trying to build this series.
> While compiling emacs-xyz, an error was raised regarding %racket-
> version not existing and and import being missing.  Assuming that you
> didn't mess up an include somewhere, there could be a cycle meaning
> we'd have to pass the version and origin by function as I originally
> said in my reply to v2.
> 
> I'll clean up my work tree and try to reapply it, but that will take
> some time.  Since you mentioned re-exports causing issues, I'd like to
> ask if you've made a similar experience, but the results got somehow
> hidden after the right incantations.

I haven't seen errors from emacs-xyz, but I have gotten errors about %racket-
version not existing: at the time I thought it was just a problem with 
incremental rebuilds while moving back and forward through history, but, 
having just refreshed my memory on more details of the cyclic issues, I think 
it may be related. I'll send another email presently with details once I've 
gathered references.

For now, I found that `rm gnu/packages/*.go` was enough to get `make` to 
succeed again.

-Philip
-----------------------------------------
  

Hi,

On Saturday, February 19, 2022 5:35:22 PM EST Philip McGrath wrote:
> On Saturday, February 19, 2022 3:46:47 PM EST you wrote:
> > Am Samstag, dem 19.02.2022 um 01:42 -0500 schrieb Philip McGrath:
> > > * gnu/packages/patches/racket-enable-scheme-backport.patch: New
> > > patch.
> > > * gnu/local.mk (dist_patch_DATA): Add it.
> > > * gnu/packages/chez-and-racket-bootstrap.scm (unbundle-chez-
> > > submodules,
> > > %racket-version, %racket-origin, racket-vm-cgc): New variables.
> > > (chez-scheme)[source]<snippet>: Use 'unbundle-chez-submodules'.
> > 
> > Something weird happened to me just now trying to build this series.
> > While compiling emacs-xyz, an error was raised regarding %racket-
> > version not existing and and import being missing.  Assuming that you
> > didn't mess up an include somewhere, there could be a cycle meaning
> > we'd have to pass the version and origin by function as I originally
> > said in my reply to v2.
> > 
> > I'll clean up my work tree and try to reapply it, but that will take
> > some time.  Since you mentioned re-exports causing issues, I'd like to
> > ask if you've made a similar experience, but the results got somehow
> > hidden after the right incantations.
> 
> I haven't seen errors from emacs-xyz, but I have gotten errors about
> %racket- version not existing: at the time I thought it was just a problem
> with incremental rebuilds while moving back and forward through history,
> but, having just refreshed my memory on more details of the cyclic issues,
> I think it may be related. I'll send another email presently with details
> once I've gathered references.
> 
> For now, I found that `rm gnu/packages/*.go` was enough to get `make` to
> succeed again.

This reminded me of one of the commits I mentioned yesterday in <https://
issues.guix.gnu.org/53878#66>:

> commit 96db2ff145ecbd962206eae815b065bda7ed3d9f
> Author: Ludovic Courtès <ludo@gnu.org>
> Date: Tue Sep 7 15:11:46 2021 +0200
>
> gnu: racket-minimal: Remove top-level reference to 'chez-scheme'.
>
> This could cause build errors; for instance, doing:
> make && touch gnu/packages/chez.scm && make
>
> would trigger a "chez-scheme: unbound variable" error.
>
> * gnu/packages/racket.scm (racket-minimal)[source]: Add 'modules'
> field. In 'snippet', remove top-level reference to CHEZ-SCHEME, which
> could cause build errors. Simplify snippet.

That symptom of an unbound variable error on re-making sounds like what's 
going on.

In <https://issues.guix.gnu.org/48682#7>, Ludo’ explained:

On Monday, May 31, 2021 12:23:27 PM EST Ludovic Courtès wrote:
> Philip McGrath <philip@philipmcgrath.com> skribis:
> > On 5/29/21 4:15 PM, Ludovic Courtès wrote:
> >> In general we cannot use #:select for (gnu packages …) modules because
> >> that doesn’t play well with circular module dependencies.
> > 
> > Ah, interesting, I'll keep that in mind. I'm used to Racket, where all
> > cyclic module dependencies cause errors at compile time.
> 
> Yeah, in hindsight, that’s probably safer…
> 
> > Do you have any advice on what would be good practice?
> 
> For package modules, the main things are:
> 
>   1. Don’t use #:select or #:autoload for (gnu packages …) modules in a
>      (gnu packages …) module.
> 
>   2. At the top level of a module, only refer to variables within that
>      module.  For instance, the following would be wrong:
> 
>        (define-module (gnu packages racket)
>          #:use-module (gnu packages chez)
>          …)
> 
>        (define whatever
>          ;; Wrong because ‘chez-scheme’ is defined in another module,
>          ;; which might be part of a cycle with this one.
>          (package (inherit chez-scheme) …))
> 
>        (define something
>          (package
>            ;; …
>            (license (package-license chez-scheme))))  ;likewise
> 
> Note that references from ‘inputs’ and ‘arguments’ fields are perfectly
> fine (fortunately!) because those fields are “thunked” (their value is
> wrapped in a thunk).
> 

My understanding of the semantics of Guile modules is strictly less than the 
content of that thread. I'm most familiar with Racket modules, and I have some 
understanding of how those semantics differ from R6RS library semantics.

I think I had seized in my memory of № 2 on  "at the top level of a module", 
and I'd filled in my usual (Racket) understanding of "at the top level of a 
module" as a less formal way of saying "in a module context", in the sense of 
"expansion contexts".[1]

But  Ludo’'s examples show that's wrong: those uses of `chez scheme` are in 
what the "expansion contexts" model would call "expression contexts".

Instead, I think rule № 2 prohibits any reference to a variable imported from 
another (gnu packages ...) module that will be evaluated when the (gnu 
packages ...) modules are—visited? instantiated? [2][3]—IDK when exactly, but, 
for practical purposes, any variable reference that is not underneath a lambda 
abstraction.

If that's right, IIUC, it would mean that:

    (define chez-scheme-for-racket
      (make-chez-scheme-for-racket ...))

would also be prohibited.

On the other hand, uses of `(racket-vm-for-system)` and `(chez-scheme-for-
system)` in an `imports` field should still be fine, thanks to the implicit 
thunks.

I think I can can make that work relatively well. The one thing I don't know 
how to avoid is defining `%racket-version` in both "chez-and-racket-
bootstrap.scm" and "racket.scm", but I'll leave a cautionary comment.

I'll try to make a v4 with the approach I have in mind. I'd be glad to hear 
better ideas, though! Especially if anyone knows more about what Guile's 
module semantics really are.

-Philip

[1]: https://docs.racket-lang.org/reference/syntax-model.html#%28part._expand-context-model%29
[2]: http://www.r6rs.org/final/html/r6rs/r6rs-Z-H-10.html#node_sec_7.2
[4]: https://docs.racket-lang.org/reference/syntax-model.html#%28part._mod-parse%29
  

Hi,

Here is v4. I believe it avoids the import cycle.

The first change from v3 is in patch 09/15, in which I changed
"gnu/packages/chez-and-racket-bootstrap.scm" no longer export
`%racket-version` or `%racket-origin`. The package `source` field is not
thunked, so exporting the latter seems like a particular trap for the unwary.

In 14/15, I looked again at the issue with `chez-scheme-for-system` that
Liliana reported in <https://issues.guix.gnu.org/53878#89>, and I realized my
intention had actually been to return `chez-scheme` if the specified system is
completely unsupported (not that it makes much difference ...). I tweaked it
slightly and added comments, since I've gotten confused about it more than
once now.

The changes to avoid the import cycle are primarily in 15/15. As I outlined
in <https://issues.guix.gnu.org/53878#93>, I duplicated the definition of
`%racket-version` in "gnu/packages/racket.scm" and added comments there and to
"gnu/packages/chez-and-racket-bootstrap.scm" warning to keep them in sync.

I also had to avoid the use of `%racket-origin` (or `(package-source
(racket-vm-for-system))`) outside of lambda abstractions, which in particular
meant that it could no longer be used in the `source` field of
`racket-minimal`. Instead, I changed `racket-minimal` to handle "base" and
"racket-lib" in the same way `racket` does for its component Racket
packages. Recalling Liliana's comment
in <https://issues.guix.gnu.org/53878#65>, since I was lifting the function
to handle Racket package origins anyway to reuse it, I changed it to produce
`computed-file`s that take care of extracting the right files from the origin,
so `racket-minimal` can now use `union-build` and `racket` no longer needs to
replace the build phase.

I can immagine a `racket-build-system` helping to improve the situation, since
it would presumably add `(racket-vm-for-system)` as an implicit input. It
might also be useful to provide special support for packages from the main
Racket Git repository: almost always, if you change transform the origin for
the Racket VM, you want all of the packages developed in the same repository
to use come from the transformed origin, too.

But for now, v4 should avoid the import cycle problem without doing anything
too ugly.

 -Philip

Philip McGrath (15):
  gnu: chez-scheme: Move to (gnu packages chez-and-racket-bootstrap).
  gnu: chez-scheme: Use "lib/chez-scheme" for search path.
  gnu: chez-scheme: Use shared zlib and lz4.
  gnu: chez-and-racket-bootstrap: Add utilities for Chez machine types.
  gnu: chez-scheme: Use new package style.
  gnu: Add stex.
  gnu: Add chez-nanopass.
  gnu: chez-scheme: Explicitly package bootstrap bootfiles.
  gnu: Add racket-vm-cgc.
  gnu: Add racket-vm-bc.
  gnu: Add chez-scheme-for-racket.
  gnu: Add racket-vm-cs.
  gnu: chez-mit: Support chez-scheme-for-racket.
  gnu: chez-and-racket-bootstrap: Add 'chez-scheme-for-system'.
  gnu: racket: Update to 8.4.

 gnu/local.mk                                  |    5 +-
 gnu/packages/chez-and-racket-bootstrap.scm    | 1077 +++++++++++
 gnu/packages/chez.scm                         |  628 +++----
 gnu/packages/emacs-xyz.scm                    |    4 +-
 gnu/packages/loko.scm                         |    4 +-
 .../racket-enable-scheme-backport.patch       |  465 +++++
 ...acket-gui-tethered-launcher-backport.patch |   26 +
 gnu/packages/racket.scm                       | 1599 +++++++++++------
 8 files changed, 2871 insertions(+), 937 deletions(-)
 create mode 100644 gnu/packages/chez-and-racket-bootstrap.scm
 create mode 100644 gnu/packages/patches/racket-enable-scheme-backport.patch
 create mode 100644 gnu/packages/patches/racket-gui-tethered-launcher-backport.patch