[bug#55030,00/30] gnu: elm: Update to 0.19.1. Add build system & importer.

Hi,

This patch series updates Elm to version 0.9.1, then adds an
'elm-build-system' and a 'guix import elm' command.

To exercise the new features, this patch series then:

  * Build the front-end for the `elm reactor` command (which is written in Elm)
    and adds a variant of Elm to Guix with the command enabled;

  * Builds 'elm-todomvc', an official example of a basic Elm application; and

  * Builds a feature-rich third-party package, "terezka/elm-charts":
    <https://elm-charts.org>.

 -Philip

Philip McGrath (30):
  gnu: elm-compiler: Update to 0.19.1.
  gnu: elm: Rename package to match the command.
  guix: Add elm-build-system and 'guix import elm'.
  gnu: Add elm-core and elm-json.
  build-system/elm: Add implicit Elm inputs.
  gnu: Add elm-virtual-dom.
  gnu: Add elm-html.
  gnu: Add elm-svg.
  gnu: Add elm-time.
  gnu: Add elm-url.
  gnu: Add elm-browser.
  gnu: Add elm-bytes.
  gnu: Add elm-file.
  gnu: Add elm-http.
  gnu: Add elm-parser.
  gnu: Add elm-project-metadata-utils.
  gnu: Add elm-explorations-markdown.
  gnu: elm: Support 'elm reactor'.
  gnu: Add elm-todomvc.
  gnu: Add elm-debois-elm-dom.
  gnu: Add elm-random.
  gnu: Add elm-explorations-test.
  gnu: Add elm-danhandrea-elm-date-format.
  gnu: Add elm-danhandrea-elm-time-extra.
  gnu: Add elm-justinmimbs-date.
  gnu: Add elm-justinmimbs-time-extra.
  gnu: Add elm-myrho-elm-round.
  gnu: Add elm-ryannhg-date-format.
  gnu: Add elm-terezka-intervals.
  gnu: Add elm-terezka-elm-charts.

 gnu/local.mk                                  |   4 +-
 gnu/packages/elm.scm                          | 767 +++++++++++++++++-
 .../elm-compiler-disable-reactor.patch        |  71 --
 .../patches/elm-compiler-fix-map-key.patch    |  38 -
 .../elm-offline-package-registry.patch        |  71 ++
 .../patches/elm-reactor-static-files.patch    | 251 ++++++
 guix/build-system/elm.scm                     | 176 ++++
 guix/build/elm-build-system.scm               | 380 +++++++++
 guix/import/elm.scm                           | 148 ++++
 guix/scripts/import.scm                       |   3 +-
 guix/scripts/import/elm.scm                   | 107 +++
 11 files changed, 1883 insertions(+), 133 deletions(-)
 delete mode 100644 gnu/packages/patches/elm-compiler-disable-reactor.patch
 delete mode 100644 gnu/packages/patches/elm-compiler-fix-map-key.patch
 create mode 100644 gnu/packages/patches/elm-offline-package-registry.patch
 create mode 100644 gnu/packages/patches/elm-reactor-static-files.patch
 create mode 100644 guix/build-system/elm.scm
 create mode 100644 guix/build/elm-build-system.scm
 create mode 100644 guix/import/elm.scm
 create mode 100644 guix/scripts/import/elm.scm
  

Hi Philip,

Philip McGrath <philip@philipmcgrath.com> skribis:

> This patch series updates Elm to version 0.9.1, then adds an
> 'elm-build-system' and a 'guix import elm' command.

Impressive!

> To exercise the new features, this patch series then:
>
>   * Build the front-end for the `elm reactor` command (which is written in Elm)
>     and adds a variant of Elm to Guix with the command enabled;
>
>   * Builds 'elm-todomvc', an official example of a basic Elm application; and
>
>   * Builds a feature-rich third-party package, "terezka/elm-charts":
>     <https://elm-charts.org>.

Woow, neat.

Annoying question that I have to ask: do these packages bundle
JavaScript libraries?  If yes, is it source or is it “minified”?

(My take is that we could tolerate some level of bundling if “doing the
right thing” is impractical, but it’d rather be source.)

Thanks,
Ludo’.
  

I sent a few comments to individual patches that probably warrant a v2,
but overall I think the patch series is almost ready to go.

Thanks!

Ludo’.
  

Hi,

Hi,

On 5/1/22 16:22, Ludovic Courtès wrote:
> Hi Philip,
> 
> Philip McGrath <philip@philipmcgrath.com> skribis:
> 
>> This patch series updates Elm to version 0.9.1, then adds an
>> 'elm-build-system' and a 'guix import elm' command.
> 
> Impressive!
> 

Thanks!

>> To exercise the new features, this patch series then:
>>
>>    * Build the front-end for the `elm reactor` command (which is written in Elm)
>>      and adds a variant of Elm to Guix with the command enabled;
>>
>>    * Builds 'elm-todomvc', an official example of a basic Elm application; and
>>
>>    * Builds a feature-rich third-party package, "terezka/elm-charts":
>>      <https://elm-charts.org>.
> 
> Woow, neat.
> 
> Annoying question that I have to ask: do these packages bundle
> JavaScript libraries?  If yes, is it source or is it “minified”?
> 
> (My take is that we could tolerate some level of bundling if “doing the
> right thing” is impractical, but it’d rather be source.)
> 

Short answer: no, they don't!

Longer answer:

Elm basically takes the view that the existing JavaScript/NPM thicket 
should be considered harmful. It imposes a lot of very strict 
requirements on Elm "packages" (vs. "applications") to avoid whole 
classes of problems. Not all of them are precisely the requirements I 
would have chosen, but I like them better than the alternative chaos.

(One reason I gave this a try was to get some hands-on experience 
writing a build system and importer in a simplified context before 
trying to write `racket-build-system`.)

In particular, allowing arbitrary JavaScript would defeat the strong 
guarantees Elm wants to offer as a statically-typed, purely-functional 
language with compiler-enforced semantic versioning (well, for a 
decidable subset of "semantics") that can make runtime errors 
vanishingly rare in practice. To that end, Elm requires that 
"packages"---the things `elm-build-system` knows how to build---be 
written in pure Elm, with no JavaScript at all. For "applications", 
interop is limited to asynchronous message passing.[1] The only two 
"applications" in this series, the `elm reactor` frontend and 
`elm-todomvc`, don't use any message passing.

Of course, Elm needs some way to implement primitives. These are 
provided by modules in the `Elm.Kernel.*` namespace, which are written 
in JavaScript with the undocumented, unsafe conventions expected by the 
Elm compiler. The Elm compiler only allows kernel modules in packages in 
the `elm/*` and `elm-explorations/*` namespaces, so users can know that 
third-party packages won't break Elm's guaranteed, and the compiler is 
free to change its internal APIs without breaking anything. (This is 
free software, so you could patch the compiler to do whatever you want: 
it's just a community norm so strong it's expressed in code.) Even this 
is all source code with whitespace and comments: it's written in a very 
stylized way, as an ASM file in another compiler implementation might 
be, but it isn't generated code.

For people who want to "minify", Elm suggests flags for UglifyJS that 
can also do otherwise-unsafe whole-program optimization of the compiled 
"application".[2] (Compiling "packages" emits only an internal 
representation, not JavaScript.) In my own projects, I've also done 
other post-processing, like adding LibreJS comments and converting the 
output to an ES6 module. I haven't tried to make `elm-build-system` do 
any of those kinds of things for "applications": I think we should find 
more examples of Elm applications people would want to package for Guix 
first, rather than trying to guess what they might need.

[1]: https://guide.elm-lang.org/interop/
[2]: https://github.com/elm/compiler/blob/master/hints/optimize.md

-Philip
  

Hi,

On 5/1/22 16:41, Ludovic Courtès wrote:
> I sent a few comments to individual patches that probably warrant a v2,
> but overall I think the patch series is almost ready to go.
> 
> Thanks!
> 
> Ludo’.

Thanks for taking a look! I have a few other things to wrap up before I 
return to this, but I'll put together a v2 relatively soon.

Tangentially, I learned after sending this series about a Rust-based 
runner [1] for "elm-explorations/test" tests, which may be easier for us 
to package than the Node.js-based runner.[2] Still, I'd probably save 
that for some future patch series.

-Philip

[1]: https://github.com/mpizenberg/elm-test-rs
[2]: https://github.com/rtfeldman/node-test-runner
  

Hi Philip,

Philip McGrath <philip@philipmcgrath.com> skribis:

> Elm basically takes the view that the existing JavaScript/NPM thicket
> should be considered harmful. It imposes a lot of very strict 
> requirements on Elm "packages" (vs. "applications") to avoid whole
> classes of problems. Not all of them are precisely the requirements I 
> would have chosen, but I like them better than the alternative chaos.
>
> (One reason I gave this a try was to get some hands-on experience
> writing a build system and importer in a simplified context before 
> trying to write `racket-build-system`.)
>
> In particular, allowing arbitrary JavaScript would defeat the strong
> guarantees Elm wants to offer as a statically-typed, purely-functional 
> language with compiler-enforced semantic versioning (well, for a
> decidable subset of "semantics") that can make runtime errors 
> vanishingly rare in practice. To that end, Elm requires that
> "packages"---the things `elm-build-system` knows how to build---be 
> written in pure Elm, with no JavaScript at all. For "applications",
> interop is limited to asynchronous message passing.[1] The only two 
> "applications" in this series, the `elm reactor` frontend and
> `elm-todomvc`, don't use any message passing.
>
> Of course, Elm needs some way to implement primitives. These are
> provided by modules in the `Elm.Kernel.*` namespace, which are written 
> in JavaScript with the undocumented, unsafe conventions expected by
> the Elm compiler. The Elm compiler only allows kernel modules in
> packages in the `elm/*` and `elm-explorations/*` namespaces, so users
> can know that third-party packages won't break Elm's guaranteed, and
> the compiler is free to change its internal APIs without breaking
> anything. (This is free software, so you could patch the compiler to
> do whatever you want: it's just a community norm so strong it's
> expressed in code.) Even this is all source code with whitespace and
> comments: it's written in a very stylized way, as an ASM file in
> another compiler implementation might be, but it isn't generated code.
>
> For people who want to "minify", Elm suggests flags for UglifyJS that
> can also do otherwise-unsafe whole-program optimization of the
> compiled "application".[2] (Compiling "packages" emits only an
> internal representation, not JavaScript.) In my own projects, I've
> also done other post-processing, like adding LibreJS comments and
> converting the output to an ES6 module. I haven't tried to make
> `elm-build-system` do any of those kinds of things for "applications":
> I think we should find more examples of Elm applications people would
> want to package for Guix first, rather than trying to guess what they
> might need.

Thanks for explaining!  Elm is fascinating in many ways.

Ludo’.
  

Hi,

Here is v2!

A few notes before the patches:

On 5/1/22 18:17, Philip McGrath wrote:
> Hi,
> 
> On 5/1/22 16:37, Ludovic Courtès wrote:
>> Philip McGrath <philip@philipmcgrath.com> skribis:
>>
>>> +  (package
>>> +    (name "elm-virtual-dom")
>>
>> [...]
>>
>>> +    (properties '((upstream-name . "elm/virtual-dom")))))
>>
>> Could/should the importer infer the upstream name from the Guix name by
>> default?
>>
>> That way, we’d only need to specify that property where the automatic
>> Guix->upstream name mapping wouldn’t work.
> 
> It could, but the heuristics seemed a bit brittle. To pick a few examples:
> 
>   1. elm-virtual-dom -> "elm/virtual-dom"
>   2. elm-explorations-markdown -> "elm-explorations/markdown"
>   2. elm-terezka-intervals -> "terezka/intervals"
> 
> We could add a special case for the "elm-explorations/*" namespace, but 
> at least one of the others would need an explicit property. I *think* 
> most of the packages in the "elm/*" namespace are single-element (e.g. 
> "elm/html"), so maybe we could require the property for e.g. 
> "elm/virtual-dom" and "elm/project-metadata-utils" ...
> 

I think I've come up with an approach to inferring upstream names that handles
most cases (everything in this series but "elm/virtual-dom" and
"elm/project-metadata-utils") while also not adding so much magic that the
behavior would be unpredictable.

I've put the functions going in both directions in `(guix build-system elm)`
rather than `(guix import elm)` (which seemed somewhat more common) because
`elm-package-origin` needs `elm->package-name`, and it seemed better overall
to define the Elm-to-Guix and Guix-to-Elm conversions in the same place.

On 5/1/22 18:03, Philip McGrath wrote:
> Hi,
> 
> On 5/1/22 16:35, Ludovic Courtès wrote:
>> Could you add an entry for the importer under “Invoking guix import”,
>> and one for the build system under “Build Systems” in guix.texi?  You
>> can follow existing entries as a template.
>>
> 
> I will give it a try! I haven't written any Texinfo before.
>

I've added documentation there, and also a section on naming conventions in
contributing.texi. I'm not sure all of my Texinfo is idiomatic (well, I
didn't find equivalents for all of the Scribble cross-reference functions I'm
used to), but it does build and work ok.

>> It would be nice to have tests for the importer.  One way to do that is
>> like ‘tests/cpan.scm’, which spawns an HTTP server that mimics the real
>> registry.
>>
> 
> I'll take a look at that.
>

I've added a number of tests. They seem to be producing more "PASS" lines than
other tests, though. It seems like nested `test-group`s may be handled
differently than I'm used to from RackUnit and (less so) Racket's `srfi/64`
implementation, but I'm not sure what the idiomatic way to structure them
would be, if this isn't it.

>>> +;; COMMENTARY:
>>
>> Nitpick: You can make that literally “;;; Commentary:”.   That’s what
>> (ice-9 documentation) expects.
>>
>>> +;; CODE:
>>
>> Likewise: “;;; Code:”.
>>
> 
> Will do.
>

Ironically, I couldn't find any documentation for `(ice-9 documentation)`
other than the comments in the source file, but I hope I've done this
properly now.

>>
>> The way the importer fiddles with alists isn’t pretty IMO.  :-)
>>
>> How about using ‘define-json-mapping’ (also from Guile-JSON) to “map”
>> JSON data structures to records?  See how pypi.scm and others do it.
>> The resulting code should be clearer.
>>
> 
> I had tried that first, but there were some problems: IIRC, there might 
> have been an issue with potentially-absent fields defaulting to 
> *unspecified*, some alist manipulation was needed anyway for fields that 
> use JSON objects as key--value maps, and, with a view toward being able 
> to process `{"type":"application"}` files some day, there didn't seem to 
> be enough ability to adapt parsing based on the value for the key. I 
> found this code less confusing. But I can try again if it seems important!
>

Most of the problems I'd run into were because I'd gotten the wrong idea from
the grammar of 'define-json-mapping`, and `define-json-mapping` didn't check
its implicit requirements: I've reported the details and potential
improvements at <https://github.com/aconchillo/guile-json/issues/79>.

The remaining, unescapable alist manipulation is for cases when JSON objects
are used as key--value maps, rather than records with some finite set of
potential keys.

>> Also, instead of or in addition to memoizing ‘elm-package-registry’,
>> would it make sense to use ‘http-fetch/cached’ to fetch that file?
>>
> 
> I'll take a look!
>

Using `http-fetch/cached` without duplicating `json-fetch` required a few
additional patches:

 - [PATCH v2 06/34] http-client: Accept '#:headers' in 'http-fetched/cached'.
 - [PATCH v2 07/34] http-client: 'http-fetch/cached' converts strings to URIs.
 - [PATCH v2 08/34] import: json: Accept '#:http-fetch' in 'json-fetch'.

They seemed small enough, and IIUC they don't affect any build-side code or
trigger rebuilds, but OTOH the actuall JSON being fetched is quite small, so
it wouldn't be a problem to drop them if there are any problems.

One other difference is that `elm-virtual-dom` had an upstream release since
the first version of this patch series, so that package is now at 1.0.3.

 -Philip

Philip McGrath (34):
  gnu: elm-compiler: Update to 0.19.1.
  gnu: elm: Rename package to match the command.
  guix: Add elm-build-system.
  gnu: Add elm-core and elm-json.
  build-system/elm: Add implicit Elm inputs.
  http-client: Accept '#:headers' in 'http-fetched/cached'.
  http-client: 'http-fetch/cached' converts strings to URIs.
  import: json: Accept '#:http-fetch' in 'json-fetch'.
  import: Add Elm importer.
  gnu: Add elm-virtual-dom.
  gnu: Add elm-html.
  gnu: Add elm-svg.
  gnu: Add elm-time.
  gnu: Add elm-url.
  gnu: Add elm-browser.
  gnu: Add elm-bytes.
  gnu: Add elm-file.
  gnu: Add elm-http.
  gnu: Add elm-parser.
  gnu: Add elm-project-metadata-utils.
  gnu: Add elm-explorations-markdown.
  gnu: elm: Support 'elm reactor'.
  gnu: Add elm-todomvc.
  gnu: Add elm-debois-elm-dom.
  gnu: Add elm-random.
  gnu: Add elm-explorations-test.
  gnu: Add elm-danhandrea-elm-date-format.
  gnu: Add elm-danhandrea-elm-time-extra.
  gnu: Add elm-justinmimbs-date.
  gnu: Add elm-justinmimbs-time-extra.
  gnu: Add elm-myrho-elm-round.
  gnu: Add elm-ryannhg-date-format.
  gnu: Add elm-terezka-intervals.
  gnu: Add elm-terezka-elm-charts.

 Makefile.am                                   |   5 +
 doc/contributing.texi                         |  82 ++
 doc/guix.texi                                 |  80 ++
 gnu/local.mk                                  |   4 +-
 gnu/packages/elm.scm                          | 749 +++++++++++++++++-
 .../elm-compiler-disable-reactor.patch        |  71 --
 .../patches/elm-compiler-fix-map-key.patch    |  38 -
 .../elm-offline-package-registry.patch        |  71 ++
 .../patches/elm-reactor-static-files.patch    | 251 ++++++
 guix/build-system/elm.scm                     | 206 +++++
 guix/build/elm-build-system.scm               | 380 +++++++++
 guix/http-client.scm                          |  24 +-
 guix/import/elm.scm                           | 210 +++++
 guix/import/json.scm                          |   9 +-
 guix/scripts/import.scm                       |   3 +-
 guix/scripts/import/elm.scm                   | 107 +++
 tests/elm.scm                                 | 268 +++++++
 17 files changed, 2414 insertions(+), 144 deletions(-)
 delete mode 100644 gnu/packages/patches/elm-compiler-disable-reactor.patch
 delete mode 100644 gnu/packages/patches/elm-compiler-fix-map-key.patch
 create mode 100644 gnu/packages/patches/elm-offline-package-registry.patch
 create mode 100644 gnu/packages/patches/elm-reactor-static-files.patch
 create mode 100644 guix/build-system/elm.scm
 create mode 100644 guix/build/elm-build-system.scm
 create mode 100644 guix/import/elm.scm
 create mode 100644 guix/scripts/import/elm.scm
 create mode 100644 tests/elm.scm


base-commit: 665dd8211cb5c7556f0fb83944d33215accf957a