| Message ID | 877d8udq44.fsf@reilysiegel.com |
|---|---|
| State | New |
| Headers | show |
| Series | None | expand |
Reily Siegel schreef op wo 16-03-2022 om 13:43 [+0100]: > + ;; Tests fail when AOT'd. > + #:aot-exclude '(#:all))) Looks like there's a bug somewhere then. Has this been reported? Could a link be added to the report so we know when the #:aot-exclude can be removed? If it's an issue with ant-build-system, can this be reported? Greetings, Maxime.
Reily Siegel schreef op wo 16-03-2022 om 13:43 [+0100]: > +First-in-first-out (clojure.core.memoize/fifo) > + > +Least-recently-used (clojure.core.memoize/lru) > + > +Least-used (clojure.core.memoize/lu) > + > +Time-to-live (clojure.core.memoize/ttl) Like in a reply to a previous package, here @itemize can be used. And @lisp or @code for clojure.core.memoize/foo. Greetings, Maxime.
Reily Siegel schreef op wo 16-03-2022 om 13:43 [+0100]:
> + (synopsis "Manipulable, pluggable, memoization framework for Clojure")
I would hope that it's manipulable, how would it be an useful library,
if it cannot be used? Also, the notion of ‘framework’ is rather vague.
Given the mentioning of various caching strategies, WDYT of:
(synopsis
"Customisable memoization library supporting automatic removal")
AFAICT, usually, memoization is presented without any automatical removal, instead
old entries are preserved forever -- that's (one of the) the difference(s) between
memoization and caching.
Greetings,
Maxime.
Reily Siegel schreef op wo 16-03-2022 om 13:43 [+0100]: > +An underlying [...] > +Memoization builders for implementations of common caching strategies, > +including: ‘An underlying foo that bars’ and ‘Memoization builders for foo of bar’ seems rather convoluted phrasing, maybe it can simplified a bit: ‘This memoization library supports various cache replacement strategies: @itemize @item @acronym{FIFO, first in first out} @item @acronym{LRU, least recently used} @item @acronym{TLL, time to live} @item no replacement, the memoisation cache can grow indefinitely @end itemize’ For the exact class and interface names, the user can look at the documentation of core.memoize. Greetings, Maxime.
Maxime Devos <maximedevos@telenet.be> writes: > Looks like there's a bug somewhere then. Has this been reported? > Could a link be added to the report so we know when the #:aot-exclude > can be removed? If it's an issue with ant-build-system, can this be > reported? Clojure libraries are usually not designed to be AOT-compiled, and are largely distributed in source form. The fact that Guix's clojure-build-system chooses to AOT all Clojure code by default is confusing to me. At some point, this default should probably be changed because many Clojure libraries rely on the fact that they are distributed uncompiled to work properly.
Reily Siegel schreef op vr 18-03-2022 om 10:01 [+0100]: > > Looks like there's a bug somewhere then. Has this been reported? > > Could a link be added to the report so we know when the #:aot- > > exclude > > can be removed? If it's an issue with ant-build-system, can this > > be > > reported? > > Clojure libraries are usually not designed to be AOT-compiled, and > are largely distributed in source form. The fact that Guix's > clojure-build-system chooses to AOT all Clojure code by default is > confusing to me. For languages for which a compiler is available, AOT is rather standard. (E.g., C, C++, Guile Scheme, Python, Java, Fortran). Apparently Clojure has a compiler as well. Basically, why not AOT when it is possible, instead of delaying compilation until runtime? > At some point, this default should probably be changed because many > Clojure libraries rely on the fact that they are distributed > uncompiled to work properly. How can a Clojure library rely on this fact in the first place? I don't quite see how -- for comparison, there are a few methods in Guile for detecting if it is has been compiled or is being interpreted, e.g. some uses of 'eval-when', '%load-compiled-path' or 'procedure- name', but they are rather convoluted and unused (except in examples). Also, if a Clojure library misbehaves when being AOT-compiled, without additional context, that seems like a bug in the Clojure library to me (or the AOT-compilation code). Greetings, Maxime.
Maxime Devos <maximedevos@telenet.be> writes: > Also, if a Clojure library misbehaves when being AOT-compiled, without > additional context, that seems like a bug in the Clojure library to me > (or the AOT-compilation code). When I first started running into these issues, I reached out to Clojure maintainers on the Clojurians Slack, and the response was that AOT compilation is only meant for complete applications to reduce startup latency, not individual libraries. Here are some of the specific responses I got. Alex Miller: > Well, guix’s system is wrong to [AOT library code]. Lots of Clojure > libs have clj files that are not part of the lib. Sean Corfield: > Packaging a library as AOT'd code is likely to cause horrible problems > for consumers of that library. You're aware that lots of Clojure > libraries are packaged as JAR files with no compilation of source > code? "Packaging" Clojure libraries seems kinda wrong on several > levels. The normal way to get Clojure libraries is as dependencies in > a project -- fetched by the Clojure tooling itself, when it is run.
Reily Siegel schreef op za 19-03-2022 om 20:07 [+0100]: > When I first started running into these issues, I reached out to Clojure > maintainers on the Clojurians Slack, and the response was that AOT > compilation is only meant for complete applications to reduce startup > latency, not individual libraries. Here are some of the specific > responses I got. [...] On which day was this, and on which channel? I'd like to have a look at the rest of the responses (on the archives at <https://clojurians-log.clojureverse.org/>) for some additional context. Greetings, Maxime.
Reily Siegel schreef op za 19-03-2022 om 20:07 [+0100]: > Sean Corfield: > > > [...] "Packaging" Clojure libraries seems kinda wrong on several > > levels. The normal way to get Clojure libraries is as dependencies in > > a project -- fetched by the Clojure tooling itself, when it is run. This defeats the point of Guix and would come with all the same problems that ‘just use the language-specific, unmoderated package registry’ (NPM, Cargo, Maven (*), ...). (*) TBC, Maven doesn't seem to cause any packaging trouble in Guix -- but from a security POV and ‘oops there are some references to binaries that need to be corrected to an absolute file name’ POV it is not sufficient. Probably likewise for whatever Clojure has in place, ignoring the occasional AOT problem. Greetings, Maxime.
Reily Siegel schreef op za 19-03-2022 om 20:07 [+0100]: > > Packaging a library as AOT'd code is likely to cause horrible > > problems for consumers of that library. This just states ‘there will be horrible problems’ but it doesn't explain at all which these problems would be. > > You're aware that lots of Clojure libraries are packaged as > > JAR files with no compilation of source code? I don't see the relation between this sentence and the previous. Also, yes, that's the idea: upstream provides the raw source code, downstream (Guix) converts it in ready-to-use pre-compiled binaries. Greetings, Maxime.
Reily Siegel schreef op za 19-03-2022 om 20:07 [+0100]: > > Also, if a Clojure library misbehaves when being AOT-compiled, > > without > > additional context, that seems like a bug in the Clojure library to > > me > > (or the AOT-compilation code). > > When I first started running into these issues, I reached out to > Clojure > maintainers on the Clojurians Slack, and the response was that AOT > compilation is only meant for complete applications to reduce startup > latency, not individual libraries. Here are some of the specific > responses I got. [...] This does not explain what these AOT issues actually are. Do you receive some error message or backtrace or something when building or running tests? Greetings, Maxime.
Reily Siegel schreef op za 19-03-2022 om 20:07 [+0100]: > When I first started running into these issues, I reached out to Clojure > maintainers on the Clojurians Slack, and the response was that AOT > compilation is only meant for complete applications to reduce startup > latency, not individual libraries. That seems backwards to me. Wouldn't AOTing compiling individual libraries be more efficient than AOTing individual applications? More concretely: Suppose app A and B consists of a single source file A.clj and B.clj respectively, and each have library C (with C.clj) as dependency. Then, what Guix currently does, is AOT'ing C.lj when building C, then AOT'ing A.clj when building A and AOT'ing B when building B. In total, AOT'ing a file happens thrice here. What seems to be implied here, is that the AOT'ing needs to be delayed to A and B. In this case, C.clj would be AOT'd twice: once for A, and once for B. In total, AOT'ing a file happens four times here. As such, wouldn't AOT'ing the applications instead of the libraries be less efficient? An additional problem here is grafting. If the AOT'ing was delayed to the application packages, then the libraries cannot be grafted, for the same reason that (C, C++, Go, ...) static libraries cannot be grafted. Greetings, Maxime.
Reily Siegel schreef op za 19-03-2022 om 20:07 [+0100]: > and the response was that AOT compilation is only meant > for complete applications to reduce startup latency, not > individual libraries. I don't see the relevancy of that what Guix does, is apparently not meant to be done. If the thing that is meant to be done is problematic and a better option exists (w.r.t. grafting and build times, see other e-mail), wouldn't it it better to ignore the officiously (*) proscribed purpose of the AOT here? (*) ‘unofficially sort-of official’ Greetings, Maxime.
Maxime Devos schreef op za 19-03-2022 om 20:46 [+0100]: > An additional problem here is grafting. If the AOT'ing was delayed > to the application packages, then the libraries cannot be grafted, > for the same reason that (C, C++, Go, ...) static libraries cannot be > grafted. Another benefit is that intermediate libraries will fail to build if there are missing dependencies or syntax errors and perhaps if the dependencies are out-of-date, which would make updating intermediate libraries because it would be easier to see if the packaged library actually works. Greetings, Maxime.
Maxime Devos <maximedevos@telenet.be> writes: > On which day was this, and on which channel? I'd like to have a look > at the rest of the responses (on the archives at > <https://clojurians-log.clojureverse.org/>) for some additional > context. https://clojurians.slack.com/archives/C6QH853H8/p1642208631059100 Here is a link to the conversation I had, everything discussed was as a reply to this message. As of now, my only goal is to package the Clojure libraries needed to make the Clojure CLI work, as that is the community standard way of accessing Clojure libraries. This seems acceptable, as NPM, Nix, etc are also packaged.
Reily Siegel schreef op ma 21-03-2022 om 09:35 [+0100]: > Maxime Devos <maximedevos@telenet.be> writes: > > > On which day was this, and on which channel? I'd like to have a look > > at the rest of the responses (on the archives at > > <https://clojurians-log.clojureverse.org/>) for some additional > > context. > > https://clojurians.slack.com/archives/C6QH853H8/p1642208631059100 That page is not publicly visible, it asks for signing in with Google or Apple. Greetings, Maxime.
Reily Siegel schreef op ma 21-03-2022 om 09:35 [+0100]: > This seems acceptable, as NPM, Nix, etc are > also packaged. NPM does not appear to be packaged in Guix?
Reily Siegel schreef op ma 21-03-2022 om 09:35 [+0100]: > As of now, my only goal is to package the Clojure libraries needed to > make the Clojure CLI work, as that is the community standard way of > accessing Clojure libraries. Ok.
Maxime Devos schreef op za 19-03-2022 om 20:15 [+0100]: > Reily Siegel schreef op za 19-03-2022 om 20:07 [+0100]: > > When I first started running into these issues, I reached out to > Clojure > > maintainers on the Clojurians Slack, and the response was that AOT > > compilation is only meant for complete applications to reduce > startup > > latency, not individual libraries. Here are some of the specific > > responses I got. [...] > > On which day was this, and on which channel? I'd like to have a look > at the rest of the responses (on the archives at > <https://clojurians-log.clojureverse.org/>) for some additional > context. More concretely, I'm looking for answers to things like ‘how to detect which files can be compiled’, ‘how to determine if a build failure is caused by AOT problems’, ‘how to know in advance if AOT might cause problems’ and ‘should AOT compilation be disabled by default for tests?’. If some answers can be found, perhaps adding #:aot-exclude can be automated in (guix build clojure-build-system), and thsese answers could help future packagers. Greetings, Maxime.
Maxime Devos <maximedevos@telenet.be> writes: > More concretely, I'm looking for answers to things like ‘how to detect > which files can be compiled’, ‘how to determine if a build failure is > caused by AOT problems’, ‘how to know in advance if AOT might cause > problems’ and ‘should AOT compilation be disabled by default for > tests?’. > > If some answers can be found, perhaps adding #:aot-exclude can be > automated in (guix build clojure-build-system), and thsese answers > could help future packagers. It seems to me that the easiest way to do this is to by default not AOT compile anything, and if a library needs a file AOT compiled, it can be set when packaged. I don't believe there is any good way to automate this, at least not with the sort of problems I have run into.
Reily Siegel schreef op ma 21-03-2022 om 12:27 [+0100]: > It seems to me that the easiest way to do this is to by default not AOT > compile anything, and if a library needs a file AOT compiled, it can be > set when packaged. As I understand it, no package strictly _needs_ to be AOT compiled. It does, however, appear to be a nice optimisation and has some tangential benefits (e.g. error out early in case of syntax errors), so to me it seems better to keep AOT on by default. Greetings, Maxime.
Reily Siegel schreef op wo 16-03-2022 om 13:43 [+0100]: > [...] > + ;; Tests fail when AOT'd. > + #:aot-exclude '(#:all))) It seems to me that the easiest way to investigate what the issue actually is, is to actually build it (without #:aot-exclude) and look at the build log, so WDYT of first reviewing the other non-reviewed patches first (ignoring AOT), and leaving the investigation of #:aot- exclude for last? Also, the build log would be useful to investigate. Greetings, Maxime.
diff --git a/gnu/packages/clojure.scm b/gnu/packages/clojure.scm index 37cea74c95..d6c8218f93 100644 --- a/gnu/packages/clojure.scm +++ b/gnu/packages/clojure.scm @@ -341,6 +341,57 @@ (define-public clojure-core-match (home-page "https://github.com/clojure/core.match") (license license:epl1.0)))) +(define-public clojure-core-memoize + (package + (name "clojure-core-memoize") + (version "1.0.253") + (home-page "https://github.com/clojure/core.memoize") + (source (origin + (method git-fetch) + (uri (git-reference + (url home-page) + (commit (string-append "v" version)))) + (file-name (git-file-name name version)) + (sha256 + (base32 + "1qky54v114sh3xn0lffwy7xx3wnnayk07fr2nvhd4lih84sv6rdz")))) + (build-system clojure-build-system) + (arguments + '(#:source-dirs '("src/main/clojure") + #:test-dirs '("src/test/clojure") + #:doc-dirs '("docs") + ;; Tests fail when AOT'd. + #:aot-exclude '(#:all))) + (propagated-inputs (list clojure-core-cache)) + (synopsis "Manipulable, pluggable, memoization framework for Clojure") + (description "@code{core.memoize} is a Clojure contrib library providing +the following features: + +An underlying PluggableMemoization protocol that allows the use of +customizable and swappable memoization caches that adhere to the synchronous +CacheProtocol found in core.cache + +Memoization builders for implementations of common caching strategies, +including: + +First-in-first-out (clojure.core.memoize/fifo) + +Least-recently-used (clojure.core.memoize/lru) + +Least-used (clojure.core.memoize/lu) + +Time-to-live (clojure.core.memoize/ttl) + +Naive cache (memo) that duplicates the functionality of Clojure's memoize +function but, unlike the built-in memoize function, ensures that in the case +of concurrent calls with the same arguments, the memoized function is only +invoked once; in addition memo can use metadata from the memoized function to +ignore certain arguments for the purpose of creating the cache key, e.g., +allowing you to memoize clojure.java.jdbc functions where the first argument +includes a (mutable) JDBC Connection object by specifying +:clojure.core.memoize/args-fn rest in the metadata") + (license license:epl1.0))) + (define-public clojure-data-codec (package (name "clojure-data-codec")