[bug#60358] gnu: Add gnulib.

* gnu/packages/build-tools.scm (gnulib-latest): New variable. This one always
tracks the latest upstream commit, so don’t use it as an input.
(gnulib): New variable. This is a "recent" snapshot.
---
 gnu/packages/build-tools.scm | 71 ++++++++++++++++++++++++++++++++++++
 1 file changed, 71 insertions(+)


base-commit: 8f93a1e01a879ae026678dd92c18e2a2a49be540
  

Hello guix,

I’m trying to provide gnulib for guix. However, when I try to use a
git-checkout as the source, to always have the latest commit, then guix
fails with: 

updating checkout of 'https://git.savannah.gnu.org/git/gnulib.git'...
guix build: error: Git failure while fetching
https://git.savannah.gnu.org/git/gnulib.git: reference
'refs/remotes/origin/HEAD' not found

I don’t fully understand what happens. Do you have an idea?

Also, when using gnulib, the whole point is to have GNULIB_SRCDIR point
to a directory where the gnulib files are, so that bootstrap srcipts
can use that instead of a git submodule (which is not possible with
guix). I tried to set it as a search-path-specification, which seems to
work, but I’m not sure that it is the correct approach.

What do you think?

Best regards,

Vivien
  

Hi.  Many packages needs a specific checkout of gnulib to work reliably,
via --gnulib-refdir= (rather than --gnulib-srcdir=), would you consider
installing the entire gnulib git archive instead of just the latest
checkout?  Since so many packages these days require different versions
of gnulib, it may make sense to mak the gnulib git archive available as
an installed Guix package, so that other packages may bootstrap build
from it and get the intended gnulib git checkout.

If a checked out gnulib git archive is available as a Guix package,
other packages could build-depend on it and use both --gnulib-refdir and
--gnulib-srcdir.  If I understand your approach now, only
--gnulib-srcdir is possible.

You could put the git archive under /share/gnulib-git or merely
/share/gnulib/ and set both GNULIB_REFDIR and GNULIB_SRCDIR.

/Simon
  

Am Dienstag, dem 27.12.2022 um 17:23 +0100 schrieb Vivien Kraus:
> * gnu/packages/build-tools.scm (gnulib-latest): New variable. This
> one always
> tracks the latest upstream commit, so don’t use it as an input.
> (gnulib): New variable. This is a "recent" snapshot.
The preferred package:patch ratio is 1:1.
> ---
>  gnu/packages/build-tools.scm | 71
> ++++++++++++++++++++++++++++++++++++
>  1 file changed, 71 insertions(+)
> 
> diff --git a/gnu/packages/build-tools.scm b/gnu/packages/build-
> tools.scm
> index 6c1350c44f..062956f685 100644
> --- a/gnu/packages/build-tools.scm
> +++ b/gnu/packages/build-tools.scm
> @@ -38,6 +38,7 @@ (define-module (gnu packages build-tools)
>    #:use-module (guix gexp)
>    #:use-module (guix download)
>    #:use-module (guix git-download)
> +  #:use-module (guix git)
>    #:use-module (guix build-system cmake)
>    #:use-module (gnu packages)
>    #:use-module (gnu packages adns)
> @@ -48,11 +49,13 @@ (define-module (gnu packages build-tools)
>    #:use-module (gnu packages cpp)
>    #:use-module (gnu packages elf)
>    #:use-module (gnu packages linux)
> +  #:use-module (gnu packages lisp)
>    #:use-module (gnu packages logging)
>    #:use-module (gnu packages lua)
>    #:use-module (gnu packages ninja)
>    #:use-module (gnu packages package-management)
>    #:use-module (gnu packages pcre)
> +  #:use-module (gnu packages perl)
>    #:use-module (gnu packages pkg-config)
>    #:use-module (gnu packages pretty-print)
>    #:use-module (gnu packages protobuf)
> @@ -803,3 +806,71 @@ (define-public genie
>  same settings to multiple projects.  It supports generating projects
> using GNU
>  Makefiles, JSON Compilation Database, and experimentally Ninja.")
>        (license license:bsd-3))))
> +
> +;; Gnulib is a source code library. The developers encourage you to
> develop
> +;; your programs with the latest commit of gnulib. However, gnulib
> is also
> +;; used to bootstrap some GNU programs. To avoid rebuilding the
> world when a
> +;; new gnulib commit is pushed, gnulib as a native input must use a
> known
> +;; snapshot of the library.
> +
> +(define-public gnulib-latest
> +  ;; Use this to develop.
> +  (package
> +    (name "gnulib-latest")
> +    (version "latest")
> +    (source
> +     (git-checkout
> +      (url "https://git.savannah.gnu.org/git/gnulib.git")))
Don't.  If you need to check out a particular commit, use the --with-
commit CLI option or a transformer.
> +    (build-system gnu-build-system)
> +    (arguments
> +     (list
> +      #:phases
> +      #~(modify-phases %standard-phases
> +          (delete 'configure)
> +          (replace 'install
> +            (lambda _
> +              (install-file "gnulib-tool"
> +                            (string-append #$output "/bin"))
> +              (copy-recursively "."
> +                                (string-append #$output
> "/share/gnulib-srcdir")))))
> +      #:tests? #f)) ;; Tests are syntax and indentation checks for
> the
> +                    ;; maintainer.
> +    (inputs ;; Shebangs for some auxiliary build files.
> +     (list python perl clisp))
> +    (home-page "https://www.gnu.org/software/gnulib/")
> +    (synopsis "GNU portability library")
> +    (description
> +     "Gnulib is a library that provides common functions from the
> glibc to your
> +programs, in order to enhance portability across operating systems.
> It also
> +provides common maintainer tools for gnu packages. This package
> always tracks
> +the latest commit in gnulib. @strong{For packages dependencies,
> please use the
> +@code{gnulib} package instead.} The latter provides a recent enough
> snapshot.")
> +    (native-search-paths
> +     (list
> +      (search-path-specification
> +       (variable "GNULIB_SRCDIR")
> +       (files (list "share/gnulib-srcdir")))))
> +    (license
> +     (list license:lgpl2.0+ license:gpl3+))))
> +
> +(define-public gnulib
> +  ;; Use this as a native input.
> +  (package
> +    (inherit gnulib-latest)
> +    (name "gnulib")
> +    (version "2022-12-27")
> +    (source
> +     (origin
> +       (method git-fetch)
> +       (uri (git-reference
> +             (url "https://git.savannah.gnu.org/git/gnulib.git")
> +             (commit "fde75446490e18d2539817ca418ab8adf73b02d3")))
Don't throw together raw commits and commit-less versions.  Use git-
version instead.
> +       (sha256
> +        (base32
> +         "0fjbdhwi9025wyq39rwc2j6aazfmagx056kkbvxx6bs97i80dcim"))))
> +    (synopsis "GNU portability library")
> +    (description
> +     "Gnulib is a library that provides common functions from the
> glibc to your
> +programs, in order to enhance portability across operating systems.
> It also
> +provides common maintainer tools for gnu packages. This package
> provides a
> +recent snapshot of gnulib.")))

Cheers
  

Hello!

Le mardi 27 décembre 2022 à 19:44 +0100, Simon Josefsson a écrit :
> Hi.  Many packages needs a specific checkout of gnulib to work
> reliably,
> via --gnulib-refdir= (rather than --gnulib-srcdir=), would you
> consider
> installing the entire gnulib git archive instead of just the latest
> checkout?  

I did not know about gnulib-refdir. Providing the entire gnulib archive
is tempting, but there might be downsides. The contents of the .git
directory depends on how trees and objects have been packed by git. As
far as I understand, it is possible that pushing a commit in gnulib
results in commit objects from previous commits to be re-packed. Thus,
if I clone gnulib from the initial commit to a specific commit, the
result might depend on other, unrelated commits. So, the content of
.git is not reproducible.

Maybe I could get around that by deleting all the refs, doing an
aggressive garbage collection and then re-packing, but I’m not sure I
would get a reproducible result.

Now, I don’t know much about gnulib-refdir. How does gnulib-refdir
work? Do you have an example of a package that uses this feature? Can I
convince the package to use a checkout instead of the gnulib
repository, if I swear I have the exact checkout it wants, maybe by
tweaking bootstrap.conf or something?

Vivien
  

Vivien Kraus via Guix-patches via <guix-patches@gnu.org> writes:

> Hello!
>
> Le mardi 27 décembre 2022 à 19:44 +0100, Simon Josefsson a écrit :
>> Hi.  Many packages needs a specific checkout of gnulib to work
>> reliably,
>> via --gnulib-refdir= (rather than --gnulib-srcdir=), would you
>> consider
>> installing the entire gnulib git archive instead of just the latest
>> checkout?  
>
> I did not know about gnulib-refdir.

It is newer than --gnulib-srcdir, but came about because --gnulib-srcdir
is often a fragile solution: you have no idea which gnulib version the
person building the package supplied.  Since gnulib is rolling,
compatibility becomes difficult.  The --gnulib-srcdir approach works if
you make sure to use the same gnulib git checkout as the project you
wish to build uses.  But then it becomes difficult to package gnulib: no
two projects are likely to rely on the same gnulib git commit.  So which
gnulib git version to package?

That's the motivation for packaging the gnulib git repository instead.
This may sound strange, but compare how gettext/autopoint ships a CVS
repository and checks out the particular files that are needed.

Yeah, I can agree that this approach is not ideal, and there are many
concerns with it.  I'm not convinced gnulib's idea of "source-level
library" is something that is viable long-term.  But it is what exists
today.

> Providing the entire gnulib archive is tempting, but there might be
> downsides. The contents of the .git directory depends on how trees and
> objects have been packed by git. As far as I understand, it is
> possible that pushing a commit in gnulib results in commit objects
> from previous commits to be re-packed. Thus, if I clone gnulib from
> the initial commit to a specific commit, the result might depend on
> other, unrelated commits. So, the content of .git is not reproducible.
>
> Maybe I could get around that by deleting all the refs, doing an
> aggressive garbage collection and then re-packing, but I’m not sure I
> would get a reproducible result.

Interesting -- I think researching this more would be useful.  It should
be possible to come up with a safe approach to produce a reproducible
checkout of a git repository.

Doesn't 'git archive' produce a reproducible output from a git
repository for a particular branch and commit?

> Now, I don’t know much about gnulib-refdir. How does gnulib-refdir
> work?

You supply --gnulib-refdir pointing to a local gnulib git repository
clone when you run ./bootstrap.  This avoid checking out the gnulib git
submodule from Savannah, and instead ./bootstrap will use the local git
repository instead.  I believe it should automatically extract the
intended gnulib git commit from the gnulib/ git submodule, and extract
that version from the local copy (please test -- may be bugs).

> Do you have an example of a package that uses this feature?

Packages wouldn't use it, but if they use gnulib's ./bootstrap script
the support this approach.  The idea is that people building projects
that use gnulib doesn't always have to fetch the gnulib git submodule,
but have a local copy for security or performance reasons.

> Can I convince the package to use a checkout instead of the gnulib
> repository, if I swear I have the exact checkout it wants, maybe by
> tweaking bootstrap.conf or something?

You can force a package to use another gnulib version by using
--gnulib-srcdir and point that to some other version of gnulib.  But the
project may not build.  The point of --gnulib-refdir is to actually get
the same version of gnulib that the project uses, why would you not want
that?  There is the GNULIB_REVISION environment variable that you can
set to something else, but this is probably not very well tested and
sounds like a bad idea (but I may be missing something).

/Simon
  

Simon Josefsson via Guix-patches via <guix-patches@gnu.org> writes:

> Doesn't 'git archive' produce a reproducible output from a git
> repository for a particular branch and commit?

Sorry, git-archive that doesn't include the .git/ sub-directory, so
that's not the command you want.  I don't know what command you would
use.  I reckon it should be a git clone, check out a particular commit,
reset the branch to that commit, filtering away all other branches, and
then do 'git gc --aggresive --prune=now'.  If that produces
unreproducable results for other unrelated changes to the repository, it
would be nice to know how to get rid of them.

/Simon
  

I’m trying to learn how to make QA happy. I have not changed anything from
last time, but now I’m trying the "deep" thread style with a cover letter for
patches on my end, so maybe it will work?

Sorry for the spam.

Vivien Kraus (4):
  gnu: libunibreak: Use ucd instead of ucd-next.
  gnu: ucd-next: Update to 15.0.0.
  gnu: Add gnulib.
  gnu: guile-gnutls: Update to 3.7.11.

 gnu/packages/build-tools.scm | 202 +++++++++++++++++++++++++++++++++++
 gnu/packages/tls.scm         |  42 +++++---
 gnu/packages/unicode.scm     |   9 +-
 3 files changed, 235 insertions(+), 18 deletions(-)


base-commit: 6d4c53893308d7db340a2978307a214103660341
  

I tried the "deep" thread style, but unfortunately it seems that Patchwork
only picked the last commit. My guess is that the email thread model that
Patchwork recognizes is one cover letter, and every patch links to the cover
letter. Maybe the deep thread style means that Patchwork recognizes the last
commit as a new commit, looks up its parent and interprets it as the cover
letter. So, maybe the penultimate patch was interpreted as the cover letter
and so only the last patch made its way.

I understand that you are upset because of the spam. I don’t like that either
when it happens to me.

I have more or less exhausted all the options that I can easily tweak in order
to make the patch work with patchwork. If this doesn’t work, then I won’t try
again and I will let the QA fail.

That’s my last "sorry for the spam" on this issue. Sorry for the spam!

Vivien Kraus (4):
  gnu: libunibreak: Use ucd instead of ucd-next.
  gnu: ucd-next: Update to 15.0.0.
  gnu: Add gnulib.
  gnu: guile-gnutls: Update to 3.7.11.

 gnu/packages/build-tools.scm | 202 +++++++++++++++++++++++++++++++++++
 gnu/packages/tls.scm         |  42 +++++---
 gnu/packages/unicode.scm     |   9 +-
 3 files changed, 235 insertions(+), 18 deletions(-)


base-commit: 6d4c53893308d7db340a2978307a214103660341
  

I added my name to the copyright line of build-tools (for gnulib) and also of
tls (for guile-gnutls).

I used the 2022 year because the initial work was done in 2022.

Vivien Kraus (4):
  gnu: libunibreak: Use ucd instead of ucd-next.
  gnu: ucd-next: Update to 15.0.0.
  gnu: Add gnulib.
  gnu: guile-gnutls: Update to 3.7.11.

 gnu/packages/build-tools.scm | 203 +++++++++++++++++++++++++++++++++++
 gnu/packages/tls.scm         |  43 +++++---
 gnu/packages/unicode.scm     |   9 +-
 3 files changed, 237 insertions(+), 18 deletions(-)


base-commit: 7efcf36e3b753a1dba6f8208f3c22d151007eaf0
  

Dear guix,

gnulib is a useful tool to recreate distribution tarballs from git on
many GNU projects. In the future, it may be a native input for many.
Should I move it to its own module, (gnu packages gnulib) for instance?
That would certainly help with forseeable mess with circular guile
module dependencies.

What do you think?

Best regards,

Vivien