mbox series

[bug#69328,00/12] Better source code recovery from SWH

Message ID cover.1708697539.git.ludo@gnu.org
Headers
Series Better source code recovery from SWH |

Message

Ludovic Courtès Feb. 23, 2024, 2:22 p.m. UTC 
Hello Guix!

This patch series improves source code recovery from SWH, as a followup
to <https://issues.guix.gnu.org/68741>.

It does several things:

  • ‘guix lint -c archival’ now emits save requests for VCSes other
    than Git.

  • Fix <https://issues.guix.gnu.org/69070>.

  • Allow content-addressed recovery of Mercurial and Subversion
    checkouts.

  • Allow Bazaar recovery using ‘download-nar’ (I didn’t bother with SWH).

  • Have all these things honor the ‘GUIX_DOWNLOAD_SEQUENCE’ environment
    variable.

You can try the various methods like this:

  GUIX_DOWNLOAD_SEQUENCE=nar ./pre-inst-env guix build -S apl --check
  GUIX_DOWNLOAD_SEQUENCE=swh ./pre-inst-env guix build -S guile-wisp --check
  GUIX_DOWNLOAD_SEQUENCE=swh ./pre-inst-env guix build -S guile-gcrypt --check

In the last case, note that you must be running guix-daemon for the checkout
since that uses “builtin:git-download”, which is implemented on the server
side.

There’s a few caveats:

  • Mercurial SWH fallback almost works, but not quite, due to this SWH bug:
    <https://gitlab.softwareheritage.org/swh/infra/sysadm-environment/-/issues/5256>.

  • Right now, no Subversion checkout has the nar-sha256 ExtID at SWH for
    unclear reasons, so retrieving the source of ‘apl’ (say) from SWH
    doesn’t work yet.

  • Multi-directory Subversion downloads (‘svn-multi-fetch’) is not supported
    yet.  For that we’ll need to arrange with our SWH friends so they
    compute nar-sha256 ExtIDs for combined directories (and we’ll have to
    include that info in ‘sources.json’).

Feedback welcome!

Ludo’.

Ludovic Courtès (12):
  lint: Switch to SRFI-71.
  lint: archival: Fix crash in non-Git case.
  lint: archival: Trigger “Save Code Now” for VCSes other than Git.
  swh: Add ‘type’ field to <visit>.
  swh: ‘origin-visits’ takes an optional ‘max’ parameter.
  swh: ‘lookup-origin-revision’ handles branches pointing to
    directories.
  hg-download: Use ‘swh-download-directory-by-nar-hash’.
  svn-download: Use ‘swh-download-directory-by-nar-hash’.
  bzr-download: Implement nar fallback.
  download-nar: Distinguish ‘output’ and ‘item’ parameter.
  perform-download: Allow use of ‘download-nar’ for ‘--check’ builds.
  download: Honor ‘GUIX_DOWNLOAD_SEQUENCE’ environment variable.

 guix/build/bzr.scm                |   3 +-
 guix/build/download-nar.scm       |  12 +--
 guix/build/download.scm           |  50 +++++++---
 guix/build/git.scm                |  27 ++++--
 guix/bzr-download.scm             |  57 ++++++++---
 guix/cvs-download.scm             |  24 +++--
 guix/download.scm                 |  53 ++++-------
 guix/git-download.scm             |  20 ++--
 guix/hg-download.scm              |  36 ++++---
 guix/lint.scm                     | 151 +++++++++++++++++++-----------
 guix/scripts/perform-download.scm |  65 +++++++------
 guix/svn-download.scm             |  84 +++++++++++------
 guix/swh.scm                      |  71 ++++++++------
 tests/lint.scm                    |  20 ++++
 tests/swh.scm                     |  74 +++++++++++++++
 15 files changed, 501 insertions(+), 246 deletions(-)


base-commit: ffcce77ec488e3c89401ad77fafa65fcd9e9f5be

Comments

Ludovic Courtès Feb. 23, 2024, 3:53 p.m. UTC | #1 
I forgot to Cc: you Timothy, but you may have useful feedback to give on
this series: <https://issues.guix.gnu.org/69328>.

(Should we create a ‘source-code-archival’ team?)
Timothy Sample March 3, 2024, 4:54 a.m. UTC | #2 
Ludovic Courtès <ludo@gnu.org> writes:

> Hello Guix!
>
> This patch series improves source code recovery from SWH, as a followup
> to <https://issues.guix.gnu.org/68741>.
>
> It does several things:
>
>   • ‘guix lint -c archival’ now emits save requests for VCSes other
>     than Git.
>
>   • Fix <https://issues.guix.gnu.org/69070>.
>
>   • Allow content-addressed recovery of Mercurial and Subversion
>     checkouts.
>
>   • Allow Bazaar recovery using ‘download-nar’ (I didn’t bother with SWH).
>
>   • Have all these things honor the ‘GUIX_DOWNLOAD_SEQUENCE’ environment
>     variable.

Very nice!  I like the design of ‘GUIX_DOWNLOAD_SEQUENCE’ compared to
‘GUIX_DOWNLOAD_FALLBACK_TEST’, but I’m not sure about the name (sorry
for bike shedding!).  In particular, the “sequences” ‘(nar swh)’ and
‘(swh nar)’ will both try ‘nar’ first and then ‘swh’.  What about
“methods” or “strategies” or something?

> You can try the various methods like this:
>
>   GUIX_DOWNLOAD_SEQUENCE=nar ./pre-inst-env guix build -S apl --check
>   GUIX_DOWNLOAD_SEQUENCE=swh ./pre-inst-env guix build -S guile-wisp --check
>   GUIX_DOWNLOAD_SEQUENCE=swh ./pre-inst-env guix build -S guile-gcrypt --check

I tried

  GUIX_DOWNLOAD_SEQUENCE=disarchive ./pre-inst-env guix build -S mes --check

and it worked like a charm.

> Feedback welcome!

Other than the name and the little separate comment on the last patch,
this all LGTM.


-- Tim
Ludovic Courtès March 5, 2024, 10:58 a.m. UTC | #3 
Hi,

Timothy Sample <samplet@ngyro.com> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:

[...]

>>   • Have all these things honor the ‘GUIX_DOWNLOAD_SEQUENCE’ environment
>>     variable.
>
> Very nice!  I like the design of ‘GUIX_DOWNLOAD_SEQUENCE’ compared to
> ‘GUIX_DOWNLOAD_FALLBACK_TEST’, but I’m not sure about the name (sorry
> for bike shedding!).  In particular, the “sequences” ‘(nar swh)’ and
> ‘(swh nar)’ will both try ‘nar’ first and then ‘swh’.  What about
> “methods” or “strategies” or something?

Good point; I like “methods”.

> Other than the name and the little separate comment on the last patch,
> this all LGTM.

Awesome; I’ll send an updated version and merge by the end of the week
if nobody objects.

Ludo’.