mbox series

[bug#48971,0/2] Add SWH support for Hg repositories

Message ID cover.1623498543.git.public@yoctocell.xyz
Headers show
Series Add SWH support for Hg repositories | expand

Message

Xinglu Chen June 12, 2021, 11:52 a.m. UTC
This series adds SWH support for Hg repositories, meaning that if the
upstream Hg repository is missing, Guix will try to fetch the source
from SWH.  SWH has archived more than 250 000 Hg repositories that were
previously hosted od BitBucket, so I think will be a useful feature.  :)

  https://www.softwareheritage.org/2020/09/17/250-000-endangered-mercurial-repositories-rescued/

Xinglu Chen (2):
  swh: Support lookup of Hg tags.
  hg-download: Support falling back to SWH.

 guix/hg-download.scm | 31 ++++++++++++++++++++++++++++---
 guix/swh.scm         | 10 ++++++++--
 2 files changed, 36 insertions(+), 5 deletions(-)


base-commit: 00727470b92e0b8ab0e00dd2cd87495c0c136341

Comments

Ludovic Courtès June 14, 2021, 4:44 p.m. UTC | #1
Hello,

Xinglu Chen <public@yoctocell.xyz> skribis:

> Xinglu Chen (2):
>   swh: Support lookup of Hg tags.
>   hg-download: Support falling back to SWH.

Pushed as 69d7333217ce85d9d1643a8349757b6b170d3b1f!

Note that currently this only works for (1) repositories that are
archived, and (2) origins that refer to hg tags, not commit IDs.

I found an example that meets these criteria: python-openpyxl.  :-)
I commented out the other fallbacks in (guix hg-download) and ran:

--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env guix build python-openpyxl -S --no-substitutes 
The following derivation will be built:
   /gnu/store/71xjlgdqw50a76ycqrmjrqh7b4mdi5s7-python-openpyxl-3.0.5-checkout.drv
building /gnu/store/71xjlgdqw50a76ycqrmjrqh7b4mdi5s7-python-openpyxl-3.0.5-checkout.drv...
guile: warning: failed to install locale
environment variable `PATH' set to `/gnu/store/378zjf2kgajcfd7mfr98jn5xyc5wa3qv-gzip-1.10/bin:/gnu/store/sf3rbvb6iqcphgm1afbplcs72hsywg25-tar-1.32/bin'
Trying to download from Software Heritage...
SWH: found revision 4d4e19e58314de481b98f4c239c3b3e9fd080613 with directory at 'https://archive.softwareheritage.org/api/1/directory/fb5bbba156e6b8e2bb94ea1dd5807ef2c8931864/'
SWH vault: requested bundle cooking, waiting for completion...
SWH vault: requested bundle cooking, waiting for completion...
SWH vault: requested bundle cooking, waiting for completion...
SWH vault: requested bundle cooking, waiting for completion...
SWH vault: requested bundle cooking, waiting for completion...
SWH vault: requested bundle cooking, waiting for completion...
SWH vault: requested bundle cooking, waiting for completion...
--8<---------------cut here---------------end--------------->8---

It’s still waiting for the bundle…

Regarding #2, there are some good news:
<https://forge.softwareheritage.org/T2849> is about implementing a
mapping from “VCS IDs” (hg commit IDs in this case) to SWHIDs.  Thus,
eventually, ‘lookup-origin-revision’ should be able to take an hg commit
ID and to map it to an SWHID.  That JSON API isn’t exposed yet.

Thanks,
Ludo’.
Xinglu Chen June 15, 2021, 5:37 a.m. UTC | #2
On Mon, Jun 14 2021, Ludovic Courtès wrote:

> Hello,
>
> Xinglu Chen <public@yoctocell.xyz> skribis:
>
>> Xinglu Chen (2):
>>   swh: Support lookup of Hg tags.
>>   hg-download: Support falling back to SWH.
>
> Pushed as 69d7333217ce85d9d1643a8349757b6b170d3b1f!
>
> Note that currently this only works for (1) repositories that are
> archived, and (2) origins that refer to hg tags, not commit IDs.
>
> I found an example that meets these criteria: python-openpyxl.  :-)
> I commented out the other fallbacks in (guix hg-download) and ran:
>
> --8<---------------cut here---------------start------------->8---
> $ ./pre-inst-env guix build python-openpyxl -S --no-substitutes 
> The following derivation will be built:
>    /gnu/store/71xjlgdqw50a76ycqrmjrqh7b4mdi5s7-python-openpyxl-3.0.5-checkout.drv
> building /gnu/store/71xjlgdqw50a76ycqrmjrqh7b4mdi5s7-python-openpyxl-3.0.5-checkout.drv...
> guile: warning: failed to install locale
> environment variable `PATH' set to `/gnu/store/378zjf2kgajcfd7mfr98jn5xyc5wa3qv-gzip-1.10/bin:/gnu/store/sf3rbvb6iqcphgm1afbplcs72hsywg25-tar-1.32/bin'
> Trying to download from Software Heritage...
> SWH: found revision 4d4e19e58314de481b98f4c239c3b3e9fd080613 with directory at 'https://archive.softwareheritage.org/api/1/directory/fb5bbba156e6b8e2bb94ea1dd5807ef2c8931864/'
> SWH vault: requested bundle cooking, waiting for completion...
> SWH vault: requested bundle cooking, waiting for completion...
> SWH vault: requested bundle cooking, waiting for completion...
> SWH vault: requested bundle cooking, waiting for completion...
> SWH vault: requested bundle cooking, waiting for completion...
> SWH vault: requested bundle cooking, waiting for completion...
> SWH vault: requested bundle cooking, waiting for completion...
> --8<---------------cut here---------------end--------------->8---
>
> It’s still waiting for the bundle…

Hmm, I tried it on a package in local config and it seemed to work fine
with commit IDs, I guess this was because I ran ‘guix hash -r’ on the
extracted tarball I downloaded from SWH, so the source was already in
the store when I tried to build it...

> Regarding #2, there are some good news:
> <https://forge.softwareheritage.org/T2849> is about implementing a
> mapping from “VCS IDs” (hg commit IDs in this case) to SWHIDs.  Thus,
> eventually, ‘lookup-origin-revision’ should be able to take an hg commit
> ID and to map it to an SWHID.  That JSON API isn’t exposed yet.

Oh, that’s great!
Simon Tournier June 16, 2021, 5:23 a.m. UTC | #3
Hi,

On Sat, 12 Jun 2021 at 13:52, Xinglu Chen <public@yoctocell.xyz> wrote:
> This series adds SWH support for Hg repositories, meaning that if the
> upstream Hg repository is missing, Guix will try to fetch the source
> from SWH.  SWH has archived more than 250 000 Hg repositories that were
> previously hosted od BitBucket, so I think will be a useful feature.  :)
>
>   https://www.softwareheritage.org/2020/09/17/250-000-endangered-mercurial-repositories-rescued/

Cool!  Thanks for working on this.


> Xinglu Chen (2):
>   swh: Support lookup of Hg tags.
>   hg-download: Support falling back to SWH.
>
>  guix/hg-download.scm | 31 ++++++++++++++++++++++++++++---
>  guix/swh.scm         | 10 ++++++++--
>  2 files changed, 36 insertions(+), 5 deletions(-)

LGTM.

All the best,
simon