mbox

[bug#45460,0/5] Zstd support for 'guix publish' and 'guix substitute'

Message ID 20201227141327.10827-1-ludo@gnu.org
Headers show

Message

Ludovic Courtès Dec. 27, 2020, 2:13 p.m. UTC
Hello Guix!

This adds zstd compression support for ‘guix publish’ and for
‘guix substitute’.

Currently ‘guix substitute’ implements the same policy has before,
which is to pick the smallest archive when several compression
methods are proposed.  The next step will be to make that
configurable.

Thoughts?

Ludo’.

Ludovic Courtès (5):
  utils: Remove 'compressed-output-port'.
  utils: Support zstd compression via Guile-zstd.
  publish: Add support for zstd compression.
  substitute: Add zstd support.
  doc: Mention optional dependency on Guile-zstd.

 doc/guix.texi               | 23 ++++++++++++-----
 guix/scripts/publish.scm    | 31 +++++++++++++----------
 guix/scripts/substitute.scm |  3 +++
 guix/utils.scm              | 23 ++++++++---------
 tests/publish.scm           | 16 ++++++++++++
 tests/utils.scm             | 49 +++++++++++++++++++++++++------------
 6 files changed, 97 insertions(+), 48 deletions(-)

Comments

Mathieu Othacehe Dec. 30, 2020, 9:30 a.m. UTC | #1
Hello Ludo,

> Currently ‘guix substitute’ implements the same policy has before,
> which is to pick the smallest archive when several compression
> methods are proposed.  The next step will be to make that
> configurable.
>
> Thoughts?

Nice one! What do you have in mind to make it configurable? Pass the
preferred compression types to the daemon and propagate it to "guix
substitute"?

Once we have better visibility on zstd, I think we should consider
making those bindings mandatory for consistency with the other
Guile compression libraries.

Otherwise, the whole patchset seems fine :).

Thanks,

Mathieu
Ludovic Courtès Jan. 3, 2021, 3:17 p.m. UTC | #2
Hi,

Mathieu Othacehe <othacehe@gnu.org> skribis:

>> Currently ‘guix substitute’ implements the same policy has before,
>> which is to pick the smallest archive when several compression
>> methods are proposed.  The next step will be to make that
>> configurable.
>>
>> Thoughts?
>
> Nice one! What do you have in mind to make it configurable? Pass the
> preferred compression types to the daemon and propagate it to "guix
> substitute"?

I’m not sure.  I thought that perhaps there could be:

  guix-daemon --download-strategy=[speed|bandwidth]

where ‘bandwidth’ would always optimize for bandwidth and thus use the
existing ‘file-size<?’ strategy in ‘narinfo-best-uri’.

For ‘speed’ it’s a bit more complicated though: you have to measure
whether downloads are actually CPU-bound to choose between (say) lzip
and zstd/gzip.  It’s doable but we’d have to say whether it can be made
robust enough.

Letting the user choose the compression type doesn’t have this drawback
but it requires expertise from the user.

> Once we have better visibility on zstd, I think we should consider
> making those bindings mandatory for consistency with the other
> Guile compression libraries.

Yes.

> Otherwise, the whole patchset seems fine :).

Cool, thanks!  I guess I’ll wait until Chris’ patch series has landed
before pushing it, or at least the bit that touches ‘guix substitute’.

Ludo’.
Ludovic Courtès Jan. 13, 2021, 10:08 p.m. UTC | #3
Pushed as e28d2cdd753f5c224853f3d9ffe63f848709b41a, along with
Jonathan’s suggestion regarding the manual.

I’ll follow up with a ‘guix’ package update and adding the dependency on
guile-zstd.

Ludo’.