mbox series

[bug#60288,RESEND,#2,v1,0/2] Start adding ZIM file(s)

Message ID 20221220052349.4965-1-GNUtoo@cyberdimension.org
Headers
Series Start adding ZIM file(s) |

Message

Denis 'GNUtoo' Carikli Dec. 23, 2022, 10:07 p.m. UTC 
Hi,

Here are two small patches. 

The first one add #:substitutable? to the copy-build system.

I don't know how to check if it works as intended though. It's
similar to the commit d0050ea8ad1c32d94cf5ba6725a0fc961bb23f38 
("build-system/go: Add #:substitutable? argument.") so normally
it shouldn't be an issue, but if someone can double check it it
would be best as it would avoid keeping around substitutes of
very big sizes.

The second patch adds a ZIM file. I'll most likely send more
patches to add additional ZIM files packages (about 10) later
on. I prefer doing it this way as it avoids having to deal with
potential rebases breaking if there is something wrong with my
second patch.

Denis 'GNUtoo' Carikli (2):
  build-system/copy: Add #:substitutable? argument.
  gnu: Add wikipedia_en_all_maxi

 gnu/local.mk               |  1 +
 gnu/packages/zim-files.scm | 86 ++++++++++++++++++++++++++++++++++++++
 guix/build-system/copy.scm |  4 +-
 3 files changed, 90 insertions(+), 1 deletion(-)
 create mode 100644 gnu/packages/zim-files.scm


base-commit: c193b5203b31246a6d74270c8086c45851561947

Comments

Christopher Baines Dec. 28, 2022, 6:10 p.m. UTC | #1 
Denis 'GNUtoo' Carikli <GNUtoo@cyberdimension.org> writes:

> Here are two small patches.
>
> The first one add #:substitutable? to the copy-build system.
>
> I don't know how to check if it works as intended though. It's
> similar to the commit d0050ea8ad1c32d94cf5ba6725a0fc961bb23f38
> ("build-system/go: Add #:substitutable? argument.") so normally
> it shouldn't be an issue, but if someone can double check it it
> would be best as it would avoid keeping around substitutes of
> very big sizes.
>
> The second patch adds a ZIM file. I'll most likely send more
> patches to add additional ZIM files packages (about 10) later
> on. I prefer doing it this way as it avoids having to deal with
> potential rebases breaking if there is something wrong with my
> second patch.
>
> Denis 'GNUtoo' Carikli (2):
>   build-system/copy: Add #:substitutable? argument.
>   gnu: Add wikipedia_en_all_maxi

I haven't looked at this in detail, but one comment on the QA
failures. Building the package for this large file involves copying it
from the store, to another place in the store. This requires 2x the
space which this large file takes up, which is a pretty wasteful
approach.

This is the reason behind the build failures I've seen, the build
machines run out of space when attempting the file copy. Maybe an
alternative if you want to have a package would be to symlink to the
source. That way, there's only a large file and a symlink in the store,
rather than two copies of the same large file.

Chris
Denis 'GNUtoo' Carikli Dec. 29, 2022, 11:19 p.m. UTC | #2 
On Wed, 28 Dec 2022 18:10:54 +0000
Christopher Baines <mail@cbaines.net> wrote:
> I haven't looked at this in detail, but one comment on the QA
> failures. Building the package for this large file involves copying it
> from the store, to another place in the store. This requires 2x the
> space which this large file takes up, which is a pretty wasteful
> approach.
Not only that but it also take a very long time to do that copy on
slower machines with an encrypted rootfs.

> This is the reason behind the build failures I've seen, the build
> machines run out of space when attempting the file copy. Maybe an
> alternative if you want to have a package would be to symlink to the
> source. That way, there's only a large file and a symlink in the
> store, rather than two copies of the same large file.
I'll try that. I hope that guix gc will not garbage collect the source
though.

Do you know if it's possible just to have a source package somehow
(and download the source to a specific filename) and not copy anything
at all?

Denis.
Denis 'GNUtoo' Carikli Jan. 2, 2023, 8:01 p.m. UTC | #3 
On Wed, 28 Dec 2022 18:10:54 +0000
Christopher Baines <mail@cbaines.net> wrote:> 
> Maybe an alternative if you want to have a package would be to
> symlink to the source.
The issue is that I don't know how to refer to the source in a
situation like that.

I didn't really find good examples of all that. So far the best
I saw was to either define (source [...]) and reuse it in multiple
packages or to reuse the source of another package with (package-source
<package name>) like in linux.scm.

With the gnu build system, it copies the source in the current
directory, so I've really no idea what to do here. We might also need
to add the source to the inputs or native-inputs or propagated-inputs
somehow so it would not garbage collect it when we install the zim. Is
propagated-inputs the way to go?

Denis.