| Message ID | 20210830163918.19419-1-jgart@dismail.de |
|---|---|
| State | New |
| Headers | show |
| Series | [bug#50274] guix: git: Adds feature to download git repository to the store. | expand |
| Context | Check | Description |
|---|---|---|
| cbaines/comparison | success | View comparision |
| cbaines/git branch | success | View Git branch |
| cbaines/applying patch | success | View Laminar job |
| cbaines/issue | success | View issue |
Hello, Thanks for the patch! I've been wanting a feature like this. I haven't tested your patch, but just reading it, I have a few comments... jgart <jgart@dismail.de> writes: > From: Julien Lepiller <julien@lepiller.eu> > > * guix/git.scm (download-git-to-store): Download Git repository from > URL at COMMIT to STORE, either under NAME or URL's basename if omitted. > Write progress reports to LOG. RECURSIVE? has the same effect as the > same-named parameter of 'git-fetch'. > > * guix/scripts/download.scm (download-git-to-store*): Adds cli option. > Examples: > guix download --git-commit=v0.1.1 github.com/anaseto/gruid-tcell > guix download -c v0.1.1 https://github.com/anaseto/gruid-tcell These examples should probably be in the main commit message, not the ChangeLog entry. > --- > guix/git.scm | 24 +++++++++++++++++- > guix/scripts/download.scm | 51 ++++++++++++++++++++++++++++++++------- > 2 files changed, 65 insertions(+), 10 deletions(-) > > diff --git a/guix/git.scm b/guix/git.scm > index 9c6f326c36..4c70782b97 100644 > --- a/guix/git.scm > +++ b/guix/git.scm > @@ -28,6 +28,7 @@ > #:use-module (gcrypt hash) > #:use-module ((guix build utils) > #:select (mkdir-p delete-file-recursively)) > + #:use-module ((guix build git) #:select (git-fetch)) > #:use-module (guix store) > #:use-module (guix utils) > #:use-module (guix records) > @@ -43,6 +44,7 @@ > #:use-module (srfi srfi-11) > #:use-module (srfi srfi-34) > #:use-module (srfi srfi-35) > + #:use-module (web uri) > #:export (%repository-cache-directory > honor-system-x509-certificates! > > @@ -61,7 +63,9 @@ > git-checkout-url > git-checkout-branch > git-checkout-commit > - git-checkout-recursive?)) > + git-checkout-recursive? > + > + download-git-to-store)) Your patch has a number of tabs in it; they should be converted to spaces. > > (define %repository-cache-directory > (make-parameter (string-append (cache-directory #:ensure? #f) > @@ -614,6 +618,24 @@ objects: 'ancestor (meaning that OLD is an ancestor of NEW), 'descendant, or > #:recursive? recursive? > #:log-port (current-error-port))))) > > +(define* (download-git-to-store store url commit > + #:optional (name (basename url)) > + #:key (log (current-error-port)) recursive?) > + "Download Git repository from URL at COMMIT to STORE, either under NAME or > +URL's basename if omitted. Write progress reports to LOG. RECURSIVE? has the > +same effect as the same-named parameter of 'git-fetch'." > + (define uri > + (string->uri url)) You've defined uri but not used it anywhere. > + > + (call-with-temporary-directory > + (lambda (temp) > + (let ((result > + (parameterize ((current-output-port log)) > + (git-fetch url commit temp > + #:recursive? recursive?)))) > + (and result > + (add-to-store store name #t "sha256" temp)))))) > + Is there a reason for not using latest-repository-commit instead of this? That already takes care of temporary directories, ignores .git, and reduces duplication if you download different tags from the same repository: guix download -c v0.1.3 example.com/repo guix download -c v2.1.0 example.com/repo > ;; Local Variables: > ;; eval: (put 'with-repository 'scheme-indent-function 2) > ;; End: > diff --git a/guix/scripts/download.scm b/guix/scripts/download.scm > index 5a91390358..6253ecaa5c 100644 > --- a/guix/scripts/download.scm > +++ b/guix/scripts/download.scm > @@ -26,15 +26,19 @@ > #:use-module (guix base32) > #:autoload (guix base64) (base64-encode) > #:use-module ((guix download) #:hide (url-fetch)) > + #:use-module ((guix git) #:select (download-git-to-store)) > #:use-module ((guix build download) > #:select (url-fetch)) > #:use-module ((guix progress) > #:select (current-terminal-columns)) > + #:use-module ((guix serialization) > + #:select (write-file)) > #:use-module ((guix build syscalls) > #:select (terminal-columns)) > #:use-module (web uri) > #:use-module (ice-9 match) > #:use-module (srfi srfi-1) > + #:use-module (srfi srfi-11) > #:use-module (srfi srfi-14) > #:use-module (srfi srfi-26) > #:use-module (srfi srfi-37) > @@ -76,12 +80,20 @@ > (ensure-valid-store-file-name (basename url)) > #:verify-certificate? verify-certificate?))) > > +(define* (download-git-to-store* url commit #:key recursive?) > + (with-store store > + (download-git-to-store store url commit > + (ensure-valid-store-file-name (basename url)) > + #:recursive? recursive?))) > + > (define %default-options > ;; Alist of default option values. > `((format . ,bytevector->nix-base32-string) > (hash-algorithm . ,(hash-algorithm sha256)) > (verify-certificate? . #t) > - (download-proc . ,download-to-store*))) > + (download-proc . ,download-to-store*) > + (git-download-proc . ,download-git-to-store*) > + (commit . #f))) > > (define (show-help) > (display (G_ "Usage: guix download [OPTION] URL > @@ -100,6 +112,9 @@ and 'base16' ('hex' and 'hexadecimal' can be used as well).\n")) > do not validate the certificate of HTTPS servers ")) > (format #t (G_ " > -o, --output=FILE download to FILE")) This option exists, so it would make sense for guix download -c v0.1 example.com/repo -o ~/src/repo to work as well. > + (format #t (G_ " > + -c, --git-commit=COMMIT > + download a Git repository")) Would it be possible to find a way to express this option so it wouldn't conflict with potential other repositories (SVN, hg, etc)? Maybe something like guix download --git -c v0.1 example.com/repo guix download --svn -c v0.1 example.com/repo But in SVN, "revisions" are used instead of "commits," so perhaps removing that terminology would be better. We could do guix download --git example.com/repo v0.1 guix download --svn example.com/repo v0.1 Or! We could use (fake) sub-commands like so: guix download git example.com/repo -c v0.1 This feels nice, since it matches e.g. 'guix import go'. See guix/import.scm for how this works. > (newline) > (display (G_ " > -h, --help display this help and exit")) > @@ -143,6 +158,9 @@ and 'base16' ('hex' and 'hexadecimal' can be used as well).\n")) > (lambda* (url #:key verify-certificate?) > (download-to-file url arg)) > (alist-delete 'download result)))) > + (option '(#\c "git-commit") #t #f > + (lambda (opt name arg result) > + (alist-cons 'commit arg result))) > > (option '(#\h "help") #f #f > (lambda args > @@ -182,16 +200,31 @@ and 'base16' ('hex' and 'hexadecimal' can be used as well).\n")) > (leave (G_ "~a: failed to parse URI~%") > arg))) > (fetch (assq-ref opts 'download-proc)) > + (git-fetch (assq-ref opts 'git-download-proc)) > + (commit (assq-ref opts 'commit)) > (path (parameterize ((current-terminal-columns > (terminal-columns))) > - (fetch (uri->string uri) > - #:verify-certificate? > - (assq-ref opts 'verify-certificate?)))) > - (hash (call-with-input-file > - (or path > - (leave (G_ "~a: download failed~%") > - arg)) > - (cute port-hash (assoc-ref opts 'hash-algorithm) <>))) > + (if commit > + (git-fetch (uri->string uri) commit) You don't actually seem to use the download-git-to-store procedure you wrote above. An oversight? > + (fetch (uri->string uri) > + #:verify-certificate? > + (assq-ref opts 'verify-certificate?))))) > + (hash (if (or (assq-ref opts 'recursive) commit) The 'recursive' option is used here, but not specified in options or the help message. I'm also not sure what it's supposed to mean here. 'Recursive' would have no meaning outside of fetching a repo, and for fetching a repo, 'recursive' should mean that the fetch is recursive, and would be applied above. > + (let-values (((port get-hash) > + (open-hash-port > + (assoc-ref opts 'hash-algorithm)))) > + (write-file path port > + #:select? > + (if commit > + (lambda (file stat) (not (equal? (basename file) ".git"))) > + (const #t))) > + (force-output port) > + (get-hash)) > + (call-with-input-file > + (or path > + (leave (G_ "~a: download failed~%") > + arg)) > + (cute port-hash (assoc-ref opts 'hash-algorithm) <>)))) Rather than special-casing repositories, perhaps you'd consider incorporating the first 1-3 patches from [0]? This would make it easier to, say, add other repository formats later. It would make all this roughly (hash (and path (file-hash* path #:algorithm (assq-ref opts 'hash-algorithm)))) modulo error reporting, of course. [0] https://issues.guix.gnu.org/50072 -- Sarah
Hi again, Sarah Morgensen <iskarian@mgsn.dev> writes: >> (fetch (assq-ref opts 'download-proc)) >> + (git-fetch (assq-ref opts 'git-download-proc)) >> + (commit (assq-ref opts 'commit)) >> (path (parameterize ((current-terminal-columns >> (terminal-columns))) >> - (fetch (uri->string uri) >> - #:verify-certificate? >> - (assq-ref opts 'verify-certificate?)))) >> - (hash (call-with-input-file >> - (or path >> - (leave (G_ "~a: download failed~%") >> - arg)) >> - (cute port-hash (assoc-ref opts 'hash-algorithm) <>))) >> + (if commit >> + (git-fetch (uri->string uri) commit) > > You don't actually seem to use the download-git-to-store procedure you > wrote above. An oversight? Please disregard this comment. I read too fast and didn't catch that git-fetch took the value of 'git-download-proc :) -- Sarah
> Please disregard this comment. I read too fast and didn't catch that > git-fetch took the value of 'git-download-proc :) No worries! :) _________________________________________ / 3B1D 7F19 E36B B60C 0F5B 2CA9 A52A A2B4 \ \ 77B6 DD35 / ----------------------------------------- \ ^__^ \ (oo)\_______ (__)\ )\/\ ||----w | || ||
On Tue, 31 Aug 2021 15:30:29 -0400 jgart <jgart@dismail.de> wrote: Hi Sarah, I think Julien will send the patch with the tabs to spaces conversion later. Thanks for the review.
Sarah Morgensen schreef op di 31-08-2021 om 11:50 [-0700]: > > Your patch has a number of tabs in it; they should be converted to spaces. Is this documented somewhere? I'm not aware of such a rule. Greetings, Maxime
Hi Maxime, Maxime Devos <maximedevos@telenet.be> writes: > Sarah Morgensen schreef op di 31-08-2021 om 11:50 [-0700]: >> >> Your patch has a number of tabs in it; they should be converted to spaces. > > Is this documented somewhere? I'm not aware of such a rule. I cannot find it explicitly stated anywhere, but etc/indent-code.el does replace tabs with spaces: --8<---------------cut here---------------start------------->8--- ;; Indent all of FILE-NAME. (find-file file-name) (let ((indent-tabs-mode nil)) (untabify (point-min) (point-max)) (indent-region (point-min) (point-max)) (save-buffer) (message "Done!"))) --8<---------------cut here---------------end--------------->8--- So I assumed it's policy. Perhaps the documentation should explicitly note that spaces are Official Policy? -- Sarah
Hi Julien & jgart, jgart <jgart@dismail.de> skribis: > From: Julien Lepiller <julien@lepiller.eu> > > * guix/git.scm (download-git-to-store): Download Git repository from > URL at COMMIT to STORE, either under NAME or URL's basename if omitted. > Write progress reports to LOG. RECURSIVE? has the same effect as the > same-named parameter of 'git-fetch'. > > * guix/scripts/download.scm (download-git-to-store*): Adds cli option. > Examples: > guix download --git-commit=v0.1.1 github.com/anaseto/gruid-tcell > guix download -c v0.1.1 https://github.com/anaseto/gruid-tcell Very useful! > +(define* (download-git-to-store store url commit > + #:optional (name (basename url)) > + #:key (log (current-error-port)) recursive?) > + "Download Git repository from URL at COMMIT to STORE, either under NAME or > +URL's basename if omitted. Write progress reports to LOG. RECURSIVE? has the > +same effect as the same-named parameter of 'git-fetch'." Can we use ‘latest-repository-commit’ instead? The difference is that it’ll populate ~/.cache/guix/checkouts, but I think that’s fine. OTOH, if we want to make it easier to support other VCSes, we can choose to not use (guix git) at all and instead use ‘git-fetch’ in (guix git-download), ‘hg-fetch’ in (guix hg-download), etc. This code would go to (guix scripts hash). WDYT? [...] > + (option '(#\c "git-commit") #t #f > + (lambda (opt name arg result) > + (alist-cons 'commit arg result))) For consistency with package transformation options, I’d call it ‘--commit’. This option would still apply if/when other VCSes are supported. This would also need an update of the manual. Thanks, Ludo’.
Hi, Ludovic Courtès <ludo@gnu.org> writes: >> +(define* (download-git-to-store store url commit >> + #:optional (name (basename url)) >> + #:key (log (current-error-port)) recursive?) >> + "Download Git repository from URL at COMMIT to STORE, either under NAME or >> +URL's basename if omitted. Write progress reports to LOG. RECURSIVE? has the >> +same effect as the same-named parameter of 'git-fetch'." > > Can we use ‘latest-repository-commit’ instead? The difference is that > it’ll populate ~/.cache/guix/checkouts, but I think that’s fine. > > OTOH, if we want to make it easier to support other VCSes, we can choose > to not use (guix git) at all and instead use ‘git-fetch’ in (guix > git-download), ‘hg-fetch’ in (guix hg-download), etc. This code would > go to (guix scripts hash). > > WDYT? Would using 'git-fetch' mean that it's already in the store (and therefore won't be redownloaded) when it's subsequently used in a source? That would be even better than latest-repository-commit! (Presumably --with-commit and friends also use 'git-fetch'?) -- Sarah
Hi, Sarah Morgensen <iskarian@mgsn.dev> skribis: > Ludovic Courtès <ludo@gnu.org> writes: > >>> +(define* (download-git-to-store store url commit >>> + #:optional (name (basename url)) >>> + #:key (log (current-error-port)) recursive?) >>> + "Download Git repository from URL at COMMIT to STORE, either under NAME or >>> +URL's basename if omitted. Write progress reports to LOG. RECURSIVE? has the >>> +same effect as the same-named parameter of 'git-fetch'." >> >> Can we use ‘latest-repository-commit’ instead? The difference is that >> it’ll populate ~/.cache/guix/checkouts, but I think that’s fine. >> >> OTOH, if we want to make it easier to support other VCSes, we can choose >> to not use (guix git) at all and instead use ‘git-fetch’ in (guix >> git-download), ‘hg-fetch’ in (guix hg-download), etc. This code would >> go to (guix scripts hash). >> >> WDYT? > > Would using 'git-fetch' mean that it's already in the store (and > therefore won't be redownloaded) when it's subsequently used in a > source? That would be even better than latest-repository-commit! Yes, you’re right. > (Presumably --with-commit and friends also use 'git-fetch'?) No, they use ‘latest-repository-commit’ (via <git-checkout>) because it’s a case where you can’t use an <origin> because the content hash is not known in advance (and the commit is also unknown when you use ‘--with-branch’). Ludo’.
Hello there! I was wondering if this patch requires any more work or attention? If the patch/bug is stale, would it be okay if I continued working on it in order to get it merge ready? I was thinking about implementing something similar, and saw this ticket, so I thought I would ask.
Hi, On Mon, 31 Jul 2023 at 18:21, TakeV <takev@disroot.org> wrote: > I was wondering if this patch requires any more work or attention? If the > patch/bug is stale, would it be okay if I continued working on it in order to > get it merge ready? I was thinking about implementing something similar, and > saw this ticket, so I thought I would ask. Well, from my understanding, it remains some work: 1. Rely on latest-repository-commit instead, 2. Do not make it Git specific but open the door for other VCSes. Do you want to give a try? Cheers, simon
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 I would absolutely love to give it a shot! I will get to work. :D - -------- Original Message -------- From: Simon Tournier <zimon.toutoune@gmail.com> Sent: August 18, 2023 10:55:53 AM PDT To: TakeV <takev@disroot.org> Cc: 50274@debbugs.gnu.org Subject: Re: bug#50274: [PATCH] guix: git: Adds feature to download git repository to the store. Hi, On Mon, 31 Jul 2023 at 18:21, TakeV <takev@disroot.org> wrote: > I was wondering if this patch requires any more work or attention? If the > patch/bug is stale, would it be okay if I continued working on it in order to > get it merge ready? I was thinking about implementing something similar, and > saw this ticket, so I thought I would ask. Well, from my understanding, it remains some work: 1. Rely on latest-repository-commit instead, 2. Do not make it Git specific but open the door for other VCSes. Do you want to give a try? Cheers, simon -----BEGIN PGP SIGNATURE----- iQG3BAEBCgAhGhxUYWtlViA8dGFrZXZAZGlzcm9vdC5vcmc+BQJk39xZAAoJEKZP QTRcdACvZB4MAKDWz4SVLj/i9aNjkUanbltiwE/EIHSb53MKxxqFlxI74JZaLwWQ yAXJaVWC5gf61HWnx9UPmjohdV9GGF2qd7K0m3z18FjsLlSJlSKLLAjbV5TM7vtg p88lwEvKNT/0zmR29OSQJUgrB4bibPHJPBXr1A5lB81L0iSXn7ku5dA3cRhKSMG9 YKL11j5EGgPwpSZjxDJgV05SDva2PK9StHF1KdajiR6C03qimU7Qi/y0QJfuXJrI FGBFy5IzVjQBBUWmUrU2XiYgW9B7Iyv5LjsAzqvTulQf0iOlV9d5ecCelvmw09c7 c93ukOrChAgWW5ZS78+OgUjlwJ/CSxFAeMI0zBSf5cwDtwMrlo71Xg3rhyO8l15s Am1FNrh7mUKGctVtHImg+cDhZcp13uXel54OXfAlSHvFDeCtxQiBdqqxhr8P2Afm 1VelHJQkgC8O3QUQVdLYRyqxXUBgeM8LQ5o4oACRubMta+Kn3au9CpOo/W5IYLN4 jEopt7z4EPRpeg== =rcuK -----END PGP SIGNATURE-----
diff --git a/guix/git.scm b/guix/git.scm index 9c6f326c36..4c70782b97 100644 --- a/guix/git.scm +++ b/guix/git.scm @@ -28,6 +28,7 @@ #:use-module (gcrypt hash) #:use-module ((guix build utils) #:select (mkdir-p delete-file-recursively)) + #:use-module ((guix build git) #:select (git-fetch)) #:use-module (guix store) #:use-module (guix utils) #:use-module (guix records) @@ -43,6 +44,7 @@ #:use-module (srfi srfi-11) #:use-module (srfi srfi-34) #:use-module (srfi srfi-35) + #:use-module (web uri) #:export (%repository-cache-directory honor-system-x509-certificates! @@ -61,7 +63,9 @@ git-checkout-url git-checkout-branch git-checkout-commit - git-checkout-recursive?)) + git-checkout-recursive? + + download-git-to-store)) (define %repository-cache-directory (make-parameter (string-append (cache-directory #:ensure? #f) @@ -614,6 +618,24 @@ objects: 'ancestor (meaning that OLD is an ancestor of NEW), 'descendant, or #:recursive? recursive? #:log-port (current-error-port))))) +(define* (download-git-to-store store url commit + #:optional (name (basename url)) + #:key (log (current-error-port)) recursive?) + "Download Git repository from URL at COMMIT to STORE, either under NAME or +URL's basename if omitted. Write progress reports to LOG. RECURSIVE? has the +same effect as the same-named parameter of 'git-fetch'." + (define uri + (string->uri url)) + + (call-with-temporary-directory + (lambda (temp) + (let ((result + (parameterize ((current-output-port log)) + (git-fetch url commit temp + #:recursive? recursive?)))) + (and result + (add-to-store store name #t "sha256" temp)))))) + ;; Local Variables: ;; eval: (put 'with-repository 'scheme-indent-function 2) ;; End: diff --git a/guix/scripts/download.scm b/guix/scripts/download.scm index 5a91390358..6253ecaa5c 100644 --- a/guix/scripts/download.scm +++ b/guix/scripts/download.scm @@ -26,15 +26,19 @@ #:use-module (guix base32) #:autoload (guix base64) (base64-encode) #:use-module ((guix download) #:hide (url-fetch)) + #:use-module ((guix git) #:select (download-git-to-store)) #:use-module ((guix build download) #:select (url-fetch)) #:use-module ((guix progress) #:select (current-terminal-columns)) + #:use-module ((guix serialization) + #:select (write-file)) #:use-module ((guix build syscalls) #:select (terminal-columns)) #:use-module (web uri) #:use-module (ice-9 match) #:use-module (srfi srfi-1) + #:use-module (srfi srfi-11) #:use-module (srfi srfi-14) #:use-module (srfi srfi-26) #:use-module (srfi srfi-37) @@ -76,12 +80,20 @@ (ensure-valid-store-file-name (basename url)) #:verify-certificate? verify-certificate?))) +(define* (download-git-to-store* url commit #:key recursive?) + (with-store store + (download-git-to-store store url commit + (ensure-valid-store-file-name (basename url)) + #:recursive? recursive?))) + (define %default-options ;; Alist of default option values. `((format . ,bytevector->nix-base32-string) (hash-algorithm . ,(hash-algorithm sha256)) (verify-certificate? . #t) - (download-proc . ,download-to-store*))) + (download-proc . ,download-to-store*) + (git-download-proc . ,download-git-to-store*) + (commit . #f))) (define (show-help) (display (G_ "Usage: guix download [OPTION] URL @@ -100,6 +112,9 @@ and 'base16' ('hex' and 'hexadecimal' can be used as well).\n")) do not validate the certificate of HTTPS servers ")) (format #t (G_ " -o, --output=FILE download to FILE")) + (format #t (G_ " + -c, --git-commit=COMMIT + download a Git repository")) (newline) (display (G_ " -h, --help display this help and exit")) @@ -143,6 +158,9 @@ and 'base16' ('hex' and 'hexadecimal' can be used as well).\n")) (lambda* (url #:key verify-certificate?) (download-to-file url arg)) (alist-delete 'download result)))) + (option '(#\c "git-commit") #t #f + (lambda (opt name arg result) + (alist-cons 'commit arg result))) (option '(#\h "help") #f #f (lambda args @@ -182,16 +200,31 @@ and 'base16' ('hex' and 'hexadecimal' can be used as well).\n")) (leave (G_ "~a: failed to parse URI~%") arg))) (fetch (assq-ref opts 'download-proc)) + (git-fetch (assq-ref opts 'git-download-proc)) + (commit (assq-ref opts 'commit)) (path (parameterize ((current-terminal-columns (terminal-columns))) - (fetch (uri->string uri) - #:verify-certificate? - (assq-ref opts 'verify-certificate?)))) - (hash (call-with-input-file - (or path - (leave (G_ "~a: download failed~%") - arg)) - (cute port-hash (assoc-ref opts 'hash-algorithm) <>))) + (if commit + (git-fetch (uri->string uri) commit) + (fetch (uri->string uri) + #:verify-certificate? + (assq-ref opts 'verify-certificate?))))) + (hash (if (or (assq-ref opts 'recursive) commit) + (let-values (((port get-hash) + (open-hash-port + (assoc-ref opts 'hash-algorithm)))) + (write-file path port + #:select? + (if commit + (lambda (file stat) (not (equal? (basename file) ".git"))) + (const #t))) + (force-output port) + (get-hash)) + (call-with-input-file + (or path + (leave (G_ "~a: download failed~%") + arg)) + (cute port-hash (assoc-ref opts 'hash-algorithm) <>)))) (fmt (assq-ref opts 'format))) (format #t "~a~%~a~%" path (fmt hash)) #t)))
From: Julien Lepiller <julien@lepiller.eu> * guix/git.scm (download-git-to-store): Download Git repository from URL at COMMIT to STORE, either under NAME or URL's basename if omitted. Write progress reports to LOG. RECURSIVE? has the same effect as the same-named parameter of 'git-fetch'. * guix/scripts/download.scm (download-git-to-store*): Adds cli option. Examples: guix download --git-commit=v0.1.1 github.com/anaseto/gruid-tcell guix download -c v0.1.1 https://github.com/anaseto/gruid-tcell --- guix/git.scm | 24 +++++++++++++++++- guix/scripts/download.scm | 51 ++++++++++++++++++++++++++++++++------- 2 files changed, 65 insertions(+), 10 deletions(-)