[bug#62202,v4,6/6] tests: juliahub: Add unit tests for (guix import juliahub).

* tests/juliahub.scm : Add unit tests juliahub-redirect,
julia-general-registry-parsing, juliahub-fetch.

Change-Id: Ief5133b49a15d0bfc72bb357321126d296ba2802
---
 tests/juliahub.scm | 185 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 185 insertions(+)
 create mode 100644 tests/juliahub.scm
  

Hi,

As part of this v4, I would recommend merging patches 2, 3, and 6, such
that there’s a single self-contained patch adding ‘guix import
juliahub’.  (That’s what we usually do and I find it clearer because we
immediately see what goes together.)

Nicolas Graves <ngraves@ngraves.fr> skribis:

> * tests/juliahub.scm : Add unit tests juliahub-redirect,
> julia-general-registry-parsing, juliahub-fetch.

Just “New file.”

Some of the other files lack a commit log; we can add it for you, but
it’d be great if you could do it upfront.

> ---
>  tests/juliahub.scm | 185 +++++++++++++++++++++++++++++++++++++++++++++

Please add it to ‘Makefile.am’.

[...]

> +(define (mock-http-fetch testcase)
> +  (lambda (url . rest)
> +    (let ((body (assoc-ref testcase url)))
> +      (if body
> +          (open-input-string body)
> +          (error "mocked http-fetch Unexpected URL: " url)))))
> +
> +(define (mock-http-get testcase)
> +  (lambda (url . rest)
> +    (let ((body (assoc-ref testcase url))
> +          (response-header
> +             (build-response
> +                #:version '(1 . 1)

I strongly encourage using ‘with-http-server’ using the same strategy
that’s used in ‘tests/pypi.scm’ and others instead of mocking.  (‘mock’
is very sensitive to inlining, plus you sorta have to make assumptions
about the code path to be able to mock the right things.)

> +(test-equal "juliahub-fetch"
> +  #t
> +  (mock ((web client) http-get
> +         (mock-http-get fixtures-juliahub-check-test))
> +        (mock ((guix http-client) http-fetch
> +               (mock-http-fetch fixtures-juliahub-check-test))
> +              (mock ((guix import utils) git->origin mock-git->origin)
> +                    ((@@ (guix import juliahub) juliahub-package?)
> +                     ((@@ (guix import juliahub) juliahub-fetch) "MyPackage"))))))

Checking for ‘juliahub-package?’ doesn’t tell us much; what about
checking the whole package, similar to what is done in other importer
tests?

Thanks,
Ludo’.
  

On 2024-04-01 22:50, Ludovic Courtès wrote:

> Hi,
>
> As part of this v4, I would recommend merging patches 2, 3, and 6, such
> that there’s a single self-contained patch adding ‘guix import
> juliahub’.  (That’s what we usually do and I find it clearer because we
> immediately see what goes together.)

Will do.
>
> Nicolas Graves <ngraves@ngraves.fr> skribis:
>
>> * tests/juliahub.scm : Add unit tests juliahub-redirect,
>> julia-general-registry-parsing, juliahub-fetch.
>
> Just “New file.”
>
> Some of the other files lack a commit log; we can add it for you, but
> it’d be great if you could do it upfront.

Sorry for these past contribution, I do it now.
>
>> ---
>>  tests/juliahub.scm | 185 +++++++++++++++++++++++++++++++++++++++++++++
>
> Please add it to ‘Makefile.am’.

Will do.
>
> [...]
>
>> +(define (mock-http-fetch testcase)
>> +  (lambda (url . rest)
>> +    (let ((body (assoc-ref testcase url)))
>> +      (if body
>> +          (open-input-string body)
>> +          (error "mocked http-fetch Unexpected URL: " url)))))
>> +
>> +(define (mock-http-get testcase)
>> +  (lambda (url . rest)
>> +    (let ((body (assoc-ref testcase url))
>> +          (response-header
>> +             (build-response
>> +                #:version '(1 . 1)
>
> I strongly encourage using ‘with-http-server’ using the same strategy
> that’s used in ‘tests/pypi.scm’ and others instead of mocking.  (‘mock’
> is very sensitive to inlining, plus you sorta have to make assumptions
> about the code path to be able to mock the right things.)
>
>> +(test-equal "juliahub-fetch"
>> +  #t
>> +  (mock ((web client) http-get
>> +         (mock-http-get fixtures-juliahub-check-test))
>> +        (mock ((guix http-client) http-fetch
>> +               (mock-http-fetch fixtures-juliahub-check-test))
>> +              (mock ((guix import utils) git->origin mock-git->origin)
>> +                    ((@@ (guix import juliahub) juliahub-package?)
>> +                     ((@@ (guix import juliahub) juliahub-fetch) "MyPackage"))))))
>
> Checking for ‘juliahub-package?’ doesn’t tell us much; what about
> checking the whole package, similar to what is done in other importer
> tests?

Couldn't manage to get it to work, don't remember why exactly but it was
related to gexps. Will retry it when moving to 'with-http-server' as
advised. 

>
> Thanks,
> Ludo’.
  

On 2024-04-01 22:50, Ludovic Courtès wrote:

> Hi,
>
> As part of this v4, I would recommend merging patches 2, 3, and 6, such
> that there’s a single self-contained patch adding ‘guix import
> juliahub’.  (That’s what we usually do and I find it clearer because we
> immediately see what goes together.)
>
> Nicolas Graves <ngraves@ngraves.fr> skribis:
>
>> * tests/juliahub.scm : Add unit tests juliahub-redirect,
>> julia-general-registry-parsing, juliahub-fetch.
>
> Just “New file.”
>
> Some of the other files lack a commit log; we can add it for you, but
> it’d be great if you could do it upfront.
>
>> ---
>>  tests/juliahub.scm | 185 +++++++++++++++++++++++++++++++++++++++++++++
>
> Please add it to ‘Makefile.am’.
>
> [...]
>
>> +(define (mock-http-fetch testcase)
>> +  (lambda (url . rest)
>> +    (let ((body (assoc-ref testcase url)))
>> +      (if body
>> +          (open-input-string body)
>> +          (error "mocked http-fetch Unexpected URL: " url)))))
>> +
>> +(define (mock-http-get testcase)
>> +  (lambda (url . rest)
>> +    (let ((body (assoc-ref testcase url))
>> +          (response-header
>> +             (build-response
>> +                #:version '(1 . 1)
>
> I strongly encourage using ‘with-http-server’ using the same strategy
> that’s used in ‘tests/pypi.scm’ and others instead of mocking.  (‘mock’
> is very sensitive to inlining, plus you sorta have to make assumptions
> about the code path to be able to mock the right things.)

I can't however mock a git server, right? I still must mock at least the
git repo instead of getting it through a custom server, or is there a
better solution here?

>
>> +(test-equal "juliahub-fetch"
>> +  #t
>> +  (mock ((web client) http-get
>> +         (mock-http-get fixtures-juliahub-check-test))
>> +        (mock ((guix http-client) http-fetch
>> +               (mock-http-fetch fixtures-juliahub-check-test))
>> +              (mock ((guix import utils) git->origin mock-git->origin)
>> +                    ((@@ (guix import juliahub) juliahub-package?)
>> +                     ((@@ (guix import juliahub) juliahub-fetch) "MyPackage"))))))
>
> Checking for ‘juliahub-package?’ doesn’t tell us much; what about
> checking the whole package, similar to what is done in other importer
> tests?
>
> Thanks,
> Ludo’.
  

On 2024-04-09 09:29, Nicolas Graves wrote:

> On 2024-04-01 22:50, Ludovic Courtès wrote:
>
>> Hi,
>>
>> As part of this v4, I would recommend merging patches 2, 3, and 6, such
>> that there’s a single self-contained patch adding ‘guix import
>> juliahub’.  (That’s what we usually do and I find it clearer because we
>> immediately see what goes together.)
>>
>> Nicolas Graves <ngraves@ngraves.fr> skribis:
>>
>>> * tests/juliahub.scm : Add unit tests juliahub-redirect,
>>> julia-general-registry-parsing, juliahub-fetch.
>>
>> Just “New file.”
>>
>> Some of the other files lack a commit log; we can add it for you, but
>> it’d be great if you could do it upfront.
>>
>>> ---
>>>  tests/juliahub.scm | 185 +++++++++++++++++++++++++++++++++++++++++++++
>>
>> Please add it to ‘Makefile.am’.
>>
>> [...]
>>
>>> +(define (mock-http-fetch testcase)
>>> +  (lambda (url . rest)
>>> +    (let ((body (assoc-ref testcase url)))
>>> +      (if body
>>> +          (open-input-string body)
>>> +          (error "mocked http-fetch Unexpected URL: " url)))))
>>> +
>>> +(define (mock-http-get testcase)
>>> +  (lambda (url . rest)
>>> +    (let ((body (assoc-ref testcase url))
>>> +          (response-header
>>> +             (build-response
>>> +                #:version '(1 . 1)
>>
>> I strongly encourage using ‘with-http-server’ using the same strategy
>> that’s used in ‘tests/pypi.scm’ and others instead of mocking.  (‘mock’
>> is very sensitive to inlining, plus you sorta have to make assumptions
>> about the code path to be able to mock the right things.)
>
> I can't however mock a git server, right? I still must mock at least the
> git repo instead of getting it through a custom server, or is there a
> better solution here?

It's actually simpler than I thought, but there's an impediment in guile
http server implementation that doesn't allow me to push this effort to
the end.

https://git-scm.com/book/en/v2/Git-Internals-Transfer-Protocols

I'm currently writing it, it'll result in a handy helper for tests, such
as :

(with-git-forge  ; spawns a dumb but functional git server
      '(("MyPackage" . ((add "a.txt" "A")
                        (commit "First commit")
                        (tag "v1.0.0" "Release 1.0"))))
    (with-julia-test-servers
        `(("/juliahub/MyPackage/" 200 ,juliahub-redirect.html)
          ("/juliahub/MyPackage/" 200 ,juliahub-redirect.html)
          ("/juliahub/MyPackage/MySlg/1.0.0/pkg.json" 200
           ,(lambda (port) (display (fixture-pkg.json) port)))
          ("/general/M/MyPackage/Package.toml" 200
           ,(lambda (port) (display (pk 'd (general-Package.toml)) port))))
      (juliahub->guix-package "MyPackage")))


However, for that I'll need the http server to be able to respond with a
         (content-type . (application/x-git-upload-pack-advertisement))
header to git. But in guile's web server implementation, this is not
possible because of sanitize-response's charset addition, which is not
configurable. 

That's outside my field, how can we progress further ? We do indeed need
such a server to properly test juliahub since we go get the tag from the
actual repo (this is justified in the patch series).
  

Hi,

Nicolas Graves <ngraves@ngraves.fr> skribis:

> I'm currently writing it, it'll result in a handy helper for tests, such
> as :
>
> (with-git-forge  ; spawns a dumb but functional git server
>       '(("MyPackage" . ((add "a.txt" "A")
>                         (commit "First commit")
>                         (tag "v1.0.0" "Release 1.0"))))
>     (with-julia-test-servers
>         `(("/juliahub/MyPackage/" 200 ,juliahub-redirect.html)
>           ("/juliahub/MyPackage/" 200 ,juliahub-redirect.html)
>           ("/juliahub/MyPackage/MySlg/1.0.0/pkg.json" 200
>            ,(lambda (port) (display (fixture-pkg.json) port)))
>           ("/general/M/MyPackage/Package.toml" 200
>            ,(lambda (port) (display (pk 'd (general-Package.toml)) port))))
>       (juliahub->guix-package "MyPackage")))

Nice!

> However, for that I'll need the http server to be able to respond with a
>          (content-type . (application/x-git-upload-pack-advertisement))
> header to git. But in guile's web server implementation, this is not
> possible because of sanitize-response's charset addition, which is not
> configurable. 

D’oh.  I’m not sure what this is about; it’s definitely possible to add
pretty much any ‘content-type’ header (we do that in ‘guix publish’ for
instance), but if it’s going too far, then maybe you should go back to
procedure mocks.

> That's outside my field, how can we progress further ? We do indeed need
> such a server to properly test juliahub since we go get the tag from the
> actual repo (this is justified in the patch series).
>
> _____________________________________________________________________________
> ;;; Git Forge = Git HTTP Server with Dump transfer protocol and repositories

This work’s not lost: we can definitely switch back to it when the
limitations you encountered has been fixed on the Guile side, and/or in
other, simpler cases.

Thanks!

Ludo’.
  

On 2024-04-16 18:52, Ludovic Courtès wrote:

> Hi,
>
> Nicolas Graves <ngraves@ngraves.fr> skribis:
>
>> I'm currently writing it, it'll result in a handy helper for tests, such
>> as :
>>
>> (with-git-forge  ; spawns a dumb but functional git server
>>       '(("MyPackage" . ((add "a.txt" "A")
>>                         (commit "First commit")
>>                         (tag "v1.0.0" "Release 1.0"))))
>>     (with-julia-test-servers
>>         `(("/juliahub/MyPackage/" 200 ,juliahub-redirect.html)
>>           ("/juliahub/MyPackage/" 200 ,juliahub-redirect.html)
>>           ("/juliahub/MyPackage/MySlg/1.0.0/pkg.json" 200
>>            ,(lambda (port) (display (fixture-pkg.json) port)))
>>           ("/general/M/MyPackage/Package.toml" 200
>>            ,(lambda (port) (display (pk 'd (general-Package.toml)) port))))
>>       (juliahub->guix-package "MyPackage")))
>
> Nice!
>
>> However, for that I'll need the http server to be able to respond with a
>>          (content-type . (application/x-git-upload-pack-advertisement))
>> header to git. But in guile's web server implementation, this is not
>> possible because of sanitize-response's charset addition, which is not
>> configurable. 
>
> D’oh.  I’m not sure what this is about; it’s definitely possible to add
> pretty much any ‘content-type’ header (we do that in ‘guix publish’ for
> instance), but if it’s going too far, then maybe you should go back to
> procedure mocks.

Just to write that in a clearer way:

libgit2, which is behind guile-git, expects this exact header or fails,
with no additional charset field. The sanitize-response function in the
guile web server implementation on its side ensures that a charset field
is added no matter what.

So indeed we can set

   (content-type . (application/x-git-upload-pack-advertisement))
    
but in reality guile-git // libgit2 will read

   (content-type . (application/x-git-upload-pack-advertisement
                     (charset . "utf-8"))
                     
and will fail in this case. 

Since it's exterior (expected by libgit2), I thought the more relevant
would be to allow more flexibility on the guile web server side. But if
you think it's way better to ask on the libgit2 or guile-git side, I'll
be happy to.  WDYT ?

>
>> That's outside my field, how can we progress further ? We do indeed need
>> such a server to properly test juliahub since we go get the tag from the
>> actual repo (this is justified in the patch series).
>>
>> _____________________________________________________________________________
>> ;;; Git Forge = Git HTTP Server with Dump transfer protocol and repositories
>
> This work’s not lost: we can definitely switch back to it when the
> limitations you encountered has been fixed on the Guile side, and/or in
> other, simpler cases.
>
> Thanks!
>
> Ludo’.
  

Nicolas Graves <ngraves@ngraves.fr> skribis:

> libgit2, which is behind guile-git, expects this exact header or fails,
> with no additional charset field. The sanitize-response function in the
> guile web server implementation on its side ensures that a charset field
> is added no matter what.
>
> So indeed we can set
>
>    (content-type . (application/x-git-upload-pack-advertisement))
>     
> but in reality guile-git // libgit2 will read
>
>    (content-type . (application/x-git-upload-pack-advertisement
>                      (charset . "utf-8"))
>                      
> and will fail in this case. 

Oh I see, I had misunderstood that.

Note that ‘sanitize-response’ does not add a ‘charset’ header when BODY
is #f or a bytevector.  Maybe we could do that?

Thanks,
Ludo’.
  

On 2024-04-17 10:51, Ludovic Courtès wrote:

> Nicolas Graves <ngraves@ngraves.fr> skribis:
>
>> libgit2, which is behind guile-git, expects this exact header or fails,
>> with no additional charset field. The sanitize-response function in the
>> guile web server implementation on its side ensures that a charset field
>> is added no matter what.
>>
>> So indeed we can set
>>
>>    (content-type . (application/x-git-upload-pack-advertisement))
>>     
>> but in reality guile-git // libgit2 will read
>>
>>    (content-type . (application/x-git-upload-pack-advertisement
>>                      (charset . "utf-8"))
>>                      
>> and will fail in this case. 
>
> Oh I see, I had misunderstood that.
>
> Note that ‘sanitize-response’ does not add a ‘charset’ header when BODY
> is #f or a bytevector.  Maybe we could do that?

I've just taken a look at your suggestion. That's possible although not
elegant.

I stumbled accross a bigger issue however: the dumb protocol is
implemented for git but not for libgit2, thus is not present in
guile-git.

https://github.com/libgit2/libgit2/issues/6609

The smart protocol is however way too complex / unadapted for such a
light use in tests, and requires more that a simple http server easily
configured.

That's sad, such a light implementation would've been quite convenient
in my juliahub case. 

>
> Thanks,
> Ludo’.
>
>
>