[bug#71697,v3,2/2] scripts: lint: Honor package property to exclude checkers.

* guix/scripts/lint.scm (exclude-package-checkers): New procedure, filter the
checker if the package is marked.
(guix-lint)[show-package-checkers]: New procedure.
* doc/guix.texi: Document it.

Change-Id: Idf8e5c67102a1701ebd917bbc6212cfeb6ea2054
---
 doc/guix.texi         | 17 ++++++++++++++++-
 guix/scripts/lint.scm | 26 +++++++++++++++++++++++++-
 2 files changed, 41 insertions(+), 2 deletions(-)
  

Hi Simon,

Simon Tournier <zimon.toutoune@gmail.com> writes:

> * guix/scripts/lint.scm (exclude-package-checkers): New procedure, filter the
> checker if the package is marked.
> (guix-lint)[show-package-checkers]: New procedure.
> * doc/guix.texi: Document it.
>
> Change-Id: Idf8e5c67102a1701ebd917bbc6212cfeb6ea2054
> ---
>  doc/guix.texi         | 17 ++++++++++++++++-
>  guix/scripts/lint.scm | 26 +++++++++++++++++++++++++-
>  2 files changed, 41 insertions(+), 2 deletions(-)
>
> diff --git a/doc/guix.texi b/doc/guix.texi
> index 037b1a2f24..1baf3fafe6 100644
> --- a/doc/guix.texi
> +++ b/doc/guix.texi
> @@ -71,7 +71,7 @@
>  Copyright @copyright{} 2019 Alex Griffin@*
>  Copyright @copyright{} 2019, 2020, 2021, 2022 Guillaume Le Vaillant@*
>  Copyright @copyright{} 2020 Liliana Marie Prikler@*
> -Copyright @copyright{} 2019, 2020, 2021, 2022, 2023 Simon Tournier@*
> +Copyright @copyright{} 2019, 2020, 2021, 2022, 2023, 2024 Simon Tournier@*
>  Copyright @copyright{} 2020 Wiktor Żelazny@*
>  Copyright @copyright{} 2020 Damien Cassou@*
>  Copyright @copyright{} 2020 Jakub Kądziołka@*
> @@ -15444,6 +15444,21 @@ Invoking guix lint
>  to the new style.
>  @end table
>
> +Sometimes it is not desired to run the same checker each time
> +@command{guix lint} is invoked---e.g., because the checker takes time or
> +to avoid to send again and again the same request for archiving.

The rationale sounds odd in the context of creating Guix packages for
Guix -- I wouldn't want someone to start adding random lint exclusions
to package properties because some check "takes time".  I think it'd be
better to give as an example which problem the mechanism was created
for, which is, to opt out of the Software Heritage archival requests.

From there the text could mention that the mechanism is general can be
used to disable other lint checks as well, such as the home page check.

> +Instead of excluding the checker at the command-line via the option
> +@code{--exclude}, the package might be marked to skip the checker by
> +honoring the property in package definition, e.g.,
> +
> +@lisp
> +(package
> +  (name "python-scikit-learn")
> +  ;; @dots{}
> +  (properties '((lint-exclude-archival? . #t)
> +                (lint-exclude-home-page? . #t))))
> +@end lisp
> +
>  The general syntax is:
>
>  @example
> diff --git a/guix/scripts/lint.scm b/guix/scripts/lint.scm
> index b98266c831..7aed467eae 100644
> --- a/guix/scripts/lint.scm
> +++ b/guix/scripts/lint.scm
> @@ -9,7 +9,7 @@
>  ;;; Copyright © 2017 Tobias Geerinckx-Rice <me@tobias.gr>
>  ;;; Copyright © 2017, 2018 Efraim Flashner <efraim@flashner.co.il>
>  ;;; Copyright © 2018, 2019 Arun Isaac <arunisaac@systemreboot.net>
> -;;; Copyright © 2019, 2020 Simon Tournier <zimon.toutoune@gmail.com>
> +;;; Copyright © 2019, 2020, 2024 Simon Tournier <zimon.toutoune@gmail.com>
>  ;;; Copyright © 2020 Brice Waegeneire <brice@waegenei.re>
>  ;;;
>  ;;; This file is part of GNU Guix.
> @@ -39,6 +39,7 @@ (define-module (guix scripts lint)
>    #:use-module (ice-9 format)
>    #:use-module (srfi srfi-1)
>    #:use-module (srfi srfi-37)
> +  #:use-module (srfi srfi-26)
>    #:export (guix-lint
>              run-checkers))
>
> @@ -59,6 +60,18 @@ (define (emit-warnings warnings)
>                 name version message))))
>     warnings))
>
> +(define (exclude-package-checkers package checkers)
> +  "Filter the CHECKERS list using PACKAGE properties field."
> +  (let ((properties (package-properties package)))
> +    (filter (lambda (checker)
> +              (not (assq-ref properties
> +                             ((compose string->symbol
> +                                       (cut string-append "lint-exclude-" <> "?")
> +                                       symbol->string
> +                                       lint-checker-name)
> +                              checker))))
> +            checkers)))

Instead of using filter + a negated test, I'd use 'remove' (from SRFI
1).

>  (define* (run-checkers package checkers #:key store)
>    "Run the given CHECKERS on PACKAGE."
>    (let ((tty? (isatty? (current-error-port))))
> @@ -223,16 +236,27 @@ (define-command (guix-lint . args)
>                  (proc store))
>                (proc #f)))
>
> +        (define (show-package-checkers package checkers)
> +          (format (current-error-port) "~a@~a checked by~{ ~a~}.~%"
> +                  (package-name package)
> +                  (package-version package)
> +                  (sort (map (compose symbol->string lint-checker-name)
> +                             (exclude-package-checkers
> +                              package checkers))
> +                   string<?)))
> +
>          (call-maybe-with-store
>           (lambda (store)
>             (cond
>              ((null? args)
>               (fold-packages (lambda (p r)
> +                              (show-package-checkers p checkers)
>                                (when (not (assoc-ref opts 'dry-run?))
>                                  (run-checkers p checkers
>                                                #:store store))) '()))
>              (else
>               (for-each (lambda (package)
> +                         (show-package-checkers package checkers)
>                           (when (not (assoc-ref opts 'dry-run?))
>                               (run-checkers package checkers
>                                             #:store store)))

I haven't tried it, but this looks reasonable to me.
  

Hi,

Simon Tournier <zimon.toutoune@gmail.com> skribis:

> +@lisp
> +(package
> +  (name "python-scikit-learn")
> +  ;; @dots{}
> +  (properties '((lint-exclude-archival? . #t)
> +                (lint-exclude-home-page? . #t))))

To complement Maxim’s review, how about:

  (properties '((lint-excluded-checkers . (archival home-page))))

?

Apart from that, the idea sounds reasonable to me.

Thanks,
Ludo’.
  

On Tue, Jun 25, 2024 at 11:15 AM Ludovic Courtès <ludo@gnu.org> wrote:
>
> Hi,
>
> Simon Tournier <zimon.toutoune@gmail.com> skribis:
>
> > +@lisp
> > +(package
> > +  (name "python-scikit-learn")
> > +  ;; @dots{}
> > +  (properties '((lint-exclude-archival? . #t)
> > +                (lint-exclude-home-page? . #t))))
>
> To complement Maxim’s review, how about:
>
>   (properties '((lint-excluded-checkers . (archival home-page))))
>
> ?
>
> Apart from that, the idea sounds reasonable to me.
>
> Thanks,
> Ludo’.

Could we not instead create a GUIX_LINT_OPTIONS, similar to
GUIX_BUILD_OPTIONS? Then anyone wishing to universally exclude certain
checkers (or disable network checks) on their own system would be free
to do so.

I find the current implementation confusing since I don't believe the
project would accept a new or modified package missing the home page
or with archiving disabled. Stated another way, to which Guix packages
are we adding lint exclusions?

Greg
  

Greg Hogan <code@greghogan.com> writes:

> I find the current implementation confusing since I don't believe the
> project would accept a new or modified package missing the home page
> or with archiving disabled. Stated another way, to which Guix packages
> are we adding lint exclusions?

To packages in your own channel.
  

Hi Greg,

Greg Hogan <code@greghogan.com> writes:

> On Tue, Jun 25, 2024 at 11:15 AM Ludovic Courtès <ludo@gnu.org> wrote:
>>
>> Hi,
>>
>> Simon Tournier <zimon.toutoune@gmail.com> skribis:
>>
>> > +@lisp
>> > +(package
>> > +  (name "python-scikit-learn")
>> > +  ;; @dots{}
>> > +  (properties '((lint-exclude-archival? . #t)
>> > +                (lint-exclude-home-page? . #t))))
>>
>> To complement Maxim’s review, how about:
>>
>>   (properties '((lint-excluded-checkers . (archival home-page))))
>>
>> ?
>>
>> Apart from that, the idea sounds reasonable to me.
>>
>> Thanks,
>> Ludo’.
>
> Could we not instead create a GUIX_LINT_OPTIONS, similar to
> GUIX_BUILD_OPTIONS? Then anyone wishing to universally exclude certain
> checkers (or disable network checks) on their own system would be free
> to do so.

That would be a good option to have too, on top of the other one.

> I find the current implementation confusing since I don't believe the
> project would accept a new or modified package missing the home page
> or with archiving disabled. Stated another way, to which Guix packages
> are we adding lint exclusions?

I don't think these exclusions should be committed in general to the
repo, except when we have for example the author of some software
explicitly requesting that SWH archival be disabled for it in Guix.

It may also be useful e.g. for some project that really don't have a
home page, to avoid a spurious lint warning in this case.
  

On Wed, Jun 26, 2024 at 3:28 PM Maxim Cournoyer
<maxim.cournoyer@gmail.com> wrote:
>
> I don't think these exclusions should be committed in general to the
> repo, except when we have for example the author of some software
> explicitly requesting that SWH archival be disabled for it in Guix.

Author requests are as problematic to a free software distribution as
the earlier demands to modify historical data are to reproducibility.

How do we authenticate authorship? Is it a single author, all authors,
majority of authorship? How would the latter be measured and valued?
Are author requests transitive? In which direction? Do the requests
propagate to dependent packages, or must a request include author
approval from all project dependencies? How do we handle cases where
copyright has not been noted as carefully as in Guix? Must the request
be made specifically to the Guix project? How do we monitor projects
for new authors or changes to requests?

We have a system for honoring author requests that resolves every
single one of these issues: software licenses. And this is not some
new issue, developers have been debating commercial use ("Micro$oft")
of their work for decades, yet here we are writing free software and
building a free Gnu/OS.

These requests to turn free software non-free are simply the tip of
the iceberg. We have always considered the artist (author) to be
separate from the art (licensed software). Now we get (from the
initiator of these demands) that "Not every political opinion should
be respected." which is a clear contradiction of the Guix Code of
Conduct's "Being respectful of differing opinions, viewpoints, and
experiences". Which individuals or demographic subgroups will be next
claimed problematic and need to have their contributions excluded?

> It may also be useful e.g. for some project that really don't have a
> home page, to avoid a spurious lint warning in this case.

If this is the best use case for a spurious feature request then I
find this a dangerous addition to the project. Those denigrading and
demanding that Guix pressure partner projects to restrict the use of
free software are unlikely to be content adding these flags to their
private packages as may exist.
  

Hi Greg,

Greg Hogan <code@greghogan.com> writes:

> On Wed, Jun 26, 2024 at 3:28 PM Maxim Cournoyer
> <maxim.cournoyer@gmail.com> wrote:
>>
>> I don't think these exclusions should be committed in general to the
>> repo, except when we have for example the author of some software
>> explicitly requesting that SWH archival be disabled for it in Guix.
>
> Author requests are as problematic to a free software distribution as
> the earlier demands to modify historical data are to reproducibility.
>
> How do we authenticate authorship? Is it a single author, all authors,
> majority of authorship? How would the latter be measured and valued?
> Are author requests transitive? In which direction? Do the requests
> propagate to dependent packages, or must a request include author
> approval from all project dependencies? How do we handle cases where
> copyright has not been noted as carefully as in Guix? Must the request
> be made specifically to the Guix project? How do we monitor projects
> for new authors or changes to requests?
>
> We have a system for honoring author requests that resolves every
> single one of these issues: software licenses. And this is not some
> new issue, developers have been debating commercial use ("Micro$oft")
> of their work for decades, yet here we are writing free software and
> building a free Gnu/OS.

You raise good questions, for which I do not have immediate answers.

> These requests to turn free software non-free are simply the tip of
> the iceberg. We have always considered the artist (author) to be
> separate from the art (licensed software). Now we get (from the
> initiator of these demands) that "Not every political opinion should
> be respected." which is a clear contradiction of the Guix Code of
> Conduct's "Being respectful of differing opinions, viewpoints, and
> experiences". Which individuals or demographic subgroups will be next
> claimed problematic and need to have their contributions excluded?
>
>> It may also be useful e.g. for some project that really don't have a
>> home page, to avoid a spurious lint warning in this case.
>
> If this is the best use case for a spurious feature request then I
> find this a dangerous addition to the project. Those denigrading and
> demanding that Guix pressure partner projects to restrict the use of
> free software are unlikely to be content adding these flags to their
> private packages as may exist.

While I dislike the attitude/approach used, I think the essence of the
complaint was that Guix, via SHW, was somehow facilitating the
scavenging of free software sources to train large language models
(LLM), with the opinion that these models do not respect the licenses of
the sources ingested for their produced output (the work is considered
new work, not a derived work, or perhaps it's still legally a gray area,
I don't know).  In this perspective, the original poster was seeking to
have the free software more protected against what they see as a loop
hole in the LLM business, as explained above.

That's an interesting legal and moral challenge/problem, but I don't
think GNU Guix is the right venue to debate it; especially not in the
way it's been attempted here.
  

On Fri, 2024-06-28 at 23:12 -0400, Maxim Cournoyer wrote:
> 
> While I dislike the attitude/approach used, I think the essence of the
> complaint was that Guix, via SHW, was somehow facilitating the
> scavenging of free software sources to train large language models
> (LLM), with the opinion that these models do not respect the licenses of
> the sources ingested for their produced output (the work is considered
> new work, not a derived work, or perhaps it's still legally a gray area,
> I don't know).  In this perspective, the original poster was seeking to
> have the free software more protected against what they see as a loop
> hole in the LLM business, as explained above.

  Original, original poster here (I'm feeling pretty awkward right now TBH, like
a bad shit-stirrer).  The point is that I use GUIX to support my own, private
projects.  It is nothing to do with licensing, I'm the only one who has ever
seen the code.  In this context it is unacceptable that GUIX should give it away
to anyone.

Dale
  

Hi Dale,

Dale Mellor <guix-devel-0brg6a@rdmp.org> writes:

> On Fri, 2024-06-28 at 23:12 -0400, Maxim Cournoyer wrote:
>> 
>> While I dislike the attitude/approach used, I think the essence of the
>> complaint was that Guix, via SHW, was somehow facilitating the
>> scavenging of free software sources to train large language models
>> (LLM), with the opinion that these models do not respect the licenses of
>> the sources ingested for their produced output (the work is considered
>> new work, not a derived work, or perhaps it's still legally a gray area,
>> I don't know).  In this perspective, the original poster was seeking to
>> have the free software more protected against what they see as a loop
>> hole in the LLM business, as explained above.
>
>   Original, original poster here (I'm feeling pretty awkward right now TBH, like
> a bad shit-stirrer).  The point is that I use GUIX to support my own, private
> projects.  It is nothing to do with licensing, I'm the only one who has ever
> seen the code.  In this context it is unacceptable that GUIX should give it away
> to anyone.

OK. From my understanding, the code is not transferred; only an archival
request to the project URL is submitted to SHW, and its up to SHW to
attempt to retrieve it (which would fail if the URL is private/protected
by some means).

Perhaps we could have a dummy 'private' or 'non-free' license added to
(guix licenses), that the 'check-archival' procedure would check to skip
the archival request?  This would need to be documented (mentioning it's
not for use for packages carried in the Guix collection, but for end
users working on a software not meant to be distributed).
  

Hi Dale,

Dale Mellor <guix-devel-0brg6a@rdmp.org> writes:

> On Mon, 2024-07-01 at 16:44 -0400, Maxim Cournoyer wrote:
>> 
>> OK. From my understanding, the code is not transferred; only an archival
>> request to the project URL is submitted to SHW, and its up to SHW to
>> attempt to retrieve it (which would fail if the URL is private/protected
>> by some means).
>> 
>> Perhaps we could have a dummy 'private' or 'non-free' license added to
>> (guix licenses), that the 'check-archival' procedure would check to skip
>> the archival request?  This would need to be documented (mentioning it's
>> not for use for packages carried in the Guix collection, but for end
>> users working on a software not meant to be distributed).
>> 
>
>   You have now brought the discussion full circle; you can pick it up at
> https://lists.gnu.org/archive/html/guix-devel/2024-06/msg00225.html.

Thanks, sorry for missing it.  I like the replies from Ricardo there.
  

Greg Hogan <code@greghogan.com> skribis:

> On Wed, Jun 26, 2024 at 3:28 PM Maxim Cournoyer
> <maxim.cournoyer@gmail.com> wrote:
>>
>> I don't think these exclusions should be committed in general to the
>> repo, except when we have for example the author of some software
>> explicitly requesting that SWH archival be disabled for it in Guix.
>
> Author requests are as problematic to a free software distribution as
> the earlier demands to modify historical data are to reproducibility.

+1

I think this would be a slippery slope.  Free software is meant to be
freely redistributable, unconditionally.

Ludo’.
  

Hi Dale,

On Sun, 30 Jun 2024 at 15:48, Dale Mellor <guix-devel-0brg6a@rdmp.org> wrote:

>   Original, original poster here (I'm feeling pretty awkward right now TBH, like
> a bad shit-stirrer).  The point is that I use GUIX to support my own, private
> projects.  It is nothing to do with licensing, I'm the only one who has ever
> seen the code.  In this context it is unacceptable that GUIX should give it away
> to anyone.

Well, from my understanding, the general case for packages included in
Guix proper is to run all checkers.

Then, my willing with this patch submission is to address side projects,
as you are pointing.  The first answer is: it’s not a problem because
“guix lint” offers the option “--exclude”.  However, as you mentioned,
it’s easy to forget and… bang.  Hence, the patch set as mitigation for
public side projects using Guix:

 + Have a --dry-run option
 + Exclude lint checkers at the package definition level
 + Still have the option --exclude option

I think, having these 3 features addresses the case of public side
projects.

Cheers,
simon
  

Hi Greg,

On Thu, 27 Jun 2024 at 12:38, Greg Hogan <code@greghogan.com> wrote:

> If this is the best use case for a spurious feature request then I
> find this a dangerous addition to the project.

Sorry, I do not see the danger.  What I see is the same policy for the
project – nothing is changed – and the patch set provides an helper for
third-party channels outside the project.

When developing or maintaining a third-party channel outside the
project, one might systematically run:

    guix lint -L . -x refresh,github-urls foobar

because of some reasons of ’foobar’.  I do not see where it is dangerous
to also have the alternative to configure this exclusion at the package
level definition.

>                                                Those denigrading and
> demanding that Guix pressure partner projects to restrict the use of
> free software are unlikely to be content adding these flags to their
> private packages as may exist.

About pressure, I will not rehash here what had been said at length
elsewhere. :-)  Let me clarify about “restrict”.

For sure, I agree that by definition of free software, one cannot
restrict its usage.  The key point here seems between a right and an
obligation.  One has the right to modify and/or share but no obligation;
it’s still free software. :-)

Therefore, if one uses Guix to develop packages, it’s up to them to
decide how they want to share their developments on free software.
However, we have the right to use these developments how we want –
limited by what the license allows.

All in all, I do not see the danger. :-)

Cheers,
simon
  

Hi,

On Wed, 26 Jun 2024 at 15:28, Maxim Cournoyer <maxim.cournoyer@gmail.com> wrote:

>> Could we not instead create a GUIX_LINT_OPTIONS, similar to
>> GUIX_BUILD_OPTIONS? Then anyone wishing to universally exclude certain
>> checkers (or disable network checks) on their own system would be free
>> to do so.
>
> That would be a good option to have too, on top of the other one.

Well, I am not convinced it would be helpful.  Because if you have:

    GUIX_LINT_EXCLUDE=archival,home-page guix lint -L . foobar

is as complicated as:

    guix lint -L . -x archival,home-page foobar

And if ones does:

    export GUIX_LINT_EXCLUDE=archival

then the checker ’archival’ would be always excluded, i.e., for the
third-party custom packages – that’s what the aim :-) – but also when
the person would contribute to Guix proper – and that’s against our
quality assurance, IMHO.

Somehow, from my point of view, the idea of the patch set is only to
provide a complementary mechanism of “--exclude”.  Consider that I am
packaging something for Guix proper and I am bored by some checker
because it takes time, or because the warning annoys me or because it
sends again and again the exact same request to SWH or because whatever
other reasons, then I can just temporarily turn off that checker when
developing and looping over “guix lint”, either via --exclude or either
via the package property.  Once the package is ready, I submit it with
all checkers turned on.

Obviously, a third-party channel could use this mechanism to turn off
the checkers.  It’s up to them. :-)

Well, maybe the wording of the manual could be tweaked if it does not
capture this idea.

Cheers,
simon
  

On Fri, Jul 12, 2024 at 1:22 PM Simon Tournier <zimon.toutoune@gmail.com> wrote:
>
> Hi Greg,
>
> On Thu, 27 Jun 2024 at 12:38, Greg Hogan <code@greghogan.com> wrote:
>
> > If this is the best use case for a spurious feature request then I
> > find this a dangerous addition to the project.
>
> Sorry, I do not see the danger.  What I see is the same policy for the
> project – nothing is changed – and the patch set provides an helper for
> third-party channels outside the project.

Please allow me to preface my response by thanking you for your
contributions to the project, and to thank all of the many illustrious
Guix contributors in this thread. There is certainly value in this
feature request and my only care is for consideration of the
implementation.

> When developing or maintaining a third-party channel outside the
> project, one might systematically run:
>
>     guix lint -L . -x refresh,github-urls foobar
>
> because of some reasons of ’foobar’.  I do not see where it is dangerous
> to also have the alternative to configure this exclusion at the package
> level definition.

In addition to adding GUIX_LINT_OPTIONS (modeled on
GUIX_BUILD_OPTIONS) could we extend the exclusions to allow
package-specific definitions as with package transformations? For
example, GUIX_LINT_OPTIONS="--exclude=archival,home-page=mypackage",
which would disable archival for all packages but the home-page check
only for "mypackage".

> >                                                Those denigrading and
> > demanding that Guix pressure partner projects to restrict the use of
> > free software are unlikely to be content adding these flags to their
> > private packages as may exist.
>
> About pressure, I will not rehash here what had been said at length
> elsewhere. :-)  Let me clarify about “restrict”.
>
> For sure, I agree that by definition of free software, one cannot
> restrict its usage.  The key point here seems between a right and an
> obligation.  One has the right to modify and/or share but no obligation;
> it’s still free software. :-)

IANAL but I do not believe this to be the case. The LICENSE does not
apply to or restrict the developer's use of the software, only the
recipient. If the software has not been distributed then it cannot be
considered free software.

> Therefore, if one uses Guix to develop packages, it’s up to them to
> decide how they want to share their developments on free software.
> However, we have the right to use these developments how we want –
> limited by what the license allows.

Guix is restricted by the GNU FSDG
[https://www.gnu.org/distros/free-system-distribution-guidelines.html].
Are the third-party channels referenced above "committed to only
including free software"?

> All in all, I do not see the danger. :-)
>
> Cheers,
> simon