mbox

[bug#37838,0/2] Rewrite (guix cve) to read NIST's JSON feed

Message ID 20191020203451.1912-1-ludo@gnu.org
Headers show

Message

Ludovic Courtès Oct. 20, 2019, 8:34 p.m. UTC 
Hello!

Last Thursday I was surprised to see that ‘guix lint -c cve’
would be redirected to:

  https://nvd.nist.gov/General/News/XML-Vulnerability-Feed-Retirement-Phase-3

… leading to a failure.

And indeed, the XML CVE feed has now been replaced by a JSON feed
(let’s hope they don’t switch to YAML next year :-)).  The JSON feed
seems to be nicer in some ways; for instance, it can specify ranges
of versions to which a given CVE applies.

The patch that follows rewrites (guix cve) so it gets info from the
JSON feed.  It does so by providing a one-to-one mapping between data
structures in JSON and Scheme records, and then converting those to
the higher-level <vulnerability> records that were already there before.

If you look at the JSON-mapped record types, there are lots of
low-hanging fruits; for instance, we could grab severity info from
the JSON feeds and use them somehow.  I’m not sure if ‘guix lint’
is the best place to display detailed CVE info, but we could/should
use that info somehow.

Feedback welcome!

Ludo’.

Ludovic Courtès (2):
  cve: Rewrite to read the JSON feed instead of the XML feed.
  lint: Re-enable CVE checker.

 Makefile.am           |    2 +-
 doc/guix.texi         |    4 +-
 guix/cve.scm          |  376 ++++++++----
 guix/lint.scm         |   16 +-
 tests/cve-sample.json | 1279 +++++++++++++++++++++++++++++++++++++++++
 tests/cve-sample.xml  |  616 --------------------
 tests/cve.scm         |   83 ++-
 7 files changed, 1610 insertions(+), 766 deletions(-)
 create mode 100644 tests/cve-sample.json
 delete mode 100644 tests/cve-sample.xml

Comments

Ludovic Courtès Oct. 23, 2019, 2:48 p.m. UTC | #1 
Hello,

Ludovic Courtès <ludo@gnu.org> skribis:

>   cve: Rewrite to read the JSON feed instead of the XML feed.
>   lint: Re-enable CVE checker.

Pushed as 9efa2c28a4f842b7ca1977e084299de441842856.

Please let me know if you notice anything fishy with ‘guix lint -c cve’:
CVEs not showing up, CVEs showing up that should not, etc.

Ludo’.
Efraim Flashner Oct. 23, 2019, 4:46 p.m. UTC | #2 
On Wed, Oct 23, 2019 at 04:48:52PM +0200, Ludovic Courtès wrote:
> Hello,
> 
> Ludovic Courtès <ludo@gnu.org> skribis:
> 
> >   cve: Rewrite to read the JSON feed instead of the XML feed.
> >   lint: Re-enable CVE checker.
> 
> Pushed as 9efa2c28a4f842b7ca1977e084299de441842856.
> 
> Please let me know if you notice anything fishy with ‘guix lint -c cve’:
> CVEs not showing up, CVEs showing up that should not, etc.
> 

Sorry to respond to a closed bug, the CVE for vim shows up as expected.
(I was actually a little worried before when it wasn't showing up)
Marius Bakke Oct. 23, 2019, 5:35 p.m. UTC | #3 
Ludovic Courtès <ludo@gnu.org> writes:

> Hello,
>
> Ludovic Courtès <ludo@gnu.org> skribis:
>
>>   cve: Rewrite to read the JSON feed instead of the XML feed.
>>   lint: Re-enable CVE checker.
>
> Pushed as 9efa2c28a4f842b7ca1977e084299de441842856.
>
> Please let me know if you notice anything fishy with ‘guix lint -c cve’:
> CVEs not showing up, CVEs showing up that should not, etc.

Here is what I get (on ee42e9f9f):

$ ./pre-inst-env guix lint -c cve ao
fetching CVE database for 2019...]...
Backtrace:
          11 (apply-smob/1 #<catch-closure 7f08d6d9d900>)
In ice-9/boot-9.scm:
    705:2 10 (call-with-prompt _ _ #<procedure default-prompt-handler (k proc)>)
In ice-9/eval.scm:
    619:8  9 (_ #(#(#<directory (guile-user) 7f08d6a23140>)))
In guix/ui.scm:
  1730:12  8 (run-guix-command _ . _)
In srfi/srfi-1.scm:
    640:9  7 (for-each #<procedure 7f08d689f3c0 at guix/scripts/lint.scm:168:16 (spec)> ("ao"))
In guix/scripts/lint.scm:
     57:4  6 (run-checkers _ _)
In srfi/srfi-1.scm:
    640:9  5 (for-each #<procedure 7f08c7706480 at guix/scripts/lint.scm:57:14 (checker)> (#<<lint-checker> name: c…>))
In guix/scripts/lint.scm:
    64:17  4 (_ _)
In guix/lint.scm:
    999:4  3 (check-vulnerabilities _)
    994:9  2 (_ _)
In unknown file:
           1 (force #<promise #<procedure 7f08d42e7928 at guix/lint.scm:982:16 ()>>)
In guix/lint.scm:
   983:24  0 (_)

guix/lint.scm:983:24: Throw to key `srfi-34' with args `(#<condition &message [message: "invalid CVE feed"] 7f08b5a39920>)'.

I tried downloading the .json.gz files manually and they seem fine.
Marius Bakke Nov. 3, 2019, 5:29 p.m. UTC | #4 
Ludovic Courtès <ludo@gnu.org> writes:

> Hi Marius,
>
> Marius Bakke <mbakke@fastmail.com> skribis:
>
>> Ludovic Courtès <ludo@gnu.org> writes:
>>
>>> Hello,
>>>
>>> Ludovic Courtès <ludo@gnu.org> skribis:
>>>
>>>>   cve: Rewrite to read the JSON feed instead of the XML feed.
>>>>   lint: Re-enable CVE checker.
>>>
>>> Pushed as 9efa2c28a4f842b7ca1977e084299de441842856.
>>>
>>> Please let me know if you notice anything fishy with ‘guix lint -c cve’:
>>> CVEs not showing up, CVEs showing up that should not, etc.
>>
>> Here is what I get (on ee42e9f9f):
>>
>> $ ./pre-inst-env guix lint -c cve ao
>> fetching CVE database for 2019...]...
>> Backtrace:
>>           11 (apply-smob/1 #<catch-closure 7f08d6d9d900>)
>> In ice-9/boot-9.scm:
>>     705:2 10 (call-with-prompt _ _ #<procedure default-prompt-handler (k proc)>)
>> In ice-9/eval.scm:
>>     619:8  9 (_ #(#(#<directory (guile-user) 7f08d6a23140>)))
>> In guix/ui.scm:
>>   1730:12  8 (run-guix-command _ . _)
>> In srfi/srfi-1.scm:
>>     640:9  7 (for-each #<procedure 7f08d689f3c0 at guix/scripts/lint.scm:168:16 (spec)> ("ao"))
>> In guix/scripts/lint.scm:
>>      57:4  6 (run-checkers _ _)
>> In srfi/srfi-1.scm:
>>     640:9  5 (for-each #<procedure 7f08c7706480 at guix/scripts/lint.scm:57:14 (checker)> (#<<lint-checker> name: c…>))
>> In guix/scripts/lint.scm:
>>     64:17  4 (_ _)
>> In guix/lint.scm:
>>     999:4  3 (check-vulnerabilities _)
>>     994:9  2 (_ _)
>> In unknown file:
>>            1 (force #<promise #<procedure 7f08d42e7928 at guix/lint.scm:982:16 ()>>)
>> In guix/lint.scm:
>>    983:24  0 (_)
>>
>> guix/lint.scm:983:24: Throw to key `srfi-34' with args `(#<condition &message [message: "invalid CVE feed"] 7f08b5a39920>)'.
>>
>> I tried downloading the .json.gz files manually and they seem fine.
>
> I don’t encounter this problem.  Is it reproducible for you?

I still get this when using './pre-inst-env', even after a 'make
clean-go'.  It works without the './pre-inst-env script'(!?).
Ludovic Courtès Nov. 4, 2019, 5:32 p.m. UTC | #5 
Hello,

Marius Bakke <mbakke@fastmail.com> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Hi Marius,
>>
>> Marius Bakke <mbakke@fastmail.com> skribis:
>>
>>> Ludovic Courtès <ludo@gnu.org> writes:
>>>
>>>> Hello,
>>>>
>>>> Ludovic Courtès <ludo@gnu.org> skribis:
>>>>
>>>>>   cve: Rewrite to read the JSON feed instead of the XML feed.
>>>>>   lint: Re-enable CVE checker.
>>>>
>>>> Pushed as 9efa2c28a4f842b7ca1977e084299de441842856.
>>>>
>>>> Please let me know if you notice anything fishy with ‘guix lint -c cve’:
>>>> CVEs not showing up, CVEs showing up that should not, etc.
>>>
>>> Here is what I get (on ee42e9f9f):
>>>
>>> $ ./pre-inst-env guix lint -c cve ao
>>> fetching CVE database for 2019...]...
>>> Backtrace:
>>>           11 (apply-smob/1 #<catch-closure 7f08d6d9d900>)
>>> In ice-9/boot-9.scm:
>>>     705:2 10 (call-with-prompt _ _ #<procedure default-prompt-handler (k proc)>)
>>> In ice-9/eval.scm:
>>>     619:8  9 (_ #(#(#<directory (guile-user) 7f08d6a23140>)))
>>> In guix/ui.scm:
>>>   1730:12  8 (run-guix-command _ . _)
>>> In srfi/srfi-1.scm:
>>>     640:9  7 (for-each #<procedure 7f08d689f3c0 at guix/scripts/lint.scm:168:16 (spec)> ("ao"))
>>> In guix/scripts/lint.scm:
>>>      57:4  6 (run-checkers _ _)
>>> In srfi/srfi-1.scm:
>>>     640:9  5 (for-each #<procedure 7f08c7706480 at guix/scripts/lint.scm:57:14 (checker)> (#<<lint-checker> name: c…>))
>>> In guix/scripts/lint.scm:
>>>     64:17  4 (_ _)
>>> In guix/lint.scm:
>>>     999:4  3 (check-vulnerabilities _)
>>>     994:9  2 (_ _)
>>> In unknown file:
>>>            1 (force #<promise #<procedure 7f08d42e7928 at guix/lint.scm:982:16 ()>>)
>>> In guix/lint.scm:
>>>    983:24  0 (_)
>>>
>>> guix/lint.scm:983:24: Throw to key `srfi-34' with args `(#<condition &message [message: "invalid CVE feed"] 7f08b5a39920>)'.
>>>
>>> I tried downloading the .json.gz files manually and they seem fine.
>>
>> I don’t encounter this problem.  Is it reproducible for you?
>
> I still get this when using './pre-inst-env', even after a 'make
> clean-go'.  It works without the './pre-inst-env script'(!?).

Hmm hmm!  Could you add some ‘pk’ calls around there in guix/lint.scm?

Ludo’.