From patchwork Mon Feb 7 09:07:07 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Xinglu Chen X-Patchwork-Id: 37052 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 2EE8D27BBEA; Mon, 7 Feb 2022 09:52:15 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-0.7 required=5.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,FROM_SUSPICIOUS_NTLD,MAILING_LIST_MULTI,PDS_OTHER_BAD_TLD, SPF_HELO_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 36F9B27BBE9 for ; Mon, 7 Feb 2022 09:52:14 +0000 (GMT) Received: from localhost ([::1]:60868 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nH0gv-0006RB-BK for patchwork@mira.cbaines.net; Mon, 07 Feb 2022 04:52:13 -0500 Received: from eggs.gnu.org ([209.51.188.92]:40376) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nH00B-00068F-R1 for guix-patches@gnu.org; Mon, 07 Feb 2022 04:08:04 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:47044) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1nH00B-0004LJ-04 for guix-patches@gnu.org; Mon, 07 Feb 2022 04:08:03 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1nH00A-0002vA-SZ; Mon, 07 Feb 2022 04:08:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#53818] [PATCH v2 5/7] import: Add 'repology' updater. Resent-From: Xinglu Chen Original-Sender: "Debbugs-submit" Resent-CC: maximedevos@telenet.be, guix-patches@gnu.org Resent-Date: Mon, 07 Feb 2022 09:08:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 53818 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 53818@debbugs.gnu.org Cc: Maxime Devos X-Debbugs-Original-Xcc: Maxime Devos Received: via spool by 53818-submit@debbugs.gnu.org id=B53818.164422484111131 (code B ref 53818); Mon, 07 Feb 2022 09:08:02 +0000 Received: (at 53818) by debbugs.gnu.org; 7 Feb 2022 09:07:21 +0000 Received: from localhost ([127.0.0.1]:40929 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nGzzT-0002tM-Vk for submit@debbugs.gnu.org; Mon, 07 Feb 2022 04:07:20 -0500 Received: from h178-251-242-94.cust.a3fiber.se ([178.251.242.94]:55300 helo=mail.yoctocell.xyz) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nGzzO-0002si-3V for 53818@debbugs.gnu.org; Mon, 07 Feb 2022 04:07:15 -0500 From: Xinglu Chen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yoctocell.xyz; s=mail; t=1644224828; bh=tM2bgZ0c95q9tm79KzPQw0F7/5o5pTKGvJyzdbNuSOQ=; h=From:To:Subject:In-Reply-To:References:Date; b=k/rcqoMhqU81atv1BjUpLmBtl6WwkMUJjnOrdcweAuAQsGnnQ9XTMbQTNKguz+MI+ rEQTYKS+ChBFYRSfA9vNuph3Dhem3f8Cksa7Xls8IPMoz38nYD0G1MdmJaPN5u3Rzy Z0QlTySUbpa7GIV3gORwRtZdEazJB9vhM/MIMftw= In-Reply-To: References: Message-Id: <98726379214702d0745f56d9f946e792e803d326.1644224421.git.public@yoctocell.xyz> Date: Mon, 07 Feb 2022 10:07:07 +0100 MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches * guix/import/repology.scm * tests/import-repology.scm: New files. * Makefile.am (MODULES): Register them. * doc/guix.texi (Invoking guix refresh): Document it. --- Makefile.am | 3 + doc/guix.texi | 7 ++ guix/import/repology.scm | 235 ++++++++++++++++++++++++++++++++++++++ tests/import-repology.scm | 145 +++++++++++++++++++++++ 4 files changed, 390 insertions(+) create mode 100644 guix/import/repology.scm create mode 100644 tests/import-repology.scm diff --git a/Makefile.am b/Makefile.am index 7463606d20..6792917b59 100644 --- a/Makefile.am +++ b/Makefile.am @@ -16,6 +16,7 @@ # Copyright © 2019 Efraim Flashner # Copyright © 2021 Chris Marusich # Copyright © 2021 Andrew Tropin +# Copyright © 2022 Xinglu Chen # # This file is part of GNU Guix. # @@ -271,6 +272,7 @@ MODULES = \ guix/import/opam.scm \ guix/import/print.scm \ guix/import/pypi.scm \ + guix/import/repology.scm \ guix/import/stackage.scm \ guix/import/texlive.scm \ guix/import/utils.scm \ @@ -488,6 +490,7 @@ SCM_TESTS = \ tests/home-import.scm \ tests/import-git.scm \ tests/import-github.scm \ + tests/import-repology.scm \ tests/import-utils.scm \ tests/inferior.scm \ tests/lint.scm \ diff --git a/doc/guix.texi b/doc/guix.texi index 0cf865a672..15d215dd48 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -12932,6 +12932,13 @@ (release-tag-version-delimiter . ":")))) @end lisp +@item repology +an updater that scans @uref{https://repology.org, Repology}, a website +that tracks packages on various package repositories, for updates. + +The name of a package in Guix is not always that same as the name on +Repology; users can set the @code{repology-name} package property to +make the updater use the correct name. @end table diff --git a/guix/import/repology.scm b/guix/import/repology.scm new file mode 100644 index 0000000000..28f3a3af5f --- /dev/null +++ b/guix/import/repology.scm @@ -0,0 +1,235 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright © 2022 Xinglu Chen +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see . + +(define-module (guix import repology) + #:use-module (guix diagnostics) + #:use-module (guix diagnostics) + #:use-module (guix git-download) + #:use-module (guix i18n) + #:use-module (guix import json) + #:use-module (guix import utils) + #:use-module (guix memoization) + #:use-module (guix packages) + #:use-module (guix upstream) + #:use-module (guix utils) + #:use-module (ice-9 match) + #:use-module (json) + #:use-module (srfi srfi-1) + #:use-module (srfi srfi-2) + #:use-module (srfi srfi-26) + #:use-module (srfi srfi-43) + #:export (repology-latest-release + %repology-updater)) + +;;; Commentary: +;;; +;;; This module provides an updater which scans Repology, a site that monitors +;;; several package repolsitories, for updates. This means that if any other +;;; package repository has a version of a package that is newer than the +;;; version in Guix, the package should be able to be updated. The updater +;;; should in theory work for all packages in Guix, but the names of some +;;; packages on Repology don't match the name in Guix. The 'repology-name' +;;; package property can be used to fix this. +;;; +;;; Guix already has many different updaters for language-specific packages, +;;; and these typically provide more accurate data, e.g., input changes, +;;; signature URLs. The Repology updater should really be treated as a last +;;; resort for those packages that don't have any other updater to rely on. +;;; +;;; See for the API. +;;; +;;; Code: + +(define %repology-url + "https://repology.org/api/v1/project") + +(define* (package-name->repology-name name #:key (attempt 1)) + "Convert NAME, the name of a Guix package, to the name of the package on +Repology. It doesn't always guess the correct name on the first attempt, so +on the second attempt it will try to guess another name." + (match attempt + (1 (cond + ((string-prefix? "ghc-" name) + (string-append "haskell:" + (string-drop name (string-length "ghc-")))) + ((string-prefix? "ocaml-" name) + (string-append "ocaml:" + (string-drop name (string-length "ocaml-")))) + ((string-prefix? "perl-" name) + (string-append "perl:" + (string-drop name (string-length "perl-")))) + ((string-prefix? "emacs-" name) + (string-append "emacs:" + (string-drop name (string-length "emacs-")))) + ((string-prefix? "go-" name) + (string-append "go:" + (string-drop name (string-length "go-")))) + ((string-prefix? "rust-" name) + (string-append "rust:" + (string-drop name (string-length "rust-")))) + ((string-prefix? "lua-" name) + (string-append "lua:" + (string-drop name (string-length "lua-")))) + ((string-prefix? "node-" name) + (string-append "node:" + (string-drop name (string-length "node-")))) + ((string-prefix? "python-" name) + (string-append "python:" + (string-drop name (string-length "python-")))) + ((string-prefix? "java-" name) + (string-append "java:" + (string-drop name (string-length "java-")))) + ((string-prefix? "r-" name) + (string-append "r:" + (string-drop name (string-length "r-")))) + ((string-prefix? "ruby-" name) + (string-append "ruby:" + (string-drop name (string-length "ruby-")))) + ((string-prefix? "xf86-" name) + (string-append "xdrv:" + (string-drop name (string-length "xf86-")))) + ((string-prefix? "font-" name) + (string-append "fonts:" + (string-drop name (string-length "font-")))) + ((string-suffix? "-minimal" name) + (string-drop-right name (string-length "-minimal"))) + (else name))) + (2 (cond + ((string-prefix? "xf86-video" name) + (string-append "xdrv:" + (string-drop name (string-length "xf86-video-")))) + ((string-prefix? "xf86-input" name) + (string-append "xdrv:" + (string-drop name (string-length "xf86-input-")))) + ((string-prefix? "minetest-" name) + (string-append "minetest-mod-" + (string-drop name (string-length "minetest-")))) + (else name))))) + + +;;; JSON mappings. + +(define-json-mapping make-repology-package + repology-package? + json->repology-package + (repository repology-package-repository "repo") + (src-name repology-package-src-name "srcname") + (binary-name repology-package-binary-name "binname") + (visible-name repology-package-visible-name "visiblename") + (version repology-package-version) + (original-version repology-package-original-version "origversion") + (status repology-package-status) + (summary repology-package-summary) + (categories repology-package-categories) + (licenses repology-package-licenses) + (maintainers repology-package-maintainers)) + + +;;; Updater. + +(define repology-fetch-info + (memoize + (lambda (package) + "Fetch information about PACKAGE using the Repology API." + (define (name->info name) + (let ((url (string-append %repology-url "/" name))) + (and=> (json-fetch url #:cached? #t) + (lambda (url) + (vector-map (lambda (a b) + (json->repology-package b)) + url))))) + + (let* ((name (or (assoc-ref (package-properties package) + 'repology-name) + (package-name->repology-name (package-name package)))) + (info (name->info name))) + (if (and info (not (vector-empty? info))) + info + (let ((info (name->info (package-name->repology-name + (package-name package) + #:attempt 2)))) + (if (and info (not (vector-empty? info))) + info + (begin + (warning (G_ "package not found on Repology: ~a\n") + (package-name package)) + #f)))))))) + +(define (update-version string old-version new-version) + "Replace OLD-VERSION in STRING with NEW-VERSION. This assumes that STRING +contains OLD-VERSION verbatim; if it doesn't, #f is returned." + (match (factorize-uri string old-version) + ((? string?) #f) + ((factorized ...) + (apply string-append + (map (lambda (component) + (match component + ('version new-version) + ((? string?) component))) + factorized))))) + +(define (package-source-urls package version) + "Return a list of URLs for PACKAGE at VERSION. If no URL was successfully constructed, return #f." + (let ((old-version (package-version package))) + ;; XXX: (guix upstream) only supports tarballs and Git repos for now. + (match (origin-uri (package-source package)) + ((? git-reference? reference) + (and-let* ((old-commit (git-reference-commit reference)) + (new-commit (if (string=? old-version old-commit) + version + (update-version old-commit + old-version + version)))) + (git-reference + (inherit reference) + (commit new-commit)))) + ((? string? url) + (list (update-version url old-version version))) + ((? list? urls) + (map (cut update-version <> old-version version) urls)) + (_ #f)))) + +(define (latest-version? repology-package) + "Return the latest released version of REPOLOGY-PACKAGE. If none are found, +return #f." + (and (or (equal? "newest" (repology-package-status repology-package)) + (equal? "unique" (repology-package-status repology-package))) + (repology-package-version repology-package))) + +;; XXX: We use 'pkg' because 'package' will clash with the 'package' field of +;; 'upstream-source'. +(define (repology-latest-release pkg) + "Return the latest release of the PKG on Repology named NAME." + (and-let* ((packages (repology-fetch-info pkg)) + (versions (filter-map latest-version? + (vector->list packages))) + (latest-version (and (pair? versions) (car versions)))) + ;; TODO: set 'signature-urls'. + (upstream-source + (package (package-name pkg)) + (version latest-version) + (urls (package-source-urls pkg latest-version))))) + +(define %repology-updater + (upstream-updater + (name 'repology) + (description "Updater for packages on Repology") + (pred (const #t)) + (latest repology-latest-release))) + +;;; repology.scm ends here diff --git a/tests/import-repology.scm b/tests/import-repology.scm new file mode 100644 index 0000000000..2d366db283 --- /dev/null +++ b/tests/import-repology.scm @@ -0,0 +1,145 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright © 2022 Xinglu Chen +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see . + +(define-module (test-import-repology) + #:use-module (guix download) + #:use-module (guix git-download) + #:use-module (guix import repology) + #:use-module (guix packages) + #:use-module (guix tests) + #:use-module (guix upstream) + #:use-module (json) + #:use-module (srfi srfi-64)) + +(test-begin "repology") + +(define package-using-git-repository + (dummy-package + "foo" + (version "1.0") + (source + (origin + (method git-fetch) + (uri (git-reference + (url "https://git.example.org/foo") + (commit "1.0"))) + (sha256 #f))))) + +(define package-using-tarball + (dummy-package + "foo" + (version "1.0") + (source + (origin + (method git-fetch) + (uri (string-append "https://example.org/foo-" version ".tar.gz")) + (sha256 #f))))) + +(define package-using-tarball-multiple-urls + (dummy-package + "foo" + (version "1.0") + (source + (origin + (method git-fetch) + (uri (list (string-append "https://example.org/foo-" + version ".tar.gz") + (string-append "https://mirror.example.org/foo-" + version ".tar.gz"))) + (sha256 #f))))) + +(define %test-json +"[ + { + \"repo\": \"aur\", + \"srcname\": \"foo\", + \"binname\": \"foo\", + \"visiblename\": \"foo\", + \"version\": \"1.0.r25.gb86405a\", + \"maintainers\": [ + \"bob@aur\" + ], + \"licenses\": [ + \"LGPL3+\" + ], + \"summary\": \"foo bar\" + \"status\": \"rolling\", + \"origversion\": \"1.0.r25.gb86405a-1\" + }, + { + \"repo\": \"gnuguix\", + \"srcname\": \"foo\", + \"binname\": \"foo\", + \"visiblename\": \"foo\", + \"version\": \"1.0\", + \"summary\": \"foo bar\", + \"status\": \"outdated\", + \"origversion\": null + }, + { + \"repo\": \"nix_unstable\", + \"name\": \"foo\", + \"visiblename\": \"foo\", + \"version\": \"2.0\", + \"maintainers\": [ + \"bob@example.org\" + ], + \"licenses\": [ + \"LGPL-3.0-or-later\" + ], + \"summary\": \"foo bar\", + \"status\": \"newest\", + \"origversion\": null + } +]") + +(define (latest-release package) + (mock ((guix import json) json-fetch + (lambda* (url #:key cached?) + (json-string->scm %test-json))) + (repology-latest-release package))) + +(test-equal "package using Git repo: version" + "2.0" + (upstream-source-version + (latest-release package-using-git-repository))) + +(test-equal "package using Git repo: git-reference" + (git-reference + (url "https://git.example.org/foo") + (commit "2.0")) + (upstream-source-urls + (latest-release package-using-git-repository))) + +(test-equal "package using tarball: version" + "2.0" + (upstream-source-version + (latest-release package-using-tarball))) + +(test-equal "package using tarball: URL" + (list "https://example.org/foo-2.0.tar.gz") + (upstream-source-urls + (latest-release package-using-tarball))) + +(test-equal "package using tarball: multiple URLs" + (list "https://example.org/foo-2.0.tar.gz" + "https://mirror.example.org/foo-2.0.tar.gz") + (upstream-source-urls + (latest-release package-using-tarball-multiple-urls))) + +(test-end "repology")