From patchwork Tue Feb 18 12:32:45 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Simon Tournier X-Patchwork-Id: 20308 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id E80CE27BBEA; Tue, 18 Feb 2020 12:34:10 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,T_DKIM_INVALID, URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTP id 5CA2227BBE4 for ; Tue, 18 Feb 2020 12:34:10 +0000 (GMT) Received: from localhost ([::1]:33634 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1j424n-0007ZE-RT for patchwork@mira.cbaines.net; Tue, 18 Feb 2020 07:34:09 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:37718) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1j424h-0007YV-NK for guix-patches@gnu.org; Tue, 18 Feb 2020 07:34:04 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1j424g-0004XH-GV for guix-patches@gnu.org; Tue, 18 Feb 2020 07:34:03 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:33649) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1j424g-0004X0-Cu for guix-patches@gnu.org; Tue, 18 Feb 2020 07:34:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1j424g-0006Tj-Ad for guix-patches@gnu.org; Tue, 18 Feb 2020 07:34:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#39547] [PATCH v2 1/2] website: Refactor and resolve mirror:// of JSON package list. References: <20200210170418.32076-1-zimon.toutoune@gmail.com> In-Reply-To: <20200210170418.32076-1-zimon.toutoune@gmail.com> Resent-From: zimoun Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Tue, 18 Feb 2020 12:34:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 39547 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 39547@debbugs.gnu.org Cc: zimoun Received: via spool by 39547-submit@debbugs.gnu.org id=B39547.158202918324825 (code B ref 39547); Tue, 18 Feb 2020 12:34:02 +0000 Received: (at 39547) by debbugs.gnu.org; 18 Feb 2020 12:33:03 +0000 Received: from localhost ([127.0.0.1]:39619 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1j423i-0006SL-Jk for submit@debbugs.gnu.org; Tue, 18 Feb 2020 07:33:02 -0500 Received: from mail-wr1-f53.google.com ([209.85.221.53]:38608) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1j423g-0006Rp-Hy for 39547@debbugs.gnu.org; Tue, 18 Feb 2020 07:33:00 -0500 Received: by mail-wr1-f53.google.com with SMTP id e8so1219492wrm.5 for <39547@debbugs.gnu.org>; Tue, 18 Feb 2020 04:33:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=7JTnTCTPn054MdGgkuwiw6bQAkCKfGWZfS315KivH6Q=; b=MlPESGYUf9GxMquSEaMdVBc4LPjgfgTQknaOWCd6WoKmVBAkGJ8uE6cosRdbhQSfxL UEFm/Tropy8mv+kDrDnpRtMPXgP+DD57DLhRyicZ7uptk4H1JgSA0NmS6slRTurqyP7e YXSWbjsXM8wKVXYf9IRZ/B4+TmPN912+rRqSer/KUc1I9b2/ktowexyhMOlZdDqAZf7g pzie7elxN8X9y+4AQzllq91tFMxPu3uLEsnwAZFw3ArgCRAMiGcg0zn198G/Xg/dO477 VrfY5JYzGI8tUZ2NyKd71nKjphAni1JGRjqQd87JvSl4P5bqX0u0R5sMCVzmsJqqXaQ8 KFWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=7JTnTCTPn054MdGgkuwiw6bQAkCKfGWZfS315KivH6Q=; b=Qyn3O/pMKrxvcVydBCNffc+K6/GWNZoZgq0HmyAY8TCdvVwzKxfA7kqqTEG4ZC+rXX UF7zU9SHX5JapDkhqZ1ZjErNTmcLJXEhASIRROaOWP3hD0POrvblRtTcKQ6kVBJy5XRW JRVEii0sNwrUiJFThKNYkgtkVBy8xyjzGsAxjQwFToNpNDRBKdJpqpP6aX6VHf7Xtj9j bMdnSNOnF4qnY82i8oRHOLoROIYOKkJmWc/5XEMixb4NXxMuNHm4tWANeVAYBOrbisbg C3yme9dDRx490tZRiLF47J0e9Ym+JwFOGtkbgHy/5OjAMvyHaEurDBUr0iRCB4ClpGos 0tcg== X-Gm-Message-State: APjAAAWHW25IQAXj3fIRsOMJV7btGQ+A7Lm+ZCwKYiEVgU7dT2uCYm5X baNcGTewz5kvkEA4K5YxWXRxxuM5 X-Google-Smtp-Source: APXvYqwoAFt/v8xHxScYKAFFQAyMwwa51YTJXoxkx0jjZ38Wfr4Civk4cFbGfpJNdmMUCcqGUDOLPA== X-Received: by 2002:adf:ea42:: with SMTP id j2mr991717wrn.139.1582029174682; Tue, 18 Feb 2020 04:32:54 -0800 (PST) Received: from pfiuh02.univ-paris-diderot.fr ([193.48.40.241]) by smtp.gmail.com with ESMTPSA id b21sm3281258wmd.37.2020.02.18.04.32.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Feb 2020 04:32:54 -0800 (PST) From: zimoun Date: Tue, 18 Feb 2020 13:32:45 +0100 Message-Id: <20200218123246.32473-1-zimon.toutoune@gmail.com> X-Mailer: git-send-email 2.25.0 MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.51.188.43 X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches * website/apps/packages/builder.scm (origin->json): New procedure. --- website/apps/packages/builder.scm | 34 ++++++++++++++++++++++--------- 1 file changed, 24 insertions(+), 10 deletions(-) diff --git a/website/apps/packages/builder.scm b/website/apps/packages/builder.scm index 9dc44c9..d3a777e 100644 --- a/website/apps/packages/builder.scm +++ b/website/apps/packages/builder.scm @@ -2,6 +2,7 @@ ;;; Copyright © 2017 Ludovic Courtès ;;; Copyright © 2019 Ricardo Wurmus ;;; Copyright © 2019 Nicolò Balzarotti +;;; Copyright © 2020 Simon Tournier ;;; ;;; Initially written by sirgazil ;;; who waives all copyright interest on this file. @@ -37,13 +38,16 @@ #:use-module (haunt page) #:use-module (haunt utils) #:use-module (srfi srfi-1) + #:use-module (srfi srfi-26) #:use-module (guix packages) #:use-module (guix download) #:use-module (guix git-download) #:use-module (guix svn-download) #:use-module (guix utils) ;location + #:use-module ((guix build download) #:select (maybe-expand-mirrors)) #:use-module (json) #:use-module (ice-9 match) + #:use-module ((web uri) #:select (string->uri uri->string)) #:export (builder)) @@ -84,33 +88,43 @@ ;; Maximum number of packages shown on /packages. 30) -(define (packages-json-builder) - "Return a JSON page listing all packages." - (define (origin->json origin) +(define (origin->json origin) (define method (origin-method origin)) + (define uri ;represented as string + (origin-uri origin)) + + (define (resolve urls) + (map uri->string + (append-map (cut maybe-expand-mirrors <> %mirrors) + (map string->uri urls)))) + `((type . ,(cond ((eq? url-fetch method) 'url) ((eq? git-fetch method) 'git) ((eq? svn-fetch method) 'svn) (else #nil))) ,@(cond ((eq? url-fetch method) - `(("url" . ,(match (origin-uri origin) - ((? string? url) (vector url)) - ((urls ...) (list->vector urls)))))) + `(("url" . ,(list->vector + (resolve + (match uri + ((? string? url) (list url)) + ((urls ...) urls))))))) ((eq? git-fetch method) - `(("git_url" . ,(git-reference-url (origin-uri origin))))) + `(("git_url" . ,(git-reference-url uri)))) ((eq? svn-fetch method) - `(("svn_url" . ,(svn-reference-url (origin-uri origin))))) + `(("svn_url" . ,(svn-reference-url uri)))) (else '())) ,@(if (eq? method git-fetch) - `(("git_ref" . ,(git-reference-commit (origin-uri origin)))) + `(("git_ref" . ,(git-reference-commit uri))) '()) ,@(if (eq? method svn-fetch) `(("svn_revision" . ,(svn-reference-revision - (origin-uri origin)))) + uri))) '()))) +(define (packages-json-builder) + "Return a JSON page listing all packages." (define (package->json package) (define cpe-name (assoc-ref (package-properties package) 'cpe-name)) From patchwork Tue Feb 18 12:32:46 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Tournier X-Patchwork-Id: 20309 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 2F44E27BBEA; Tue, 18 Feb 2020 12:35:01 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,T_DKIM_INVALID, URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTP id BD72327BBE4 for ; Tue, 18 Feb 2020 12:35:00 +0000 (GMT) Received: from localhost ([::1]:33642 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1j425c-0007rl-BS for patchwork@mira.cbaines.net; Tue, 18 Feb 2020 07:35:00 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:37720) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1j424i-0007YY-0R for guix-patches@gnu.org; Tue, 18 Feb 2020 07:34:05 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1j424g-0004Xu-TG for guix-patches@gnu.org; Tue, 18 Feb 2020 07:34:03 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:33650) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1j424g-0004Xh-PT for guix-patches@gnu.org; Tue, 18 Feb 2020 07:34:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1j424g-0006Tr-NZ for guix-patches@gnu.org; Tue, 18 Feb 2020 07:34:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#39547] [PATCH v2 2/2] website: Provide JSON sources list used by Software Heritage. Resent-From: zimoun Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Tue, 18 Feb 2020 12:34:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 39547 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 39547@debbugs.gnu.org Cc: zimoun Received: via spool by 39547-submit@debbugs.gnu.org id=B39547.158202918924844 (code B ref 39547); Tue, 18 Feb 2020 12:34:02 +0000 Received: (at 39547) by debbugs.gnu.org; 18 Feb 2020 12:33:09 +0000 Received: from localhost ([127.0.0.1]:39622 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1j423o-0006Sd-WD for submit@debbugs.gnu.org; Tue, 18 Feb 2020 07:33:09 -0500 Received: from mail-wr1-f54.google.com ([209.85.221.54]:45938) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1j423m-0006S2-U5 for 39547@debbugs.gnu.org; Tue, 18 Feb 2020 07:33:07 -0500 Received: by mail-wr1-f54.google.com with SMTP id g3so23678964wrs.12 for <39547@debbugs.gnu.org>; Tue, 18 Feb 2020 04:33:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=2zgWveQl2Gffl3h7QHE+zVyAFsJc309pFQQOjmOQdZo=; b=sPe9dWajkDVeE6kJmFhCJ+CyyNlMchO8En4/4BWiWZAV4Q9Mhn8M5xcvioo+KzrUve 2Cj1c/IR2tOifKwSlb2SN2lNf42jdhlEHxHkv76DQesFiRYow3sNApvZ/lZG5bwQSMuQ lhmVq8uaAj08teAeMl5KRUY5YvZsegp0H3XlLWMfoku61+QS189X1cSv9ej7YXIIxEZr hM6YrWmO9oH1tHZpRtV5TS5XlD77nOgPA5jywQt0k/Fm0pwDE1Ryfot+PI+4IRq7m2gG XQ0bUYYwZeFFtuHLrlUUfq+mleyXDmSgIiP/AgJw+v3GC3c7FYAb2ouJyZD26XVF7aLJ idoQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=2zgWveQl2Gffl3h7QHE+zVyAFsJc309pFQQOjmOQdZo=; b=hvG6zadhtpycGJGp3kWQ3K/F9pic2uh6TKhSDs2WtiX5TtbmO3jrOCOoxQSPkpB2xR L8qUhN4eXAhWMAvAnuqy9FvjVbz9vjdVFRMP2dpaWrzY5jz4dZMXiYuuekFpFZpjj6ib obXWNWn+6DmYfB9h4fXCNgoGqUgguKyHjOA4vnWF3jS0IMQPgSFqR+rkA/wHgO4+QIre GPCFi2uFWIszpbf/KzerDTWa2Ui/wqn+MlG9UFDhuc5oq9qd9FoXezllu6JnXn57/eNm 9Lq+b6vO4Xf740ykkLbcHeplgTLEX5aRiN9YbWHQout4gHwPsGv1Td9C4qEu8rokLpC2 kqLw== X-Gm-Message-State: APjAAAWN/xCj09GC7/4nTA2eHw8zj5iODdM9GuHta9K16/ePRquEP3Eb rLlLmuxf53IB/2VHMGVcSxjlADb+ X-Google-Smtp-Source: APXvYqzhVafkKIlY7BwXpV6ncY8ulgwemsrqfqgd8H0wRgu7jhiyQQ1gspGJsjEzH26beppEuhSBuQ== X-Received: by 2002:adf:db48:: with SMTP id f8mr27267999wrj.146.1582029181140; Tue, 18 Feb 2020 04:33:01 -0800 (PST) Received: from pfiuh02.univ-paris-diderot.fr ([193.48.40.241]) by smtp.gmail.com with ESMTPSA id b21sm3281258wmd.37.2020.02.18.04.33.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Feb 2020 04:33:00 -0800 (PST) From: zimoun Date: Tue, 18 Feb 2020 13:32:46 +0100 Message-Id: <20200218123246.32473-2-zimon.toutoune@gmail.com> X-Mailer: git-send-email 2.25.0 In-Reply-To: <20200218123246.32473-1-zimon.toutoune@gmail.com> References: <20200218123246.32473-1-zimon.toutoune@gmail.com> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.51.188.43 X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches Format discussed here . * website/apps/packages/builder.scm (origin->json): Add list modifier. * website/apps/packages/builder.scm (sources-json-builder): New procedure. --- website/apps/packages/builder.scm | 22 +++++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-) diff --git a/website/apps/packages/builder.scm b/website/apps/packages/builder.scm index d3a777e..49721f6 100644 --- a/website/apps/packages/builder.scm +++ b/website/apps/packages/builder.scm @@ -74,6 +74,7 @@ (flatten (list (index-builder) + (sources-json-builder) (packages-json-builder) (packages-builder) (package-list-builder)))) @@ -88,7 +89,7 @@ ;; Maximum number of packages shown on /packages. 30) -(define (origin->json origin) +(define (origin->json origin transformer) (define method (origin-method origin)) @@ -105,7 +106,7 @@ ((eq? svn-fetch method) 'svn) (else #nil))) ,@(cond ((eq? url-fetch method) - `(("url" . ,(list->vector + `(("url" . ,(transformer (resolve (match uri ((? string? url) (list url)) @@ -136,7 +137,7 @@ ,@(if cpe-name `(("cpe_name" . ,cpe-name)) '()) ,@(if cpe-version `(("cpe_version" . ,cpe-version)) '()) ,@(if (origin? (package-source package)) - `(("source" . ,(origin->json (package-source package)))) + `(("source" . ,(origin->json (package-source package) list->vector))) '()) ("synopsis" . ,(package-synopsis package)) ,@(if (package-home-page package) @@ -155,6 +156,21 @@ (list->vector (map package->json (all-packages))) scm->json)) +(define (sources-json-builder) + "Return a JSON page listing all the sources. + +See ." + (define (package->json package) + `(,@(if (origin? (package-source package)) + (origin->json (package-source package) car) + `(("type" . "no-origin") + ("name" . ,(package-name package)))))) + + (make-page "sources.json" + `(("sources" . ,(list->vector (map package->json (all-packages)))) + ("version" . "1")) + scm->json)) + (define (index-builder) "Return a Haunt page listing some random packages." (define (sample n from)