From patchwork Thu Apr 14 12:18:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Guillaume Le Vaillant X-Patchwork-Id: 38586 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id A4FEA27BBEA; Thu, 14 Apr 2022 13:31:34 +0100 (BST) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.7 required=5.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,SPF_HELO_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 06BFE27BBE9 for ; Thu, 14 Apr 2022 13:31:34 +0100 (BST) Received: from localhost ([::1]:58952 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1neydI-0006Sz-VM for patchwork@mira.cbaines.net; Thu, 14 Apr 2022 08:31:33 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:46324) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1neyco-0006RU-Qm for guix-patches@gnu.org; Thu, 14 Apr 2022 08:31:03 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:33422) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1neyco-0006tq-I9 for guix-patches@gnu.org; Thu, 14 Apr 2022 08:31:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1neyco-0001cf-EI for guix-patches@gnu.org; Thu, 14 Apr 2022 08:31:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#54723] [PATCH] Check URI when verifying narinfo validity. Resent-From: Guillaume Le Vaillant Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 14 Apr 2022 12:31:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 54723 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Ludovic =?utf-8?q?Court=C3=A8s?= Cc: Simon Streit , 54723@debbugs.gnu.org, Maxim Cournoyer Received: via spool by 54723-submit@debbugs.gnu.org id=B54723.16499394146179 (code B ref 54723); Thu, 14 Apr 2022 12:31:02 +0000 Received: (at 54723) by debbugs.gnu.org; 14 Apr 2022 12:30:14 +0000 Received: from localhost ([127.0.0.1]:55552 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1neyc1-0001bb-Qg for submit@debbugs.gnu.org; Thu, 14 Apr 2022 08:30:14 -0400 Received: from mout01.posteo.de ([185.67.36.65]:56173) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1neybz-0001aD-5U for 54723@debbugs.gnu.org; Thu, 14 Apr 2022 08:30:12 -0400 Received: from submission (posteo.de [185.67.36.169]) by mout01.posteo.de (Postfix) with ESMTPS id E68DC240028 for <54723@debbugs.gnu.org>; Thu, 14 Apr 2022 14:30:04 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017; t=1649939404; bh=ISuXq+Qj+Xx8yHd10e00hZulWYB/o1MFjTzZUlh/f50=; h=From:To:Cc:Subject:Date:From; b=l2y323fCU1TqALOslMr3QIEiPUP+jrFOSZxgcfOv9a/fHsNxPwfbRVUCylY4rIJqo 4Q8Gsn7rjvfcHwUZeaObcGrWA3QBXzEWubBI0yfRFkemGokmo9T5xL37BrHq6ByO24 X2wTfMLiv8QM9EVXZrdMcvUw9MyhaSITV4agryl/DopG52YVMM3Q67y8YAh8gXckva sSfAwVJKv0K+ZTI6JFoP39/hiBSqx2ogCO/W3IuWSpx7OY3FDU1SrVZLGFUq5m1hMD puIFaR6tWbhcp+mA1HuK6R35nP8OU6uZcapAzUdphIy/aWBS0R5yMNzYHo6pQdI2OC zmO/Mv55W+9pg== Received: from customer (localhost [127.0.0.1]) by submission (posteo.de) with ESMTPSA id 4KfJgt39MYz9rxW; Thu, 14 Apr 2022 14:30:02 +0200 (CEST) References: <87a6czbzvh.fsf@kitej> <877d83lapv.fsf@gnu.org> <875ynnbe65.fsf@kitej> <87pmlq6lqx.fsf_-_@gnu.org> <877d7ydjwk.fsf@kitej> <87wnfv90cl.fsf@kitej> <871qy2g2v3.fsf@gnu.org> <87o816wt36.fsf@kitej> From: Guillaume Le Vaillant Date: Thu, 14 Apr 2022 12:18:51 +0000 In-reply-to: <87o816wt36.fsf@kitej> Message-ID: <87sfqfc0fq.fsf@kitej> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches Guillaume Le Vaillant skribis: > Ludovic Courtès skribis: > >> Woow. How do you build and run ‘guix publish’? Is it a distro package >> or is it coming straight from Guix? What command-line options are you >> passing? >> >> I’ve never seen this, although we have it running on several servers, >> notably ci.guix. I wonder what could cause this. >> >> Thanks, >> Ludo’. > > I'm using guix-publish-service-type in the operating-system definition > to manage the "guix publish" server, using the on-the-fly mode and > fast Zstandard compression: > > (service guix-publish-service-type > (guix-publish-configuration > (host "0.0.0.0") > (port 8080) > (compression '(("zstd" 3))) > (advertise? #t))) > > > When booting the machine, shepherd starts the server with the following > command-line options: > > /gnu/store/059svbd32i4s0l9s5i7z0krcnl666bjy-guix-1.3.0-24.2fb4304/libexec/guix/guile \ /gnu/store/059svbd32i4s0l9s5i7z0krcnl666bjy-guix-1.3.0-24.2fb4304/bin/guix publish -u guix-publish -p 8080 -C zstd:3 --nar-path=nar --listen=0.0.0.0 --advertise > > There's another report about this at > I had forgotten about, where Simon Streit and Maxim Cournoyer indicate > that they have seen this issue too. I tested a few things, and it looks like there is a concurrency issue in the 'http-write' function of the publish server. After removing the 'call-with-new-thread' in the part of the function sending narinfos, I can't reproduce the "broken pipe" error anymore. --8<---------------cut here---------------start------------->8--- --8<---------------cut here---------------end--------------->8--- diff --git a/guix/scripts/publish.scm b/guix/scripts/publish.scm index 870dfc11e9..e9fad5845a 100644 --- a/guix/scripts/publish.scm +++ b/guix/scripts/publish.scm @@ -1002,28 +1002,25 @@ (define compression (begin (when (keep-alive? response) (keep-alive client)) - (call-with-new-thread - (lambda () - (set-thread-name "publish narinfo") - (let* ((narinfo - (with-store store - (narinfo-string store (utf8->string body) - #:nar-path nar-path - #:compressions compressions))) - (narinfo-bv (string->bytevector narinfo "UTF-8")) - (narinfo-length - (bytevector-length narinfo-bv)) - (response (write-response - (with-content-length response - narinfo-length) - client)) - (output (response-port response))) - (configure-socket client) - (put-bytevector output narinfo-bv) - (force-output output) - (unless (keep-alive? response) - (close-port output)) - (values))))) + (let* ((narinfo + (with-store store + (narinfo-string store (utf8->string body) + #:nar-path nar-path + #:compressions compressions))) + (narinfo-bv (string->bytevector narinfo "UTF-8")) + (narinfo-length + (bytevector-length narinfo-bv)) + (response (write-response + (with-content-length response + narinfo-length) + client)) + (output (response-port response))) + (configure-socket client) + (put-bytevector output narinfo-bv) + (force-output output) + (unless (keep-alive? response) + (close-port output)) + (values))) (%http-write server client response body)))) (_ (match (assoc-ref (response-headers response) 'x-raw-file)