Message ID | 87zgsbuss7.fsf@gnu.org |
---|---|
State | New |
Headers | show |
Hey, The guix-publish daemon has been updated on berlin to include this fix: 222f4661ed11b225f458cbe495a296f233129bec. Sadly, it looks like it doesn't improve the situation. I observed that the builds of the following evaluation failed because of the typical: "cannot build missing derivation ?/gnu/store/4p6hgfkj11q5mqk39lb3kxczwqswzh97-iso-image-installer.drv?". https://ci.guix.gnu.org/eval/27520 When looking at the Nginx logs, we have those concomitant errors: --8<---------------cut here---------------start------------->8--- 021/10/05 10:52:44 [info] 129925#0: *19790054 client 94.237.57.97 closed keepalive connection 2021/10/05 10:52:44 [error] 129925#0: *19787848 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 141.80.167.181, server: ci.guix.gnu.org, request: "GET /bhjk59akv02qibag0pxx6cjlrlgs1gjm.narinfo HTTP/1.1", upstream: "http://127.0.0. 1:3000/bhjk59akv02qibag0pxx6cjlrlgs1gjm.narinfo", host: "141.80.167.131" 2021/10/05 10:52:44 [error] 129925#0: *19788215 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 141.80.167.183, server: ci.guix.gnu.org, request: "GET /g243k85y7z7j6dq7nx74gp3d28hhad71.narinfo HTTP/1.1", upstream: "http://127.0.0. 1:3000/g243k85y7z7j6dq7nx74gp3d28hhad71.narinfo", host: "141.80.167.131" 2021/10/05 10:52:44 [error] 129925#0: *19787890 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 141.80.167.160, server: ci.guix.gnu.org, request: "GET /gxp6fhj7mszs90b24k630wprrkwsm5vb.narinfo HTTP/1.1", upstream: "http://127.0.0. 1:3000/gxp6fhj7mszs90b24k630wprrkwsm5vb.narinfo", host: "141.80.167.131" 2021/10/05 10:52:44 [error] 129925#0: *19788116 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 141.80.167.163, server: ci.guix.gnu.org, request: "GET /hyjiw3cnvsn2k5mpfafmczy68jmpb71x.narinfo HTTP/1.1", upstream: "http://127.0.0. 1:3000/hyjiw3cnvsn2k5mpfafmczy68jmpb71x.narinfo", host: "141.80.167.131" 2021/10/05 10:52:44 [error] 129925#0: *19788032 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 141.80.167.167, server: ci.guix.gnu.org, request: "GET /ngnnppl9ncixxjid0crfhwjn4h0xyg8c.narinfo HTTP/1.1", upstream: "http://127.0.0. 1:3000/ngnnppl9ncixxjid0crfhwjn4h0xyg8c.narinfo", host: "141.80.167.131" 2021/10/05 10:52:44 [error] 129925#0: *19787887 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 141.80.167.181, server: ci.guix.gnu.org, request: "GET /lvpmiq4p8wqyxi34q99qm7d0xfwq484a.narinfo HTTP/1.1", upstream: "http://127.0.0. 1:3000/lvpmiq4p8wqyxi34q99qm7d0xfwq484a.narinfo", host: "141.80.167.131" 2021/10/05 10:52:44 [error] 129925#0: *19788074 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 141.80.167.186, server: ci.guix.gnu.org, request: "GET /wxv8a2845d5zxq0yaxd57arjvdvhppib.narinfo HTTP/1.1", upstream: "http://127.0.0. 1:3000/wxv8a2845d5zxq0yaxd57arjvdvhppib.narinfo", host: "141.80.167.131" 2021/10/05 10:52:44 [error] 129925#0: *19787681 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 141.80.167.165, server: ci.guix.gnu.org, request: "GET /8kmc1fznb29rmbjiyvkbkixhzm2287al.narinfo HTTP/1.1", upstream: "http://127.0.0. 1:3000/8kmc1fznb29rmbjiyvkbkixhzm2287al.narinfo", host: "141.80.167.131" 2021/10/05 10:52:44 [error] 129925#0: *19787714 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 141.80.167.180, server: ci.guix.gnu.org, request: "GET /795mhps0sdli2ndvcv1aq54b7fg59d97.narinfo HTTP/1.1", upstream: "http://127.0.0. 1:3000/795mhps0sdli2ndvcv1aq54b7fg59d97.narinfo", host: "141.80.167.131" 2021/10/05 10:52:44 [error] 129925#0: *19788327 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 141.80.167.165, server: ci.guix.gnu.org, request: "GET /zsv7zg4xxr2hlirsbifka5y62394zzl8.narinfo HTTP/1.1", upstream: "http://127.0.0. 1:3000/zsv7zg4xxr2hlirsbifka5y62394zzl8.narinfo", host: "141.80.167.131" 2021/10/05 10:52:44 [error] 129925#0: *19787809 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 141.80.167.169, server: ci.guix.gnu.org, request: "GET /a8npwin71gbn9fprn18i5j26cl91l3rn.narinfo HTTP/1.1", upstream: "http://127.0.0. --8<---------------cut here---------------end--------------->8--- Thanks, Mathieu
diff --git a/hydra/nginx/berlin.scm b/hydra/nginx/berlin.scm index 44ff28e..7a085e5 100644 --- a/hydra/nginx/berlin.scm +++ b/hydra/nginx/berlin.scm @@ -97,9 +97,9 @@ ;; Do not tolerate slowness of hydra.gnu.org when fetching ;; narinfos: better return 504 quickly than wait forever. - "proxy_connect_timeout 2s;" - "proxy_read_timeout 2s;" - "proxy_send_timeout 2s;" + "proxy_connect_timeout 10s;" + "proxy_read_timeout 10s;" + "proxy_send_timeout 10s;" --8<---------------cut here---------------end--------------->8--- I'll see if I can reproduce the I/O pressure on demand, to be able to have more accurate strace/perf investigations.