diff mbox series

[bug#76811] services: nginx: Replace invoke with spawn-command.

Message ID e944b76a55fdad935058ac12f2fd19ecfc17ebd5.1741352535.git.mail@cbaines.net
State New
Headers
Series [bug#76811] services: nginx: Replace invoke with spawn-command. |

Commit Message

Christopher Baines March 7, 2025, 1:02 p.m. UTC 
  I'm not sure where invoke is coming from here, but it could be from (guix
build utils), that uses system* which uses waitpid, which might cause problems
with recent versions of the shepherd?

At least I'm seeing issues on multiple machines where attempting to restart
the nginx service sometimes causes the shepherd to hang.

* gnu/services/web.scm (nginx-shepherd-service): Replace invoke with
spawn-command.

Change-Id: Ie9ce4be9a4df121465b28148612b4fbc45fb5126
---
 gnu/services/web.scm | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)


base-commit: 9bc4c9f521caab8aa8d88aa948a650945bb55838

Comments

Ludovic Courtès March 7, 2025, 1:56 p.m. UTC | #1 
Christopher Baines <mail@cbaines.net> skribis:

> I'm not sure where invoke is coming from here, but it could be from (guix
> build utils), that uses system* which uses waitpid, which might cause problems
> with recent versions of the shepherd?
>
> At least I'm seeing issues on multiple machines where attempting to restart
> the nginx service sometimes causes the shepherd to hang.
>
> * gnu/services/web.scm (nginx-shepherd-service): Replace invoke with
> spawn-command.
>
> Change-Id: Ie9ce4be9a4df121465b28148612b4fbc45fb5126

Hi!  ‘invoke’ uses ‘system*’, which is an alias for ‘spawn-command’ (see
‘replace-core-bindings!’ in ‘shepherd.scm’) so the only effect of this
patch is that errors from “nginx -c nginx.conf …” would be ignored.

I think we need a reproducer for the hang so we can pinpoint the
problem because it’s a pretty serious bug!

Ludo’.
Christopher Baines March 7, 2025, 5:18 p.m. UTC | #2 
Ludovic Courtès <ludo@gnu.org> writes:

> Christopher Baines <mail@cbaines.net> skribis:
>
>> I'm not sure where invoke is coming from here, but it could be from (guix
>> build utils), that uses system* which uses waitpid, which might cause problems
>> with recent versions of the shepherd?
>>
>> At least I'm seeing issues on multiple machines where attempting to restart
>> the nginx service sometimes causes the shepherd to hang.
>>
>> * gnu/services/web.scm (nginx-shepherd-service): Replace invoke with
>> spawn-command.
>>
>> Change-Id: Ie9ce4be9a4df121465b28148612b4fbc45fb5126
>
> Hi!  ‘invoke’ uses ‘system*’, which is an alias for ‘spawn-command’ (see
> ‘replace-core-bindings!’ in ‘shepherd.scm’) so the only effect of this
> patch is that errors from “nginx -c nginx.conf …” would be ignored.

Ah, yes, I see, I've tried to verify this and it does seem that the
nginx server is using this system* replacement.

> I think we need a reproducer for the hang so we can pinpoint the
> problem because it’s a pretty serious bug!

I did try restarting nginx over and over again in the system test os,
but that seemed to work.

On a VM I have though, it only takes a few restarts for it to hang, I'm
not sure why though.
Ludovic Courtès March 7, 2025, 11:08 p.m. UTC | #3 
Hey,

> I did try restarting nginx over and over again in the system test os,
> but that seemed to work.
>
> On a VM I have though, it only takes a few restarts for it to hang, I'm
> not sure why though.

It could be that nginx alone would work well, but combining it with
another service that does unusual things triggers the bug.

One way to investigate would be to start from ‘bayfront.scm’ (which I
think has that problem) and boil it down until we have something that
can run in a VM and reproduces the problem.

I don’t see any ‘waitpid’ uses left in Shepherd services under
gnu/services/*.scm.

One thing I found that is risky is ‘guix-data-service-setup-database’:
it loads a bunch of (guix-data-service …) modules into PID 1 and runs
non-trivial code in there; I strongly recommend doing this in a separate
process, similar to how ‘bffe-shepherd-services’ does it.

Ludo’.
Ludovic Courtès March 10, 2025, 10:04 p.m. UTC | #4 
Ludovic Courtès <ludo@gnu.org> skribis:

> One thing I found that is risky is ‘guix-data-service-setup-database’:
> it loads a bunch of (guix-data-service …) modules into PID 1 and runs
> non-trivial code in there; I strongly recommend doing this in a separate
> process, similar to how ‘bffe-shepherd-services’ does it.

The guix-data-service tests shows that:

  https://ci.guix.gnu.org/build/9557216/log

Namely:

--8<---------------cut here---------------start------------->8---
[    5.788985] shepherd[1]: Service loopback started.
[    5.790001] shepherd[1]: Service loopback running with value #t.
Uncaught exception in task:
In fibers.scm:
    172:8  6 (_)
In shepherd/service/system-log.scm:
   180:10  5 (run-system-log #<<channel> getq: #<atomic-box 7fcc8e8?> ?)
In srfi/srfi-1.scm:
   586:17  4 (map1 (#<input-output: socket 17> #<input: /proc/kmsg?>))
In shepherd/service/system-log.scm:
   181:33  3 (_ #<input-output: socket 17>)
In fibers/io-wakeup.scm:
    72:13  2 (make-wait-operation #<procedure 7fcc8e8b0300 at fiber?> ?)
    72:13  1 (make-wait-operation #f #<procedure 7fcc8e8b0300 at fi?> ?)
In ice-9/boot-9.scm:
  1685:16  0 (raise-exception _ #:continuable? _)

ice-9/boot-9.scm:1685:16: In procedure raise-exception:
Wrong type to apply: #<syntax-transformer make-base-operation>
--8<---------------cut here---------------end--------------->8---

Here bindings in (fibers io-wakeup) are likely “polluted” by loading
guile-fibers-next via the (guix-data-service …) modules.

Ludo’.
diff mbox series

Patch

diff --git a/gnu/services/web.scm b/gnu/services/web.scm
index 7593cd2eaa..b46a4db73f 100644
--- a/gnu/services/web.scm
+++ b/gnu/services/web.scm
@@ -870,7 +870,8 @@  (define (nginx-shepherd-service config)
           (nginx-action
            (lambda args
              #~(lambda _
-                 (invoke #$nginx-binary "-c" #$config-file #$@args)
+                 (spawn-command
+                  (list #$nginx-binary "-c" #$config-file #$@args))
                  (match '#$args
                    (("-s" . _) #t)
                    (_