When I stop all services on my foreman server, most of the time the foreman.service will automatically start again even while the service stop is still running.
# foreman-maintain service stop
Journal for foreman.service:
Jun 07 10:40:06 foreman.example.com foreman:  - Gracefully shutting down workers...
Jun 07 10:40:06 foreman.example.com foreman: Exiting
Jun 07 10:40:06 foreman.example.com systemd: Stopped Foreman.
Jun 07 10:40:12 foreman.example.com systemd: Starting Foreman...
Maybe that’s due to this message which pops up occasionally during stop:
Warning: Stopping foreman.service, but it can still be activated by:
foreman.service should be stopped and remain stopped.
Foreman and Proxy versions:
Foreman 2.3.4, Katello 3.18.2
Distribution and version:
@upadhyeammit would this be possible to add to the
foreman-maintain? I think it’s reasonable expectation. TBH even when I do
systemctl stop foreman I would prefer it really stops Foreman and won’t allow it to start if something hits its url. I guess there’s some reason for it. Perhaps @ekohl or @ehelms know.
I thought that was fixed as part of this:
I wonder if it’s a bug and a race condition where the service is started again (due to socket activation) between stopping
foreman.socket. AFAIK that shouldn’t happen if you run
systemctl stop foreman.service foreman.socket but I don’t know if foreman-maintain combines the two or calls
systemctl stop twice.
Well, if I look at the journal log for foreman.socket I don’t see any attempt to stop the socket:
# journalctl -u foreman.socket
May 26 09:26:55 foreman.example.com systemd: Listening on Foreman HTTP Server Accept Sockets.
May 28 11:26:46 foreman.example.com systemd: Closed Foreman HTTP Server Accept Sockets.
-- Reboot --
May 28 11:28:23 foreman.example.com systemd: Listening on Foreman HTTP Server Accept Sockets.
Jun 01 16:03:57 foreman.example.com systemd: Closed Foreman HTTP Server Accept Sockets.
-- Reboot --
Jun 01 16:04:53 foreman.example.com systemd: Listening on Foreman HTTP Server Accept Sockets.
Jun 03 16:03:45 foreman.example.com systemd: Closed Foreman HTTP Server Accept Sockets.
-- Reboot --
Jun 06 13:23:56 foreman.example.com systemd: Listening on Foreman HTTP Server Accept Sockets.
My last stop was today, June 7th, 10:40. No message logged there.
If I run a manual
# systemctl stop foreman.socket
I’ll get a
Closed Foreman HTTP Server Accept Sockets in the journal.
So according to the journal, the socket is never stopped.
There is also no stop command in /var/log/foreman-maintain/foreman-maintain.log, only for foreman.service, not for foreman.socket. Maybe because it’s set to disabled?
# systemctl status foreman.socket
● foreman.socket - Foreman HTTP Server Accept Sockets
Loaded: loaded (/usr/lib/systemd/system/foreman.socket; disabled; vendor preset: disabled)
Active: active (running) since Sun 2021-06-06 13:23:56 CEST; 23h ago
Listen: 127.0.0.1:3000 (Stream)
That’s quite possible. I also see it’s still listening on
127.0.0.1:3000 rather than
/run/foreman.sock. This was a change we did in the installer (and started to explicitly manage the service) in Foreman 2.4 (so Katello 4.0). That could very well be the root cause of the issue here.
If you use
systemctl enable foreman.socket, does
foreman-maintain try to stop the socket as well?
I have just tested it once, and yes: if I enable foreman.socket foreman-maintain will stop it. A grep fore ‘stop foreman’ in /var/log/foreman-maintain/foreman-maintain.log:
D, [2021-06-07 10:40:03+0200 #47657] DEBUG -- : Running command systemctl stop foreman-proxy with stdin nil
D, [2021-06-07 10:40:06+0200 #47657] DEBUG -- : Running command systemctl stop foreman with stdin nil
D, [2021-06-07 13:45:31+0200 #25064] DEBUG -- : Running command systemctl stop foreman-proxy with stdin nil
D, [2021-06-07 13:45:33+0200 #25064] DEBUG -- : Running command systemctl stop foreman with stdin nil
D, [2021-06-07 13:45:33+0200 #25064] DEBUG -- : Running command systemctl stop foreman.socket with stdin nil
It stops the service first, though, thus there may be a race condition. If you stop foreman.socket it also stops the service as the service requires foreman.socket. Thus stopping foreman.socket should be enough to stop both and without any race condition.
FYI, the change to use a unix socket is:
And since this commit disabled services are no longer managed:
So I do believe this is fixed for 4.0 and we’re unlikely to fix it in Foreman 2.3. Now that Foreman 2.5 is out, 2.3 won’t really receive more updates (there will be a last one for CentOS 8.4 compatibility).
So the manual
systemctl enable foreman.socket workaround is for now the only thing we have.
As for running some systemd commands concurrently, that’s something I’ve been hoping foreman-maintain would do for a long time. I thought this commit would at least partially do it: