When I stop all services on my foreman server, most of the time the foreman.service will automatically start again even while the service stop is still running.
# foreman-maintain service stop
Journal for foreman.service:
Jun 07 10:40:06 foreman.example.com foreman[1287]: [1287] - Gracefully shutting down workers...
Jun 07 10:40:06 foreman.example.com foreman[1287]: Exiting
Jun 07 10:40:06 foreman.example.com systemd[1]: Stopped Foreman.
Jun 07 10:40:12 foreman.example.com systemd[1]: Starting Foreman...
Maybe that’s due to this message which pops up occasionally during stop:
Warning: Stopping foreman.service, but it can still be activated by:
foreman.socket
Expected outcome:
foreman.service should be stopped and remain stopped.
@upadhyeammit would this be possible to add to the foreman-maintain? I think it’s reasonable expectation. TBH even when I do systemctl stop foreman I would prefer it really stops Foreman and won’t allow it to start if something hits its url. I guess there’s some reason for it. Perhaps @ekohl or @ehelms know.
I wonder if it’s a bug and a race condition where the service is started again (due to socket activation) between stopping foreman.service and foreman.socket. AFAIK that shouldn’t happen if you run systemctl stop foreman.service foreman.socket but I don’t know if foreman-maintain combines the two or calls systemctl stop twice.
There is also no stop command in /var/log/foreman-maintain/foreman-maintain.log, only for foreman.service, not for foreman.socket. Maybe because it’s set to disabled?
# systemctl status foreman.socket
● foreman.socket - Foreman HTTP Server Accept Sockets
Loaded: loaded (/usr/lib/systemd/system/foreman.socket; disabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/foreman.socket.d
└─installer.conf
Active: active (running) since Sun 2021-06-06 13:23:56 CEST; 23h ago
Listen: 127.0.0.1:3000 (Stream)
That’s quite possible. I also see it’s still listening on 127.0.0.1:3000 rather than /run/foreman.sock. This was a change we did in the installer (and started to explicitly manage the service) in Foreman 2.4 (so Katello 4.0). That could very well be the root cause of the issue here.
If you use systemctl enable foreman.socket, does foreman-maintain try to stop the socket as well?
I have just tested it once, and yes: if I enable foreman.socket foreman-maintain will stop it. A grep fore ‘stop foreman’ in /var/log/foreman-maintain/foreman-maintain.log:
It stops the service first, though, thus there may be a race condition. If you stop foreman.socket it also stops the service as the service requires foreman.socket. Thus stopping foreman.socket should be enough to stop both and without any race condition.
And since this commit disabled services are no longer managed:
So I do believe this is fixed for 4.0 and we’re unlikely to fix it in Foreman 2.3. Now that Foreman 2.5 is out, 2.3 won’t really receive more updates (there will be a last one for CentOS 8.4 compatibility).
So the manual systemctl enable foreman.socket workaround is for now the only thing we have.
As for running some systemd commands concurrently, that’s something I’ve been hoping foreman-maintain would do for a long time. I thought this commit would at least partially do it:
