I created a before_provision foreman hook python script. This python script fires off a long running job supposedly in the background and I don’t wish to wait for it to complete before moving on (it includes a few sleep commands). When I run this hook from the command line it runs as expected, the parent python script exits and the child process is running in the background. However, when it runs as part of the provisioning workflow, it waits on that child process finishing, I don’t understand why.
Expected outcome:
Child process runs in background and workflow continues without waiting for it to complete
Foreman and Proxy versions:
1.24
Distribution and version:
CentOS 7
Other relevant data:
Sanitized version of scripts:
The hook:
#!/usr/bin/python
import sys
import json
import tempfile
import requests
import subprocess
import os
import time
sys.path.append('/usr/share/foreman-community/hooks')
from functions import \
(HOOK_EVENT, HOOK_OBJECT, HOOK_TEMP_DIR, get_json_hook)
PREFIX = "created_by_hook-{}".format(sys.argv[0].split('/')[-1])
HOOK_JSON = get_json_hook()
# read the information received
if HOOK_JSON.get('host'):
hostname = HOOK_JSON.get('host').get('name', None)
subprocess.Popen(['/opt/child.py'],shell=False,stdin=None,stdout=None,stderr=None,close_fds=True)
sys.exit(0)
When you want to just spawn a subprocess, why do you use popen? I am not a Python expert, but I am pretty sure this language has spawn call which you can use with P_NOWAIT argument which should do the trick.
If you need those two processes to communicate via stdin/stdout, then you need both of them obviously. If you only need to send data, then make sure to close input/output, it can block.
Thanks for the response. I did try with both popen and spawn functions. Popen is considered the morer “proper” way these days (pretty much all python docs on spawn advise using the subprocess module instead).
It seems likely I just forgot to close stdin, thanks
Or not, ensured I closed all input/output, tried using popen, spawnl, system. Every case it waits for the child process to finish before moving on. I don’t understand why it behaves like this when running as a hook but has no issue when running from command line.
I don’t think foreman_hooks was designed as something async. It’s designed to run something synchronous. I wouldn’t be surprised if it waited for all children and didn’t properly support forking off new processes.
As for closing stdin/stdout/stderr, I see you are using =None in the example. That doesn’t work. I haven’t seen your updated code, but perhaps using /dev/null would be better?
Thanks for the tip, but it didn’t appear to change anything. I’m considering going route of writing a plugin instead, but my Ruby skills aren’t great.
@lzap, is there any rough timeline on foreman_webhooks you mentioned? Just trying to determine if it’s worth pursuing getting this worth working with foreman_hooks or not.
[Unit]
Description=Echo server service
[Service]
ExecStart=/path/to/echo.py
StandardInput=socket
And /path/to/echo.py
#!/usr/bin/python
from time import sleep
import sys
sys.stdout.write(sys.stdin.readline().strip().upper() + 'rn')
sys.stdin.close()
sleep(10)
# Do other work
Then running systemctl daemon-reload && systemctl enable --now echo.socket will start the socket and let it listen on /run/echo.sock. You should confirm it’s there with ls -l /run/echo.sock and that it’s owned by Foreman with mode 0600.
Now you can write to it:
echo test | socat - /run/echo.sock
You should note that echo.py sticks around for 10 seconds and you can launch multiple processes. This is fairly trivial job queuing, but probably sufficient for your use case. The original hook is closed fast and the actual work happens in the background.
Maybe a FIFO socket works better:
Note that example also writes the output to the journal rather than on the socket, which may be much better.
This post may not work entirely, but should at least guide you to a working solution without having to wait for and set up webhooks.
Also interesting is that you can run the socket under one user but the service as another. That can give you proper isolation and sandboxing.
Keep in mind if you have SELinux turned on (enforcing) foreman domain will not be allowed to connect to a socket. You need to add a rule to allow this.
