Foreman hooks, running script in background


I created a before_provision foreman hook python script. This python script fires off a long running job supposedly in the background and I don’t wish to wait for it to complete before moving on (it includes a few sleep commands). When I run this hook from the command line it runs as expected, the parent python script exits and the child process is running in the background. However, when it runs as part of the provisioning workflow, it waits on that child process finishing, I don’t understand why.

Expected outcome:

Child process runs in background and workflow continues without waiting for it to complete

Foreman and Proxy versions:


Distribution and version:

CentOS 7

Other relevant data:
Sanitized version of scripts:

The hook:

    import sys
    import json
    import tempfile
    import requests
    import subprocess
    import os
    import time


    from functions import \
      (HOOK_EVENT, HOOK_OBJECT, HOOK_TEMP_DIR, get_json_hook)

    PREFIX = "created_by_hook-{}".format(sys.argv[0].split('/')[-1])

    HOOK_JSON = get_json_hook()

    # read the information received
    if HOOK_JSON.get('host'):
        hostname = HOOK_JSON.get('host').get('name', None)



The child:

import sys
import json
import tempfile
import requests
import subprocess
import os
import time

hostname = sys.argv[1]

When you want to just spawn a subprocess, why do you use popen? I am not a Python expert, but I am pretty sure this language has spawn call which you can use with P_NOWAIT argument which should do the trick.

If you need those two processes to communicate via stdin/stdout, then you need both of them obviously. If you only need to send data, then make sure to close input/output, it can block.

Thanks for the response. I did try with both popen and spawn functions. Popen is considered the morer “proper” way these days (pretty much all python docs on spawn advise using the subprocess module instead).

It seems likely I just forgot to close stdin, thanks

Or not, ensured I closed all input/output, tried using popen, spawnl, system. Every case it waits for the child process to finish before moving on. I don’t understand why it behaves like this when running as a hook but has no issue when running from command line.

I have no idea, we are about to replace foreman_hooks with foreman_webhooks pretty soon.

I don’t think foreman_hooks was designed as something async. It’s designed to run something synchronous. I wouldn’t be surprised if it waited for all children and didn’t properly support forking off new processes.

As for closing stdin/stdout/stderr, I see you are using =None in the example. That doesn’t work. I haven’t seen your updated code, but perhaps using /dev/null would be better?

Thanks for the tip, but it didn’t appear to change anything. I’m considering going route of writing a plugin instead, but my Ruby skills aren’t great.

@lzap, is there any rough timeline on foreman_webhooks you mentioned? Just trying to determine if it’s worth pursuing getting this worth working with foreman_hooks or not.


Webhooks should be part of the next major release, 2.4.

If you only want to trigger something async you can consider running a systemd activated socket.

I’d recommend the first example. It means systemd opens a unix socket to listen on. Based on that:

You would create /etc/systemd/system/echo.socket

Description = Echo server

ListenStream = %t/echo.sock
Accept = yes

WantedBy =

And /etc/systemd/system/echo@.service

Description=Echo server service


And /path/to/

from time import sleep
import sys
sys.stdout.write(sys.stdin.readline().strip().upper() + 'rn')
# Do other work

Then running systemctl daemon-reload && systemctl enable --now echo.socket will start the socket and let it listen on /run/echo.sock. You should confirm it’s there with ls -l /run/echo.sock and that it’s owned by Foreman with mode 0600.

Now you can write to it:

echo test | socat - /run/echo.sock

You should note that sticks around for 10 seconds and you can launch multiple processes. This is fairly trivial job queuing, but probably sufficient for your use case. The original hook is closed fast and the actual work happens in the background.

Maybe a FIFO socket works better:

Note that example also writes the output to the journal rather than on the socket, which may be much better.

This post may not work entirely, but should at least guide you to a working solution without having to wait for and set up webhooks.

Also interesting is that you can run the socket under one user but the service as another. That can give you proper isolation and sandboxing.

Brilliant, not the most elegant code I’ve written, but it does what it needs to do. I never think to leverage systemd to its potential

Keep in mind if you have SELinux turned on (enforcing) foreman domain will not be allowed to connect to a socket. You need to add a rule to allow this.

Isn’t that allowed since Puma also needs sockets?

Foreman can only connect to one particularly (labelled) socket, not all sockets.