Hey there,
I’ve been using ansible with Foreman for quite a while now and it seems to be working fine. But I have an issue that could lead to a few Problems for me. When I run a single Playbook on one or a few hosts it runs great. But if I now scale the receiving hosts up to 50 or 100 Hosts the Process slows down immensely. My Enviroment could scale up to 500 Hosts and such bad Parallel Performance could really be a Problem.
Does anybody have Experience with this Issue or are there any guides to Increase Ansible Performance?
Could you quantify what you mean by “runs great”, “slows down immensely” and “bad performance”?
At 5-10 Hosts it takes about 2-5mins for all Hosts to successfully finish the job what I would call great.
At 50 Hosts the same job takes about 15-25min and it gets gradually longer if I add more Hosts.
Some other thoughts:
If I have a look while the jobs are running it seems like that the Hosts are waiting for each other to start the next Step in the Playbook what could possibly hinder speed.
Additionally it seems like not every host receives the steps at the same time. Not all hosts take the same time to finish their steps even though the Performance of all hosts is the exact same.
I could see that some Hosts were Finished with their step but would not start the next step until all hosts were finished with the same step.
And to establish a baseline, what happens if you run the same thing against the same set of hosts with ansible outside of Foreman?
Well, yes, that’s what ansible does by default. It takes the first task from the playbook and applies it to all the hosts, then it takes the next task and so on.
At the ansible level, ansible doesn’t apply the tasks to all the hosts at once, but it applies them to up to (by default) 5 at most. This should be controlled by ansible’s forks tunable Ansible Configuration Settings — Ansible Community Documentation.
I think the forks tunable is the solution to my Problem thank you very much for that. Locally on the Smartproxy it runs now very smoothly after adjusting the cfg.
The only Problem I have now is that Foreman isn’t using my ansible.cfg. If I look at the output with verbose activtated it says there is no config file used and its using defaults what means only 5 forks.
I don’t know where my error is, locally it takes my cfg but in foreman it doesn’t. But in order to make it work locally I also had to use a env Variable for the location of the cfg export ANSIBLE_CONFIG=/etc/foreman-proxy/ansible.cfg
I think the expectation was that user changes would go to /etc/ansible/ansible.cfg and those would be picked implicitly. Would that work?
On my Smartproxy it didn’t even create a /etc/ansible/* directory. I only have the /etc/foreman-proxy/ansible.cfg for configuration. I don’t know if I did something wrong in the setup process but it still worked so far.
Can I somehow pass foreman what config file to use?
You could just create the directory and file - it’s the default location for Ansible.
HTH