Hello,
I need some help from you guys that automated their patch workflow. I want to start to automate my patching of my CentOS servers. Currently we have about 40 CentOS systems (i know its a very small enviroment), but its possible that our farm is growing in near future. So i want to automate it as soon as possible.
My plan:
Foreman is syncing the CentOS repos on saturday. After that a new CV version will be published and the dev/test env. is getting promoted into it. On sunday all servers in dev/test env. are getting automatically patched. Now our devs are getting 2 weeks to test the updates. After 2 weeks the prod. env. is getting promoted to the new CV version, and will be patched after that. Now I dont really now how i should start to automate it, because of the many ways (cronjobs, ansible, cvmanager etc.)
How did you guys automate your workflow? Im thankful for any tip and message!
So I would not use cvmanager as this is a deprecated project as far as I know.
In the end you need to find a way to write your automation - this could be a script using hammer-commands - but I think that Ansible is the best way to go.
With Ansible you have the way to create your logic and you can run it from your foreman-host but also from other hosts or instances - and you can do the scheduling with a cronjob or with another logic (even with a recurring job inside foreman as remote execution - or AWX or a suitable CI/CD-pipeline)
I would opt for Ansible nowadays and scheduling done by Foreman itself, gives most simple code and maximum transparency.
One tip for the prod update: Create a simple way to block updates, per node and for all. It should not be needed, but better have it and it saves your production once than not having it and needing it.
We use a combination of foreman jobs, bash scripts, and cron.
We have two foreman jobs - one to run yum/dnf update, and one to reboot the servers.
Then we have bash scripts that are run on a schedule - one script calls hammer to run the patch job, and one to reboot the servers.
The patch script takes the return code of the foreman job to determine if a server succeeded or failed. If it failed, it will rerun the job on any failed servers. At the end, it emails us with a note that it completed successfully or completed with failures.
The reboot script does essentially the same thing, also sending an email with completion notification.
This is very close to how we are doing it. I tried a weekly cadence like you are doing, but with ~200 systems, it turned out to be a recipe for insanity. We now sync repos (automatically with a Foreman sync job) on a particular day of the month (hint: avoid the first few days of the month to avoid the stampeding herd problem), and automatically publish new CVs the next day (from a cron-driven bash script using the hammer CLI. Although if I had to redo it today, I would avoid hammer like the plague).
As in your environment, this newly-published CV becomes available to dev systems immediately. Those dev systems run yum update in the middle of the night, again driven by cron. This is actually a fairly complex script for us since it needs to serialize multiple maintenance tasks. For instance, you don’t want to reboot after yum update in the middle of auto-renewing certificate.
From here on, we manually promote the CVs to the next lifecycle environments. Automating this can be problematic because if you do discover a problem, you don’t want to auto-promote the problematic updates. Also, sometimes our production team won’t allow updates due to certain peak seasons or software updates.
So my recommendation is:
Overall, your process looks good to me.
Move from weekly to monthly syncs
Automate the publishing of a new CV, but keep the promotions manual.
