Hopefully, everyone is already aware of Foreman Maintain tool. Those who don’t, please check here.
I am working on a task where it introduces maintenance-mode sub-command in foreman-maintain with start/stop/status/is-enabled.
Currently, foreman-maintain adds/removes iptables chain to have maintenance-mode on/off during upgrade.
But with this sub-command change, it will also going to consider two additional steps i.e sync-plans disable/enable and cron jobs stop/start under maintenance-mode.
Depending upon whether user wants to manage cron job stop/start during maintenance-mode, he can configure an option i.e :manage_crond in foreman-maintain. By default value is true for this option.
To achieve cron jobs stop/start, foreman-miantain deals with “crond” service.
Query:
When :manage_crond = true then user will see crond service under sub-command i.e foreman-maintain service list.
Note that - katello-service is replaced by command foreman-maintain service
This means that crond will restart with every foreman-maintain service restart.
So, I would like to hear from you that does restarting crond with other Satellite related services bring any harm?
The only thing I could see here is that whenever command foreman-maintain service restart will run, it won’t run jobs scheduled at that moment.
Any suggestions or concerns on whether to consider crond service under service sub-command during :manage_crond = true.
Or its better to limit a crond service stop/start behaviour only for maintenance-mode on/off.
I’m ok with managing crond with foreman-maintain, for the reasons to control it during the maintenance mode, but I would be interested into others’ opinions.
I’d be OK with it, but I do want to question if we should move our cronjobs to systemd timers and explicitly manage those. On the other hand, that wouldn’t serve BSD users.
Another point you need configurable is the actual service name. On my CentOS 7 server it’s crond while on my Debian 9 it cron. Given we also have deployments on FreeBSD it’s likely yet something else.
@kgaikwad: If you are saying that you’d completely disable crond, I’m wondering if this isn’t too invasive as it will also stop all other cron jobs on the host. Could we just create a file somewhere and check in Foreman cron tasks if the file exists (and skip the job, then)? Similar to how puppet agent --disable works?
That is doable by setting manage_crond: false and testing result of foreman-maintain maintenance-mode is-enabled in the cron task (the command returns exit code 1/0 indicating whether the maint mode os on or not).
Taking this a step further: do we need a generic “allow cronjobs” setting. In a clustered setup where you have multiple machines you may only want to run cronjobs on foreman01 but not foreman02.
Cronjobs can be simply disabled by removing execute flag from /etc/cron.*/script files. That sounds like reasonable solution, we would need to enforce all jobs to be listed somewhere, maybe some pattern like /etc/cron.*/foreman.job_name would be appropriate, then enable/disable is as easy as one command (GNU find, chown).
It is important that you re-enable those you disabled. With stopping a daemon you can still be sure that even if you forget it, after a reboot it’ll be fixed. With changing files there’s a much higher chance something fails and you’ve now disabled them permanently.
Note that some users who run Foreman on multiple load balancers could have changed the permissions to disable cronjobs themselves and blindly enabling them could have a bad effect.
Yes, I do agree with @ekohl . Previously we thought of checking file presence as @TimoGoebel suggested.
To avoid changing cron files and instead of having dependency on other file presence, we introduces config option i.e manage_crond and foreman-maintain maintenance-mode is-enabled subcommand which simply returns 1/0.
If users don’t want to manage cron service then they can easily disable this configuration and manage their cron jobs behaviour using this subcommand.
And I say this is actually unwanted behavior, during maintenance you may want to reboot the system without bringing service up until you finish the process. If this is coded properly, it can be actually better than shutting down xinetd.
I dislike shutting a service on puppet-managed system which Foreman is by default. We will see puppet agent fighting the maintain tool bringing the service up while my suggestion actually prevents from that.
Oneliner example was an extreme, of course this is doable with some extra effort checking desired state before modifying.
But I found one more agrument against changing permissions - RPM file ownership. A yum upgrade would change permissions, we would need to make them configuration files.
Yet another argument for checking for some marker that crons should be disabled. This does require jobs to be modified so it’s a bigger change, but I’m sure that long term it’s easier to maintain because you don’t fight with other tools/systems/admins that may interfere in a way we didn’t anticipate.
You’re right though a quick grep through packaging tells me they should all be installed as config files. We’re not very consistent with replacement behavior though:
It seems that making the cron jobs maint-mode aware is the most safe way to go. It could help avoiding other changes in the f-m logic when the jobs are converted in the timers.
This way we are not able to cover custom cron jobs though. Would a detection (f-m check) of foreman related jobs that lack the main-mode clause be way to handle it?
