State in Foreman 2.0
In Foreman 2.0 we’ve introduced a refactoring of Dynflow to use Sidekiq. This has allowed us to introduce separate services:
- dynflow-sidekiq@orchestrator - special worker with a single thread
- dynflow-sidekiq@worker - worker that by default has 5 threads and monitors the
- dynflow-sidekiq@worker-hosts-queue - Only on Katello - monitors the
hosts_queuequeue (yes, the naming looks redundant). Configured by the installer and shares the threads variable with
In practice this can be configured (values are defaults):
foreman-installer \ --foreman-dynflow-pool-size 5 \ --foreman-jobs-manage-service true \ --foreman-jobs-service-ensure running \ --foreman-jobs-service-enable true \ --foreman-jobs-sidekiq-redis-url undef
--foreman-dynflow-pool-size 5configures the threads for a
worker. In Katello you have 2 worker processes which means 10 threads in total.
--foreman-jobs-manage-service truetells the installer whether to manage the services or not. Generally not advisable, but can be used to work around installer limitations.
--foreman-jobs-service-ensure runningensures the state of the jobs services. Can be changed to
stopped. This may be useful on setups where you run the job services on different physical servers
--foreman-jobs-service-enable truesimilar to the ensure parameter, but this is for enable/disable starting at boot. Should generally match the ensure
--foreman-jobs-sidekiq-redis-url undefcan be used to point to a non-local Redis instance. Useful in load balanced setups or other split setups.
Note there is no knob to change the number of threads independently for
Adding a pool
Threading in Ruby has its limitations and at some point you may need to use multiple processes. The systemd job definition easily allows this, but the question is how we configure this.
The idea is that you have
dynflow-sidekiq@worker-n. A concrete example:
- dynflow-sidekiq@orchestrator has 1 thread
- dynflow-sidekiq@worker-1 has 5 threads
- dynflow-sidekiq@worker-2 has 5 threads
- dynflow-sidekiq@worker-hosts-queue-1 has 5 threads
- dynflow-sidekiq@worker-hosts-queue-2 has 5 threads
I started a PR to implement this just to test out the likely issues:
It turns out my idea generally works, but there are some issues we should discuss.
First of all, bike shedding around naming.
Do we always name them
worker-n or do we use
My suggestion is to always use
worker-n and start 1-based.
The implication is a migration from 2.0 where we had
worker. This should also be changed in packaging which ships a default
worker.yml file. While the installer can and will remove this config file, any
yum update will replace it since yum isn’t smart enough to track file deletions (apt is).
For reference, Pulp 3 also uses 1-based worker names according to https://docs.pulpproject.org/installation/instructions.html#systemd.
The current PR only implements a parameter
--foreman-dynflow-pool-size and keeps
--foreman-dynflow-workers. This may be confusing. Especially if Katello doesn’t have its own tuning available.
What would users expect here?
While it’s very late, I would like to suggest this for Foreman 2.1. The actual changes are not large and it either deploys + runs the service, or it doesn’t. Sidekiq workers themselves require no code changes. That should make it fairly safe.