This is a follow up to RFC: Redis in Foreman.
Up till now, Redis has been present in the Foreman project as:
- The database for Sidekiq
- Cache for Pulp content locations
Previously, Redis was also used by Pulp for it’s tasking system until it was re-written into a simpler, Postgresql based solution.
When we first brought Redis into the ecosystem it was being used primarily by tasking systems that needed Redis to be configured with persistent storage. Pulp content caching piggy backed on the existing Redis instance utilizing it’s own database within the same instance.
Redis has been available as an option for the Rails cache for a number of years, with Foreman defaulting to the file-based cache. The file based cache has been shown to fall over at scale, https://bugzilla.redhat.com/show_bug.cgi?id=2063717.
Additionally worth noting is that if using performance co-pilot with a local Grafana, pcp will use Redis as a cache.
The proposed change is to switch to using Redis as the default cache for Foreman.
In order to do this, we will need to make a change to our deployment architecture to support two instances of Redis. The reason for this is so that we have one instance deployed with persistence which is required by Sidekiq and separate instance deployed for caching. To quote Sidekiq documentation:
it’s important that Sidekiq be run against a Redis instance that is not configured as a cache but as a persistent store.
To that end the proposal is to have two Redis instances.
- Keep default Redis instance as a pure cache
The current, default Redis instance will be kept but configured for caching and used by Foreman and Pulp. On foreman-proxy / Capsule this same default Redis will be used for caching there as well.
- Introduce a new Redis instance configured for persistence
A second instance of Redis will be deployed configured in persistence mode to be used by Sidekiq. This will require defining and managing our own systemd unit file for the second Redis instance and will need a name to differentiate it from the default Redis.