But memcached does not support anything like that. I’d rather avoid tight integration with specific broker, Foreman is not the kind of app where you have tens of thousands active users at one time with millions of notifications per minute. I think storing them in RDBM is just fine for our use case.
Redis is fairly light weight. For Pulp tasking they chose to use python-rq with Pulp 3. That requires Redis because you don’t have to reinvent queueing. Memcache is just a key-value store which has no queueing functionality.
Also note that Redis offers some features like persistent caching so that you don’t lose all your user sessions after a reboot.
I think that it shouldn’t be hard to support both Memcache and Redis. Giving the users choice would be a good thing. However, we can prefer Redis. In the Katello case it leads to a simpler stack because of Pulp. For vanilla Foreman there’s little difference but it’s good to align the stacks.
I would prefer just to use redis, here are my reasons:
- lets be opinionated, reduce the size of the stack (why would we want the add more pain for supporting our stack)?
- using redis, you can get a simplified message bus, it has a lot of benefits imho, ranging from web ui live notifications (without the need to do continues pulling, and without the need to increase the db pool size)
- its a common / reliable software stack, we can have (in some day) non ruby code putting data on redis, and have some ruby respond to it?
- async job processing tools such as sidekiq rely on redis, maybe we can use that to reduce db /maintenance load.
- we know the developers
1 Like
First of all, I am not strictly against sticking with Redis, I just want to discuss this properly before we do the decision. I think you are actually showing with examples my biggest concern - let’s not get hard dependency on caching component. Finding common denominator between memcached and Redis is what I would like us to do, but what you actually propose is to go full-throttle Redis which is actually quite the opposite.
Because we already have foreman_memcached in our theforeman.org? I have to admit that migration should be pretty easy (well users will need to drop sessions/cache I guess). But this is by far the least significant point in the discussion IMHO.
As always, there are drawbacks. Polling is one way of doing this, many TCP connections is the other way which is exactly what Redis Pub/Sub does. There are many articles in the wild saying connecting directly from JS is a bad idea and WS should be used instead. That’s another component in our stack since Passenger/Ruby/Rails is not good at WS.
Beware! Redis was built as messaging for background processing and UI. It has no notion of durability and guaranteed delivery and it is far from being a full-blown messaging broker. Let’s not build more features than background processing or simple UI messaging on top of that or we will hit walls very soon.
Not something you wrote so I will leave a warning here: nobody should think about using Redis as enterprise service bus between (containerized) components (e.g. smart-proxy and foreman). That would be terrible design mistake to make.
What you are saying is “let’s have some more complex environment in the future”. I actually treat this one as anti-argument Unified stack with RDBMS is actually a good thing!
And we don’t use sidekiq, this is a weird statement. We do have foreman-tasks and we won’t be starting with sidekiq only because it’s using Redis which we could have. That does not make sense to me.
my point is why support memcache and redis (and non memcache like today) and that imho we should pick one way and move forward (similar to @Gwmngilfen comment about multi distro support).
I would support WS solution going forward (infact, thats what I was imagining from the beginning), and you are right, we would need another WS server that is most likely not ruby (maybe something like anycable).
the truth is that many times we wished we had that kind of infrastructure, but we often skip it because limitation in our current architecture.
further, this would allow us to stop using the python ws just for vnc/spice and can be a generic solution for all WS traffic.
true, having said that, it can still be used in many ways without the need for a full message broker.
it is no secret that I’m in favor of using off the shelf solution for background processing, hence this comment.
At the end, if we are considering adding a required additional service, and redis is as simple as memcache but offer more flexibility and features (while being stable), I’m in favor if redis.
I think my preferences are sort of captured in this: off the shelf solutions in general and the more flexible solutions.
With that in mind, is there no generic Rails caching solution out there? Currently foreman_memcache implies it’s very Foreman specific, but http://guides.rubyonrails.org/caching_with_rails.html tells me there are out of the box Memcache and Redis cache stores out there. It means the user has full flexibility in their caching solution and we can use an off the shelf solution.
Thanks for the replies and digging into this thus far. Recapping where I believe this has landed so far:
Recapping Key points:
- Pulp 3 is using RQ and it’s default backend is Redis, therefore we get stack re-usability if we defer to Redis when content services are enabled
- Redis as a default provides other potential use benefits and is a well-maintained off the shelf solution
- This RFC does not intend to drop memcached support
- This RFC does not intend to require a cache store by default
This is where I see things currently:
- General agreement to move configuration of cache store into Foreman core
- When the Foreman project is deploying Foreman and requires a cache store to use Redis (e.g. Forklift environments, blog posts, containers, etc.)
- Maintain support of memcached through configuration options
3 Likes
ehelms https://community.theforeman.org/u/ehelms Leader
July 2Thanks for the replies and digging into this thus far. Recapping where I
believe this has landed so far:Recapping Key points:
- Pulp 3 is using RQ and it’s default backend is Redis, therefore we
get stack re-usability if we defer to Redis when content services are
enabled- Redis as a default provides other potential use benefits and is a
well-maintained off the shelf solution- This RFC does not intend to drop memcached support
- This RFC does not intend to require a cache store by default
This is where I see things currently:
- General agreement to move configuration of cache store into Foreman
core- When the Foreman project is deploying Foreman and requires a cache
store to use Redis (e.g. Forklift environments, blog posts, containers,
etc.)- Maintain support of memcached through configuration options
My proposal was to deprecate memcache support, and introduce in core
support for redis as a required service, initially for cache, later on for
UI WS and other services.
I think this is a bit premature. For pure caching I don’t see a difference between memcache and redis. The case of UI WS is still quite complex and I think we need to do this in a separate RFC.
While I fully agree, my motivation here is that we will be opinionated to reduce the permutation we have for our users. of course that if someone really wants to keep memcache, there is a way (or even keep the memcache plugin) but by default we should not enable that (e.g. in the installer)
I am sensing two separate questions that are worth breaking out specifically to answer:
- Redis vs. memache in general for caching and which we should prefer
- Whether the Foreman application, should by default, configure and use a Redis backend for caching? Rather than the current local disk caching that occurs.
This just makes it harder making Katello running on existing Foreman. If we can, we should unity on one solution that fits all, unless there is really bit reason for not oding so.
I am fine with that as long as we support switching cache backends back and forth via our installer.
We already talked about introducing a message broker for dynflow? Could we use redis for that? @iNecas?
In general, both memcached and redis are liked by admins. We have had good experiences with both. I believe that redis gives use more features and I think we should make use of them and introduce them as a default stack.
Can we get rid of qpid completely in favour of redis?
2 Likes
Currently both Pulp (2) and Candlepin use qpid. With Pulp 3 moving to Redis we still have Candlepin using it to notify Katello about {entitlement,pool}.{created,deleted} + compliance.created events. I do not know if this can easily be converted to Redis or some other technology as well.
Then there is the whole qdrouter/gofer stack for clients that I know nothing about, but I believe that there is an alternative of using a yum/dnf tracer plugin for reporting and using REX for active actions.
IMHO this is a good simplification in the stack and something we should aim for.
2 Likes
This is one question of cross-process communication we need to figure out before we move to multi-workers setup.
@lzap raised his concerns, that redis was not build to provide enterprise-level messaging, due to limitations in durability and guaranteeing of delivery. On the other hand, since we serialize most of the data to database, using the pub/sub just for poking the workers when needed might be potential way to go.
The main issue with message brokers (at least in my experience) is, that they work nicely, until they don’t. And they require additional skills on those who maintain the infrastructure to keep the broker happy. It might be just our experience with specific implementation though. Not having something we could rely on in terms of durability and guarantee of delivery, and forcing us on handling that on higher levels, might be one way to go forward.
However, this requires separate investigation. What we tent to do in Dynflow is leveraging existing infra to do our job, rather than introducing new stack just for our purposes. Therefore, presence of Redis that we could rely on might influence, how we think about cross-process communication and how much we expect from it. If nothing else, it would allow us looking at Sidekiq for the tasks processing.
Anyway, I don’t want to derail this conversation too far.
Big caution here, Redis messaging is not persistent at all and the protocol does not allow client-to-client reliability. Dynflow was built on notification concept - fire and forget. As soon as the broken confirms, dynflow developer assumes the work will be done (now or later).
Redis is not the right tool for dynflow’s job.
Yet it can be useful for Foreman, some of my ideas include offloading facts from RDBM or aggregating telemetry data for Prometheus Ruby client (currently single-process only).
Of course Dynflow could workaround lack of durability/reliability, although I think it’s bad idea to reinvent the wheel. Let’s not build another messaging system, we already built our own tasking system and if you peek into dynflow code, you will see a lot of code there.
1 Like
lzap https://community.theforeman.org/u/lzap Discovery
July 3lzap:
Redis is not the right tool for dynflow’s job.
Yet it can be useful for Foreman, some of my ideas include offloading
facts from RDBM or aggregating telemetry data for Prometheus Ruby client
(currently single-process only). And dynflow could workaround that,
although I think it’s bad idea to reinvent the wheel.
with regards to facts, you could have a rails web process to receive the
facts, store them in redis (as strings), and pass the uuid to a remote
worker without a need to seralize/deserilize and store it in a db.
You are right that its not great if you lose that kind of data, but imho
its OK in some cases (e.g. for facts you can request them again).
Regarding good fit or not for dynflow, IMHO if its good enough for large
community projects such as sidekiq, it should probably be good enough for
our usage cases too.
1 Like
The candlepin folks are aware of the change, and moving to redis for that communication stack would be easy for them to do. They are moving to artemis for their own messaging, but the broker will be internal to candlepin so it will not be an impact on the service architecture.