This document aims to capture the strategy for existing plugins, and next generation plugins designed to act as services. This will be dependent on and drive the overall architecture.
Existing Plugin Design
Plugin Design Proposal
Plugins as Services Design Proposal
Existing Plugin Design
Developed as Rails engines in Ruby
Declare plugin properties through plugin API
Add API and UI through gem packaging enabled through bundler
Bring along database tables and migrations
Add attributes to existing core tables tightly coupling
Add code to existing models, controller through Ruby-isms and not defined API
Include
Extend
Addition of plugin requires server process restart
Plugin Design Proposal
This design is centered around allowing existing plugins to undergo little to no change but still be delivered independently of a base image.
Design
Core image:
Contains only what is needed to run vanilla Foreman
Plugin Images:
Contain plugin code and dependencies
Contain bundler.d file to activate plugin
Contain any settings or configuration files
Deployment
Core image is deployed initially and considered running as a service
Plugins are added by deploying their images as either a deployment or job or added as a sidecar container
Plugin deployments mount a shared volume with core container
Copy plugin code, dependencies, bundler.d and settings file to shared volume
End of deployment triggers restart of core container deployment
Pros
Foreman core container can focus on vanilla Foreman
Plugins require little change to continue working
Plugins can be delivered asynchronously to Foreman
Cons
Requires shared storage for runtime code
Plugins cannot be removed or turned off easily
Addition of plugins require server restart
Open Questions
How to disable a plugin
Plugins as Services Design Proposal
This design proposal aims to move plugins towards a service oriented design focusing on independent plugin release, and core service(s) with the ability to scale plugins independently.
Design
Core Image:
Contain what is needed to run Foreman core service
Plugin image
Contains all plugin code, settings, and dependencies
Runs a stand-alone webserver
Provides API and UI
Apache frontend handles routing to services
Pros
Plugins can be added, removed and scaled independently
Plugins can have independent release cycles
Plugins could be written in any language potentially
Core image focuses on core service(s) and stable API
Core image can rev independently
Easy for plugins to bring along dependent backend services
Cons
Requires redesign of plugins
Plugins must provide full stack for webserver
Using existing APIs, communication is JSON over HTTP(s) protocol which potentially has speed problems at scale (see open questions about RPC over HTTP2)
Open Questions
How to handle UI?
Cohesive UI with plugins providing UI
Use common framework for header and navigation
Use state service (or make part of core) to mark things like active page
Master service UI page that routes to individual independent UIs
Should Istio be used for service routing?
Investigate gRPC for inter-service communication
HTTP2 can speed up HTTP calls – changes needed?
Should plugins be allowed to modify core database tables?
Migration, evolution path?
Should we encourage the use of standardized webserver stacks and provide templates or library code to enable service creation?
This isn’t well supported today. It should be doable, I could imagine us modifying the plugin template to show how to be a good citizen and clean-up upon removal. But, I don’t think you necessarily need to solve that problem as part of containerization, at least not now.
I could imagine more things being turned into Foreman::Plugin extension points, with support for plugin things being DRb objects. It might give more of a chance to allow refactoring existing plugins without rewriting everything.
Fair point. If you look at the architecture discussion, and use of the operator we may in fact need some, even naive, solution at the outset. If we provide a way to set a plugin to true, then the reconciliation needs to be able to handle setting that same value to false. At the outset, this could be as simple as dropping the bundler.d file so at a minimum the code is not loaded.
Thanks for bringing this to my attention. I’ll reach out to ManageIQ developers to dig into their use of it, experience with it and any pros/cons they have found with it and report back here as a viability option.
Removing the code could cause problems, for example anything built with STI will make rails fail if it’s removed (try taking Katello out of your bundler, start Rails, and visit the web UI). But that’s probably easy to fix. Not sure if there’s any other gotchas.
Remember CORBA? Then the industry shifted towards onestacks like EJB or Spring. The rest is history. Could not resist, now on the serious note
I am not convinced that Foreman is a good subject for containerization. The app will always be “pet server” kind, no matter if it is running in container, VM or bare-metal. I am very concerned about trying to break Foreman into microservices. Rails is by design one monolithic application and doing it differently is swimming against current. I remember the battle with MongoDB on Rails.
For this reason, I prefer option (A) - Rails in a single container. I suggest being opinionated about plugins shipping vanilla Foreman and all-plugins Foreman but nothing in between. No runtime support for enabling plugins, containers should be immutable. I believe that ability to install/uninstall plugins (tinkering with database schema) degrades containerization to weird-VM level, this is not containers how they are supposed to be.
Having that said, we need to give our users ability to install their very own plugin set or custom plugins. To do this, we need to have documented and tested container build process, so users would essentially build their own containers on top of our vanilla or all-plugins base images. We could provide automation for that, possibilities are to use the vanilla Foreman all-in-one setup to actually bootstrap Foreman with own plugins container (aka Undercloud and Overcloud in OpenStack) so we have better control of the process.
What’s important is that option (A) does not close doors to option (B) - full microservice architecture some day. But I believe this is out of table technically today. This smells like a complete rewrite of architecture and lot of core code.
I have thought a bit about this and it’s the hardest part. I think you never really get away from it. Being pluggable means giving the admin a choice on how to deploy their application. That’s why I’m currently thinking about a way to easily customize the build process to list plugins that should be available.
IMHO the hardest part is that plugins have full control over the database. They can modify existing tables and you have no idea what they’ll do. In a system like Django apps clearly define their own tables and are “namespaced” (prefixed). That isn’t a full safe guard because of foreign keys but not modifying existing tables is a good start. I don’t know how often this happens, but it’d be good to investigate this.
In the past @Dmitri_Dolguikh has suggested to fully separate databases to avoid this problem. When you suggest plugins as services that’s what I think about. I like this idea, but is a huge mind shift and might not be practical (due to performance, complexity).
if we target (B) from the beginning, the changes are that we will never finish, as the chunk of work is just too much to be done in one release. It would actually mean throwing away the whole plugin ecosystem Foreman has today and starting from scratch, with many open questions to start with.
The proper path for (B) would be starting with some new functionality that we want to add to Foreman using this model (or select one/limited set of plugins first) and answer all the answers there. Then, we could evaluate if that approach is suitable for the whole ecosystem. But since (A) is much smaller effort and can still enable (B), I don’t see a reason to choose (B) to begin with.
There are other services that can benefit from containerization (Pulp, dynflow workers, candlepin, ansible runner, additional monitoring services etc.) that can leverage the k8s environment, without touching the plugins ecosystem at first.
Absolutely, just to clarify by “Foreman” in my initial post I though purely the Rails App and engines, not the whole ecosystem with backend systems. Natural break up into containers (e.g. dynflow, cron jobs, candlepin, pulp) does make much more sense.
This implies all plugins are enabled all the time which could get to be a bit much for users and put a greater burden community to ensure all plugins work all the time together. I do get the sentiment though. I just worry about practicality when compared to configuration options or users building their own images.
One key to remember is, the Foreman application itself is just a Rails monolith at present but the runtime requirements and ecosystem around it are a variety of services. Everything from databases, to cache store, to async task runner to smart proxy and plugins that bring along external services. Micro services do provide their own risks and challenges. I don’t want to disrupt the ecosystem for plugins, which is why I am glad we are discussing evolutionary patterns.
It would be interesting to see what other Rails apps do for plugins in containerized deployments. Discourse is one example of this, and we have @Gwmngilfen who already played with it quite a bit. Other one that pops to mind is Redmine, while we don’t run it in containers afaik I think it is possible and it also has plugins. There are probably others that can be investigated so we don’t have to reinvent the wheel here.
The main container thread suggested an approach of starting with a simpler approach of running containers via systemd. The first step to this is the RFC and ramping up work to run Foreman as a systemd service behind an Apache proxy. This opens the door to replace the system process with a container instead. However, the discussions in this thread are still very relevant. How do we handle plugins?
Let’s assume we are dealing with a container running via Docker. We still have to ask ourselves how we enable plugins. Someone asked how Discourse handles this. @Gwmngilfen can correct me where I go wrong, when we investigated this I believe we arrived at Discourse rebuilding the image locally based upon the set of plugins enabled and a command ran by the operator.
Summarizing options so far:
Single container for Foreman with all plugins installed but disabled by default
Single core Foreman container that is rebuilt whenever a plugin is enabled
Single core Foreman container, each plugin is a container that runs, adds the code to a shared directory on the host, then shuts down making it now available to Foreman container
Operators of a Foreman, rebuild the Foreman image configured per their own requirements enabling plugins as they need and creating a custom image
The little I have researched seems to indicate that 3 as a sidecar is the preferred model. (4) sounds nice if we deliver scripting to help rebuild it and do the migrations.
Add git clone https://<repo-url-for-plugin> to a container config file
Rebuild main container via /var/discourse/launcher app rebuild
I think that probably gives us the least headaches (“only” some script tooling to make this smooth), provided we’re OK with asking users to rebuild containers. Based on my experience with Discourse, I don’t think it’s any more complex than yum install plugin - both involve a small amount of command line use, and a small amount of service downtime (Foreman has to restart to enable the plugin in any case).
Looking at the Discourse launcher script [link] might be a good start for such tooling? Apologies if that’s in progress, I’m out of the loop
I like this one, what are pros and cons? Makes sense to me.
This has one big drawback - it nullifies one important aspect of container delivery: immutable artifact. But I understand we need to enable users to install their own plugins somehow. Therefore I lean towards having this as the second option with a visible flag in UI/API/logs showing the container was custom-made as a warning for us when providing help.
What about plugins we’re not aware of? Either 3rd party plugins or locally-developed ones. Making it hard to install plugins other than ones we’ve “sanctioned” in some way seems like increasing the barrier-to-entry for both users (who might need a plugin we haven’t included), and for plugin authors (who’ll want to test their plugin on their system). That’s not a good idea.
Also we have 90+ plugins that we know of - just keeping them up to date given they all have their own release cadence seems like a recipe for disaster.
I think focussing on the consistent runtime is more important from our point of view. Much of our time is spent on dealing with inconsistencies between OS versions - that all goes away, and the user is highly unlikely to mess with that. If we provide good tooling, they’ll have no reason to log in to the image, only to rebuild it via our tools (which should work in predictable ways, since we wrote them).
It’s clear we need still to provide a way to deploy custom plugins, there is no doubt about it. This does not rule out this option, we have the numbers from The Survey and we can say with some degree of confidence (I am sure you know how to even measure it ) that our container covers 90 % of users.
I don’t see a value in trying to decide how to curate which plugins should or should not be in the container - we already have many discussions over this for what should be included in the installer, and I’m not convinced it adds value. Let’s treat all plugins equally, and they can all have the same install method.
Actually, that’s quite hard to do, since we don’t collect stats from running Foreman instances. We certainly don’t have any view of locally developed plugins that are not released to the community.
It is reasonable to assume that users with larger deployments which also have their own custom plugins, so they’re affected more strongly by this two-tier approach. I’d like to get rid of that. Our plugins, other plugins, in-development plugins - all should have the same deployment method.
I see huge benefit in covering large portion of our user with an immutable pre-populated container with most useful plugins. We also do something similar today - RPM/DEB ship only portion the plugins. You like to type “90+ plugins” in these containers discussions a lot, but it’s fair to mention that we also ship a fraction of this in linux packages. The remaining 80+ plugins are DIY installations, people are deploying Foreman from sources because of that sometimes. It is very similar situation, yet the added value of having bunch of plugins being released in the same distribution format with the same pace as Foreman core is clear.
We actually are already opinionated, and if we choose the one opinionated container it will be no drastic change.
if we target (B) from the beginning, the changes are that we will never finish, as the chunk of work is just too much to be done in one release. It would actually mean throwing away the whole plugin ecosystem Foreman has today and starting from scratch, with many open questions to start with.