Containers: Plugins

containers
rfc

#1

Plugin/Service Design

This document aims to capture the strategy for existing plugins, and next generation plugins designed to act as services. This will be dependent on and drive the overall architecture.

  • Existing Plugin Design
  • Plugin Design Proposal
  • Plugins as Services Design Proposal

Existing Plugin Design

  • Developed as Rails engines in Ruby
  • Declare plugin properties through plugin API
  • Add API and UI through gem packaging enabled through bundler
  • Bring along database tables and migrations
  • Add attributes to existing core tables tightly coupling
  • Add code to existing models, controller through Ruby-isms and not defined API
  • Include
  • Extend
  • Addition of plugin requires server process restart

Plugin Design Proposal

This design is centered around allowing existing plugins to undergo little to no change but still be delivered independently of a base image.

Design

  • Core image:
    • Contains only what is needed to run vanilla Foreman
  • Plugin Images:
    • Contain plugin code and dependencies
    • Contain bundler.d file to activate plugin
    • Contain any settings or configuration files

Deployment

  • Core image is deployed initially and considered running as a service
  • Plugins are added by deploying their images as either a deployment or job or added as a sidecar container
  • Plugin deployments mount a shared volume with core container
  • Copy plugin code, dependencies, bundler.d and settings file to shared volume
  • End of deployment triggers restart of core container deployment

Pros

  • Foreman core container can focus on vanilla Foreman
  • Plugins require little change to continue working
  • Plugins can be delivered asynchronously to Foreman

Cons

  • Requires shared storage for runtime code
  • Plugins cannot be removed or turned off easily
  • Addition of plugins require server restart

Open Questions

  • How to disable a plugin

Plugins as Services Design Proposal

This design proposal aims to move plugins towards a service oriented design focusing on independent plugin release, and core service(s) with the ability to scale plugins independently.

Design

  • Core Image:
    • Contain what is needed to run Foreman core service
  • Plugin image
    • Contains all plugin code, settings, and dependencies
    • Runs a stand-alone webserver
    • Provides API and UI
    • Apache frontend handles routing to services

Pros

  • Plugins can be added, removed and scaled independently
  • Plugins can have independent release cycles
  • Plugins could be written in any language potentially
  • Core image focuses on core service(s) and stable API
  • Core image can rev independently
  • Easy for plugins to bring along dependent backend services

Cons

  • Requires redesign of plugins
  • Plugins must provide full stack for webserver
  • Using existing APIs, communication is JSON over HTTP(s) protocol which potentially has speed problems at scale (see open questions about RPC over HTTP2)

Open Questions

  • How to handle UI?
  • Cohesive UI with plugins providing UI
  • Use common framework for header and navigation
  • Use state service (or make part of core) to mark things like active page
  • Master service UI page that routes to individual independent UIs
  • Should Istio be used for service routing?
  • Investigate gRPC for inter-service communication
  • HTTP2 can speed up HTTP calls – changes needed?
  • Should plugins be allowed to modify core database tables?
  • Migration, evolution path?
  • Should we encourage the use of standardized webserver stacks and provide templates or library code to enable service creation?

Containerizing the Foreman Ecosystem
#2

This isn’t well supported today. It should be doable, I could imagine us modifying the plugin template to show how to be a good citizen and clean-up upon removal. But, I don’t think you necessarily need to solve that problem as part of containerization, at least not now.

There is distributed ruby: http://nithinbekal.com/posts/distributed-ruby/. I have zero experience with it but I do know ManageIQ uses it. Has this been investigated at all?

I could imagine more things being turned into Foreman::Plugin extension points, with support for plugin things being DRb objects. It might give more of a chance to allow refactoring existing plugins without rewriting everything.


#3

Fair point. If you look at the architecture discussion, and use of the operator we may in fact need some, even naive, solution at the outset. If we provide a way to set a plugin to true, then the reconciliation needs to be able to handle setting that same value to false. At the outset, this could be as simple as dropping the bundler.d file so at a minimum the code is not loaded.

Thanks for bringing this to my attention. I’ll reach out to ManageIQ developers to dig into their use of it, experience with it and any pros/cons they have found with it and report back here as a viability option.


#4

Removing the code could cause problems, for example anything built with STI will make rails fail if it’s removed (try taking Katello out of your bundler, start Rails, and visit the web UI). But that’s probably easy to fix. Not sure if there’s any other gotchas.


#5

Remember CORBA? Then the industry shifted towards onestacks like EJB or Spring. The rest is history. Could not resist, now on the serious note :slight_smile:

I am not convinced that Foreman is a good subject for containerization. The app will always be “pet server” kind, no matter if it is running in container, VM or bare-metal. I am very concerned about trying to break Foreman into microservices. Rails is by design one monolithic application and doing it differently is swimming against current. I remember the battle with MongoDB on Rails.

For this reason, I prefer option (A) - Rails in a single container. I suggest being opinionated about plugins shipping vanilla Foreman and all-plugins Foreman but nothing in between. No runtime support for enabling plugins, containers should be immutable. I believe that ability to install/uninstall plugins (tinkering with database schema) degrades containerization to weird-VM level, this is not containers how they are supposed to be.

Having that said, we need to give our users ability to install their very own plugin set or custom plugins. To do this, we need to have documented and tested container build process, so users would essentially build their own containers on top of our vanilla or all-plugins base images. We could provide automation for that, possibilities are to use the vanilla Foreman all-in-one setup to actually bootstrap Foreman with own plugins container (aka Undercloud and Overcloud in OpenStack) so we have better control of the process.

What’s important is that option (A) does not close doors to option (B) - full microservice architecture some day. But I believe this is out of table technically today. This smells like a complete rewrite of architecture and lot of core code.


#6

I have thought a bit about this and it’s the hardest part. I think you never really get away from it. Being pluggable means giving the admin a choice on how to deploy their application. That’s why I’m currently thinking about a way to easily customize the build process to list plugins that should be available.

IMHO the hardest part is that plugins have full control over the database. They can modify existing tables and you have no idea what they’ll do. In a system like Django apps clearly define their own tables and are “namespaced” (prefixed). That isn’t a full safe guard because of foreign keys but not modifying existing tables is a good start. I don’t know how often this happens, but it’d be good to investigate this.

In the past @Dmitri_Dolguikh has suggested to fully separate databases to avoid this problem. When you suggest plugins as services that’s what I think about. I like this idea, but is a huge mind shift and might not be practical (due to performance, complexity).


#7

:+1: if we target (B) from the beginning, the changes are that we will never finish, as the chunk of work is just too much to be done in one release. It would actually mean throwing away the whole plugin ecosystem Foreman has today and starting from scratch, with many open questions to start with.

The proper path for (B) would be starting with some new functionality that we want to add to Foreman using this model (or select one/limited set of plugins first) and answer all the answers there. Then, we could evaluate if that approach is suitable for the whole ecosystem. But since (A) is much smaller effort and can still enable (B), I don’t see a reason to choose (B) to begin with.

There are other services that can benefit from containerization (Pulp, dynflow workers, candlepin, ansible runner, additional monitoring services etc.) that can leverage the k8s environment, without touching the plugins ecosystem at first.


#8

Absolutely, just to clarify by “Foreman” in my initial post I though purely the Rails App and engines, not the whole ecosystem with backend systems. Natural break up into containers (e.g. dynflow, cron jobs, candlepin, pulp) does make much more sense.


#9

Related to this, there is a part of the RFC related to container image build I did not send along yet. This can currently be found at: https://github.com/theforeman/forklift/pull/831/files#diff-e07f67c313ca192a3243f21b6b518fd4

This implies all plugins are enabled all the time which could get to be a bit much for users and put a greater burden community to ensure all plugins work all the time together. I do get the sentiment though. I just worry about practicality when compared to configuration options or users building their own images.

One key to remember is, the Foreman application itself is just a Rails monolith at present but the runtime requirements and ecosystem around it are a variety of services. Everything from databases, to cache store, to async task runner to smart proxy and plugins that bring along external services. Micro services do provide their own risks and challenges. I don’t want to disrupt the ecosystem for plugins, which is why I am glad we are discussing evolutionary patterns.


#10

Apologies, I should have kept reading before fully replying!


#11

It would be interesting to see what other Rails apps do for plugins in containerized deployments. Discourse is one example of this, and we have @Gwmngilfen who already played with it quite a bit. Other one that pops to mind is Redmine, while we don’t run it in containers afaik I think it is possible and it also has plugins. There are probably others that can be investigated so we don’t have to reinvent the wheel here.


#12

What if there is a service that can build you container according to your needs? A hosted SaaS build your container, like:

https://rom-o-matic.eu/


#13

The main container thread suggested an approach of starting with a simpler approach of running containers via systemd. The first step to this is the RFC and ramping up work to run Foreman as a systemd service behind an Apache proxy. This opens the door to replace the system process with a container instead. However, the discussions in this thread are still very relevant. How do we handle plugins?

Let’s assume we are dealing with a container running via Docker. We still have to ask ourselves how we enable plugins. Someone asked how Discourse handles this. @Gwmngilfen can correct me where I go wrong, when we investigated this I believe we arrived at Discourse rebuilding the image locally based upon the set of plugins enabled and a command ran by the operator.

Summarizing options so far:

  1. Single container for Foreman with all plugins installed but disabled by default
  2. Single core Foreman container that is rebuilt whenever a plugin is enabled
  3. Single core Foreman container, each plugin is a container that runs, adds the code to a shared directory on the host, then shuts down making it now available to Foreman container
  4. Operators of a Foreman, rebuild the Foreman image configured per their own requirements enabling plugins as they need and creating a custom image

Containers: Image Build Strategy
#14

The little I have researched seems to indicate that 3 as a sidecar is the preferred model. (4) sounds nice if we deliver scripting to help rebuild it and do the migrations.


#15

@ehelms yes, (4) is what Discourse does. Their process is

  • Add git clone https://<repo-url-for-plugin> to a container config file
  • Rebuild main container via /var/discourse/launcher app rebuild

I think that probably gives us the least headaches (“only” some script tooling to make this smooth), provided we’re OK with asking users to rebuild containers. Based on my experience with Discourse, I don’t think it’s any more complex than yum install plugin - both involve a small amount of command line use, and a small amount of service downtime (Foreman has to restart to enable the plugin in any case).

Looking at the Discourse launcher script [link] might be a good start for such tooling? Apologies if that’s in progress, I’m out of the loop :wink:


#16

I like this one, what are pros and cons? Makes sense to me.

This has one big drawback - it nullifies one important aspect of container delivery: immutable artifact. But I understand we need to enable users to install their own plugins somehow. Therefore I lean towards having this as the second option with a visible flag in UI/API/logs showing the container was custom-made as a warning for us when providing help.


#17

What about plugins we’re not aware of? Either 3rd party plugins or locally-developed ones. Making it hard to install plugins other than ones we’ve “sanctioned” in some way seems like increasing the barrier-to-entry for both users (who might need a plugin we haven’t included), and for plugin authors (who’ll want to test their plugin on their system). That’s not a good idea.

Also we have 90+ plugins that we know of - just keeping them up to date given they all have their own release cadence seems like a recipe for disaster.

I think focussing on the consistent runtime is more important from our point of view. Much of our time is spent on dealing with inconsistencies between OS versions - that all goes away, and the user is highly unlikely to mess with that. If we provide good tooling, they’ll have no reason to log in to the image, only to rebuild it via our tools (which should work in predictable ways, since we wrote them).


#18

It’s clear we need still to provide a way to deploy custom plugins, there is no doubt about it. This does not rule out this option, we have the numbers from The Survey and we can say with some degree of confidence (I am sure you know how to even measure it :slight_smile: ) that our container covers 90 % of users.


#19

I don’t see a value in trying to decide how to curate which plugins should or should not be in the container - we already have many discussions over this for what should be included in the installer, and I’m not convinced it adds value. Let’s treat all plugins equally, and they can all have the same install method.

Actually, that’s quite hard to do, since we don’t collect stats from running Foreman instances. We certainly don’t have any view of locally developed plugins that are not released to the community.

It is reasonable to assume that users with larger deployments which also have their own custom plugins, so they’re affected more strongly by this two-tier approach. I’d like to get rid of that. Our plugins, other plugins, in-development plugins - all should have the same deployment method.


#20

I see huge benefit in covering large portion of our user with an immutable pre-populated container with most useful plugins. We also do something similar today - RPM/DEB ship only portion the plugins. You like to type “90+ plugins” in these containers discussions a lot, but it’s fair to mention that we also ship a fraction of this in linux packages. The remaining 80+ plugins are DIY installations, people are deploying Foreman from sources because of that sometimes. It is very similar situation, yet the added value of having bunch of plugins being released in the same distribution format with the same pace as Foreman core is clear.

We actually are already opinionated, and if we choose the one opinionated container it will be no drastic change.