RFC: Make Foreman easy to deploy and maintain

[Marek_Hulan]

Hello,

I’ve seen several threads recently discussing how hard it is to deploy Foreman. I also feel upgrades are not always as smooth as one would hope, especially when installed with many plugins. So I thought it would be a good time to share what I’ve been thinking about for a while. First, let me try to describe the state I’d like to get to.

Installing Foreman should be as easy as running an installer with not more than 5-10 parameters. Installation will provide minimal setup of Foreman that will allow to install and deploy additional features. The configuration and further maintenance is made from this Foreman instance. Few examples - user decides to configure provisioning. They fill in required fields in UI, e.g. type - network based provisioning, smart proxy used for such setup, IP range, domain, etc and click submit. That will ensure the correct setup of dhcp, dns, tftp services and also creates required objects such as Subnet and Domain in Foreman DB. Later after Foreman is upgraded, we should also automatically detect, whether a smart proxy is compatible with current version and if not, we should provide a way to update it, again with an easy of clicking a single button. The same concepts should apply to installing additional plugins (including Katello) and upgrading Foreman itself.

After several discussions we had on the provisioning setup, you may now think, well it’s more complicated than that. There are many variations of provisioning setup that users do. I’d propose focus on those we know are most widely used and provide setup helpers only for them. While we shouldn’t prevent users to reconfigure their instances according to their needs, we should become more opinionated on how we recommend to setup Foreman and potentially what we support.

Good thing is, we have all we need. I see Foreman as a great framework to build and configure basically everything. We just need to identify most common use cases and make them simple to achieve. We have powerful orchestration framework, we have integration with several configuration management tools, we can provision additional hosts so all we need is to start using them and wrap it in nice UI. So how to get there?

Step 1: Installer

Our existing installer has huge amount of parameters. Yet it’s limited e.g. in terms of configuring provisioning, it only support single network. Hiera configs can be perhaps used for other, but it does not feel as “easy way to configure”. Adding more parameters to installer does not make it easier either, today we have several hundreds of them and we see users struggle with it. It also often cause confusion when we rerun it during upgrade, as it overrides configuration files user modified, without any warning.

This approach worked for a small app we started years ago, but it may not be the best option anymore. It’s hard to configure everything correctly from the very beginning. So I’d like to offer alternative which should help with initial deploy.

I’d like to have a simple, very opinionated, quick install tool, that will install Foreman core, internal smart proxy with just basic set of features and plugins required for further configuration. I’m thinking of foreman_ansible - in fact that brings also rex and tasks as dependencies and next generation of foreman_setup. It would only ask for SSL certs paths, admin credentials and that should be it. This should give user a minimal setup, that can be used for adding more features to the Foreman.

This does not aim to replace existing foreman-installer which is still a great way to do e.g. automated setups for users, who know all answers before they start. This aims to make it easy to deploy Foreman, start with simple setup, experiment with additional features. Mainly new users and users with an easy use case, such as cfgmgmt dashboard, should benefit from this mostly.

Step 2: Configuration of additional features

It’s not easy today to configure advanced features, such as provisioning. Users run the installer and need to figure ~15 parameters the hard way in advance, then they need to replicate some information in Foreman UI too. Also installing a plugin is advised to do via another installer run while the configuration then usually happens inside of Foreman. We should focus on make this as easy as possible.

There is a foreman_setup plugin which aimed to help with provisioning setup. It was built before rex/ansible integration was available so it ask user to run some steps manually on the Foreman host. We can improve this flow and never ask user to leave the UI in order to finish the configuration. Also the UX of wizard could be improved with technology we have available in core today.

The similar aproach from foreman_setup should be added for other things, such as adding new plugins. Usually that shouldn’t be more complicated then installing few packages, doing some configuration and restarting Foreman. Obviously the hardest plugin to add to existing setup is Katello, because of many backend services, but it’s not impossible. The catch might be SSL setup as we wouldn’t be able to rely on puppet CA anymore and we’d probably need to orchestrate certificate requests and signing somehow, but that’s another thing that deserves more improvements for a long time. Given plugin installation requires Foreman restart we should also look at how to improve maintenance mode of the app.

We can start easily with shipping some playbooks implementing the above. Then we can extend foreman_setup that will add nice UI on top of it.

Step 3: Maintenance of Foreman installation

Today if some backend service such as smart proxy does not respond, we usually display ugly 500. This could happen due to network issue, old version of some smart proxy plugin, bug in code etc. We should change our approach and always write a robust code that would detect issue before it tries to do something with the backend. In case the backed does not work, we should try to detect the cause and provide hints to user or even better remediation. We should always check the backend version and ensure it’s compatible. And if it’s not, we should ideally offer a way to upgrade it. Imagine a button that would trigger rex/ansible job to install newer version of smart proxy plugin and restart the proxy. If we take similar approach for every isolated part of the system, we could do the full maintenance within Foreman only. The only service that probably can’t be easily maintained this way is Foreman itself.

Conclusion

While all steps can be worked on in paralel, I would personally like to start with Step 1 soon as I see it as relatively low hanging fruit, with benefits even without Step 2 and 3 finished. Of course, implementing all of steps is a huge amount of work that I will never be able to finish myself. It’s not something that can happen right away either. It’s something we first need to agree on and incorporate into our decision making. If we agree the above is a good direction, I’d need each developer to do a mindset shift to focus first on user experience and ease of use. I’d ask every reviewer to start asking for such things in reviews. I’d need help from UX team to build intuitive wizards. I’d need plugin maintainers to replicate such configuration patterns for their plugins.

So for now, I’d like to hear what you all think. Is this the right path forward?

[lzap]

Is this the right path forward?

I don’t see anything wrong with your proposal and I particularly like that it can work in parallel with our current installer. Some users can decide to go full with puppet, others might try the new method.

But I feel like we can still do better with the current installer. Let me say it and let me say it loud: I think our problem is lack of high quality documentation. If we find a way to improve it substantially this might be solution to the problem.

Key pain points in our current documentation:

1. Almost zero info on deployment plan. Users need to figure out if they want Katello or not, plugins, how many smart proxies, memory and CPU requirements and other things themselves.

2. Content is sparse, we are missing many areas and also what we have only scratches the surface.

3. It mostly covers success scenarios but we do not add content on what to do when things break. We do have ERF Foreman error codes wiki which is useful tho.

4. Bad navigation and confusion. We have separate Foreman Core, Katello and individual plugin manuals. Newcomers do not even know where to start. We should have a single site, a book, that covers everything.

5. Documentation is not accessible for changes. While it reads fine - just clone a git repo it’s a markdown, in practice the story is a bit more complicated due to how chapters are structured, it’s not easy to add new chapter or restructure flow. And technically we are not able easily to generate PDF.

Now, Foreman does have a much improved documentation and this is called Red Hat Satellite 6 documentation. It features several books of much improved content, however this is copyrighted material by Red Hat. If we had a chance to move the content to github and start from there, I believe we should be able to start improving right away. One difference between Satellite and Foreman is that the Red Hat product is opinionated about plugins - Katello is always installed. This would give us documentation with Katello, but we should be able to solve this by adding some boxes like “Skip this section if Katello was not installed”.

[John_Mitsch]

Adding more parameters to installer does not make it easier either, today we have several hundreds of them and we see users struggle with it.

This does seem like a large area for improvement, it is nice to see this topic brought up. I think the most user-friendly software has sane default options that are easily changed, this seems to be along the lines of what you are proposing. I don’t know enough about the installer to comment more about the logistics, but a general +1 to improving this area!

This would give us documentation with Katello, but we should be able to solve this by adding some boxes like “Skip this section if Katello was not installed”.

I like this idea. Perhaps it could even be dynamic and have a checkbox saying “Include Katello docs” or something along those lines. Currently, it is three clicks to get to Katello documentation, even to install it, which (as we know) many users install Foreman first and then say “now how do I install Katello?”. This is another area we could definitely improve on, maybe it warrants a separate topic

[mbacovsky]

Installing Foreman should be as easy as running an installer with not more than 5-10 parameters. Installation will provide minimal setup of Foreman that will allow to install and deploy additional features.

I like this approach a lot and I think this is a way to go especially if there are plans to deploy Foreman in containerized environment.

This does not aim to replace existing foreman-installer which is still a great way to do e.g. automated setups for users, who know all answers before they start.

I agree the current installer should exist in parallel. As addition to that I see big potential in foreman-maintain which could provide a wizard for configuring installer parameters which could replace quite generic and limited interactive mode of foreman-installer.

[mason]

But I feel like we can still do better with the current installer. Let me say it and let me say it loud: I think our problem is lack of high quality documentation. If we find a way to improve it substantially this might be solution to the problem.

@lzap, I am so happy to see this discussion happening! While I do agree that the existing documentation needs some serious work, I think that @Marek_Hulan’s proposal is where energy really needs to be spent. The documentation task will be far more manageable once the installation process is more friendly.

Key pain points in our current documentation:

1. Almost zero info on deployment plan. Users need to figure out if they want Katello or not, plugins, how many smart proxies, memory and CPU requirements and other things themselves.

Yes! As a new user, I have found the process of trying to get my head around all the components, how they fit together, what depends on what, etc… to be a very challenging trial and error process. Especially if you want to automate the installation process and not just fumble around in the UI. To have a few common deployment scenarios, graphically depicted, with the associated hammer commands to implement it, would be fantastic!

1. Content is sparse, we are missing many areas and also what we have only scratches the surface.

My impression isn’t that the content is sparse, so much as it’s just not organized in a way that clearly leads someone from point A to point B. The current content is more like a mix of reference material and tutorial all jammed in together. If tutorial could be separated from reference material, I think that would be a good start.

Now, Foreman does have a much improved documentation and this is called Red Hat Satellite 6 documentation. It features several books of much improved content, however this is copyrighted material by Red Hat. If we had a chance to move the content to github and start from there, I believe we should be able to start improving right away.

Once I learned that it’s possible to get a developer account, I was thrilled to find the Satellite 6 documentation. Still, it manages to be voluminous while being short on implementation details. Again, it’s the difference between reference material and tutorial articles. I think Foreman/Katello would greatly benefit from having more of the latter. Here’s a great example, which really helped me to start getting my head around things - https://www.lisenet.com/2018/homelab-project-with-kvm-katello-and-puppet/

One difference between Satellite and Foreman is that the Red Hat product is opinionated about plugins - Katello is always installed. This would give us documentation with Katello, but we should be able to solve this by adding some boxes like “Skip this section if Katello was not installed”.

When one is just starting out with something as complex as Foreman/Katello, I think it would be very helpful to have a strongly opinionated deployment guide and installer. The flexibility isn’t helpful at the beginning, it just gets in the way of learning the basics. Of course, once you have some experience and you want to start fitting the system to your own environment and needs, then the flexibility of the system and the reference material style of documentation are more helpful.

[mason]

@Marek_Hulan, if you want to see Foreman/Katello usage increase, then I think you’re absolutely on the right path.

After several discussions we had on the provisioning setup, you may now think, well it’s more complicated than that. There are many variations of provisioning setup that users do. I’d propose focus on those we know are most widely used and provide setup helpers only for them. While we shouldn’t prevent users to reconfigure their instances according to their needs, we should become more opinionated on how we recommend to setup Foreman and potentially what we support.

I agree that a more opinionated setup would dramatically reduce the barrier to entry.

[lzap]

just not organized

That is the proper phrase, thanks.

[iNecas]

In general, I like the idea of setting up just bare minimal to get the stuff up and running and mange as much as possible later from the infrastructure itself (we see similar things in OpenStack or Kubernetes). I like to think about how other on-premise services (have some experience with OwnCloud or Plex, but I’m sure there are others using similar mechanism. Basically the flow is:

1. get the bare usable minimum up and running quickly
2. let the rest be configured as you go

It will for sure require revisiting few things we do/don’t do right now (such as error handling). For example, dealing with the fact that DB is not available for some reason, and showing some maintenance page with more details on what’s wrong, and also perform more health status reporting.

The question is how to start so that we get something useful in timely manner.

[Peek]

Here’s a great example, which really helped me to start getting my head around things - https://www.lisenet.com/2018/homelab-project-with-kvm-katello-and-puppet/

Repeat - GREAT EXAMPLE !

Excuse me, I have quite a bit of reading to catch up with.

[lzap]

Since there are no opposing opinions in this tread, for healthful discussion let me throw one. If we go this path, we must be absolutely sure that this is worth the effort because today users are able to deploy 90% of the Foreman/Katello with our puppet-based installer. We’ve invested lot of effort into it, millions of things were parametrized. This looks like we are throwing this away. Sure, you said let’s do this in parallel, but it does not make much sense to me to develop and maintain two options of deploying Foreman. So I read that as a migration, switch or rewrite.

As much as I like Ansible more for this task of installing Foreman, I’d rather see us simplifying what we have today (Puppet) rather than making something brand new.

[mcorr]

Now, Foreman does have a much improved documentation and this is called Red Hat Satellite 6 documentation. It features several books of much improved content, however this is copyrighted material by Red Hat. If we had a chance to move the content to github and start from there, I believe we should be able to start improving right away. One difference between Satellite and Foreman is that the Red Hat product is opinionated about plugins - Katello is always installed. This would give us documentation with Katello, but we should be able to solve this by adding some boxes like “Skip this section if Katello was not installed”.

With regards to using the Satellite 6 documentation, I think there are a number of considerations we need to take into account before proceeding. However, as a pilot, and because this is specifically about how difficult it is to deploy as well as maintain Foreman, could we start with looking at one guide, an installation guide, and what we would need to do to make the current Satellite 6 installation guide usable as a Foreman guide. Having a well-structured guide might help with the complexities.

Could we look at the exiting Installation Documentation for Satellite, then look at the main causes of users to struggle and document a set of commands that is generic enough to get to a fresh install of Foreman?

Or am I dreaming?

[areyus]

I also would agree that what is needed most is not a tool to replace/hide foreman-installer.

In my opinion, here are some things that would probably be doable with less effort or at least to a more productive effect:

1. I do agree with Marek_Hulan on the fact that katello finally needs to be installable on top of an existing Foreman setup. I do not know why this is currently not possible, but having to throw your whole setup away after maybe weeks of work just because you did not look at Katello docs beforehand is a huge frustration factor. Until that works, I would suggest at least putting some sort of disclaimer in the main Foreman installation manual warning people about that facts. Currently this disclaimer resides in the katello documentation, and to be honest that is not where you need it. By the time you think about Katello, you have either already setup a complete Foreman environment and want to extend it (in that case, it is to late) or do not have anything up yet but already know you want katello (in which case, you do not need it).

2. Having better documentation (as was mentioned here already quite often) would also greatly increase the experience. I just scrolled through the Foreman Manuals installation section and to be honest, this feels more like an unorganized info dump mixed together with some more-or-less random scenarios for usage of foreman-installer (not to be confused with “foreman-installer --scenario”-like scenarios). I think there are quite some improvements that can be done here:
   a) Organize the docs in a more aproachable way. Having to sections right after each other explaining how to install Foreman is weird. Why is there a help on setting up random puppet modules in the quickstart guide? Put howtos on installing the software in front of using the installer thats hidden in the rpm, not behind that. I know it links to the quickstart guide, but if that quickstart guide is comprehensive enough, why do we need to have a second “install from packages” section? Why is the explanation about tfm-SCL packages on the same level as RPM and DEB install instructions?
   b) Get rid of the infodumps like the mile-long foreman-installer --help output. Move that to the wiki or some place where new users are not so overwhelmed at the first glance at the manual.
   c) What would probably help a lot more than having those infodumps would be an expansion of the “Installation Scenarios” section with some real-world setups. Maybe this could be based on infos from community survey or something like that?

3. Acompanying 2c, I would guess a stronger usage of scenarios (like “foreman-installer --scenario”) based on those real-world scenarios would also help a lot, especially on the proxy side. I am thinking of things like foreman-proxy-puppetserver, maybe foreman-proxy-dhcp-isc. Depending on the actual usage of those, having scenarios for Foreman with ansible/salt/idk without puppet could also be considered. Maybe adding a way to use more than one scenario at once would strengthen this approach.
   While the approach to deploy a all-in-one solution out of the box sounds nice in theory and probably is for some usecases, most companies already have infrastructure in place they need to work with and so will have to work with smartproxies for say DHCP and DNS.

4. What I could imagine, and this is probably something that would take up a lot of time, is adding certain wizzards to the UI helping with more complicated tasks to wrap your head around when you are getting started. I would imagine something like a “infrastructure wizzard” that guides you through all the required steps to integrate with existing infrstructure, checks wether smartproxys for Networks are connected etc., or some “add operatingsystem” wizzard that step by step walks you through the process of creating OS entries, products and reposiries, and whatelse not.

Finally I want to add my opinion regarding the Satellite documentation: Those are in fact quite high quality documents for a lot of common usecases. There are some downsides though I see when leening to much towards that:
a) As already mentioned, Sattelite docs are copyrighted material by RedHat. This will make it a leagal problem relying on those if nothing else. Still I do not know copyright law all to well and could imagine there might be chances something can get worked out.
b) Satellite docs are badly opinionated about what you can and can’t do with you setup and how your setup should look like. We are talking about everything from the plugins that can be used, over how smartproxys are set up, up to always installing only on RHEL. This could be a major downside.
c) To my experience, Satellite is (like most enterprise products) a wild mess of the original upstream product in some older version, mixed with some backports and additional theming. This means that Satellite docs will nearly always be outdated compared to current Foreman or Katello releases.
If those problems can be worked out, relying on those docs might indeed be the way to go. If not, we could at least take a peak an learn some things about how they aproach certain topics.

I just realized that I, once again, wrote a huge wall of text just to make some basic points. Sorry for that

[lzap]

Could we look at the exiting Installation Documentation for Satellite, then look at the main causes of users to struggle and document a set of commands that is generic enough to get to a fresh install of Foreman?

We kind of cover that in our documentation already, installer commands are not the main issue, it’s all the content that is provided in Sat docs on top of that: planning info, SELinux and NFS considerations, requirements, detailed firewall instructions and stuff like that. The core is the same:

# satellite-installer --scenario satellite \
--foreman-initial-organization "initial_organization_name" \
--foreman-initial-location "initial_location_name" \
--foreman-admin-username admin_user_name \
--foreman-admin-password admin_password \
--foreman-proxy-dns-managed=false \
--foreman-proxy-dhcp-managed=false

One of the biggest difference is that Satellite install guide is very opinionated - it’s basically only Katello installation with selected plugins preinstalled by default. We have a installer scenario called “satellite” which has some plugins already turned in. I think upstreamed documentation we are talking about should stick with being opinionated. There would be a challenge providing some useful info and navigation to users who are interested in vanilla Foreman. It’s a huge challenge, there are diagrams and lot of stuff which is only relevant for RHEL (subscriptions, CDN).

[lzap]

a) As already mentioned, Sattelite docs are copyrighted material by RedHat.

I was assuming that RH legal would work it out and upstreamed the docs under some FOSS license.

b) Satellite docs are badly opinionated

Two things. We need to be more opinionated, that’s the core of this tread. We should learn something from this. On the other side, it’s pretty clear that Satellite docs cannot serve as Foreman/Katello documentation without further changes. My idea was to upstream documentation and work on improving it - obviously what’s missing is installation on non-RHEL platforms, more plugins.

My thinking was to use some macros to hide this content during downstream process so both upstream and downstream could benefit from this.

c) … older version, mixed with some backports and additional theming

Surely Satellite 6 is older, it’s enterprise. Again, the idea was to “fork” the documentation and start maintaining upstream version of it. So eventually, Foreman “new docs” would become the new upstream (ahead of stable Satellite versions).

[areyus]

Two things. We need to be more opinionated, that’s the core of this tread. We should learn something from this. On the other side, it’s pretty clear that Satellite docs cannot serve as Foreman/Katello documentation without further changes. My idea was to upstream documentation and work on improving it - obviously what’s missing is installation on non-RHEL platforms, more plugins.

My thinking was to use some macros to hide this content during downstream process so both upstream and downstream could benefit from this.

This indeed sounds like a way to go if that can be worked out with RedHat legal department. In that case, I would definetly agree this could be a very good way forward. I agree that Foreman and Katello docs need to be more opinionated (Probably did not stretch that enough in my post), just not as much as satellite docs are.
I simply wanted to point out that despite beeing of a high quality, taking over Satellite docs would not be some sort of “low effort” solution to all the problems with the current docs. Still, it would surely be less effort than having to build everything from scratch.

[mcorr]

We kind of cover that in our documentation already, installer commands are not the main issue, it’s all the content that is provided in Sat docs on top of that: planning info, SELinux and NFS considerations, requirements, detailed firewall instructions and stuff like that. The core is the same:

Yes, that is what I meant - not the command but all the small steps a user must take to prepare the environment, open the right ports and security settings. One end-to-end tested and verified method of deploying Foreman fully.

One of the biggest difference is that Satellite install guide is very opinionated

One of the greatest difficulty of having good docs is sticking to one narrative, one scenario, and not adding in all possibilities. It is also the greatest chance of success.

I also think the upstream docs could do with being broken into more clearly defined sections, similar to the Satellite documentation, where you are guided through all the different preparations for installation, then the installation command, then some post-installation tasks. All the material you need for day 1 of Foreman on one page, then the rest chopped up neatly and placed somewhere else.

[viwon]

One thing that burned me in the past was the update process, and the fact that you must not skip versions, eg must go 3.12, 3.13, 3.14 etc. and you can’t go from say 3.12 to 3.14.

It is hard to keep up with every release of foreman in the enterprise, I am grateful for the health of the project, but often we have to limit our changes and may be running one version while the project actually goes through a couple of releases, and then when we do get a chance to upgrade we have to step through all the versions one at a one all at once, which is a painful upgrade process.

I assume this is because of rails, and the data model changes that can happen each version via rails.

But it would be nice if you could update foreman with a simple yum update from any version to any version, and not have to also run rake commands, foreman-installer or anything else to patch/update your foreman installation.

[lzap]

I assume this is because of rails, and the data model changes that can happen each version via rails.

Yes, mostly. If we let this train to leave, we might not be able to catch up. Such a fast-paced project, both Ruby and Rails.

update foreman with a simple yum update

This will never happen, we are simply too complex setup and we actually also want to break things intentionally. This enables us to do some design changes or important decisions. For example dropping MySQL has been discussed and there’s a lot of content about it. This cannot be a “yum update”.