RFC: Drop reliance on puppet-agent AIO for foreman-installer

Development RFCs
1

The foreman-installer package today, through the kafo library, defaults to relying on the puppet-agent AIO package to handle parsing parameters and running puppet apply. This enforces a heavy reliance on production of the puppet-agent AIO package by Puppetlabs and leads to Ruby environment issues if attempting to use an SCL for newer Ruby versions.

Proposal:

  • Package puppet gem for SCL on EL7
  • Switch installer to use puppet from RPM install or gem in Debian case

This would enable:

  • Drops reliance on puppet-agent packaging for newer versions of puppet, and for OSes that there is no puppet-agent package for (e.g. Z systems)
  • Simplifies installer to rely on packages we package only
  • Allows moving installer to SCL on EL7 systems and use newer Ruby

Some Technical

The current design of the installer relies on kafo via the kafo_parsers project to determine the puppet bin path to use when parsing parameters or running puppet apply. In particular, this bin path calculation checks for existence of the AIO puppet package and then defaults to system puppet.

  • Couldn’t we just install and use system puppet then?

If Puppetserver is being used on the same box as Foreman or a Smart Proxy, the current installer would default to using the installed puppet-agent package instead of the intended puppet gem install. This could lead to differences in versions of Puppet used by the installer and generate bug reports.

  • Can we just always use system puppet for a minimum version of puppet?

On Debian this might would work, but on EL7 the system Ruby is 2.0 which is too old for newer Puppet. The Puppet gem would need to be packaged for an SCL Ruby the same as the installer, Foreman, and the Proxy.

  • Sounds simple enough, what’s the rub?

SCLs create a context that must be enabled to ensure that the pathing is correct for a given binfile. Further, the way kafo_parsers and kafo use puppet as an executable to shell out to requires that simple puppet cannot be used. Similar to how we have to handle commands like rake on a Foreman install on EL7, we need a wrapper on El7. The wrapper would like something like:

#!/bin/bash

source scl_source enable tfm
exec puppet "$@"

This could be named and deployed as kafo-puppet or foreman-installer-puppet and relied upon. This would, however, create a difference between EL7 and Debian in terms of the executable being relied upon. We ought to be able to detect this and have fall backs such that this can work in all environments.

Adding a link to some technical implementation I used for testing this strategy could actually work. This is a link to an RPM packaging PR and some kafo updates to choose this path if its available:

https://github.com/theforeman/foreman-packaging/pull/4614

2

I have a practical question concerning this:
Afaik, the installer sets certain Puppet related things according to the Puppet version used for the installer. I have seen some support requests on the forum where people used different puppetserver and puppetclient versions which led to very obscure errors like erroneous DB seeding (from what I understood).
Wouldn’t packaging Puppet with the installer lead to the exact same problems every time someone uses a different puppetserver version than the one shipped with the installer? I know there is an installer option to force the settings for a specific puppetserver version, but that one is currently buried in the “–full-help” part of the installer. If I am correct, this would probably need to become a “common setting” and we need to explicitly outline that in installation documentation.

1 Like
3

Would this mean I could easily install/register my Foreman or Smart Proxy to Puppet Enterprise without a conflicting puppet-agent? (Puppet Enterprise ships its own puppet-agent RPM) If so, that’s exciting

4

The idea with this proposal is that the puppet used by the installer would be isolated and in no way affect the puppet used by the Puppetserver. If installing Puppetserver, puppet-agent AIO would be installed and used. The puppet gem packaged for use by the installer would not deploy the puppet bin file system wide.

5

Correct. The proposal is intended to let puppet-agent be installed and handle being the puppet agent for the Foreman or Smart Proxy and the packaged puppet gem would be present only to support runs of foreman-installer.

6

I got this, and to be clear: I am all in favor of the change if it makes developing and maintaining the installer easier.
My only point is to raise awareness that this change might have implications for the way the targeted puppet version is determined. I don’t have enough knowledge about the inner workings of the installer (or probably kafo in this case) to know for sure. I would be happy to learn that I am wrong :wink:

7

@mmoll Is this strategy more complicated on Debian due to the way SCLs on EL7 isolate environments from the system bin files ?

I think EL8 would introduce a similar need whereby the puppet bin from the packaged gems would need to be available at a known path to Kafo but not available via the standard system paths.

8

My initial feeling is that I don’t like it. Let me be clear, I don’t like the bundled AIO install from puppet-agent but I dislike the maintenance burden more. My suspicion is also that (as @areyus pointed out) that it will be more complex to deploy a Puppetserver since we can no longer count on coupling of versions (in itself flawed, but good enough).

It would hugely increase the maintenance burden we have for systems where puppet-agent is available. For better or worse, a lot of users have started to rely on paths in /opt/puppetlabs, even if it’s just for debugging. This increases the chances of bugs due to our modified environment. Since puppet 6, various modules that used to be bundled are now standalone. For reference, from puppet-agent 6.12.0:

Contains the following components:
cleanup
cpp-hocon 0.2.0
cpp-pcp-client 1.6.1
dmidecode 2.12
facter 3.14.7
facter-ng 0.0.10
hiera 3.6.0
leatherman 1.10.0
libwhereami 0.4.0
module-puppetlabs-augeas_core 1.0.5
module-puppetlabs-cron_core 1.0.3
module-puppetlabs-host_core 1.0.3
module-puppetlabs-mount_core 1.0.4
module-puppetlabs-scheduled_task 1.0.0
module-puppetlabs-selinux_core 1.0.4
module-puppetlabs-sshkeys_core 1.0.3
module-puppetlabs-yumrepo_core 1.0.6
module-puppetlabs-zfs_core 1.0.4
module-puppetlabs-zone_core 1.0.3
puppet 6.12.0
puppet-resource_api 1.8.11
puppet-runtime 202001090
pxp-agent 1.15.1
runtime
shellpath 2015-09-18
virt-what 1.18
wrapper-script

I have also seen Arch users (who do split it up) complain that running Puppet on the latest rubies doesn’t always work. This means that Foreman can’t upgrade to the latest Ruby version unless Puppet is also updated. Right now we’re isolated from that.

Providing a different set also means we must make sure our installer modules work with the combinations of software we provide. That includes third party modules that may not test with the newest Ruby versions. In the community we’ve seen some combinations where there’s a Facter 2 with a Puppet 4 (Debian did this) or Facter 2 with Puppet 5 (Solaris) and suddenly things break in very subtle ways.

We already have isolation between the two environments and if anything doesn’t work due to the installer itself running in SCL, that’s a bug. We call /opt/puppetlabs/puppet/bin/puppet and that should work. If the SCL environment leaks into it and breaks things, I’m sure there’s better ways to call the binary for proper isolation.

I never looked at this, but I can’t see a reason why this doesn’t work today. Perhaps the package names are different. In which case foreman-installer should depend on something common. Anyone with a Puppet Enterprise that can see what the package provides? For reference, the open source shows this:

$ rpm -q --provides puppet-agent
puppet >= 4.0.0
facter >= 1:3.0.0
cfacter >= 0.5.0
hiera >= 2.0.0
puppet-agent = 6.12.0-1.el7
puppet-agent(x86-64) = 6.12.0-1.el7
1 Like
9

This part I have still not grasped due to lack of knowledge in this area. If this SCL puppet is isolated from the system, why would it affect a Puppetserver installation?

You will have to educate me more here. Right now this adds 12 gems to our build process which doesn’t feel like a lot of burden. I get that Facter would add more due to its more complex nature, Fedora does package this and it’s dependencies so we have something to look at.

How do you mean? When using just the installer users are digging into /opt/puppetlabs ?

I read this as adding an increase to our need to declare our dependencies explicitly and include them in the installer. This does add overhead but allows makes things clearer which isn’t necessarily a bad thing.

I would think this means that the foreman-installer can’t upgrade to the latest Ruby, not Foreman. Last I checked, we live in this world already :slight_smile:

Admittedly this one confuses me due to lack of knowledge. If I look at puppet-foreman, this is being tested against the puppet gem already. I would think that this would align us closer then.

Check out this comment RFC: Moving Foreman Proxy, Installer and NodeJS to SCLs - #10 by ehelms

10

I meant to also ask. Are there any alternative solutions folks can think to support operating systems for which puppet-agent does not exist but that the community would like to support installable Foreman on? For example, the community members that want to build Foreman for Z systems.

11

Where does this connection come from? I thought the point of the whole Puppet 4 work was to remove the dependency onto a Puppet that’s required to be in the same Ruby environment as the installer, so I’d expect that SCLing would already be possible today.

I hope the only dependency at the moment would be any Puppet. At least I think with the native Puppet 5.x package of Debian it was possible to run the installer without problems, only the fact that no native Java Puppetserver is available on Debian made it impossible to have Puppetmaster install on such a box.

For RHEL/CentOS on such platforms there can be a SCL Puppet and if that’s required, the installer can probably be worked on to supported such installs.

If the dependency will be that the installer and Puppet will need to run in the same Ruby environment, we’d need to create some special foreman-puppet DEB, which will be like the native Debian package, but with the binary paths of puppet, facter, hiera etc. not included, to not clash with the puppet-agent package, which still would be kind of required. This is IMHO doable, but not my favorite solution.

12

The problem is the following: Currently, when the installer is called, it generates all the answers for the not explicitly set parameters based on a set of rules. Most of these are static or only OS dependent, but that is not true for all of them. The decision of the Puppetserver version that should be installed and configured by foreman-installer is based on the version of Puppet that runs the installer.
Let me give an example of the workflows: A user want’s to setup a Foreman + Puppet 5 environment.
Currently, the workflow is: Install puppet-agent 5, install Foreman RPMs, run foreman-installer. The user now has a Foreman instance + working Puppet 5 server matching their puppet-agent version.
If you use this workflow after implementing your suggestion, this would result in them having a Foreman instance, a Puppet 5 client and a Puppetserver matching the version of whatever Puppet gem we would ship with the Foreman installer. This would require users to always explicitly give the desired version as an argument to the installer, and tbh, figuring out which parameters you need to hand to the installer is already more then enough trouble for everyone that does not do this on a regular basis (so, most users).

I have no experience with installing Puppet on platforms with no officially built puppet-agent, but could those platforms not simply install the puppet gems instead? That would only require a change in the install docs, where those platforms just do “gem install puppet” (or whatever) instead of “yum/apt install puppet-agent”. Maybe packaging the gems for only those platforms could be an option, but I don’t know the implications on the build process and maintainability of packaging etc.

13

To be more explicit: if no explicit puppetserver version is passed in, we use the puppetversion fact to guess what it is. This is because puppetserver is rather particular about which options it accepts and fails to start if you set an option it doesn’t know about.

As @areyus said: when we ship our own Puppet 6.x and the user actually wants Puppetserver 5.x, this logic no longer works and we need to come up with a new solution.