Puppet with splay=true

gvde · October 7, 2020, 8:25am

I have just noticed that the foreman-installer uses the splay setting from puppet.conf. I have splay=true set on all our servers including a server running a katello content proxy (only the main foreman/katello server has splay=false). I was wondering why the foreman-installer run on the proxy server wasn’t progressing until I have noticed that it is “hanging” on a puppet call. Due to the splay setting in puppet.conf it was sleeping.

I guess it would be good if foreman-installer would use the --no-splay option for all puppet runs…

Dirk · October 7, 2020, 9:06am

Perhaps this is a feature noone noticed yet as it came with puppet 4.2 that apply takes splay into account.

ekohl · October 7, 2020, 10:35am

Interesting and a good find. Which version of rubygem-kafo (or tfm-rubygem-kafo) is installed? I thought I had isolated it by using a separate conf file, but this would suggest it’s not.

gvde · October 7, 2020, 11:11am

tfm-rubygem-kafo-4.1.0-3.el7.noarch

ekohl · October 7, 2020, 2:44pm

That should include:

https://github.com/theforeman/kafo/commit/29999419d8ef5288c3a87708e3cbabd896422576

I wonder why the default puppet.conf still leaks.

If you want to always configure it, I think here would be the correct place to do so:

github.com

theforeman/kafo/blob/374a420b25dbdbad6e1e6b727e18e94e76576b13/lib/kafo/execution_environment.rb#L28-L43


      
          def configure_puppet(settings = {})
            @logger.debug("Configuring Puppet in #{directory}")
          
            @logger.debug("Writing facts to #{factpath}")
            FactWriter.write_facts(facts, factpath)
          
            hiera_config = configure_hiera
          
            settings = {
              'environmentpath' => environmentpath,
              'factpath'        => factpath,
              'hiera_config'    => hiera_config,
            }.merge(settings)
          
            PuppetConfigurer.new(puppet_conf, settings)
          end

Otherwise here:

github.com

theforeman/kafo/blob/374a420b25dbdbad6e1e6b727e18e94e76576b13/lib/kafo/puppet_command.rb#L9-L14


      
          if puppet_config
            puppet_config['basemodulepath'] = modules_path.join(':')
            @options = options.push("--config=#{puppet_config.config_path}")
          else
            @options = options.push("--modulepath #{modules_path.join(':')}")
          end

Perhaps in the latter it always needs to pass --config but point it to /dev/null?

gvde · October 7, 2020, 3:35pm

Just made a test run with splay set. First time it “hangs”, these processes are running:

root      79916  79886  4 17:32 pts/1    00:00:01 ruby /sbin/foreman-installer
root      80289  79916  0 17:32 pts/1    00:00:00 sh -c echo "package { ['crane-selinux']: ensure => installed }" | /opt/puppetlabs/bin/puppet apply --detailed-exitcodes
root      80294  80289  9 17:32 pts/1    00:00:01 /opt/puppetlabs/puppet/bin/ruby /opt/puppetlabs/puppet/bin/puppet apply --detailed-exitcodes

ekohl · October 7, 2020, 3:57pm

Aha! It’s here:

github.com

theforeman/foreman-installer/blob/20fc903858d643daecf95d794a97e53cb9c8115f/hooks/boot/01-kafo-hook-extensions.rb#L34-L37


def apply_puppet_code(code)
  bin_path = Kafo::PuppetCommand.search_puppet_path('puppet')
  Open3.capture3(*Kafo::PuppetCommand.format_command("echo \"#{code}\" | #{bin_path} apply --detailed-exitcodes"))
end

That doesn’t use the isolated execution environment so it ‘leaks’.

ekohl · October 7, 2020, 5:36pm

A 100% untested patch:
https://github.com/theforeman/foreman-installer/pull/584

If nobody beat me to it, I’ll try to test this tomorrow.

gvde · August 26, 2021, 5:07am

Sorry to bother: any chance to get this into the installer? I am still having this problem with my content proxy which has splay=true set in puppet.conf.

ekohl · August 31, 2021, 3:32pm

@ehelms what are your thoughts here? It may be easier (and safer) to always configure kafo’s Puppet run with splay=false. We never want that so we might as well be explicit about it. It’d be a safe cherry pick as well.