Boltello - requesting feedback

gerard.ryan · October 7, 2020, 1:15pm

Hello All,

I hope this post finds readers in the Foreman community healthy and well. These are challenging times, indeed.

I’m Gerard Ryan. As a long time community lurker – and occasional question-asker in IRC – I’d like to formally introduce myself. I’ve been developing with Puppet for 9 years and have followed Foreman development for nearly as long.

My post today announces a project I’ve tinkered wth over the years called “boltello”.

Boltello started out as a simple BASH script which used a collection of puppet modules to build the Master of Masters and shipped tarballs of proxy certificates around with SSH. Once certificates were generated and distributed, the script would then bootstrap each compile master to the MoM to receive its configuration. That solution developed into its own puppet module over time and when Bolt was first announced I realized I could accomplish more powerful and more controlled orchestration with ease. Bolt is agent-less, yet natively understands Puppet – it seemed like a great match.

I’m seeking community feedback on this project in its current state and any suggestions or critiques would be helpful!

Thanks for reading my post!

My best,
Gerard

mcorr · October 7, 2020, 1:38pm

Hi Gerard,
First of all, welcome to the community.
We have a community demo tomorrow, would you like to come along and demo this / talk about this for a few minutes? Foreman Community Demo #85

gerard.ryan · October 7, 2020, 2:21pm

Hi and yes!
How do I join?

mcorr · October 7, 2020, 2:23pm

Excellent!

Hit the edit button and add yourself to the demo:

gerard.ryan · October 7, 2020, 2:28pm

New user, not sure if I’m able to edit that post… however, I’ll definitely be present!

mcorr · October 7, 2020, 2:30pm

Thanks @gerard.ryan I will add you to the agenda and DM you a meeting link in 2 mins.

wbclark · October 8, 2020, 4:03pm

Thanks for the demo today @gerard.ryan ! I enjoyed seeing what you have done with boltello, it’s a neat project and powerful tool for sure.

gerard.ryan · October 9, 2020, 8:11am

Thank you WB, that’s encouraging!

ekohl · October 12, 2020, 10:06am

@gerard.ryan thanks for that. I only just watched the recording, but it looks very nice.

This does fill a gap that we’ve never been able to fill. I’ve talked about this in some presentations, but I’ll repeat it here. In theory all the installer modules present the basic module layer and the foreman-installer fills the role layer. Then the answers define the profile layer. What you’ve done here, is essentially write a different profile + role layer. This is very much what I like to see and something I often encourage. Using the modules directly is IMHO a supported use case.

Scaling the PuppetCA that way is nice and also how I used to do it back in the day when I still ran production environments (something I still miss because it’s super useful to know as an installer maintainer). IMHO it should be the default, but I never had time to implement it. I see you’ve used nginx but Apache is already present on Capsules. Any thoughts on using that? I also assume (didn’t check it) that you enable the PuppetCA feature on the Smart Proxy. Does that still work with signing?

I’ve opened a few issues and PRs in the repo itself, but I also would like to ask you what we can do in the modules themselves to better support this.

For what it’s worth: puppet-katello and puppet-foreman_proxy_content are being refactored to slowly allow more modular installations. It’s still a long term goal for me to be able to deploy Candlepin, Foreman+Katello and Pulp all on different servers to allow independent scaling. I’d also very much like to get your opinion on that.

gerard.ryan · October 14, 2020, 5:30pm

Hi ekohl, thanks for your thoughtful and insightful note.

I agree that nginx is superfluous with all the apache in the stack and I’ve got a little research and testing to do in order to port the nginx reverse proxy to apache… but it’s straightforward enough.

As far as the abstractions of role/profile, I was really hoping the developer team would see value in my approach. The core strength of the foreman-installer is the collection of puppet modules that comprise the installer… Don’t get me wrong, Kafo is amazing; however, there’s an enormous gap between:

the published “scenarios”
the atomic controls available on the command line
the exposed puppet class parameters and attendant hiera data available via custom-hiera.yaml

Exposing the complete Hiera backend by using “fall-through/override” configurations (e.g., “profiles”) offers lots of benefits for me as an end user as I get more flexibility in how I deliver katello. I can also preview new releases in nonprod before rolling changes out to prod… complete with any tweaks changes.

One issue I’ve had is capturing a complete and accurate metadata dump from the installer. I’ve found that I cannot entirely rely on the log dump to /var/log/foreman-installer/{katello,foreman-proxy-content}.log as some of the data is incorrect (certs module: ‘generate’ boolean; foreman_proxy_content module: pulp resources for pulpnode, as a couple examples).

Despite that challenge, I’ve created a vanilla “capsule” role, which overrides the default roles/proxy.yaml file in boltello. This “profile” delivers a proxy server just as it would be configured by foreman-installer. I thought it would be helpful in two ways: it provides an opportunity to mix up the deployment toplogy, as well as providing a comparative example for creating additional profiles, if anyone wants to do that

PuppetCA feature on the Smart Proxy. Does that still work with signing?

I flag that feature off in Hiera and allow the Master of Masters to sign certificates natively. The ca requests are proxied back to the MoM and all other puppet traffic managed by the compile master. I initially tried to make this work, but got frustrated with certificate issues and dropped it. I have always wanted to make this work.

Thanks for the Apache suggestion; I am actively working on it.

gerard.ryan · October 14, 2020, 6:52pm

Another use case I’ve wanted to tackle is standalone PostGreSQL.
Implementing it as a role should be easy.

ekohl · October 27, 2020, 10:52am

Agreed.

Yes, the challenge is to find a balance between being able to completely tweak all details and keeping it simple for new users. As you’ve discovered, the path we’ve taken is to make the modules very complete, but hide some options via Hiera. For this reason alone I consider the use of our modules outside our installer a supported use case.

gerard.ryan · October 30, 2020, 10:52am

I’ve added a Vagrantfile to the project to make it easier to spin up katello resources.

Encountering a Tomcat issue where candlepin throws a port 8443 error, using katello 3.17/foreman 2.2. it’s resolved with an additional puppet run; however it is unclear why.

ekohl · October 30, 2020, 11:37am

My guess would be SELinux due to https://tickets.puppetlabs.com/browse/PUP-10548. The short summary is that Puppet calculates the SELinux contexts before the main run. It doesn’t see any contexts that were installed via packages during the run. See our hook for a workaround.

github.com

theforeman/foreman-installer/blob/878459361b92fa00a8d954304a08f09faeb03bd6/hooks/pre/32-install_selinux_packages.rb

# Working around https://tickets.puppetlabs.com/browse/PUP-10548
if facts.dig(:os, :selinux, :enabled)
  packages = []

  if el7?
    packages << 'rh-postgresql12-postgresql-server' if local_postgresql?
    packages << 'rh-redis5-redis' if local_redis?
  end

  packages << 'foreman-selinux' if foreman_server?
  packages << 'katello-selinux' if katello_enabled?
  packages << 'candlepin-selinux' if katello_enabled?
  packages << 'pulpcore-selinux' if pulpcore_enabled?
  packages << 'crane-selinux' if pulp_enabled?

  ensure_packages(packages, 'installed')
end

Various conditionals are defined here:

github.com

theforeman/foreman-installer/blob/878459361b92fa00a8d954304a08f09faeb03bd6/hooks/boot/01-kafo-hook-extensions.rb#L44-L90


def foreman_server?
  module_enabled?('foreman')
end


def katello_enabled?
  module_enabled?('katello')
end


def katello_present?
  module_present?('katello')
end


def devel_scenario?
  module_enabled?('katello_devel')
end


def local_foreman_db?
  foreman_server? && param_value('foreman', 'db_manage')
end

This file has been truncated. show original

gerard.ryan · October 30, 2020, 12:23pm

I haven’t looked too deep into the ticket but would deferred functions in Puppet6 resolve this?

Example in use:

github.com

superfantasticawesome/boltello/blob/7cd2e61cf7eaaa9e5594a44da496487cc9fc9be8/modules/boltello_builder/manifests/versionlock.pp#L33


Concat::Fragment {
  target => "${target}",
  notify => Exec['refresh_yum']
}


concat::fragment { 'versionlock_header':
  content => "# Managed by Puppet\n",
  order   => -1,
}


# Deferred values: https://puppet.com/docs/puppet/6.19/deferring_functions.html
# Wish I had a nickel for every Deferred()...
# Not actually useful here as this subclass is only invoked 
# by apply() on localhost... still cool, right?
#
$versionlock_packages.each |Tuple $package_data| {
  Hash($package_data).each |String $package, String $lock_level| {
    $versionlock_string_ = Deferred('boltello_builder::compute_lock_pattern', ["$package", "$lock_level"])
    $versionlock_string = $versionlock_string_.call()
    unless $versionlock_string.empty() {
      $versionlock = {

ekohl · October 30, 2020, 12:31pm

No, it’s really a bug in the file type.

gerard.ryan · November 19, 2020, 1:08pm

Remote databases capability added. Currently installs postgresql and mongodb on one instance; however, I will likely add the ability to split these roles out to separate instances. I believe pulp will dropping mongodb in the near future and migrating wholly to postgresql; but I have not run database system/services under any significant load to determine if it’s really necessary.

ekohl · November 20, 2020, 10:18am

Nice! It’s still on my agenda to properly support a remote database server as a scenario in the regular installer.

gerard.ryan · November 20, 2020, 10:35pm

Starting the following service(s):
rh-mongodb34-mongod, postgresql (foreman), postgresql (candlepin), qdrouterd, qpidd, rh-redis5-redis, squid, pulp_celerybeat, pulp_resource_manager, pulp_streamer, pulp_workers, smart_proxy_dynflow_core, tomcat, dynflow-sidekiq@orchestrator, foreman, httpd, puppetserver, dynflow-sidekiq@worker, dynflow-sidekiq@worker-hosts-queue, foreman-proxy
\ starting rh-mongodb34-mongod
rh-mongodb34-mongod is remote and is UP.
| starting postgresql (foreman)
postgresql (foreman) is remote and is UP.
/ starting postgresql (candlepin)
postgresql (candlepin) is remote and is UP