Foreman 3.11 Installer + Puppet 8 Client (not server) compatible?

Problem:
foreman-installer does not appear to be able to complete install with puppet-8 agent
Expected outcome:
foreman-installer completed with a puppet 8 server and puppet 8 client - not expected, testing, so prepared for ‘won’t work’
Foreman and Proxy versions:

Foreman and Proxy plugin versions:
3.11.1
Distribution and version:
RHEL9.4
Other relevant data:
I’d been testing a foreman 3.11.1 install with puppet 8 repos, with solid success. On moving to a clean machine to allow for tighter integration testing, I discovered my bootstrap for foreman 3.11.1 was setting up both puppet 8 and 7 repo’s and my testing that had been successful was with a puppet 8 server, but still using the puppet 7 agent. I’ve clean installed RHEL 9 and disabled the ability for my bootstrap to pull puppet 7 agent, so only puppet 8 is available, and straight away the foreman-installer failed.

2024-08-05 20:02:20 [NOTICE] [root] Loading installer configuration. This will take some time.
Could not get default values, check log file at /var/log/foreman-installer/foreman.log for more information
2024-08-05 20:02:21 [ERROR ] [root] echo '
        $kafo_config_file="/etc/foreman-installer/scenarios.d/foreman.yaml"
                kafo_configure::puppet_version_semver { "puppetlabs-apache":
          requirement => ">= 7.9.0 < 9.0.0",
        }

        kafo_configure::puppet_version_semver { "puppetlabs-apt":
          requirement => ">= 7.0.0 < 9.0.0",
        }

        kafo_configure::puppet_version_semver { "katello-candlepin":
          requirement => ">= 7.0.0 < 9.0.0",
        }

        kafo_configure::puppet_version_semver { "katello-certs":
          requirement => ">= 7.0.0 < 9.0.0",
        }

        kafo_configure::puppet_version_semver { "puppetlabs-concat":
          requirement => ">= 7.0.0 < 9.0.0",
        }

        kafo_configure::puppet_version_semver { "theforeman-dhcp":
          requirement => ">= 7.0.0 < 9.0.0",
        }

        kafo_configure::puppet_version_semver { "theforeman-dns":
          requirement => ">= 7.0.0 < 9.0.0",
        }

        kafo_configure::puppet_version_semver { "puppet-extlib":
          requirement => ">= 7.0.0 < 9.0.0",
        }

        kafo_configure::puppet_version_semver { "theforeman-foreman":
          requirement => ">= 7.0.0 < 9.0.0",
        }

        kafo_configure::puppet_version_semver { "theforeman-foreman_proxy":
          requirement => ">= 7.0.0 < 9.0.0",
        }

        kafo_configure::puppet_version_semver { "katello-foreman_proxy_content":
          requirement => ">= 7.0.0 < 9.0.0",
        }

        kafo_configure::puppet_version_semver { "puppetlabs-inifile":
          requirement => ">= 7.0.0 < 9.0.0",
        }

        kafo_configure::puppet_version_semver { "katello-katello":
          requirement => ">= 7.0.0 < 9.0.0",
        }

        kafo_configure::puppet_version_semver { "puppet-mosquitto":
          requirement => ">= 7.0.0 < 9.0.0",
        }

        kafo_configure::puppet_version_semver { "puppetlabs-postgresql":
          requirement => ">= 7.0.0 < 9.0.0",
        }

        kafo_configure::puppet_version_semver { "theforeman-pulpcore":
          requirement => ">= 7.0.0 < 9.0.0",
        }

        kafo_configure::puppet_version_semver { "theforeman-puppet":
          requirement => ">= 7.0.0 < 9.0.0",
        }

        kafo_configure::puppet_version_semver { "theforeman-puppetserver_foreman":
          requirement => ">= 7.0.0 < 9.0.0",
        }

        kafo_configure::puppet_version_semver { "puppet-redis":
          requirement => ">= 7.0.0 < 9.0.0",
        }

        kafo_configure::puppet_version_semver { "puppetlabs-stdlib":
          requirement => ">= 7.0.0 < 9.0.0",
        }

        kafo_configure::puppet_version_semver { "puppet-systemd":
          requirement => ">= 7.9.0 < 9.0.0",
        }

        kafo_configure::puppet_version_semver { "theforeman-tftp":
          requirement => ">= 7.0.0 < 9.0.0",
        }

        kafo_configure::puppet_version_semver { "puppet-trusted_ca":
          requirement => ">= 7.0.0 < 9.0.0",
        }

        kafo_configure::puppet_version_semver { "puppetlabs-vcsrepo":
          requirement => ">= 7.0.0 < 9.0.0",
        }

        kafo_configure::puppet_version_semver { "theforeman-kafo_configure":
          requirement => ">= 4.5.0 < 9.0.0",
        }

                  include foreman::params include foreman::cli::params include foreman_proxy::params include puppet::params include foreman_proxy::plugin::ansible::params include foreman_proxy::plugin::discovery::params include foreman_proxy::plugin::dynflow::params include foreman_proxy::plugin::salt::params
          class { '::kafo_configure::dump_values':
            lookups   => ["apache::mod::status::requires","foreman::unattended_url","foreman::db_host","foreman::db_port","foreman::db_sslmode","foreman::db_root_cert","foreman::db_pool","foreman::server_ssl_protocol","foreman::initial_admin_first_name","foreman::initial_admin_last_name","foreman::initial_admin_email","foreman::initial_admin_locale","foreman::initial_admin_timezone","foreman::initial_organization","foreman::initial_location","foreman::http_keytab","foreman::ipa_sssd_default_realm","foreman::websockets_ssl_key","foreman::websockets_ssl_cert","foreman::logging_layout","foreman::email_delivery_method","foreman::email_sendmail_location","foreman::email_sendmail_arguments","foreman::email_smtp_address","foreman::email_smtp_domain","foreman::email_smtp_user_name","foreman::email_smtp_password","foreman::email_reply_address","foreman::email_subject_prefix","foreman::dynflow_redis_url","foreman::foreman_service_puma_threads_min","foreman::foreman_service_puma_workers","foreman::provisioning_ct_location","foreman::provisioning_fcct_location","foreman::cli::foreman_url","foreman::cli::username","foreman::cli::password","foreman::cli::ssl_ca_file","foreman_proxy::foreman_ssl_ca","foreman_proxy::foreman_ssl_cert","foreman_proxy::foreman_ssl_key","foreman_proxy::puppetca_certificate","foreman_proxy::tftp_dirs","foreman_proxy::tftp_servername","foreman_proxy::dhcp_search_domains","foreman_proxy::dhcp_gateway","foreman_proxy::dhcp_range","foreman_proxy::dhcp_pxeserver","foreman_proxy::dhcp_ipxefilename","foreman_proxy::dhcp_network","foreman_proxy::dhcp_netmask","foreman_proxy::dhcp_key_name","foreman_proxy::dhcp_key_secret","foreman_proxy::dhcp_peer_address","foreman_proxy::dns_reverse","foreman_proxy::registered_proxy_url","foreman_proxy::registration_url","puppet::run_hour","puppet::run_minute","puppet::server_jvm_java_bin","puppet::server_puppetserver_telemetry","puppet::server_acceptor_threads","puppet::server_selector_threads","puppet::server_ssl_acceptor_threads","puppet::server_ssl_selector_threads","puppet::server_max_threads","puppet::server_versioned_code_id","puppet::server_versioned_code_content","foreman::plugin::remote_execution::cockpit::ensure","foreman_proxy::plugin::acd::version","foreman_proxy::plugin::dhcp::infoblox::username","foreman_proxy::plugin::dhcp::infoblox::password","foreman_proxy::plugin::dhcp::remote_isc::key_name","foreman_proxy::plugin::dhcp::remote_isc::key_secret","foreman_proxy::plugin::discovery::version","foreman_proxy::plugin::dns::infoblox::dns_server","foreman_proxy::plugin::dns::infoblox::username","foreman_proxy::plugin::dns::infoblox::password","foreman_proxy::plugin::dns::powerdns::rest_api_key","foreman_proxy::plugin::dns::route53::aws_access_key","foreman_proxy::plugin::dns::route53::aws_secret_key","foreman_proxy::plugin::monitoring::version","foreman_proxy::plugin::omaha::http_proxy","foreman_proxy::plugin::omaha::version","foreman_proxy::plugin::omaha::distribution","foreman_proxy::plugin::openscap::version","foreman_proxy::plugin::openscap::proxy_name","foreman_proxy::plugin::remote_execution::script::ssh_log_level","foreman_proxy::plugin::remote_execution::script::mqtt_ttl","foreman_proxy::plugin::remote_execution::script::mqtt_rate_limit","foreman_proxy::plugin::remote_execution::script::mqtt_resend_interval","foreman_proxy::plugin::salt::group","foreman_proxy::plugin::shellhooks::version"],
            variables => ["foreman::params::foreman_url","foreman::params::servername","foreman::params::db_password","foreman::params::server_ssl_ca","foreman::params::server_ssl_chain","foreman::params::server_ssl_cert","foreman::params::server_ssl_key","foreman::params::server_ssl_crl","foreman::params::client_ssl_ca","foreman::params::client_ssl_cert","foreman::params::client_ssl_key","foreman::params::oauth_consumer_key","foreman::params::oauth_consumer_secret","foreman::params::oauth_effective_user","foreman::params::initial_admin_password","foreman_proxy::params::ssl_ca","foreman_proxy::params::ssl_cert","foreman_proxy::params::ssl_key","foreman_proxy::params::trusted_hosts","foreman_proxy::params::ssldir","foreman_proxy::params::puppetdir","foreman_proxy::params::autosignfile","foreman_proxy::params::puppet_url","foreman_proxy::params::ssl_ca","foreman_proxy::params::ssl_cert","foreman_proxy::params::ssl_key","foreman_proxy::params::template_url","foreman_proxy::params::tftp_root","foreman_proxy::params::dhcp_option_domain","foreman_proxy::params::dhcp_interface","foreman_proxy::params::dhcp_config","foreman_proxy::params::dhcp_leases","foreman_proxy::params::dhcp_failover_address","foreman_proxy::params::dhcp_manage_acls","foreman_proxy::params::dns_interface","foreman_proxy::params::dns_zone","foreman_proxy::params::dns_tsig_keytab","foreman_proxy::params::dns_tsig_principal","foreman_proxy::params::realm_keytab","foreman_proxy::params::realm_principal","foreman_proxy::params::keyfile","foreman_proxy::params::foreman_base_url","foreman_proxy::params::registered_name","foreman_proxy::params::oauth_consumer_key","foreman_proxy::params::oauth_consumer_secret","puppet::params::version","puppet::params::user","puppet::params::group","puppet::params::dir","puppet::params::codedir","puppet::params::vardir","puppet::params::logdir","puppet::params::rundir","puppet::params::ssldir","puppet::params::sharedir","puppet::params::manage_packages","puppet::params::dir_owner","puppet::params::dir_group","puppet::params::package_provider","puppet::params::package_install_options","puppet::params::package_source","puppet::params::agent_server_port","puppet::params::splay","puppet::params::splaylimit","puppet::params::autosign","puppet::params::autosign_entries","puppet::params::autosign_mode","puppet::params::autosign_content","puppet::params::autosign_source","puppet::params::runinterval","puppet::params::usecacheonfailure","puppet::params::runmode","puppet::params::unavailable_runmodes","puppet::params::cron_cmd","puppet::params::systemd_cmd","puppet::params::systemd_randomizeddelaysec","puppet::params::agent_noop","puppet::params::agent_default_schedules","puppet::params::show_diff","puppet::params::module_repository","puppet::params::http_connect_timeout","puppet::params::http_read_timeout","puppet::params::ca_server","puppet::params::ca_port","puppet::params::ca_crl_filepath","puppet::params::certificate_revocation","puppet::params::prerun_command","puppet::params::postrun_command","puppet::params::dns_alt_names","puppet::params::use_srv_records","puppet::params::srv_domain","puppet::params::pluginsource","puppet::params::pluginfactsource","puppet::params::additional_settings","puppet::params::agent_additional_settings","puppet::params::agent_restart_command","puppet::params::classfile","puppet::params::hiera_config","puppet::params::localconfig","puppet::params::allow_any_crl_auth","puppet::params::auth_allowed","puppet::params::client_package","puppet::params::agent","puppet::params::report","puppet::params::client_certname","puppet::params::agent_server_hostname","puppet::params::systemd_unit_name","puppet::params::service_name","puppet::params::syslogfacility","puppet::params::environment","puppet::params::server","puppet::params::server_admin_api_allowlist","puppet::params::manage_user","puppet::params::user","puppet::params::group","puppet::params::dir","puppet::params::ip","puppet::params::agent_server_port","puppet::params::server_ca","puppet::params::server_ca_crl_sync","puppet::params::server_crl_enable","puppet::params::server_ca_auth_required","puppet::params::server_ca_client_self_delete","puppet::params::server_ca_client_allowlist","puppet::params::server_custom_trusted_oid_mapping","puppet::params::server_http","puppet::params::server_http_port","puppet::params::server_reports","puppet::params::server_puppetserver_dir","puppet::params::server_puppetserver_vardir","puppet::params::server_puppetserver_rundir","puppet::params::server_puppetserver_logdir","puppet::params::server_puppetserver_version","puppet::params::server_external_nodes","puppet::params::server_trusted_external_command","puppet::params::server_cipher_suites","puppet::params::server_connect_timeout","puppet::params::server_git_repo","puppet::params::server_default_manifest","puppet::params::server_default_manifest_path","puppet::params::server_default_manifest_content","puppet::params::server_environments_owner","puppet::params::server_environments_group","puppet::params::server_environments_mode","puppet::params::server_environments_recurse","puppet::params::server_envs_dir","puppet::params::server_envs_target","puppet::params::server_common_modules_path","puppet::params::server_git_repo_hook_mode","puppet::params::server_git_repo_path","puppet::params::server_git_repo_umask","puppet::params::server_git_repo_group","puppet::params::server_git_repo_user","puppet::params::server_git_branch_map","puppet::params::server_idle_timeout","puppet::params::server_post_hook_content","puppet::params::server_post_hook_name","puppet::params::server_storeconfigs","puppet::params::server_ruby_load_paths","puppet::params::server_ssl_dir","puppet::params::server_ssl_dir_manage","puppet::params::server_ssl_key_manage","puppet::params::server_ssl_protocols","puppet::params::server_ssl_chain_filepath","puppet::params::server_package","puppet::params::server_version","puppet::params::server_certname","puppet::params::server_request_timeout","puppet::params::server_strict_variables","puppet::params::server_additional_settings","puppet::params::server_foreman","puppet::params::server_foreman_url","puppet::params::server_foreman_ssl_ca","puppet::params::server_foreman_ssl_cert","puppet::params::server_foreman_ssl_key","puppet::params::server_foreman_facts","puppet::params::server_puppet_basedir","puppet::params::server_parser","puppet::params::server_environment_timeout","puppet::params::server_jvm_config","puppet::params::server_jvm_min_heap_size","puppet::params::server_jvm_max_heap_size","puppet::params::server_jvm_extra_args","puppet::params::server_jvm_cli_args","puppet::params::server_jruby_gem_home","puppet::params::server_environment_vars","puppet::params::server_max_active_instances","puppet::params::server_max_requests_per_instance","puppet::params::server_max_queued_requests","puppet::params::server_max_retry_delay","puppet::params::server_multithreaded","puppet::params::server_check_for_updates","puppet::params::server_environment_class_cache_enabled","puppet::params::server_allow_header_cert_info","puppet::params::server_web_idle_timeout","puppet::params::server_metrics_jmx_enable","puppet::params::server_metrics_graphite_enable","puppet::params::server_metrics_graphite_host","puppet::params::server_metrics_graphite_port","puppet::params::server_metrics_server_id","puppet::params::server_metrics_graphite_interval","puppet::params::server_metrics_allowed","puppet::params::server_puppetserver_experimental","puppet::params::server_puppetserver_auth_template","puppet::params::server_puppetserver_trusted_agents","puppet::params::server_puppetserver_trusted_certificate_extensions","puppet::params::server_compile_mode","puppet::params::server_ca_allow_sans","puppet::params::server_ca_allow_auth_extensions","puppet::params::server_ca_enable_infra_crl","puppet::params::server_ca_allow_auto_renewal","puppet::params::server_ca_allow_auto_renewal_cert_ttl","puppet::params::server_max_open_files","puppet::params::puppetconf_mode","foreman::params::client_ssl_ca","foreman::params::client_ssl_cert","foreman::params::client_ssl_key","foreman_proxy::plugin::ansible::params::enabled","foreman_proxy::plugin::ansible::params::listen_on","foreman_proxy::plugin::ansible::params::ansible_dir","foreman_proxy::plugin::ansible::params::working_dir","foreman_proxy::plugin::ansible::params::host_key_checking","foreman_proxy::plugin::ansible::params::roles_path","foreman_proxy::plugin::ansible::params::ssh_args","foreman_proxy::plugin::ansible::params::install_runner","foreman_proxy::plugin::ansible::params::callback","foreman_proxy::plugin::ansible::params::runner_package_name","foreman_proxy::plugin::ansible::params::collections_paths","foreman_proxy::plugin::discovery::params::install_images","foreman_proxy::plugin::discovery::params::tftp_root","foreman_proxy::plugin::discovery::params::source_url","foreman_proxy::plugin::discovery::params::image_name","foreman_proxy::plugin::dynflow::params::enabled","foreman_proxy::plugin::dynflow::params::listen_on","foreman_proxy::plugin::dynflow::params::database_path","foreman_proxy::plugin::dynflow::params::console_auth","foreman_proxy::plugin::dynflow::params::ssl_disabled_ciphers","foreman_proxy::plugin::dynflow::params::tls_disabled_versions","foreman_proxy::plugin::dynflow::params::open_file_limit","foreman_proxy::plugin::salt::params::autosign_file","foreman_proxy::plugin::salt::params::autosign_key_file","foreman_proxy::plugin::salt::params::enabled","foreman_proxy::plugin::salt::params::listen_on","foreman_proxy::plugin::salt::params::user","foreman_proxy::plugin::salt::params::api","foreman_proxy::plugin::salt::params::api_url","foreman_proxy::plugin::salt::params::api_auth","foreman_proxy::plugin::salt::params::api_username","foreman_proxy::plugin::salt::params::api_password","foreman_proxy::plugin::salt::params::saltfile"],
          }

      ' | RUBYLIB=/usr/share/gems/gems/kafo-7.4.0/lib/kafo/../..//modules: /opt/puppetlabs/bin/puppet apply --config=/tmp/kafo_installation20240805-27974-vhr4ec/puppet.conf 
2024-08-05 20:02:21 [ERROR ] [root] Error: Could not create resources for managing Puppet's files and directories in sections [:main, :agent, :ssl]: Cannot get default context with nil handle
Error: Could not prepare for execution: Could not create resources for managing Puppet's files and directories in sections [:main, :agent, :ssl]: Cannot get default context with nil handle
Cannot get default context with nil handle

2024-08-05 20:02:21 [ERROR ] [root] Could not get default values, cannot continue

looking at the output the depends are greater than puppet 7 and less than puppet 9 so all should be satifed at a basic level, however the kafo gem config (I can’t seem to find this when the install exits) seems to not like puppet 8 doing a puppet apply against it’s installer generated config.

input to puppet 8 compatibility at a client level would be apprecaoted.

I think we can consider this a duplicate of Puppet 8 support.

sorry, I’d been following that thread and some of the issues raised on it, and most seemed in a good shape, but where specific to the puppet-server component. I asked in a separate thread for two reasons

a.) I didn’t see much talk about the puppet agent / client being supported/unsupported, but I do appreciate there is a level of implication that comes with ‘puppet 8 support’

b.) reading through the bugs, it looks like it’s mostly working for the server part, yet, when I discovered my mistake (using puppet 7 agent during the installer) and swapped to puppet 8 agent, it failed straight away

This made me question if the support of puppet 8’s agent had been considered and tested and how all the work that had gone into puppet 8 support so far had got so far, to the point of mostly supported if it’s failing at such an early stage of the installer when you do use the puppet 8 agent as part of the installer.