Upgrade to Foreman 2.1 - Katello 3.16 rc3

Support
,
1

Problem:

Foreman Installer Fails to run

Expected outcome:

Foreman installer completes the upgrade task

Foreman and Proxy versions:

Foreman 2.1
Katello 3.16rc3

Distribution and version:

RHEL 7.8

Other relevant data:

[ERROR 2020-07-06T20:27:33 main] Ignoring bcrypt-3.1.12 because its extensions are not built. Try: gem pristine bcrypt --version 3.1.12
Ignoring ffi-1.12.2 because its extensions are not built. Try: gem pristine ffi --version 1.12.2
Ignoring nio4r-2.5.2 because its extensions are not built. Try: gem pristine nio4r --version 2.5.2
Ignoring nokogiri-1.10.9 because its extensions are not built. Try: gem pristine nokogiri --version 1.10.9
Ignoring passenger-4.0.18 because its extensions are not built. Try: gem pristine passenger --version 4.0.18
Ignoring pg-1.1.4 because its extensions are not built. Try: gem pristine pg --version 1.1.4
Ignoring qpid_messaging-1.38.0 because its extensions are not built. Try: gem pristine qpid_messaging --version 1.38.0
Ignoring rkerberos-0.1.5 because its extensions are not built. Try: gem pristine rkerberos --version 0.1.5
Ignoring ruby-libvirt-0.7.1 because its extensions are not built. Try: gem pristine ruby-libvirt --version 0.7.1
Ignoring sqlite3-1.3.13 because its extensions are not built. Try: gem pristine sqlite3 --version 1.3.13
Ignoring unf_ext-0.0.7.2 because its extensions are not built. Try: gem pristine unf_ext --version 0.0.7.2
Ignoring unicode-0.4.4.4 because its extensions are not built. Try: gem pristine unicode --version 0.4.4.4
Ignoring websocket-driver-0.7.1 because its extensions are not built. Try: gem pristine websocket-driver --version 0.7.1
Ignoring bcrypt-3.1.12 because its extensions are not built. Try: gem pristine bcrypt --version 3.1.12
Ignoring ffi-1.12.2 because its extensions are not built. Try: gem pristine ffi --version 1.12.2
Ignoring nio4r-2.5.2 because its extensions are not built. Try: gem pristine nio4r --version 2.5.2
Ignoring nokogiri-1.10.9 because its extensions are not built. Try: gem pristine nokogiri --version 1.10.9
Ignoring passenger-4.0.18 because its extensions are not built. Try: gem pristine passenger --version 4.0.18
Ignoring pg-1.1.4 because its extensions are not built. Try: gem pristine pg --version 1.1.4
Ignoring qpid_messaging-1.38.0 because its extensions are not built. Try: gem pristine qpid_messaging --version 1.38.0
Ignoring rkerberos-0.1.5 because its extensions are not built. Try: gem pristine rkerberos --version 0.1.5
Ignoring ruby-libvirt-0.7.1 because its extensions are not built. Try: gem pristine ruby-libvirt --version 0.7.1
Ignoring sqlite3-1.3.13 because its extensions are not built. Try: gem pristine sqlite3 --version 1.3.13
Ignoring unf_ext-0.0.7.2 because its extensions are not built. Try: gem pristine unf_ext --version 0.0.7.2
Ignoring unicode-0.4.4.4 because its extensions are not built. Try: gem pristine unicode --version 0.4.4.4
Ignoring websocket-driver-0.7.1 because its extensions are not built. Try: gem pristine websocket-driver --version 0.7.1
cannot load such file – semantic_puppet
[ERROR 2020-07-06T20:27:33 main] Could not get default values, cannot continue

2

Just to verify, did you follow the Katello upgrade instructions?

3

I did. I can’t even get the installer to run with just the help flags, as it comes back with the default values error.

4

Can you share logs? When the installer is unable to get its default values, it indicates a problem and suggests to look at the installer log.

Out of experience, I’m guessing the installer is unable to determine the domain for the server though that’s unlikely on upgrades.

5

Which logs would you need?

6

Just some added info, I did not include this bit in the excerpt from the initial post:

[ERROR 2020-07-06T20:27:33 main] echo ’
$kafo_config_file="/etc/foreman-installer/scenarios.d/katello.yaml"
$kafo_add_progress=true
kafo_configure::puppet_version_semver { “puppetlabs-apache”:
requirement => “>= 5.5.10 < 7.0.0”,
}

    kafo_configure::puppet_version_semver { "puppetlabs-apt":
      requirement => ">= 5.5.10 < 7.0.0",
    }

    kafo_configure::puppet_version_semver { "herculesteam-augeasproviders_core":
      requirement => ">= 5.0.0 < 7.0.0",
    }

    kafo_configure::puppet_version_semver { "herculesteam-augeasproviders_sysctl":
      requirement => ">= 5.0.0 < 7.0.0",
    }

    kafo_configure::puppet_version_semver { "katello-candlepin":
      requirement => ">= 5.5.8 < 7.0.0",
    }

    kafo_configure::puppet_version_semver { "katello-certs":
      requirement => ">= 5.5.8 < 7.0.0",
    }

    kafo_configure::puppet_version_semver { "puppetlabs-concat":
      requirement => ">= 5.5.10 < 7.0.0",
    }

    kafo_configure::puppet_version_semver { "theforeman-dhcp":
      requirement => ">= 5.5.8 < 7.0.0",
    }

    kafo_configure::puppet_version_semver { "theforeman-dns":
      requirement => ">= 5.5.8 < 7.0.0",
    }

    kafo_configure::puppet_version_semver { "puppet-extlib":
      requirement => ">= 5.5.8 < 7.0.0",
    }

    kafo_configure::puppet_version_semver { "theforeman-foreman":
      requirement => ">= 5.5.8 < 7.0.0",
    }

    kafo_configure::puppet_version_semver { "theforeman-foreman_proxy":
      requirement => ">= 5.5.8 < 7.0.0",
    }

    kafo_configure::puppet_version_semver { "katello-foreman_proxy_content":
      requirement => ">= 5.5.8 < 7.0.0",
    }

    kafo_configure::puppet_version_semver { "theforeman-git":
      requirement => ">= 5.5.8 < 7.0.0",
    }

    kafo_configure::puppet_version_semver { "puppetlabs-inifile":
      requirement => ">= 5.5.10 < 7.0.0",
    }

    kafo_configure::puppet_version_semver { "katello-katello":
      requirement => ">= 5.5.8 < 7.0.0",
    }

    kafo_configure::puppet_version_semver { "puppet-mongodb":
      requirement => ">= 5.5.8 < 7.0.0",
    }

    kafo_configure::puppet_version_semver { "puppetlabs-postgresql":
      requirement => ">= 5.5.10 < 7.0.0",
    }

    kafo_configure::puppet_version_semver { "katello-pulp":
      requirement => ">= 5.5.8 < 7.0.0",
    }

    kafo_configure::puppet_version_semver { "theforeman-pulpcore":
      requirement => ">= 5.5.8 < 7.0.0",
    }

    kafo_configure::puppet_version_semver { "theforeman-puppet":
      requirement => ">= 5.5.8 < 7.0.0",
    }

    kafo_configure::puppet_version_semver { "katello-qpid":
      requirement => ">= 5.5.8 < 7.0.0",
    }

    kafo_configure::puppet_version_semver { "puppet-redis":
      requirement => ">= 5.5.8 < 7.0.0",
    }

    kafo_configure::puppet_version_semver { "puppet-selinux":
      requirement => ">= 5.5.8 < 7.0.0",
    }

    kafo_configure::puppet_version_semver { "puppet-squid":
      requirement => ">= 5.5.8 < 7.0.0",
    }

    kafo_configure::puppet_version_semver { "puppetlabs-stdlib":
      requirement => ">= 5.5.10 < 7.0.0",
    }

    kafo_configure::puppet_version_semver { "camptocamp-systemd":
      requirement => ">= 4.10.10 < 7.0.0",
    }

    kafo_configure::puppet_version_semver { "theforeman-tftp":
      requirement => ">= 5.5.8 < 7.0.0",
    }

    kafo_configure::puppet_version_semver { "puppetlabs-transition":
      requirement => ">= 3.0.0 < 7.0.0",
    }

    kafo_configure::puppet_version_semver { "puppetlabs-translate":
      requirement => ">= 5.5.10 < 7.0.0",
    }

    kafo_configure::puppet_version_semver { "puppet-trusted_ca":
      requirement => ">= 5.5.8 < 7.0.0",
    }

    kafo_configure::puppet_version_semver { "puppetlabs-xinetd":
      requirement => ">= 4.7.1 < 7.0.0",
    }

    kafo_configure::puppet_version_semver { "theforeman-kafo_configure":
      requirement => ">= 4.5.0 < 7.0.0",
    }

              include certs::params include foreman::params include foreman::cli::params include foreman_proxy::params include foreman_proxy_content::params include foreman_proxy::plugin::ansible::params include foreman_proxy::plugin::chef::params include foreman_proxy::plugin::discovery::params include foreman_proxy::plugin::pulp::params include foreman_proxy::plugin::salt::params include katello::params include puppet::params
      class { '::kafo_configure::dump_values':
        lookups   => ["foreman::db_host","foreman::db_port","foreman::db_database","foreman::db_sslmode","foreman::db_root_cert","foreman_proxy::plugin::dhcp::infoblox::username","foreman_proxy::plugin::dhcp::infoblox::password","foreman_proxy::plugin::dhcp::remote_isc::key_name","foreman_proxy::plugin::dhcp::remote_isc::key_secret","foreman_proxy::plugin::dns::infoblox::dns_server","foreman_proxy::plugin::dns::infoblox::username","foreman_proxy::plugin::dns::infoblox::password","foreman_proxy::plugin::monitoring::version","foreman_proxy::plugin::openscap::version","foreman_proxy::plugin::openscap::proxy_name","katello::candlepin_oauth_key","katello::candlepin_oauth_secret","katello::num_pulp_workers","katello::pulp_max_speed","katello::candlepin_db_port","katello::candlepin_db_password","katello::pulp_db_username","katello::pulp_db_password","katello::pulp_db_replica_set","katello::pulp_db_ssl_keyfile","katello::pulp_db_ssl_certfile","katello::pulp_db_write_concern","puppet::run_hour","puppet::run_minute","puppet::server_acceptor_threads","puppet::server_selector_threads","puppet::server_ssl_acceptor_threads","puppet::server_ssl_selector_threads","puppet::server_max_threads","puppet::server_versioned_code_id","puppet::server_versioned_code_content"],
        variables => ["certs::params::log_dir","certs::params::node_fqdn","certs::params::cname","certs::params::generate","certs::params::regenerate","certs::params::deploy","certs::params::ca_common_name","certs::params::country","certs::params::state","certs::params::city","certs::params::org","certs::params::org_unit","certs::params::expiration","certs::params::ca_expiration","certs::params::server_cert","certs::params::server_key","certs::params::server_cert_req","certs::params::server_ca_cert","certs::params::pki_dir","certs::params::ssl_build_dir","certs::params::user","certs::params::group","certs::params::default_ca_name","certs::params::server_ca_name","certs::params::tar_file","foreman::params::foreman_url","foreman::params::unattended","foreman::params::unattended_url","foreman::params::apache","foreman::params::passenger","foreman::params::passenger_ruby","foreman::params::passenger_ruby_package","foreman::params::plugin_prefix","foreman::params::servername","foreman::params::serveraliases","foreman::params::ssl","foreman::params::version","foreman::params::plugin_version","foreman::params::db_manage","foreman::params::db_username","foreman::params::db_password","foreman::params::db_pool","foreman::params::db_manage_rake","foreman::params::app_root","foreman::params::manage_user","foreman::params::user","foreman::params::group","foreman::params::user_groups","foreman::params::rails_env","foreman::params::vhost_priority","foreman::params::server_port","foreman::params::server_ssl_port","foreman::params::server_ssl_ca","foreman::params::server_ssl_chain","foreman::params::server_ssl_cert","foreman::params::server_ssl_certs_dir","foreman::params::server_ssl_key","foreman::params::server_ssl_crl","foreman::params::server_ssl_protocol","foreman::params::server_ssl_verify_client","foreman::params::client_ssl_ca","foreman::params::client_ssl_cert","foreman::params::client_ssl_key","foreman::params::oauth_active","foreman::params::oauth_map_users","foreman::params::oauth_consumer_key","foreman::params::oauth_consumer_secret","foreman::params::passenger_prestart","foreman::params::passenger_min_instances","foreman::params::passenger_start_timeout","foreman::params::initial_admin_username","foreman::params::initial_admin_password","foreman::params::initial_admin_first_name","foreman::params::initial_admin_last_name","foreman::params::initial_admin_email","foreman::params::initial_admin_locale","foreman::params::initial_admin_timezone","foreman::params::initial_organization","foreman::params::initial_location","foreman::params::ipa_authentication","foreman::params::http_keytab","foreman::params::pam_service","foreman::params::ipa_manage_sssd","foreman::params::websockets_encrypt","foreman::params::websockets_ssl_key","foreman::params::websockets_ssl_cert","foreman::params::logging_level","foreman::params::logging_type","foreman::params::logging_layout","foreman::params::loggers","foreman::params::email_delivery_method","foreman::params::email_smtp_address","foreman::params::email_smtp_port","foreman::params::email_smtp_domain","foreman::params::email_smtp_authentication","foreman::params::email_smtp_user_name","foreman::params::email_smtp_password","foreman::params::telemetry_prefix","foreman::params::telemetry_prometheus_enabled","foreman::params::telemetry_statsd_enabled","foreman::params::telemetry_statsd_host","foreman::params::telemetry_statsd_protocol","foreman::params::telemetry_logger_enabled","foreman::params::telemetry_logger_level","foreman::params::dynflow_pool_size","foreman::params::jobs_manage_service","foreman::params::jobs_service_ensure","foreman::params::jobs_service_enable","foreman::params::jobs_sidekiq_redis_url","foreman::params::hsts_enabled","foreman::params::cors_domains","foreman::params::foreman_service_puma_threads_min","foreman::params::foreman_service_puma_threads_max","foreman::params::foreman_service_puma_workers","foreman::params::rails_cache_store","foreman::params::keycloak","foreman::params::keycloak_app_name","foreman::params::keycloak_realm","foreman::cli::params::foreman_url","foreman::cli::params::version","foreman::cli::params::manage_root_config","foreman::cli::params::username","foreman::cli::params::password","foreman::cli::params::refresh_cache","foreman::cli::params::request_timeout","foreman::cli::params::ssl_ca_file","foreman::cli::params::hammer_plugin_prefix","foreman::params::client_ssl_ca","foreman::params::client_ssl_cert","foreman::params::client_ssl_key","foreman_proxy::params::repo","foreman_proxy::params::gpgcheck","foreman_proxy::params::version","foreman_proxy::params::ensure_packages_version","foreman_proxy::params::bind_host","foreman_proxy::params::http_port","foreman_proxy::params::ssl_port","foreman_proxy::params::groups","foreman_proxy::params::log","foreman_proxy::params::log_level","foreman_proxy::params::log_buffer","foreman_proxy::params::log_buffer_errors","foreman_proxy::params::http","foreman_proxy::params::ssl","foreman_proxy::params::ssl_ca","foreman_proxy::params::ssl_cert","foreman_proxy::params::ssl_key","foreman_proxy::params::foreman_ssl_ca","foreman_proxy::params::foreman_ssl_cert","foreman_proxy::params::foreman_ssl_key","foreman_proxy::params::trusted_hosts","foreman_proxy::params::ssl_disabled_ciphers","foreman_proxy::params::tls_disabled_versions","foreman_proxy::params::manage_sudoersd","foreman_proxy::params::use_sudoersd","foreman_proxy::params::use_sudoers","foreman_proxy::params::puppetca","foreman_proxy::params::puppetca_split_configs","foreman_proxy::params::puppetca_listen_on","foreman_proxy::params::ssldir","foreman_proxy::params::puppetdir","foreman_proxy::params::puppetca_cmd","foreman_proxy::params::puppet_group","foreman_proxy::params::puppetca_provider","foreman_proxy::params::autosignfile","foreman_proxy::params::puppetca_sign_all","foreman_proxy::params::puppetca_tokens_file","foreman_proxy::params::puppetca_token_ttl","foreman_proxy::params::puppetca_certificate","foreman_proxy::params::manage_puppet_group","foreman_proxy::params::puppet","foreman_proxy::params::puppet_listen_on","foreman_proxy::params::puppetrun_provider","foreman_proxy::params::customrun_cmd","foreman_proxy::params::customrun_args","foreman_proxy::params::mcollective_user","foreman_proxy::params::puppetssh_sudo","foreman_proxy::params::puppetssh_command","foreman_proxy::params::puppetssh_user","foreman_proxy::params::puppetssh_keyfile","foreman_proxy::params::puppetssh_wait","foreman_proxy::params::salt_puppetrun_cmd","foreman_proxy::params::puppet_user","foreman_proxy::params::puppet_url","foreman_proxy::params::ssl_ca","foreman_proxy::params::ssl_cert","foreman_proxy::params::ssl_key","foreman_proxy::params::puppet_api_timeout","foreman_proxy::params::templates","foreman_proxy::params::templates_listen_on","foreman_proxy::params::template_url","foreman_proxy::params::logs","foreman_proxy::params::logs_listen_on","foreman_proxy::params::httpboot","foreman_proxy::params::httpboot_listen_on","foreman_proxy::params::tftp","foreman_proxy::params::tftp_listen_on","foreman_proxy::params::tftp_managed","foreman_proxy::params::tftp_manage_wget","foreman_proxy::params::tftp_syslinux_filenames","foreman_proxy::params::tftp_root","foreman_proxy::params::tftp_dirs","foreman_proxy::params::tftp_servername","foreman_proxy::params::tftp_replace_grub2_cfg","foreman_proxy::params::dhcp","foreman_proxy::params::dhcp_listen_on","foreman_proxy::params::dhcp_managed","foreman_proxy::params::dhcp_provider","foreman_proxy::params::dhcp_subnets","foreman_proxy::params::dhcp_option_domain","foreman_proxy::params::dhcp_search_domains","foreman_proxy::params::dhcp_interface","foreman_proxy::params::dhcp_additional_interfaces","foreman_proxy::params::dhcp_gateway","foreman_proxy::params::dhcp_range","foreman_proxy::params::dhcp_pxeserver","foreman_proxy::params::dhcp_pxefilename","foreman_proxy::params::dhcp_network","foreman_proxy::params::dhcp_netmask","foreman_proxy::params::dhcp_nameservers","foreman_proxy::params::dhcp_server","foreman_proxy::params::dhcp_config","foreman_proxy::params::dhcp_leases","foreman_proxy::params::dhcp_key_name","foreman_proxy::params::dhcp_key_secret","foreman_proxy::params::dhcp_omapi_port","foreman_proxy::params::dhcp_peer_address","foreman_proxy::params::dhcp_node_type","foreman_proxy::params::dhcp_failover_address","foreman_proxy::params::dhcp_failover_port","foreman_proxy::params::dhcp_max_response_delay","foreman_proxy::params::dhcp_max_unacked_updates","foreman_proxy::params::dhcp_mclt","foreman_proxy::params::dhcp_load_split","foreman_proxy::params::dhcp_load_balance","foreman_proxy::params::dhcp_manage_acls","foreman_proxy::params::dns","foreman_proxy::params::dns_listen_on","foreman_proxy::params::dns_managed","foreman_proxy::params::dns_provider","foreman_proxy::params::dns_interface","foreman_proxy::params::dns_zone","foreman_proxy::params::dns_reverse","foreman_proxy::params::dns_server","foreman_proxy::params::dns_ttl","foreman_proxy::params::dns_tsig_keytab","foreman_proxy::params::dns_tsig_principal","foreman_proxy::params::dns_forwarders","foreman_proxy::params::libvirt_network","foreman_proxy::params::libvirt_connection","foreman_proxy::params::bmc","foreman_proxy::params::bmc_listen_on","foreman_proxy::params::bmc_default_provider","foreman_proxy::params::bmc_ssh_user","foreman_proxy::params::bmc_ssh_key","foreman_proxy::params::bmc_ssh_powerstatus","foreman_proxy::params::bmc_ssh_powercycle","foreman_proxy::params::bmc_ssh_poweroff","foreman_proxy::params::bmc_ssh_poweron","foreman_proxy::params::realm","foreman_proxy::params::realm_listen_on","foreman_proxy::params::realm_provider","foreman_proxy::params::realm_keytab","foreman_proxy::params::realm_principal","foreman_proxy::params::freeipa_config","foreman_proxy::params::freeipa_remove_dns","foreman_proxy::params::keyfile","foreman_proxy::params::register_in_foreman","foreman_proxy::params::foreman_base_url","foreman_proxy::params::registered_name","foreman_proxy::params::registered_proxy_url","foreman_proxy::params::oauth_effective_user","foreman_proxy::params::oauth_consumer_key","foreman_proxy::params::oauth_consumer_secret","foreman_proxy_content::params::parent_fqdn","foreman_proxy_content::params::pulp_admin_password","foreman_proxy_content::params::pulp_max_speed","foreman_proxy_content::params::pulp_num_workers","foreman_proxy_content::params::pulp_proxy_password","foreman_proxy_content::params::pulp_proxy_port","foreman_proxy_content::params::pulp_proxy_url","foreman_proxy_content::params::pulp_proxy_username","foreman_proxy_content::params::pulp_puppet_wsgi_processes","foreman_proxy_content::params::pulp_ca_cert","foreman_proxy_content::params::pulp_worker_timeout","foreman_proxy_content::params::puppet","foreman_proxy_content::params::reverse_proxy","foreman_proxy_content::params::reverse_proxy_port","foreman_proxy_content::params::ssl_protocol","foreman_proxy_content::params::rhsm_hostname","foreman_proxy_content::params::rhsm_url","foreman_proxy_content::params::qpid_router","foreman_proxy_content::params::qpid_router_hub_addr","foreman_proxy_content::params::qpid_router_hub_port","foreman_proxy_content::params::qpid_router_agent_addr","foreman_proxy_content::params::qpid_router_agent_port","foreman_proxy_content::params::qpid_router_broker_addr","foreman_proxy_content::params::qpid_router_broker_port","foreman_proxy_content::params::qpid_router_logging_level","foreman_proxy_content::params::qpid_router_logging","foreman_proxy_content::params::qpid_router_logging_path","foreman_proxy_content::params::qpid_router_ssl_ciphers","foreman_proxy_content::params::qpid_router_ssl_protocols","foreman_proxy_content::params::qpid_router_sasl_mech","foreman_proxy_content::params::qpid_router_sasl_username","foreman_proxy_content::params::qpid_router_sasl_password","foreman_proxy_content::params::enable_ostree","foreman_proxy_content::params::enable_yum","foreman_proxy_content::params::enable_file","foreman_proxy_content::params::proxy_pulp_isos_to_pulpcore","foreman_proxy_content::params::proxy_pulp_yum_to_pulpcore","foreman_proxy_content::params::enable_puppet","foreman_proxy_content::params::enable_docker","foreman_proxy_content::params::enable_deb","foreman_proxy_content::params::manage_broker","foreman_proxy_content::params::pulpcore_manage_postgresql","foreman_proxy_content::params::pulpcore_postgresql_host","foreman_proxy_content::params::pulpcore_postgresql_port","foreman_proxy_content::params::pulpcore_postgresql_user","foreman_proxy_content::params::pulpcore_postgresql_password","foreman_proxy_content::params::pulpcore_postgresql_db_name","foreman_proxy_content::params::pulpcore_postgresql_ssl","foreman_proxy_content::params::pulpcore_postgresql_ssl_require","foreman_proxy_content::params::pulpcore_postgresql_ssl_cert","foreman_proxy_content::params::pulpcore_postgresql_ssl_key","foreman_proxy_content::params::pulpcore_postgresql_ssl_root_ca","foreman_proxy::plugin::ansible::params::enabled","foreman_proxy::plugin::ansible::params::listen_on","foreman_proxy::plugin::ansible::params::ansible_dir","foreman_proxy::plugin::ansible::params::working_dir","foreman_proxy::plugin::ansible::params::host_key_checking","foreman_proxy::plugin::ansible::params::stdout_callback","foreman_proxy::plugin::ansible::params::roles_path","foreman_proxy::plugin::ansible::params::ssh_args","foreman_proxy::plugin::ansible::params::install_runner","foreman_proxy::plugin::ansible::params::manage_runner_repo","foreman_proxy::plugin::chef::params::enabled","foreman_proxy::plugin::chef::params::listen_on","foreman_proxy::plugin::chef::params::version","foreman_proxy::plugin::chef::params::server_url","foreman_proxy::plugin::chef::params::client_name","foreman_proxy::plugin::chef::params::private_key","foreman_proxy::plugin::chef::params::ssl_verify","foreman_proxy::plugin::chef::params::ssl_pem_file","foreman_proxy::plugin::discovery::params::install_images","foreman_proxy::plugin::discovery::params::tftp_root","foreman_proxy::plugin::discovery::params::source_url","foreman_proxy::plugin::discovery::params::image_name","foreman_proxy::plugin::pulp::params::enabled","foreman_proxy::plugin::pulp::params::listen_on","foreman_proxy::plugin::pulp::params::pulpnode_enabled","foreman_proxy::plugin::pulp::params::pulpcore_enabled","foreman_proxy::plugin::pulp::params::pulpcore_api_url","foreman_proxy::plugin::pulp::params::pulpcore_content_url","foreman_proxy::plugin::pulp::params::pulpcore_mirror","foreman_proxy::plugin::pulp::params::version","foreman_proxy::plugin::pulp::params::pulp_url","foreman_proxy::plugin::pulp::params::pulp_dir","foreman_proxy::plugin::pulp::params::pulp_content_dir","foreman_proxy::plugin::pulp::params::puppet_content_dir","foreman_proxy::plugin::pulp::params::mongodb_dir","foreman_proxy::plugin::salt::params::autosign_file","foreman_proxy::plugin::salt::params::enabled","foreman_proxy::plugin::salt::params::listen_on","foreman_proxy::plugin::salt::params::user","foreman_proxy::plugin::salt::params::api","foreman_proxy::plugin::salt::params::api_url","foreman_proxy::plugin::salt::params::api_auth","foreman_proxy::plugin::salt::params::api_username","foreman_proxy::plugin::salt::params::api_password","foreman_proxy::plugin::salt::params::saltfile","puppet::params::version","puppet::params::user","puppet::params::group","puppet::params::dir","puppet::params::codedir","puppet::params::vardir","puppet::params::logdir","puppet::params::rundir","puppet::params::ssldir","puppet::params::sharedir","puppet::params::manage_packages","puppet::params::dir_owner","puppet::params::dir_group","puppet::params::package_provider","puppet::params::package_source","puppet::params::port","puppet::params::pluginsync","puppet::params::splay","puppet::params::splaylimit","puppet::params::autosign","puppet::params::autosign_entries","puppet::params::autosign_mode","puppet::params::autosign_content","puppet::params::autosign_source","puppet::params::runinterval","puppet::params::usecacheonfailure","puppet::params::runmode","puppet::params::unavailable_runmodes","puppet::params::cron_cmd","puppet::params::systemd_cmd","puppet::params::systemd_randomizeddelaysec","puppet::params::agent_noop","puppet::params::show_diff","puppet::params::module_repository","puppet::params::http_connect_timeout","puppet::params::http_read_timeout","puppet::params::ca_server","puppet::params::ca_port","puppet::params::ca_crl_filepath","puppet::params::prerun_command","puppet::params::postrun_command","puppet::params::dns_alt_names","puppet::params::use_srv_records","puppet::params::srv_domain","puppet::params::pluginsource","puppet::params::pluginfactsource","puppet::params::additional_settings","puppet::params::agent_additional_settings","puppet::params::agent_restart_command","puppet::params::classfile","puppet::params::hiera_config","puppet::params::auth_template","puppet::params::allow_any_crl_auth","puppet::params::auth_allowed","puppet::params::client_package","puppet::params::agent","puppet::params::remove_lock","puppet::params::report","puppet::params::client_certname","puppet::params::puppetmaster","puppet::params::systemd_unit_name","puppet::params::service_name","puppet::params::syslogfacility","puppet::params::environment","puppet::params::server","puppet::params::server_admin_api_whitelist","puppet::params::manage_user","puppet::params::user","puppet::params::group","puppet::params::dir","puppet::params::ip","puppet::params::port","puppet::params::server_ca","puppet::params::server_ca_crl_sync","puppet::params::server_crl_enable","puppet::params::server_ca_auth_required","puppet::params::server_ca_client_self_delete","puppet::params::server_ca_client_whitelist","puppet::params::server_custom_trusted_oid_mapping","puppet::params::server_http","puppet::params::server_http_port","puppet::params::server_reports","puppet::params::server_puppetserver_dir","puppet::params::server_puppetserver_vardir","puppet::params::server_puppetserver_rundir","puppet::params::server_puppetserver_logdir","puppet::params::server_puppetserver_version","puppet::params::server_external_nodes","puppet::params::server_trusted_external_command","puppet::params::server_cipher_suites","puppet::params::server_config_version","puppet::params::server_connect_timeout","puppet::params::server_git_repo","puppet::params::server_default_manifest","puppet::params::server_default_manifest_path","puppet::params::server_default_manifest_content","puppet::params::server_environments_owner","puppet::params::server_environments_group","puppet::params::server_environments_mode","puppet::params::server_envs_dir","puppet::params::server_envs_target","puppet::params::server_common_modules_path","puppet::params::server_git_repo_mode","puppet::params::server_git_repo_path","puppet::params::server_git_repo_group","puppet::params::server_git_repo_user","puppet::params::server_git_branch_map","puppet::params::server_idle_timeout","puppet::params::server_post_hook_content","puppet::params::server_post_hook_name","puppet::params::server_storeconfigs","puppet::params::server_ruby_load_paths","puppet::params::server_ssl_dir","puppet::params::server_ssl_dir_manage","puppet::params::server_ssl_key_manage","puppet::params::server_ssl_protocols","puppet::params::server_ssl_chain_filepath","puppet::params::server_package","puppet::params::server_version","puppet::params::server_certname","puppet::params::server_request_timeout","puppet::params::server_strict_variables","puppet::params::server_additional_settings","puppet::params::server_foreman","puppet::params::server_foreman_url","puppet::params::server_foreman_ssl_ca","puppet::params::server_foreman_ssl_cert","puppet::params::server_foreman_ssl_key","puppet::params::server_foreman_facts","puppet::params::server_puppet_basedir","puppet::params::server_parser","puppet::params::server_environment_timeout","puppet::params::server_jvm_java_bin","puppet::params::server_jvm_config","puppet::params::server_jvm_min_heap_size","puppet::params::server_jvm_max_heap_size","puppet::params::server_jvm_extra_args","puppet::params::server_jvm_cli_args","puppet::params::server_jruby_gem_home","puppet::params::server_max_active_instances","puppet::params::server_max_requests_per_instance","puppet::params::server_max_queued_requests","puppet::params::server_max_retry_delay","puppet::params::server_multithreaded","puppet::params::server_use_legacy_auth_conf","puppet::params::server_check_for_updates","puppet::params::server_environment_class_cache_enabled","puppet::params::server_allow_header_cert_info","puppet::params::server_web_idle_timeout","puppet::params::server_puppetserver_jruby9k","puppet::params::server_puppetserver_metrics","puppet::params::server_metrics_jmx_enable","puppet::params::server_metrics_graphite_enable","puppet::params::server_metrics_graphite_host","puppet::params::server_metrics_graphite_port","puppet::params::server_metrics_server_id","puppet::params::server_metrics_graphite_interval","puppet::params::server_metrics_allowed","puppet::params::server_puppetserver_experimental","puppet::params::server_puppetserver_trusted_agents","puppet::params::server_compile_mode","puppet::params::server_ca_allow_sans","puppet::params::server_ca_allow_auth_extensions","puppet::params::server_ca_enable_infra_crl","puppet::params::server_max_open_files"],
      }

  ' | RUBYLIB=/opt/theforeman/tfm/root/usr/share/gems/gems/kafo-4.1.0/lib/kafo/../..//modules: /usr/local/bin/puppet apply --config=/tmp/kafo_installation20200706-31587-1sdcvb6/puppet.conf

I attempted to complete this command manually in an effort to get some more info about what was happening, and I got this error:

Error: Evaluation Error: Error while evaluating a Resource Statement, Unknown resource type: ‘kafo_configure::puppet_version_semver’ (line: 4, column: 17) on node SANITIZED

7

To help us debug, you mentioned this was a RHEL 7.8 system. Was this a clean, no other software had been installed on it RHEL 7.8?

8

There are some monitoring tools installed, but nothing beyond that and Foreman/Katello. This server didn’t start its life on RHEL 7.8 though. I think it was initially built with 7.4. The Foreman/Katello installation has also been through several upgrades during its lifespan.

9

That’s an odd one - it can’t find the kafo_configure module. Can you verify /opt/theforeman/tfm/root/usr/share/gems/gems/kafo-4.1.0/modules does exist and lists the kafo_configure module?

10

It’s there:

ls -l /opt/theforeman/tfm/root/usr/share/gems/gems/kafo-4.1.0/modules
total 4
drwxr-xr-x 5 root root 4096 Jul 6 18:15 kafo_configure

I had started chasing this down, but sadly I just don’t know enough about how these are being called to work it out. It looked to me like puppet_version_semver IS a defined resource type, so I didn’t understand why it wasn’t found:

ls -l /opt/theforeman/tfm/root/usr/share/gems/gems/kafo-4.1.0/modules/kafo_configure/manifests/
total 12
-rw-r–r-- 1 root root 763 Apr 6 14:35 dump_values.pp
-rw-r–r-- 1 root root 526 Apr 6 14:35 init.pp
-rw-r–r-- 1 root root 281 Apr 6 14:35 puppet_version_semver.pp

cat /opt/theforeman/tfm/root/usr/share/gems/gems/kafo-
4.1.0/modules/kafo_configure/manifests/init.pp
# This class is called from the kafo configure script
# and expects a yaml file to exist at either:
# optional $answers class parameter
# $modulepath/config/answers.yaml
# /etc/kafo-configure/answers.yaml
#
# @param add_progress
# Whether to add a progress bar. Only works on Puppet < 6.
class kafo_configure(
Boolean $add_progress = $::kafo_add_progress,
) {
if $add_progress and SemVer($facts[‘puppetversion’]) =~ SemVerRange(’< 6.0.0’) {
add_progress()
}
lookup(‘classes’, {merge => unique}).include
}

cat /opt/theforeman/tfm/root/usr/share/gems/gems/kafo-
4.1.0/modules/kafo_configure/manifests/puppet_version_semver.pp
define
kafo_configure::puppet_version_semver($requirement) {
unless SemVer($facts[‘puppetversion’]) =~ SemVerRange($requirement) {
fail(“kafo_configure::puppet_version_failure: Puppet ${facts[‘puppetversion’]} does not meet
requirements for ${title} (${requirement})”)
}
}

11

Are you by any chance using RVM or other similar technologies?

12

I’m not using RVM

13

Any possibility this is a path problem? I see semantic_puppet on the system, but it seems like it isn’t being found.

find / -name semantic_puppet
/opt/puppetlabs/server/data/puppetserver/vendored-jruby-gems/gems/semantic_puppet-1.0.2/lib/semantic_puppet
/opt/puppetlabs/server/data/puppetserver/vendored-jruby-gems/gems/semantic_puppet-1.0.2/spec/unit/semantic_puppet
/opt/puppetlabs/puppet/lib/ruby/vendor_gems/gems/semantic_puppet-1.0.2/lib/semantic_puppet
/opt/puppetlabs/puppet/lib/ruby/vendor_gems/gems/semantic_puppet-1.0.2/spec/unit/semantic_puppet

14

OK, I’ve managed to get a little further down the path. I looked through some of the older logs from successful upgrades, and it looks to me like it’s trying to use a puppet.conf file that isn’t being written out for some reason. I can find older kafo directories in /tmp, but none for any of the runs that I have done since installing the rpms for foreman 2.1 and Katello 3.16rc3

15

I’ve pinned down the problem, and it’s due to the GEM_PATH configuration in the tfm scl environment. As it is currently configured, puppet is unable to find its built-in gems once the tfm environment is enabled. Adding /opt/puppetlabs/puppet/lib/ruby/vendor_gems to GEM_PATH got the installer to run once more.
Since this is being set by a wrapper, I updated the /opt/theforeman/tfm/enable file to include the additional path. The file now looks like this:

. scl_source enable rh-ruby25
if [ -e /opt/theforeman/tfm/enable_assets ]; then
. /opt/theforeman/tfm/enable_assets
fi
export PATH=/opt/theforeman/tfm/root/usr/bin${PATH:+:${PATH}}
export LIBRARY_PATH=/opt/theforeman/tfm/root/usr/lib64${LIBRARY_PATH:+:${LIBRARY_PATH}}
export LD_LIBRARY_PATH=/opt/theforeman/tfm/root/usr/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
export MANPATH=/opt/theforeman/tfm/root/usr/share/man:${MANPATH}
export CPATH=/opt/theforeman/tfm/root/usr/include${CPATH:+:${CPATH}}
export PKG_CONFIG_PATH=/opt/theforeman/tfm/root/usr/lib64/pkgconfig${PKG_CONFIG_PATH:+:${PKG_CONFIG_PATH}}
export GEM_PATH=/opt/theforeman/tfm/root/usr/share/gems:/opt/puppetlabs/puppet/lib/ruby/vendor_gems:${GEM_PATH:+${GEM_PATH}}${GEM_PATH:-scl enable rh-ruby25 -- ruby -e "print Gem.path.join(':')"}
export GEM_HOME="/opt/theforeman/tfm/root/usr/share/gems"

16

This should not be needed. Foreman should never need any gems from /opt/puppetlabs/puppet/lib/ruby/vendor_gem. Puppet should be isolated not use anything from tfm.

We have done some fixes around this in 2.1.0 GA. I think 3.16 RC4 is also on the mirrors (but not yet officially announced). Would you mind testing with that?

17

I set the enable file back to tfm-runtime default, updated to katello 3.16rc4, and am back to “Could not get default values, check log file at /var/log/foreman-installer/katello.log for more information”

I did a couple of additional tests to make some comparisons:

[root@myserver ~]# puppet --version
6.16.0
[root@myserver ~]# scl enable rh-ruby25 bash
[root@myserver ~]# puppet --version
6.16.0
[root@myserver ~]# exit
[root@myserver ~]# scl enable tfm bash
[root@myserver ~]# puppet --version
Ignoring bcrypt-3.1.12 because its extensions are not built. Try: gem pristine bcrypt --version 3.1.12
Ignoring ffi-1.12.2 because its extensions are not built. Try: gem pristine ffi --version 1.12.2
Ignoring nio4r-2.5.2 because its extensions are not built. Try: gem pristine nio4r --version 2.5.2
Ignoring nokogiri-1.10.9 because its extensions are not built. Try: gem pristine nokogiri --version 1.10.9
Ignoring passenger-4.0.18 because its extensions are not built. Try: gem pristine passenger --version 4.0.18
Ignoring pg-1.1.4 because its extensions are not built. Try: gem pristine pg --version 1.1.4
Ignoring rkerberos-0.1.5 because its extensions are not built. Try: gem pristine rkerberos --version 0.1.5
Ignoring ruby-libvirt-0.7.1 because its extensions are not built. Try: gem pristine ruby-libvirt --version 0.7.1
Ignoring sqlite3-1.3.13 because its extensions are not built. Try: gem pristine sqlite3 --version 1.3.13
Ignoring unf_ext-0.0.7.2 because its extensions are not built. Try: gem pristine unf_ext --version 0.0.7.2
Ignoring unicode-0.4.4.4 because its extensions are not built. Try: gem pristine unicode --version 0.4.4.4
Ignoring websocket-driver-0.7.1 because its extensions are not built. Try: gem pristine websocket-driver --version 0.7.1
Ignoring bcrypt-3.1.12 because its extensions are not built. Try: gem pristine bcrypt --version 3.1.12
Ignoring ffi-1.12.2 because its extensions are not built. Try: gem pristine ffi --version 1.12.2
Ignoring nio4r-2.5.2 because its extensions are not built. Try: gem pristine nio4r --version 2.5.2
Ignoring nokogiri-1.10.9 because its extensions are not built. Try: gem pristine nokogiri --version 1.10.9
Ignoring passenger-4.0.18 because its extensions are not built. Try: gem pristine passenger --version 4.0.18
Ignoring pg-1.1.4 because its extensions are not built. Try: gem pristine pg --version 1.1.4
Ignoring rkerberos-0.1.5 because its extensions are not built. Try: gem pristine rkerberos --version 0.1.5
Ignoring ruby-libvirt-0.7.1 because its extensions are not built. Try: gem pristine ruby-libvirt --version 0.7.1
Ignoring sqlite3-1.3.13 because its extensions are not built. Try: gem pristine sqlite3 --version 1.3.13
Ignoring unf_ext-0.0.7.2 because its extensions are not built. Try: gem pristine unf_ext --version 0.0.7.2
Ignoring unicode-0.4.4.4 because its extensions are not built. Try: gem pristine unicode --version 0.4.4.4
Ignoring websocket-driver-0.7.1 because its extensions are not built. Try: gem pristine websocket-driver --version 0.7.1
cannot load such file – semantic_puppet

18

Puppet is not expected to function with the SCL enabled. The installer should clear the environment prior to calling it.

It would be really useful if you shared the actual installer log. Without it, it’s really hard to say anything.

19

katello-installer.log (29.9 KB)
Log attached

20

foreman-installer.log (2.9 KB)

Realized I didn’t include the log for foreman