Problem with Foreman 2.1rc2 with Remote Execution on CentOS 8

aruzicka · June 15, 2020, 7:36am

I did a bit more debugging. The actual error does not show up in logs (a bug), I had to run smart_proxy_dynflow_core with logging to console to actually be able to see it.

E, [2020-06-15T03:20:30.722368 #7382] ERROR -- : OpenSSH keys only supported if ED25519 is available
net-ssh requires the following gems for ed25519 support:
 * rbnacl (>= 3.2, < 5.0)
 * rbnacl-libsodium, if your system doesn't have libsodium installed.
 * bcrypt_pbkdf (>= 1.0, < 2.0)
See https://github.com/net-ssh/net-ssh/issues/478 for more information
Gem::MissingSpecError : "Could not find 'rbnacl' (< 5.0, >= 3.2.0) among 177 total gem(s)
Checked in 'GEM_PATH=/usr/share/foreman-proxy/.gem/ruby:/usr/share/gems:/usr/local/share/gems', execute `gem env` for more information"
 (NotImplementedError)
/usr/share/gems/gems/net-ssh-4.2.0/lib/net/ssh/authentication/ed25519_loader.rb:19:in `raiseUnlessLoaded'
/usr/share/gems/gems/net-ssh-4.2.0/lib/net/ssh/key_factory.rb:112:in `classify_key'
/usr/share/gems/gems/net-ssh-4.2.0/lib/net/ssh/key_factory.rb:52:in `load_data_private_key'
/usr/share/gems/gems/net-ssh-4.2.0/lib/net/ssh/key_factory.rb:43:in `load_private_key'
/usr/share/gems/gems/net-ssh-4.2.0/lib/net/ssh/authentication/key_manager.rb:142:in `sign'
/usr/share/gems/gems/net-ssh-4.2.0/lib/net/ssh/authentication/methods/publickey.rb:62:in `authenticate_with'
/usr/share/gems/gems/net-ssh-4.2.0/lib/net/ssh/authentication/methods/publickey.rb:20:in `block in authenticate'
/usr/share/gems/gems/net-ssh-4.2.0/lib/net/ssh/authentication/key_manager.rb:122:in `block in each_identity'
/usr/share/gems/gems/net-ssh-4.2.0/lib/net/ssh/authentication/key_manager.rb:119:in `each'
/usr/share/gems/gems/net-ssh-4.2.0/lib/net/ssh/authentication/key_manager.rb:119:in `each_identity'
/usr/share/gems/gems/net-ssh-4.2.0/lib/net/ssh/authentication/methods/publickey.rb:19:in `authenticate'
/usr/share/gems/gems/net-ssh-4.2.0/lib/net/ssh/authentication/session.rb:80:in `block in authenticate'
/usr/share/gems/gems/net-ssh-4.2.0/lib/net/ssh/authentication/session.rb:66:in `each'
/usr/share/gems/gems/net-ssh-4.2.0/lib/net/ssh/authentication/session.rb:66:in `authenticate'
/usr/share/gems/gems/net-ssh-4.2.0/lib/net/ssh.rb:241:in `start'
/usr/share/gems/gems/foreman_remote_execution_core-1.3.0/lib/foreman_remote_execution_core/script_runner.rb:267:in `session'
/usr/share/gems/gems/foreman_remote_execution_core-1.3.0/lib/foreman_remote_execution_core/script_runner.rb:337:in `run_sync'
/usr/share/gems/gems/foreman_remote_execution_core-1.3.0/lib/foreman_remote_execution_core/script_runner.rb:426:in `ensure_remote_directory'
/usr/share/gems/gems/foreman_remote_execution_core-1.3.0/lib/foreman_remote_execution_core/script_runner.rb:403:in `upload_data'
/usr/share/gems/gems/foreman_remote_execution_core-1.3.0/lib/foreman_remote_execution_core/script_runner.rb:399:in `cp_script_to_remote'
/usr/share/gems/gems/foreman_remote_execution_core-1.3.0/lib/foreman_remote_execution_core/script_runner.rb:167:in `prepare_start'
/usr/share/gems/gems/foreman_remote_execution_core-1.3.0/lib/foreman_remote_execution_core/script_runner.rb:153:in `start'
/usr/share/gems/gems/foreman-tasks-core-0.3.4/lib/foreman_tasks_core/runner/dispatcher.rb:32:in `start_runner'

Looking into why it only happens on EL8.

aruzicka · June 15, 2020, 7:59am

Oh, I see.

The net-ssh library we use uses the first line of the private key to determine which type of key (RSA, ed25519 and so on) it is. The version of ssh shipped with EL7 generates keys in the PEM format, where private keys contain the string BEGIN RSA PRIVATE KEY on the first line of the key. The version shipped with EL8 generates keys in RFC4716 which contain BEGIN OPENSSH PRIVATE KEY and which are considered to be ed25519 keys by the library we use. Hence the error is fired because we don’t ship the dependencies it needs for handling ed25519 keys.

If you don’t mind regenerating your keys, then it is easy. Just regenerate them in the right format with the following command and then redeploy the public key to the target hosts.

sudo -u foreman-proxy ssh-keygen -f ~foreman-proxy/.ssh/id_rsa_foreman_proxy -m PEM -N '' -t rsa -b 4096

We will need to be explicit about the format we require in the installer.

illumino · June 16, 2020, 8:30am

Sorry for the delay in responding.

Recreated ssh keys and updated authorized keys for root.
Tested Remote Execution (again just to the foreman host itself) - No Problems
sudo dnf upgrade moved system from CentOS 8.1 -> CentOS 8.2, Rebooted.
Tested Remote Execution (ditto above aside)- No Problems

Many Thanks,
Peter

PS: On to the ansible …

illumino · June 16, 2020, 11:24am

Continuing the adventure into unknown territory…

The Good:

Remote Execution and Tasks Hammer CLI are functional on both CentOS 7.8 and 8.2

sudo foreman-installer --enable-foreman-cli-remote-execution --enable-foreman-cli-tasks
hammer remote-execution-feature list
hammer remote-execution-feature info --id 1
hammer task list
Ansible features install on CentOS 7.8 without issue

sudo foreman-installer --enable-foreman-cli-ansible --enable-foreman-plugin-ansible --enable-foreman-proxy-plugin-ansible

The Bad: Ansible features install on CentOS 8 fails (same command line as above)

Execution of '/bin/dnf -d 0 -e 1 -y install rubygem-smart_proxy_ansible' returned 1: Error: 
Problem: cannot install the best candidate for the job
.- nothing provides ansible >= 2.2 needed by rubygem-smart_proxy_ansible-3.0.1-5.fm2_1.el8.noarch
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/execution.rb:297:in `execute'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/provider.rb:101:in `execute'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/provider/package/yum.rb:303:in `install'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type/package.rb:98:in `block (3 levels) in <module:Puppet>'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/property.rb:490:in `set'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/property.rb:570:in `sync'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:241:in `sync'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:136:in `sync_if_needed'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:82:in `perform_changes'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:21:in `evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:267:in `apply'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:287:in `eval_resource'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:191:in `call'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:191:in `block (2 levels) in evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:521:in `block in thinmark'
/opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:520:in `thinmark'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:191:in `block in evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:122:in `traverse'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:178:in `evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:240:in `block (2 levels) in apply'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:521:in `block in thinmark'
/opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:520:in `thinmark'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:239:in `block in apply'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/log.rb:161:in `with_destination'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/report.rb:146:in `as_logging_destination'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:238:in `apply'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:185:in `block (2 levels) in apply_catalog'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:521:in `block in thinmark'
/opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:520:in `thinmark'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:184:in `block in apply_catalog'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:233:in `block in benchmark'
/opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:232:in `benchmark'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:183:in `apply_catalog'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:391:in `run_internal'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:227:in `block in run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:314:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:210:in `run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:343:in `apply_catalog'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:260:in `block (2 levels) in main'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:314:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:243:in `block in main'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:314:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:207:in `main'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:177:in `run_command'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:382:in `block in run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:710:in `exit_on_fail'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:382:in `run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:143:in `run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:77:in `execute'
/opt/puppetlabs/puppet/bin/puppet:5:in `<main>'
 Problem: cannot install the best candidate for the job
  - nothing provides ansible >= 2.2 needed by rubygem-smart_proxy_ansible-3.0.1-5.fm2_1.el8.noarch
Preparing installation Done                                              
  Something went wrong! Check the log for ERROR-level output
  The full log is at /var/log/foreman-installer/foreman.log

On CentOS 7, EPEL provides ansible as can be seen in:

sudo yum list installed | grep ansible
ansible.noarch                          2.9.9-1.el7             @epel           
ansible-runner.noarch                   1.4.6-1.el7             @ansible-runner 
python2-ansible-runner.noarch           1.4.6-1.el7             @ansible-runner 
python2-daemon.noarch                   2.1.2-7.el7at           @ansible-runner 
python2-pexpect.noarch                  4.6-1.el7at             @ansible-runner 
python2-ptyprocess.noarch               0.5.2-3.el7at           @ansible-runner 
tfm-rubygem-foreman_ansible.noarch      5.0.1-1.fm2_1.el7       @foreman-plugins
tfm-rubygem-foreman_ansible_core.noarch 3.0.3-1.fm2_1.el7       @foreman-plugins
tfm-rubygem-hammer_cli_foreman_ansible.noarch
tfm-rubygem-smart_proxy_ansible.noarch  3.0.1-5.fm2_1.el7       @foreman-plugins

But EPEL can not be installed on CentOS 8.x without causing problems (as noted in attachment CmdLineHostory…). But I’ll include the observed problem here as I didn’t include the details in the attachment.

The commands are:

sudo yum -y upgrade
sudo yum -y install epel-release
sudo yum -y upgrade
sudo yum -y erase epel-release
sudo yum -y upgrade

The output is:

[admin@virtualcentos8 ~] CentOS 8 $ sudo yum -y upgrade
Last metadata expiration check: 0:28:23 ago on Tue 16 Jun 2020 20:46:03 AEST.
Dependencies resolved.
Nothing to do.
Complete!
[admin@virtualcentos8 ~] CentOS 8 $ sudo yum -y install epel-release
Last metadata expiration check: 0:28:31 ago on Tue 16 Jun 2020 20:46:03 AEST.
Dependencies resolved.
================================================================================
 Package               Architecture    Version            Repository       Size
================================================================================
Installing:
 epel-release          noarch          8-8.el8            extras           23 k

Transaction Summary
================================================================================
Install  1 Package

Total download size: 23 k
Installed size: 32 k
Downloading Packages:
epel-release-8-8.el8.noarch.rpm                 299 kB/s |  23 kB     00:00    
--------------------------------------------------------------------------------
Total                                            22 kB/s |  23 kB     00:01     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                        1/1 
  Installing       : epel-release-8-8.el8.noarch                            1/1 
  Running scriptlet: epel-release-8-8.el8.noarch                            1/1 
  Verifying        : epel-release-8-8.el8.noarch                            1/1 

Installed:
  epel-release-8-8.el8.noarch                                                   

Complete!
[admin@virtualcentos8 ~] CentOS 8 $ sudo yum -y upgrade
Last metadata expiration check: 0:04:47 ago on Tue 16 Jun 2020 21:09:56 AEST.
Error: 
 Problem: package foreman-2.1.0-0.20.rc2.el8.noarch requires rubygem(net-ssh) = 4.2.0, but none of the providers can be installed
  - cannot install both rubygem-net-ssh-5.1.0-2.el8.noarch and rubygem-net-ssh-4.2.0-2.el8.noarch
  - cannot install both rubygem-net-ssh-4.2.0-2.el8.noarch and rubygem-net-ssh-5.1.0-2.el8.noarch
  - cannot install the best update candidate for package rubygem-net-ssh-4.2.0-2.el8.noarch
  - cannot install the best update candidate for package foreman-2.1.0-0.20.rc2.el8.noarch
(try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
[admin@virtualcentos8 ~] CentOS 8 $ sudo yum -y erase epel-release
Dependencies resolved.
================================================================================
 Package               Architecture    Version           Repository        Size
================================================================================
Removing:
 epel-release          noarch          8-8.el8           @extras           32 k

Transaction Summary
================================================================================
Remove  1 Package

Freed space: 32 k
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                        1/1 
  Running scriptlet: epel-release-8-8.el8.noarch                            1/1 
  Erasing          : epel-release-8-8.el8.noarch                            1/1 
  Running scriptlet: epel-release-8-8.el8.noarch                            1/1 
  Verifying        : epel-release-8-8.el8.noarch                            1/1 

Removed:
  epel-release-8-8.el8.noarch                                                   

Complete!
[admin@virtualcentos8 ~] CentOS 8 $ sudo yum -y upgrade
Last metadata expiration check: 0:28:53 ago on Tue 16 Jun 2020 20:46:03 AEST.
Dependencies resolved.
Nothing to do.
Complete!
[admin@virtualcentos8 ~] CentOS 8 $

This will be a much harder problem to solve as it affects even the most basic Foreman install on CentOS 8, if I recall correctly!

tbrisker · June 16, 2020, 12:13pm

Following up from the other thread, EPEL shouldn’t be required on EL8. @aruzicka are we missing some requirement of rubygem-smart_proxy_ansible or a repo in the install instruction? where do we get ansible from on el8?

aruzicka · June 16, 2020, 12:58pm

When checking repoclosure in packaging we’re using the repository at http://mirrorlist.centos.org/?release=8&arch=x86_64&repo=configmanagement-ansible-29.

When you install centos-release-ansible-29 on EL8 it will set things up for you to be able to pull ansible from there. This should probably be mentioned in the docs and maybe the installer should be able to deal with it?

illumino · June 16, 2020, 1:55pm

This definitely needs to be in the documentation for CentOS 8, as I just did a quick google search (to conform my suspicion), and the first half-a-dozen how-to links on Ansible on RHEL8/CentOS8 said enable the EPEL repo, and than either dnf install ansible or install python3-pip and pip3 install ansible.

So documentation and/or installer changes will be required to ensure a good first experience - IMHO.

illumino · June 17, 2020, 12:54am

I can report that with the changes for remote execution (discussed above) and the CentOS configuration management ansible dnf repository (discussed below), I was able to install the Foreman Ansible features on CentOS 8.2 / Foreman 2.1rc2, as follows:

sudo vi /etc/yum.repos.d/CentOS-ansible.repo
cat     /etc/yum.repos.d/CentOS-ansible.repo
[ansible]
name=CentOS-$releasever - Ansible 2.9
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=configmanagement-ansible-29&infra=$infra
gpgcheck=1
enabled=1
gpgkey=https://www.centos.org/keys/RPM-GPG-KEY-CentOS-SIG-ConfigManagement

sudo yum list ansible

# Enable Foreman's Ansible Features

	sudo foreman-installer --enable-foreman-cli-ansible --enable-foreman-plugin-ansible --enable-foreman-proxy-plugin-ansible

Both the WebUI and CLI ansible features are present, I’ll look into testing these once I have finished reading the relevant documentation

illumino · June 17, 2020, 2:22am

Sorry, I misunderstood part of Adam’s earlier response (I’ll blame the headache I had ). The procedure should be as suggested by Adam:

sudo dnf install centos-release-ansible-29
sudo dnf list ansible
sudo foreman-installer --enable-foreman-cli-ansible --enable-foreman-plugin-ansible --enable-foreman-proxy-plugin-ansible

There is no need to manually configure the dnf repository.

illumino · June 18, 2020, 4:58am

As the saying goes: If it is not tested, it doesn’t work
and another saying goes: Testing can only show the presence of bugs, never there absence

Summary: Ansible (“Run all ansible roles”) fails on CentOS 8

After installing the ansible features as described above on both CentOS 7.8 and CentOS 8.2 VM previously mentioned in the posting, we have the following installed:

CentOS 7:

sudo yum list installed | grep ansible

ansible.noarch                          2.9.9-1.el7             @epel
ansible-runner.noarch                   1.4.6-1.el7             @ansible-runner
python2-ansible-runner.noarch           1.4.6-1.el7             @ansible-runner
python2-daemon.noarch                   2.1.2-7.el7at           @ansible-runner
python2-pexpect.noarch                  4.6-1.el7at             @ansible-runner
python2-ptyprocess.noarch               0.5.2-3.el7at           @ansible-runner
tfm-rubygem-foreman_ansible.noarch      5.0.1-1.fm2_1.el7       @foreman-plugins
tfm-rubygem-foreman_ansible_core.noarch 3.0.3-1.fm2_1.el7       @foreman-plugins
tfm-rubygem-hammer_cli_foreman_ansible.noarch
tfm-rubygem-smart_proxy_ansible.noarch  3.0.1-5.fm2_1.el7       @foreman-plugins

CentOS 8:

sudo yum list installed | grep ansible

ansible.noarch                                     2.9.9-2.el8                                      @ansible
ansible-runner.noarch                              1.4.6-1.el8                                      @ansible-runner
centos-release-ansible-29.noarch                   1-2.el8                                          @extras
python3-ansible-runner.noarch                      1.4.6-1.el8                                      @ansible-runner
python3-daemon.noarch                              2.1.2-9.el8ar                                    @ansible-runner
python3-lockfile.noarch                            1:0.11.0-8.el8ar                                 @ansible-runner
python3-pexpect.noarch                             4.6-2.el8ar                                      @ansible-runner
rubygem-foreman_ansible.noarch                     5.1.1-1.fm2_1.el8                                @foreman-plugins
rubygem-foreman_ansible_core.noarch                3.0.3-1.fm2_1.el8                                @foreman-plugins
rubygem-hammer_cli_foreman_ansible.noarch          0.3.2-1.fm2_1.el8                                @foreman-plugins
rubygem-smart_proxy_ansible.noarch                 3.0.1-5.fm2_1.el8                                @foreman-plugins
sshpass.x86_64                                     1.06-8.el8                                       @ansible

Unless stated otherwise, commands are being performed on both VMs.

sudo find /etc -name ansible.cfg
/etc/foreman-proxy/ansible.cfg
/etc/ansible/ansible.cfg

sudo egrep -v '^#' /etc/foreman-proxy/ansible.cfg
sudo egrep -v '^#' /etc/ansible/ansible.cfg  | uniq

As section heading can only appear once in ansible.cfg, and there is no (actual) default configuration for ansible, lets just replace the package manager provided configuration, with the one foreman-installer generated.

sudo cp /etc/ansible/ansible.cfg /etc/ansible/ansible.cfg.orig
sudo cp /etc/foreman-proxy/ansible.cfg /etc/ansible/ansible.cfg
ls -l   /etc/foreman-proxy/ansible.cfg /etc/ansible/ansible.cfg

# Verify command line ansible is functional
ansible -m ping localhost
localhost | SUCCESS => {
    "changed": false,
    "ping": "pong"
}

ansible -a 'cat /etc/redhat-release' localhost
localhost | CHANGED | rc=0 >>

CentOS 7: CentOS Linux release 7.8.2003 (Core)
CentOS 8: CentOS Linux release 8.2.2004 (Core)

I than selected the WebUI -> Hosts -> All Hosts -> Select Action: Run all ansible roles
(which is running against the Foreman host itself)

This completed successfully after a few seconds on CentOS 7 but failed on CentOS 8.

Given I had not updated either system for a few days, I did a sudo yum upgrade on both systems. Only CentOS 7 had updates to apply and they are listed at the end of this posting. I followed this by a sudo foreman-maintain service restart on BOTH systems.

The test was repeated with the same results (CentOS 7: Success & CentOS 8: Failed). I further tested just remote execution, and it was functional, as can be seen in the following screen snap (from the CentOS 8 WebUI):

I reviewed the backtrace / stacktrace in each failure and they are the same (no differences at all), and it is attached here:
CentOS8-Run1.stacktrace.log (9.4 KB)

The following log files don’t add any value, that I can see:

/var/log/foreman-proxy/smart_proxy_dynflow_core.log
/var/log/foreman-proxy/proxy.log
/var/log/foreman/production.log

Regards,
Peter

The CentOS 7 packages updated

 Package                                              Arch                   Version                           Repository                       Size
=====================================================================================================================================================
Updating:
 tfm-rubygem-foreman-tasks                            noarch                 2.0.0-1.fm2_1.el7                 foreman-plugins                 2.2 M
 tfm-rubygem-foreman_ansible                          noarch                 5.1.1-1.fm2_1.el7                 foreman-plugins                 2.0 M
 tfm-rubygem-foreman_remote_execution                 noarch                 3.3.1-1.fm2_1.el7                 foreman-plugins                 1.6 M

Transaction Summary
=====================================================================================================================================================
Upgrade  3 Packages

aruzicka · June 18, 2020, 7:18am

The stacktrace you provided doesn’t contain anything interesting as it will always contain exactly the same thing if at least one host in the job fails. Could you provide output for the host the failed?

illumino · June 18, 2020, 12:14pm

Adam, Sorry about that - brain fade - attached is the requested SubTask stacktrace + error - Peter
CentOS8-Run2-Subtask-Stacktrace.log (12.2 KB)

aruzicka · June 18, 2020, 12:31pm

Still not quite what I’m after. Either go to the job details and click the hostname in the table at the bottom. Alternatively going to job details > job task > sub tasks > click any of the tasks > dynflow console > run tab > proxy action and get everything from there.

illumino · June 18, 2020, 1:35pm

Adam, Thanks for pointing me in the right direction, the referenced logs are very useful and good to known that level of detail exists.

I’ll attach both the functional CentOS7 and non-functional CentOS8 logs.

My analysis is: the CentOS 8 system tries to run the “script” from the sub-task definition as a shell script (generating a lot of command not found errors). The task definitions are essentially identical (save system name/IP/MAC/SSHkeys differences). The output of course is very different.

CentOS7-DynFlow-SubTask-Redacted.log (7.7 KB)
CentOS8-DynFlow-SubTask-Redacted.log (6.6 KB)

CentOS 7 $ which  ansible
/usr/bin/ansible
CentOS 7 $ ansible --version
ansible 2.9.9
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/home/admin/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.5 (default, Apr  2 2020, 13:16:51) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]

CentOS 8 $ which  ansible
/usr/bin/ansible
CentOS 8 $ ansible --version
ansible 2.9.9
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/admin/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.6/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 3.6.8 (default, Apr 16 2020, 01:36:27) [GCC 8.3.1 20191121 (Red Hat 8.3.1-5)]

aruzicka · June 19, 2020, 8:37am

This is the same issue as in comment 13. Wrong macro expansion puts a needed file into a wrong location. Here’s a workaround for now

 ln -s /opt/theforeman/tfm/root/usr/share/smart_proxy_dynflow_core/bundler.d/foreman_ansible_core.rb /usr/share/smart_proxy_dynflow_core/bundler.d/

illumino · June 19, 2020, 11:19am

Failed the first test, restarted using sudo foreman-maintain service restart and repeated the test without issue.

BTW: I yum/dnf upgraded in-place to 2.1.0rc3 earlier today, so that is what is being tested.

sudo  ln -s /opt/theforeman/tfm/root/usr/share/smart_proxy_dynflow_core/bundler.d/foreman_ansible_core.rb /usr/share/smart_proxy_dynflow_core/bundler.d/
ls -l /usr/share/smart_proxy_dynflow_core/bundler.d/
    total 0
    lrwxrwxrwx. 1 root root  93 Jun 19 20:57 foreman_ansible_core.rb -> /opt/theforeman/tfm/root/usr/share/smart_proxy_dynflow_core/bundler.d/foreman_ansible_core.rb
    lrwxrwxrwx. 1 root root 102 Jun 12 16:25 foreman_remote_execution_core.rb -> /opt/theforeman/tfm/root/usr/share/smart_proxy_dynflow_core/bundler.d/foreman_remote_execution_core.rb
ls -lL /usr/share/smart_proxy_dynflow_core/bundler.d/
    total 8
    -rw-r--r--. 1 root root 27 Jun  1 22:49 foreman_ansible_core.rb
    -rw-r--r--. 1 root root 36 Jun  1 23:54 foreman_remote_execution_core.rb
sudo foreman-maintain service restart

Again, Many Thanks!!!
Peter

PS: I notice that Katello repositories exist the EL8 packages - Is it worth trying the 3.16.0rc3 on CentOS 8, at this stage, or is it a little early for that?

illumino · June 20, 2020, 11:59am

Topic Summary + Remaining TO-DOs

Note: I have not jumped into the code base to answer the questions I’m raising and the answers may be contained in the code base. I’m coming to Foreman as a new user, not a developer.

Note: Also the Low Priority section ended up longer than expected, you may wish to jump to the Foreman 2.1.0rc* specifics after that section, which summarizes the problems and fixes and if they have a known Foreman Bug Ids (at the time of writing).

Low Priority

The warning generated by smart_proxy_dynflow_core.service:
Can't open PID file /var/run/foreman-proxy/smart_proxy_dynflow_core.pid (yet?) after start: No such file or directory
Question: Does this problem exist because the /var/run/foreman-proxy directory does not exist when the service starts? That is, should the service add a dependency on systemd-tmpfiles-setup.service? (that service does create this directory)
Question: Is this actually a complaint that the file smart_proxy_dynflow_core.pid does not exist? As noted in the previous question, the directory it is located in does not exist until created on each system boot by the systemd-tmpfiles-setup.service. So the question, than I believe becomes …
Question: Should the PID file exist at service startup?
If so, that can be accommodated by systemd-tmpfiles-setup.service creating any empty file for the code base, and adding the dependency on that service.
Alternatively, if the file does not need to exist (IMHO the likely case), should the warning message be changed, or the logic that generated it should be changed?
Commentary: As I long time developer and system administrator, I would suggest that the PID file is likely used, to check if another instance of the process is already running. This check is often done to prevent other problems being generated and reported, like trying to listen on port that is already in use, etc. In my experience this rarely done well, especially on older systems with non-volatile storage areas for PID files. As the PID file may contain out-of-date data, and checking if a process with that PID simply exists, does not mean it is an already running instance of this process (two reasons: non-volatile PID storage areas especially on boot, or system Process IDs have wrapped on a long running systems). So what is done with the contents of the PID file (if non-empty) needs to be investigated.
Depending on the outcome of the investigation and the future directions of the project, one or more of the following may be viable:
1. Add a dependency on systemd-tmpfiles-setup.service;
2. Use systemd-tmpfiles-setup.service to create an empty PID file;
3. Only generate a (warning) message with the PID to be checked, when the PID file exists and contains a valid PID (c.f. an empty file, or a file with nonsensical data);
4. Drop the check and rely on the service management software (systemd(1), …) to ensure a single instance of the service - N.B. developers may push for its retention, on:
  - dev-systems when debugging a non-installed instance of the service;
  - if the code is ever deployed on systems without modern service management software - given the code is part of smart-proxy and not dynflow that seems unlikely;
5. Something I haven’t discussed guided by the outcome of the investigation;
Ansible Plugin Documentation
Latest Documentation: Foreman :: Plugin Manuals

Fixable: The page is named ‘Foreman :: Plugin Manuals’, when its the ‘Ansible Plugin 3.x Manual’, given this would likely be generated, it could contain the actual plugin name and version, as this would improve tab names and bookmarks in your browser.

Fixable: The Ansible Plugin Versioning for End-Users
- The rubygem-foreman_ansible version is 5.1.1;
- The rubygem-foreman_ansible_core version is 3.0.3;
- The WebUI -> Infrastructure -> Smart Proxies -> (selected proxy), tab labeled ‘Services’ states Ansible is version 3.0.1;
So I’m at a bit lost as to the version of the Ansible Plugin Manual, that I should be reading. I assume there should be a 5.X manual rather than the 3.X one provided by the project’s web site. If not, maybe a sentence could be added to the Ansible Plugin manual explaining the plugin version versus the packaging versions.

Fixable: In Section 2.1 Ansible callback, it documents the changes required to your ansible configuration, without mentioning that the installer will generate a sample and usual configuration (/etc/foreman-proxy/ansible.cfg) during plugin installation when using foreman-installer. I, in fact, used the generated configuration for my testing (by copying it to /etc/ansible/ansible.cfg), so I did not have the changes suggested in-place before running the foreman-installer. I would suggest that the plugin maintainer, review if the best way to document the sequence is to run foreman-installer first, and than configure ansible, as a second step.

Fixable: Given the Section 2.1 Ansible callback documentation, the generated configuration /etc/foreman-proxy/ansible.cfg has additional configuration specifications. Maybe the documentation should suggest reading the generated file for the latest Foreman Ansible Plugin recommended ansible.cfg configuration, as this would save having to document everything in the written documentation. The generated file, could also document in comments, the mandatory configuration versus the recommended / optional configuration. Also being generated, the file could account for Ansible version, OS and OS version and other factors in what it generates, meaning that the options required on older ansible versions or platforms would not be generated on newer (remediated) platforms. This could reduce updates to the documentation, leaving them in generated comments, that will only appear when necessary. If necessary, point the reader to the template used to generate the sample configuration. The ansible.cfg template way wish to document in comments the key template variable values used to generate it, so users don’t widely assume it is generic.
Foreman Documentation for New Users

Being new to Foreman, I feel that the Foreman 2.X manual, could probably be enhanced by starting with a “What is Foreman?” section, summarizing the high level features in a paragraph or two, and suggest that the reader review Foreman :: Introduction, before continuing.

After the Smart-Proxy description in Section 1 of the Foreman Manual, it would be useful to have a high-level set of scenarios and what components would be used in such a scenario. For example, some provisioning scenarios might include:
- Single Site - One Subnet (Simplest Case)
- Multiple Sites - One Subnet at each site
- Single Site - Multiple Subnets
- Multiple Sites - Multiple Subnets (I would hope documenting this would be unnecessary - if I understand Foreman correctly, as its a combination of approach of the last two)
These scenarios could discuss basic architectural requirements, like type of servers needed (by this I mean foreman and smart-proxy instances and their high level configuration). Details like how the foreman-installer only supports a single subnet configuration via the command line, but does support multiple subnet via confgiuration files, with references to appropriate part of the manual.

Importantly, this introduction should probably discribe what Katello adds to the Foreman architectural picture (high level), and how to architect the above scenarios (say assuming limited bandwidth inter-site links), if Katello is part of the picture. Also if my understanding is correct - that if you wish to use Katello, you need to follow the Katello installation documentation, and use this documentation as a referenece for the Foreman and Smart-Proxy components of the system.

Finally in section 1 of the manual, I feel references to the other resources available to assist in your understanding of Foreman / Smart-Proxy / Katello / CandlePin. A brief (say one paragraph) introduction or history of each and maybe a reference to the commerical products they appear in. But I feel the key to the last part of section 1 should be an appropriately qualified link to work-in-progress and existing documentation available for Foreman and its derived products http://docs.theforeman.org/web/. The guides on these pages greatly assisted in my understanding of the system. I feel it’s fine to say in the qualifications that: other parties may have authored some guides (e.g. RedHat); they may be guides to existing commerical derivates which have yet to be updated the Foreman 2.X; and such. But because they are classed a work-in-progress they weren’t the easiest to fine (for me at least), but IMHO they need to read as part of introduction or guide to new users. The “Provisioning Guide” was well written, and showed off the [very cool IMHO] hammer(1) CLI interface. Qualify the link appropriately, WIP/out-of-date/etc but link to that resource, and maybe suggest a reading order for some of the guides, or update the referenced page, to provide additional information like which open source products a guide covers, and what major versions its has been written for, and if it’s worthwhile reading for the current open-source revisions of these products.

Fixes for 2.1.0rc* before its General Availability release (IMHO)

Packaging Fixes for Smart Proxy Modules
The installation of files into their older installation location (/opt/theforeman): I discovered two instances during my testing on CentOS 8, which @aruzicka offered simple workarounds for:
sudo ln -s /opt/theforeman/tfm/root/usr/share/smart_proxy_dynflow_core/bundler.d/foreman_remote_execution_core.rb /usr/share/smart_proxy_dynflow_core/bundler.d/
sudo ln -s /opt/theforeman/tfm/root/usr/share/smart_proxy_dynflow_core/bundler.d/foreman_ansible_core.rb /usr/share/smart_proxy_dynflow_core/bundler.d/
sudo systemctl restart smart_proxy_dynflow_core.service
Today, I downloaded all Foreman and Foreman Plugin 2.1.0rc* packages for EL8, and found another instance in rubygem-smart-proxy-probing for the file foreman_probing_core.rb.
I’ll also note that every RPM (all 661 of then) in the main foreman repository generated a warning of the form: warning: FILENAME.rpm: Header V4 RSA/SHA256 Signature, key ID b6f08ccf: NOKEY. No foreman-plugin RPMs generated this warning. Fedora 32 rpm(8) was used to check the installation paths via rpm -qlp $rpm | grep '^/opt/' in a command line for loop.
Affected Packages:
- rubygem-smart_proxy_ansible
- rubygem-smart-proxy-probing
- rubygem-smart_proxy_remote_execution_ssh
Foreman Bug Id: Unknown
Compatibility RHEL/CentOS 8 and net-ssh

Problem 1: ssh-keygen(1) default output format on EL8 - as noted in Comment 18.
Workaround: sudo -u foreman-proxy ssh-keygen -f ~foreman-proxy/.ssh/id_rsa_foreman_proxy -m PEM -N '' -t rsa -b 4096
Foreman Bug Id: Bug #30121: ssh-keygen on EL8 generates private keys in RFC4716 format, net-ssh requires PEM format - Installer - Foreman

Problem 2: Use of rubygem-net-ssh-4.2.0 blocking use of EPEL on CentOS 8
Further Discussion: Foreman 2.1.0-rc3 release process
Workaround: See post 9 in the above discussion.
Foreman Bug Id: Bug #30162: Allow newer Net::SSH - Foreman
Ansible - Which RPM repsoitory to use.

Ansible is available from at least two repsoitories on CentOS 8: EPEL and CentOS ConfigManagement SIG. As discussed above, EPEL repository usage currently causes problems. This topic showed that in fact the Foreman team (or at least Adam) use the CentOS ConfigManagement SIG repository, that fact should be documented.
Solution: sudo dnf install centos-release-ansible-29 ; sudo dnf list ansible
Foreman Bug Id: Unknown - @mcorr stated she has raised an issue for this in the rc3 checklist topic - Needs documenting in the Quick Start Guide for 2.1.0 and CentOS 8

Finally, as someone completely new to Foreman, I would like to acknowledge the patience and helpfulness of @aruzicka! And the welcoming and friendly responses from Adam, @ekohl, @evgeni, @tbrisker and @mcorr. After all, jumping in as a newbie to Foreman using a release candidate on a newly supported OS - was probably not the simplest route to understanding the product! But having to write-up/debug issues really does assist in that initial understanding of a product, especially with such a helpful community!

Regards,
Peter

mcorr · June 22, 2020, 8:25am

Thank you so much for your feedback. From a documentation perspective alone, this will provide great guidance on how to make improvements in the immediate term that will make things better for new users who come after you.

This is on my to-do list to check through all the plugins and ensure there is an up-to-date manual for each, and if not, gently guide the original author towards providing one

My colleague @tahliar is currently migrating a “Planning” guide from the downstream Red Hat docs that is a deep dive into the architecture and concepts of Foreman. I will link to this from the existing Foreman guides as we move towards consolidating old and new documentation.

aruzicka · June 22, 2020, 8:30am

I can’t speak for Katello, but maybe @iballou could chime in?

This was tracked as Bug #27184: [smart_proxy_dynflow_core] "PID file /var/run/foreman-proxy/smart_proxy_dynflow_core.pid not readable (yet?) after start" - foreman-tasks - Foreman and should be fixed in smart_proxy_dynflow_core-0.2.6.

All the things you mentioned are valid, thank you for bearing with us and pointing those things out

evgeni · June 22, 2020, 10:32am

Ad Katello: 3.16 is not (yet) fully available on EL8 - and that’s why we’ve not been advetising it yet. @ehelms and @Justin_Sherrill are working on it.

Ad Ansible repo: yes, we’d prefer if you’d use the config management repo – but epel should work too. I had raised this point in https://github.com/theforeman/theforeman.org/pull/1612 when we documented EL8 things, but we didn’t add it due to not being a dependency of Foreman itself. Wonder what would be a good place to document optional things. @ehelms @mcorr what do you think?

Ad /opt/theforeman/ in EL8 packages: I’ll take a look – that’s a bug.

Ad Ansible version: the version you see in “Infra → Proxies” is the proxy plugin, and that is really 3.0.1, while the Foreman plugin is 5.1.1 (and 3.0.3 for “core”). A tad irritating, I agree. @aruzicka any way we could “fix” that nicely?