When checking repoclosure in packaging we’re using the repository at http://mirrorlist.centos.org/?release=8&arch=x86_64&repo=configmanagement-ansible-29.
When you install centos-release-ansible-29 on EL8 it will set things up for you to be able to pull ansible from there. This should probably be mentioned in the docs and maybe the installer should be able to deal with it?
This definitely needs to be in the documentation for CentOS 8, as I just did a quick google search (to conform my suspicion), and the first half-a-dozen how-to links on Ansible on RHEL8/CentOS8 said enable the EPEL repo, and than either dnf install ansible or install python3-pip and pip3 install ansible.
So documentation and/or installer changes will be required to ensure a good first experience - IMHO.
I can report that with the changes for remote execution (discussed above) and the CentOS configuration management ansible dnf repository (discussed below), I was able to install the Foreman Ansible features on CentOS 8.2 / Foreman 2.1rc2, as follows:
sudo vi /etc/yum.repos.d/CentOS-ansible.repo
cat /etc/yum.repos.d/CentOS-ansible.repo
[ansible]
name=CentOS-$releasever - Ansible 2.9
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=configmanagement-ansible-29&infra=$infra
gpgcheck=1
enabled=1
gpgkey=https://www.centos.org/keys/RPM-GPG-KEY-CentOS-SIG-ConfigManagement
sudo yum list ansible
# Enable Foreman's Ansible Features
sudo foreman-installer --enable-foreman-cli-ansible --enable-foreman-plugin-ansible --enable-foreman-proxy-plugin-ansible
Both the WebUI and CLI ansible features are present, I’ll look into testing these once I have finished reading the relevant documentation 
Sorry, I misunderstood part of Adam’s earlier response (I’ll blame the headache I had 
). The procedure should be as suggested by Adam:
sudo dnf install centos-release-ansible-29
sudo dnf list ansible
sudo foreman-installer --enable-foreman-cli-ansible --enable-foreman-plugin-ansible --enable-foreman-proxy-plugin-ansible
There is no need to manually configure the dnf repository.
As the saying goes: If it is not tested, it doesn’t work
and another saying goes: Testing can only show the presence of bugs, never there absence
Summary: Ansible (“Run all ansible roles”) fails on CentOS 8
After installing the ansible features as described above on both CentOS 7.8 and CentOS 8.2 VM previously mentioned in the posting, we have the following installed:
CentOS 7:
sudo yum list installed | grep ansible
ansible.noarch 2.9.9-1.el7 @epel
ansible-runner.noarch 1.4.6-1.el7 @ansible-runner
python2-ansible-runner.noarch 1.4.6-1.el7 @ansible-runner
python2-daemon.noarch 2.1.2-7.el7at @ansible-runner
python2-pexpect.noarch 4.6-1.el7at @ansible-runner
python2-ptyprocess.noarch 0.5.2-3.el7at @ansible-runner
tfm-rubygem-foreman_ansible.noarch 5.0.1-1.fm2_1.el7 @foreman-plugins
tfm-rubygem-foreman_ansible_core.noarch 3.0.3-1.fm2_1.el7 @foreman-plugins
tfm-rubygem-hammer_cli_foreman_ansible.noarch
tfm-rubygem-smart_proxy_ansible.noarch 3.0.1-5.fm2_1.el7 @foreman-plugins
CentOS 8:
sudo yum list installed | grep ansible
ansible.noarch 2.9.9-2.el8 @ansible
ansible-runner.noarch 1.4.6-1.el8 @ansible-runner
centos-release-ansible-29.noarch 1-2.el8 @extras
python3-ansible-runner.noarch 1.4.6-1.el8 @ansible-runner
python3-daemon.noarch 2.1.2-9.el8ar @ansible-runner
python3-lockfile.noarch 1:0.11.0-8.el8ar @ansible-runner
python3-pexpect.noarch 4.6-2.el8ar @ansible-runner
rubygem-foreman_ansible.noarch 5.1.1-1.fm2_1.el8 @foreman-plugins
rubygem-foreman_ansible_core.noarch 3.0.3-1.fm2_1.el8 @foreman-plugins
rubygem-hammer_cli_foreman_ansible.noarch 0.3.2-1.fm2_1.el8 @foreman-plugins
rubygem-smart_proxy_ansible.noarch 3.0.1-5.fm2_1.el8 @foreman-plugins
sshpass.x86_64 1.06-8.el8 @ansible
Unless stated otherwise, commands are being performed on both VMs.
sudo find /etc -name ansible.cfg
/etc/foreman-proxy/ansible.cfg
/etc/ansible/ansible.cfg
sudo egrep -v '^#' /etc/foreman-proxy/ansible.cfg
sudo egrep -v '^#' /etc/ansible/ansible.cfg | uniq
As section heading can only appear once in ansible.cfg, and there is no (actual) default configuration for ansible, lets just replace the package manager provided configuration, with the one foreman-installer generated.
sudo cp /etc/ansible/ansible.cfg /etc/ansible/ansible.cfg.orig
sudo cp /etc/foreman-proxy/ansible.cfg /etc/ansible/ansible.cfg
ls -l /etc/foreman-proxy/ansible.cfg /etc/ansible/ansible.cfg
# Verify command line ansible is functional
ansible -m ping localhost
localhost | SUCCESS => {
"changed": false,
"ping": "pong"
}
ansible -a 'cat /etc/redhat-release' localhost
localhost | CHANGED | rc=0 >>
CentOS 7: CentOS Linux release 7.8.2003 (Core)
CentOS 8: CentOS Linux release 8.2.2004 (Core)
I than selected the WebUI → Hosts → All Hosts → Select Action: Run all ansible roles
(which is running against the Foreman host itself)
This completed successfully after a few seconds on CentOS 7 but failed on CentOS 8.
Given I had not updated either system for a few days, I did a sudo yum upgrade on both systems. Only CentOS 7 had updates to apply and they are listed at the end of this posting. I followed this by a sudo foreman-maintain service restart on BOTH systems.
The test was repeated with the same results (CentOS 7: Success & CentOS 8: Failed). I further tested just remote execution, and it was functional, as can be seen in the following screen snap (from the CentOS 8 WebUI):
I reviewed the backtrace / stacktrace in each failure and they are the same (no differences at all), and it is attached here:
CentOS8-Run1.stacktrace.log (9.4 KB)
The following log files don’t add any value, that I can see:
/var/log/foreman-proxy/smart_proxy_dynflow_core.log
/var/log/foreman-proxy/proxy.log
/var/log/foreman/production.log
Regards,
Peter
Package Arch Version Repository Size
=====================================================================================================================================================
Updating:
tfm-rubygem-foreman-tasks noarch 2.0.0-1.fm2_1.el7 foreman-plugins 2.2 M
tfm-rubygem-foreman_ansible noarch 5.1.1-1.fm2_1.el7 foreman-plugins 2.0 M
tfm-rubygem-foreman_remote_execution noarch 3.3.1-1.fm2_1.el7 foreman-plugins 1.6 M
Transaction Summary
=====================================================================================================================================================
Upgrade 3 Packages
The stacktrace you provided doesn’t contain anything interesting as it will always contain exactly the same thing if at least one host in the job fails. Could you provide output for the host the failed?
Adam, Sorry about that - brain fade - attached is the requested SubTask stacktrace + error - Peter
CentOS8-Run2-Subtask-Stacktrace.log (12.2 KB)
Still not quite what I’m after. Either go to the job details and click the hostname in the table at the bottom. Alternatively going to job details > job task > sub tasks > click any of the tasks > dynflow console > run tab > proxy action and get everything from there.
Adam, Thanks for pointing me in the right direction, the referenced logs are very useful and good to known that level of detail exists.
I’ll attach both the functional CentOS7 and non-functional CentOS8 logs.
My analysis is: the CentOS 8 system tries to run the “script” from the sub-task definition as a shell script (generating a lot of command not found errors). The task definitions are essentially identical (save system name/IP/MAC/SSHkeys differences). The output of course is very different.
CentOS7-DynFlow-SubTask-Redacted.log (7.7 KB)
CentOS8-DynFlow-SubTask-Redacted.log (6.6 KB)
CentOS 7 $ which ansible
/usr/bin/ansible
CentOS 7 $ ansible --version
ansible 2.9.9
config file = /etc/ansible/ansible.cfg
configured module search path = [u'/home/admin/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
ansible python module location = /usr/lib/python2.7/site-packages/ansible
executable location = /usr/bin/ansible
python version = 2.7.5 (default, Apr 2 2020, 13:16:51) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]
CentOS 8 $ which ansible
/usr/bin/ansible
CentOS 8 $ ansible --version
ansible 2.9.9
config file = /etc/ansible/ansible.cfg
configured module search path = ['/home/admin/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /usr/lib/python3.6/site-packages/ansible
executable location = /usr/bin/ansible
python version = 3.6.8 (default, Apr 16 2020, 01:36:27) [GCC 8.3.1 20191121 (Red Hat 8.3.1-5)]
This is the same issue as in comment 13. Wrong macro expansion puts a needed file into a wrong location. Here’s a workaround for now
ln -s /opt/theforeman/tfm/root/usr/share/smart_proxy_dynflow_core/bundler.d/foreman_ansible_core.rb /usr/share/smart_proxy_dynflow_core/bundler.d/
Failed the first test, restarted using sudo foreman-maintain service restart and repeated the test without issue. 
BTW: I yum/dnf upgraded in-place to 2.1.0rc3 earlier today, so that is what is being tested.
sudo ln -s /opt/theforeman/tfm/root/usr/share/smart_proxy_dynflow_core/bundler.d/foreman_ansible_core.rb /usr/share/smart_proxy_dynflow_core/bundler.d/
ls -l /usr/share/smart_proxy_dynflow_core/bundler.d/
total 0
lrwxrwxrwx. 1 root root 93 Jun 19 20:57 foreman_ansible_core.rb -> /opt/theforeman/tfm/root/usr/share/smart_proxy_dynflow_core/bundler.d/foreman_ansible_core.rb
lrwxrwxrwx. 1 root root 102 Jun 12 16:25 foreman_remote_execution_core.rb -> /opt/theforeman/tfm/root/usr/share/smart_proxy_dynflow_core/bundler.d/foreman_remote_execution_core.rb
ls -lL /usr/share/smart_proxy_dynflow_core/bundler.d/
total 8
-rw-r--r--. 1 root root 27 Jun 1 22:49 foreman_ansible_core.rb
-rw-r--r--. 1 root root 36 Jun 1 23:54 foreman_remote_execution_core.rb
sudo foreman-maintain service restart
Again, Many Thanks!!!
Peter
PS: I notice that Katello repositories exist the EL8 packages - Is it worth trying the 3.16.0rc3 on CentOS 8, at this stage, or is it a little early for that?
Topic Summary + Remaining TO-DOs
Note: I have not jumped into the code base to answer the questions I’m raising and the answers may be contained in the code base. I’m coming to Foreman as a new user, not a developer.
Note: Also the Low Priority section ended up longer than expected, you may wish to jump to the Foreman 2.1.0rc* specifics after that section, which summarizes the problems and fixes and if they have a known Foreman Bug Ids (at the time of writing).
Low Priority
The warning generated by smart_proxy_dynflow_core.service:
Can't open PID file /var/run/foreman-proxy/smart_proxy_dynflow_core.pid (yet?) after start: No such file or directory
Question: Does this problem exist because the /var/run/foreman-proxy directory does not exist when the service starts? That is, should the service add a dependency on systemd-tmpfiles-setup.service? (that service does create this directory)
Question: Is this actually a complaint that the file smart_proxy_dynflow_core.pid does not exist? As noted in the previous question, the directory it is located in does not exist until created on each system boot by the systemd-tmpfiles-setup.service. So the question, than I believe becomes …
Question: Should the PID file exist at service startup?
If so, that can be accommodated by systemd-tmpfiles-setup.service creating any empty file for the code base, and adding the dependency on that service.
Alternatively, if the file does not need to exist (IMHO the likely case), should the warning message be changed, or the logic that generated it should be changed?
Commentary: As I long time developer and system administrator, I would suggest that the PID file is likely used, to check if another instance of the process is already running. This check is often done to prevent other problems being generated and reported, like trying to listen on port that is already in use, etc. In my experience this rarely done well, especially on older systems with non-volatile storage areas for PID files. As the PID file may contain out-of-date data, and checking if a process with that PID simply exists, does not mean it is an already running instance of this process (two reasons: non-volatile PID storage areas especially on boot, or system Process IDs have wrapped on a long running systems). So what is done with the contents of the PID file (if non-empty) needs to be investigated.
Depending on the outcome of the investigation and the future directions of the project, one or more of the following may be viable:
systemd-tmpfiles-setup.service;systemd-tmpfiles-setup.service to create an empty PID file;systemd(1), …) to ensure a single instance of the service - N.B. developers may push for its retention, on:
Ansible Plugin Documentation
Latest Documentation: Foreman :: Plugin Manuals
Fixable: The page is named ‘Foreman :: Plugin Manuals’, when its the ‘Ansible Plugin 3.x Manual’, given this would likely be generated, it could contain the actual plugin name and version, as this would improve tab names and bookmarks in your browser.
Fixable: The Ansible Plugin Versioning for End-Users
rubygem-foreman_ansible version is 5.1.1;rubygem-foreman_ansible_core version is 3.0.3;So I’m at a bit lost as to the version of the Ansible Plugin Manual, that I should be reading. I assume there should be a 5.X manual rather than the 3.X one provided by the project’s web site. If not, maybe a sentence could be added to the Ansible Plugin manual explaining the plugin version versus the packaging versions.
Fixable: In Section 2.1 Ansible callback, it documents the changes required to your ansible configuration, without mentioning that the installer will generate a sample and usual configuration (/etc/foreman-proxy/ansible.cfg) during plugin installation when using foreman-installer. I, in fact, used the generated configuration for my testing (by copying it to /etc/ansible/ansible.cfg), so I did not have the changes suggested in-place before running the foreman-installer. I would suggest that the plugin maintainer, review if the best way to document the sequence is to run foreman-installer first, and than configure ansible, as a second step.
Fixable: Given the Section 2.1 Ansible callback documentation, the generated configuration /etc/foreman-proxy/ansible.cfg has additional configuration specifications. Maybe the documentation should suggest reading the generated file for the latest Foreman Ansible Plugin recommended ansible.cfg configuration, as this would save having to document everything in the written documentation. The generated file, could also document in comments, the mandatory configuration versus the recommended / optional configuration. Also being generated, the file could account for Ansible version, OS and OS version and other factors in what it generates, meaning that the options required on older ansible versions or platforms would not be generated on newer (remediated) platforms. This could reduce updates to the documentation, leaving them in generated comments, that will only appear when necessary. If necessary, point the reader to the template used to generate the sample configuration. The ansible.cfg template way wish to document in comments the key template variable values used to generate it, so users don’t widely assume it is generic.
Foreman Documentation for New Users
Being new to Foreman, I feel that the Foreman 2.X manual, could probably be enhanced by starting with a “What is Foreman?” section, summarizing the high level features in a paragraph or two, and suggest that the reader review Foreman :: Introduction, before continuing.
After the Smart-Proxy description in Section 1 of the Foreman Manual, it would be useful to have a high-level set of scenarios and what components would be used in such a scenario. For example, some provisioning scenarios might include:
These scenarios could discuss basic architectural requirements, like type of servers needed (by this I mean foreman and smart-proxy instances and their high level configuration). Details like how the foreman-installer only supports a single subnet configuration via the command line, but does support multiple subnet via confgiuration files, with references to appropriate part of the manual.
Importantly, this introduction should probably discribe what Katello adds to the Foreman architectural picture (high level), and how to architect the above scenarios (say assuming limited bandwidth inter-site links), if Katello is part of the picture. Also if my understanding is correct - that if you wish to use Katello, you need to follow the Katello installation documentation, and use this documentation as a referenece for the Foreman and Smart-Proxy components of the system.
Finally in section 1 of the manual, I feel references to the other resources available to assist in your understanding of Foreman / Smart-Proxy / Katello / CandlePin. A brief (say one paragraph) introduction or history of each and maybe a reference to the commerical products they appear in. But I feel the key to the last part of section 1 should be an appropriately qualified link to work-in-progress and existing documentation available for Foreman and its derived products http://docs.theforeman.org/web/. The guides on these pages greatly assisted in my understanding of the system. I feel it’s fine to say in the qualifications that: other parties may have authored some guides (e.g. RedHat); they may be guides to existing commerical derivates which have yet to be updated the Foreman 2.X; and such. But because they are classed a work-in-progress they weren’t the easiest to fine (for me at least), but IMHO they need to read as part of introduction or guide to new users. The “Provisioning Guide” was well written, and showed off the [very cool IMHO] hammer(1) CLI interface. Qualify the link appropriately, WIP/out-of-date/etc but link to that resource, and maybe suggest a reading order for some of the guides, or update the referenced page, to provide additional information like which open source products a guide covers, and what major versions its has been written for, and if it’s worthwhile reading for the current open-source revisions of these products.
Fixes for 2.1.0rc* before its General Availability release (IMHO)
Packaging Fixes for Smart Proxy Modules
The installation of files into their older installation location (/opt/theforeman): I discovered two instances during my testing on CentOS 8, which @aruzicka offered simple workarounds for:
sudo ln -s /opt/theforeman/tfm/root/usr/share/smart_proxy_dynflow_core/bundler.d/foreman_remote_execution_core.rb /usr/share/smart_proxy_dynflow_core/bundler.d/
sudo ln -s /opt/theforeman/tfm/root/usr/share/smart_proxy_dynflow_core/bundler.d/foreman_ansible_core.rb /usr/share/smart_proxy_dynflow_core/bundler.d/
sudo systemctl restart smart_proxy_dynflow_core.service
Today, I downloaded all Foreman and Foreman Plugin 2.1.0rc* packages for EL8, and found another instance in rubygem-smart-proxy-probing for the file foreman_probing_core.rb.
I’ll also note that every RPM (all 661 of then) in the main foreman repository generated a warning of the form: warning: FILENAME.rpm: Header V4 RSA/SHA256 Signature, key ID b6f08ccf: NOKEY. No foreman-plugin RPMs generated this warning. Fedora 32 rpm(8) was used to check the installation paths via rpm -qlp $rpm | grep '^/opt/' in a command line for loop.
Affected Packages:
- rubygem-smart_proxy_ansible
- rubygem-smart-proxy-probing
- rubygem-smart_proxy_remote_execution_ssh
Foreman Bug Id: Unknown
Compatibility RHEL/CentOS 8 and net-ssh
Problem 1: ssh-keygen(1) default output format on EL8 - as noted in Comment 18.
Workaround: sudo -u foreman-proxy ssh-keygen -f ~foreman-proxy/.ssh/id_rsa_foreman_proxy -m PEM -N '' -t rsa -b 4096
Foreman Bug Id: Bug #30121: ssh-keygen on EL8 generates private keys in RFC4716 format, net-ssh requires PEM format - Installer - Foreman
Problem 2: Use of rubygem-net-ssh-4.2.0 blocking use of EPEL on CentOS 8
Further Discussion: Foreman 2.1.0-rc3 release process
Workaround: See post 9 in the above discussion.
Foreman Bug Id: Bug #30162: Allow newer Net::SSH - Foreman
Ansible - Which RPM repsoitory to use.
Ansible is available from at least two repsoitories on CentOS 8: EPEL and CentOS ConfigManagement SIG. As discussed above, EPEL repository usage currently causes problems. This topic showed that in fact the Foreman team (or at least Adam) use the CentOS ConfigManagement SIG repository, that fact should be documented.
Solution: sudo dnf install centos-release-ansible-29 ; sudo dnf list ansible
Foreman Bug Id: Unknown - @mcorr stated she has raised an issue for this in the rc3 checklist topic - Needs documenting in the Quick Start Guide for 2.1.0 and CentOS 8
Finally, as someone completely new to Foreman, I would like to acknowledge the patience and helpfulness of @aruzicka! And the welcoming and friendly responses from Adam, @ekohl, @evgeni, @tbrisker and @mcorr. After all, jumping in as a newbie to Foreman using a release candidate on a newly supported OS - was probably not the simplest route to understanding the product! But having to write-up/debug issues really does assist in that initial understanding of a product, especially with such a helpful community!
Regards,
Peter
Thank you so much for your feedback. From a documentation perspective alone, this will provide great guidance on how to make improvements in the immediate term that will make things better for new users who come after you.
This is on my to-do list to check through all the plugins and ensure there is an up-to-date manual for each, and if not, gently guide the original author towards providing one
My colleague @tahliar is currently migrating a “Planning” guide from the downstream Red Hat docs that is a deep dive into the architecture and concepts of Foreman. I will link to this from the existing Foreman guides as we move towards consolidating old and new documentation.
I can’t speak for Katello, but maybe @iballou could chime in?
This was tracked as Bug #27184: [smart_proxy_dynflow_core] "PID file /var/run/foreman-proxy/smart_proxy_dynflow_core.pid not readable (yet?) after start" - foreman-tasks - Foreman and should be fixed in smart_proxy_dynflow_core-0.2.6.
All the things you mentioned are valid, thank you for bearing with us and pointing those things out
Ad Katello: 3.16 is not (yet) fully available on EL8 - and that’s why we’ve not been advetising it yet. @ehelms and @Justin_Sherrill are working on it.
Ad Ansible repo: yes, we’d prefer if you’d use the config management repo – but epel should work too. I had raised this point in https://github.com/theforeman/theforeman.org/pull/1612 when we documented EL8 things, but we didn’t add it due to not being a dependency of Foreman itself. Wonder what would be a good place to document optional things. @ehelms @mcorr what do you think?
Ad /opt/theforeman/ in EL8 packages: I’ll take a look – that’s a bug.
Ad Ansible version: the version you see in “Infra → Proxies” is the proxy plugin, and that is really 3.0.1, while the Foreman plugin is 5.1.1 (and 3.0.3 for “core”). A tad irritating, I agree. @aruzicka any way we could “fix” that nicely?
that depends, what would you like to see there?
If only I’d knew.
Just releasing version bumps so that they match to the Foreman plugin sounds silly.
Maybe that page just needs a small info box saying “the versions listed here represent the proxy plugin versions and don’t necessarily match the Foreman plugin versions”?
Mostly brainstorming