Problem with Foreman 2.1rc2 with Remote Execution on CentOS 8

Support
33

Topic Summary + Remaining TO-DOs

Note: I have not jumped into the code base to answer the questions I’m raising and the answers may be contained in the code base. I’m coming to Foreman as a new user, not a developer.

Note: Also the Low Priority section ended up longer than expected, you may wish to jump to the Foreman 2.1.0rc* specifics after that section, which summarizes the problems and fixes and if they have a known Foreman Bug Ids (at the time of writing).

Low Priority

  • The warning generated by smart_proxy_dynflow_core.service:
    Can't open PID file /var/run/foreman-proxy/smart_proxy_dynflow_core.pid (yet?) after start: No such file or directory
    Question: Does this problem exist because the /var/run/foreman-proxy directory does not exist when the service starts? That is, should the service add a dependency on systemd-tmpfiles-setup.service? (that service does create this directory)
    Question: Is this actually a complaint that the file smart_proxy_dynflow_core.pid does not exist? As noted in the previous question, the directory it is located in does not exist until created on each system boot by the systemd-tmpfiles-setup.service. So the question, than I believe becomes …
    Question: Should the PID file exist at service startup?
    If so, that can be accommodated by systemd-tmpfiles-setup.service creating any empty file for the code base, and adding the dependency on that service.
    Alternatively, if the file does not need to exist (IMHO the likely case), should the warning message be changed, or the logic that generated it should be changed?
    Commentary: As I long time developer and system administrator, I would suggest that the PID file is likely used, to check if another instance of the process is already running. This check is often done to prevent other problems being generated and reported, like trying to listen on port that is already in use, etc. In my experience this rarely done well, especially on older systems with non-volatile storage areas for PID files. As the PID file may contain out-of-date data, and checking if a process with that PID simply exists, does not mean it is an already running instance of this process (two reasons: non-volatile PID storage areas especially on boot, or system Process IDs have wrapped on a long running systems). So what is done with the contents of the PID file (if non-empty) needs to be investigated.
    Depending on the outcome of the investigation and the future directions of the project, one or more of the following may be viable:

    1. Add a dependency on systemd-tmpfiles-setup.service;
    2. Use systemd-tmpfiles-setup.service to create an empty PID file;
    3. Only generate a (warning) message with the PID to be checked, when the PID file exists and contains a valid PID (c.f. an empty file, or a file with nonsensical data);
    4. Drop the check and rely on the service management software (systemd(1), …) to ensure a single instance of the service - N.B. developers may push for its retention, on:
      • dev-systems when debugging a non-installed instance of the service;
      • if the code is ever deployed on systems without modern service management software - given the code is part of smart-proxy and not dynflow that seems unlikely;
    5. Something I haven’t discussed guided by the outcome of the investigation;

  • Ansible Plugin Documentation
    Latest Documentation: Foreman :: Plugin Manuals

    Fixable: The page is named ‘Foreman :: Plugin Manuals’, when its the ‘Ansible Plugin 3.x Manual’, given this would likely be generated, it could contain the actual plugin name and version, as this would improve tab names and bookmarks in your browser.

    Fixable: The Ansible Plugin Versioning for End-Users

    • The rubygem-foreman_ansible version is 5.1.1;
    • The rubygem-foreman_ansible_core version is 3.0.3;
    • The WebUI -> Infrastructure -> Smart Proxies -> (selected proxy), tab labeled ‘Services’ states Ansible is version 3.0.1;

    So I’m at a bit lost as to the version of the Ansible Plugin Manual, that I should be reading. I assume there should be a 5.X manual rather than the 3.X one provided by the project’s web site. If not, maybe a sentence could be added to the Ansible Plugin manual explaining the plugin version versus the packaging versions.

    Fixable: In Section 2.1 Ansible callback, it documents the changes required to your ansible configuration, without mentioning that the installer will generate a sample and usual configuration (/etc/foreman-proxy/ansible.cfg) during plugin installation when using foreman-installer. I, in fact, used the generated configuration for my testing (by copying it to /etc/ansible/ansible.cfg), so I did not have the changes suggested in-place before running the foreman-installer. I would suggest that the plugin maintainer, review if the best way to document the sequence is to run foreman-installer first, and than configure ansible, as a second step.

    Fixable: Given the Section 2.1 Ansible callback documentation, the generated configuration /etc/foreman-proxy/ansible.cfg has additional configuration specifications. Maybe the documentation should suggest reading the generated file for the latest Foreman Ansible Plugin recommended ansible.cfg configuration, as this would save having to document everything in the written documentation. The generated file, could also document in comments, the mandatory configuration versus the recommended / optional configuration. Also being generated, the file could account for Ansible version, OS and OS version and other factors in what it generates, meaning that the options required on older ansible versions or platforms would not be generated on newer (remediated) platforms. This could reduce updates to the documentation, leaving them in generated comments, that will only appear when necessary. If necessary, point the reader to the template used to generate the sample configuration. The ansible.cfg template way wish to document in comments the key template variable values used to generate it, so users don’t widely assume it is generic.

  • Foreman Documentation for New Users

    Being new to Foreman, I feel that the Foreman 2.X manual, could probably be enhanced by starting with a “What is Foreman?” section, summarizing the high level features in a paragraph or two, and suggest that the reader review Foreman :: Introduction, before continuing.

    After the Smart-Proxy description in Section 1 of the Foreman Manual, it would be useful to have a high-level set of scenarios and what components would be used in such a scenario. For example, some provisioning scenarios might include:

    • Single Site - One Subnet (Simplest Case)
    • Multiple Sites - One Subnet at each site
    • Single Site - Multiple Subnets
    • Multiple Sites - Multiple Subnets (I would hope documenting this would be unnecessary - if I understand Foreman correctly, as its a combination of approach of the last two)

    These scenarios could discuss basic architectural requirements, like type of servers needed (by this I mean foreman and smart-proxy instances and their high level configuration). Details like how the foreman-installer only supports a single subnet configuration via the command line, but does support multiple subnet via confgiuration files, with references to appropriate part of the manual.

    Importantly, this introduction should probably discribe what Katello adds to the Foreman architectural picture (high level), and how to architect the above scenarios (say assuming limited bandwidth inter-site links), if Katello is part of the picture. Also if my understanding is correct - that if you wish to use Katello, you need to follow the Katello installation documentation, and use this documentation as a referenece for the Foreman and Smart-Proxy components of the system.

    Finally in section 1 of the manual, I feel references to the other resources available to assist in your understanding of Foreman / Smart-Proxy / Katello / CandlePin. A brief (say one paragraph) introduction or history of each and maybe a reference to the commerical products they appear in. But I feel the key to the last part of section 1 should be an appropriately qualified link to work-in-progress and existing documentation available for Foreman and its derived products http://docs.theforeman.org/web/. The guides on these pages greatly assisted in my understanding of the system. I feel it’s fine to say in the qualifications that: other parties may have authored some guides (e.g. RedHat); they may be guides to existing commerical derivates which have yet to be updated the Foreman 2.X; and such. But because they are classed a work-in-progress they weren’t the easiest to fine (for me at least), but IMHO they need to read as part of introduction or guide to new users. The “Provisioning Guide” was well written, and showed off the [very cool IMHO] hammer(1) CLI interface. Qualify the link appropriately, WIP/out-of-date/etc but link to that resource, and maybe suggest a reading order for some of the guides, or update the referenced page, to provide additional information like which open source products a guide covers, and what major versions its has been written for, and if it’s worthwhile reading for the current open-source revisions of these products.

Fixes for 2.1.0rc* before its General Availability release (IMHO)

  • Packaging Fixes for Smart Proxy Modules
    The installation of files into their older installation location (/opt/theforeman): I discovered two instances during my testing on CentOS 8, which @aruzicka offered simple workarounds for:
    sudo ln -s /opt/theforeman/tfm/root/usr/share/smart_proxy_dynflow_core/bundler.d/foreman_remote_execution_core.rb /usr/share/smart_proxy_dynflow_core/bundler.d/
    sudo ln -s /opt/theforeman/tfm/root/usr/share/smart_proxy_dynflow_core/bundler.d/foreman_ansible_core.rb /usr/share/smart_proxy_dynflow_core/bundler.d/
    sudo systemctl restart smart_proxy_dynflow_core.service
    Today, I downloaded all Foreman and Foreman Plugin 2.1.0rc* packages for EL8, and found another instance in rubygem-smart-proxy-probing for the file foreman_probing_core.rb.
    I’ll also note that every RPM (all 661 of then) in the main foreman repository generated a warning of the form: warning: FILENAME.rpm: Header V4 RSA/SHA256 Signature, key ID b6f08ccf: NOKEY. No foreman-plugin RPMs generated this warning. Fedora 32 rpm(8) was used to check the installation paths via rpm -qlp $rpm | grep '^/opt/' in a command line for loop.
    Affected Packages:
    - rubygem-smart_proxy_ansible
    - rubygem-smart-proxy-probing
    - rubygem-smart_proxy_remote_execution_ssh
    Foreman Bug Id: Unknown

  • Compatibility RHEL/CentOS 8 and net-ssh

    Problem 1: ssh-keygen(1) default output format on EL8 - as noted in Comment 18.
    Workaround: sudo -u foreman-proxy ssh-keygen -f ~foreman-proxy/.ssh/id_rsa_foreman_proxy -m PEM -N '' -t rsa -b 4096
    Foreman Bug Id: Bug #30121: ssh-keygen on EL8 generates private keys in RFC4716 format, net-ssh requires PEM format - Installer - Foreman

    Problem 2: Use of rubygem-net-ssh-4.2.0 blocking use of EPEL on CentOS 8
    Further Discussion: Foreman 2.1.0-rc3 release process
    Workaround: See post 9 in the above discussion.
    Foreman Bug Id: Bug #30162: Allow newer Net::SSH - Foreman

  • Ansible - Which RPM repsoitory to use.

    Ansible is available from at least two repsoitories on CentOS 8: EPEL and CentOS ConfigManagement SIG. As discussed above, EPEL repository usage currently causes problems. This topic showed that in fact the Foreman team (or at least Adam) use the CentOS ConfigManagement SIG repository, that fact should be documented.
    Solution: sudo dnf install centos-release-ansible-29 ; sudo dnf list ansible
    Foreman Bug Id: Unknown - @mcorr stated she has raised an issue for this in the rc3 checklist topic - Needs documenting in the Quick Start Guide for 2.1.0 and CentOS 8

Finally, as someone completely new to Foreman, I would like to acknowledge the patience and helpfulness of @aruzicka! And the welcoming and friendly responses from Adam, @ekohl, @evgeni, @tbrisker and @mcorr. After all, jumping in as a newbie to Foreman using a release candidate on a newly supported OS - was probably not the simplest route to understanding the product! But having to write-up/debug issues really does assist in that initial understanding of a product, especially with such a helpful community!

Regards,
Peter

5 Likes