Foreman 1.24.3 - smart-proxy failing to start dhcp_isc

Support
1

Problem:
working foreman 1.24.3 install on CentOS 7.8.2003 current patch set using puppet CA on same host as foreman to talk to isc dhcp on same host as foreman, currently managing ~30 hosts no problem working for well over a year stable. Not been used for approx ~10 days. Deleted 2 hosts from foreman interface, recieved error could not be deleted, looked at the logs and saw that the dhcp entry could not be removed and the API call to the foremanhost:8443/dhcp was 404. checking the connection to the dhcp api I saw the puppet CA certificate was still valid and had over 2 years left to run on it, but the actual foreman proxy had failed to start the dhcp_isc component. Looking in the foreman service browser there was two error lines with a backtrace failing to start the dhcp_isc service.

/usr/share/foreman-proxy/modules/dhcp_common/isc/configuration_parser.rb:415:in []' /usr/share/foreman-proxy/modules/dhcp_common/isc/configuration_parser.rb:415:in literal_to_filename’ /usr/share/foreman-proxy/modules/dhcp_common/isc/configuration_parser.rb:410:in block in include_file' /usr/local/share/gems/gems/rsec-1.0.0/lib/rsec/parsers/misc.rb:8:in ’ /usr/local/share/gems/gems/rsec-1.0.0/lib/rsec/parsers/misc.rb:8:in _parse' /usr/local/share/gems/gems/rsec-1.0.0/lib/rsec/parsers/misc.rb:79:in block in _parse’ /usr/local/share/gems/gems/rsec-1.0.0/lib/rsec/parsers/misc.rb:78:in each' /usr/local/share/gems/gems/rsec-1.0.0/lib/rsec/parsers/misc.rb:78:in _parse’ /usr/local/share/gems/gems/rsec-1.0.0/lib/rsec/parsers/join.rb:67:in block in _parse' /usr/local/share/gems/gems/rsec-1.0.0/lib/rsec/parsers/join.rb:65:in loop’ /usr/local/share/gems/gems/rsec-1.0.0/lib/rsec/parsers/join.rb:65:in _parse' /usr/local/share/gems/gems/rsec-1.0.0/lib/rsec/parsers/misc.rb:120:in _parse’ /usr/local/share/gems/gems/rsec-1.0.0/lib/rsec/parser.rb:17:in parse!' /usr/share/foreman-proxy/modules/dhcp_common/isc/configuration_parser.rb:448:in subnets_hosts_and_leases’ /usr/share/foreman-proxy/modules/dhcp_common/isc/subnet_service_initialization.rb:11:in load_configuration_file' /usr/share/foreman-proxy/modules/dhcp_isc/isc_state_changes_observer.rb:152:in load_configuration_file’ /usr/share/foreman-proxy/modules/dhcp_isc/isc_state_changes_observer.rb:129:in block in do_start' /usr/share/foreman-proxy/modules/dhcp_common/subnet_service.rb:154:in block in group_changes’ /usr/share/ruby/monitor.rb:211:in mon_synchronize' /usr/share/foreman-proxy/modules/dhcp_common/subnet_service.rb:154:in group_changes’ /usr/share/foreman-proxy/modules/dhcp_isc/isc_state_changes_observer.rb:128:in do_start' /usr/share/foreman-proxy/modules/dhcp_isc/isc_state_changes_observer.rb:80:in start’ /usr/share/foreman-proxy/modules/dhcp_isc/inotify_leases_file_observer.rb:42:in start' /usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:257:in block in start_services’ /usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:255:in each' /usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:255:in start_services’ /usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:240:in configure_plugin' /usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:109:in block in configure’ /usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:109:in each' /usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:109:in configure’ /usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:187:in block in initialize_plugins' /usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:187:in each’ /usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:187:in initialize_plugins' /usr/share/foreman-proxy/lib/launcher.rb:168:in launch’ /usr/share/foreman-proxy/bin/smart-proxy:6:in `’

the isc dhcp configuration file has not been touched for months and has been working fine, I have put the foreman proxy log to debug in the hope of getting more output but still only get the following lines

2020-06-18T21:14:24 [D] 'dhcp_isc' settings: 'blacklist_duration_minutes': 1800 (default), 'config': /etc/dhcp/dhcpd.conf (default), 'key_name': jarvis_omapi.key, 'key_secret': <redacted>, 'leases': /var/lib/dhcpd/dhcpd.leases (default), 'leases_file_observer': inotify_leases_file_observer, 'omapi_port': 7911, 'ping_free_ip': true, 'server': 127.0.0.1, 'subnets': [], 'use_provider': dhcp_isc

2020-06-18T21:14:24  [E] Couldn't enable 'dhcp_isc'
2020-06-18T21:14:24  [E] Disabling all modules in the group ['dhcp_isc', 'dhcp'] due to a failure in one of them: no implicit conversion of Range into Integer

Ok

Expected outcome:

smart proxy to load isc_dhcp and make API available allowing foreman to delete host

Foreman and Proxy versions:
1.24.3

Foreman and Proxy plugin versions:

tfm-rubygem-foreman_dhcp_browser-0.0.8-3.fm1_23.el7.noarch
foreman-proxy-1.24.3-1.el7.noarch

Distribution and version:
Centos 7.8.2003 x86_64
Other relevant data:

detail in message body with context and preformatted as required,

2

It looks like you have rsec 1.0 installed in /usr/local/share but it’s incompatible with that version. Not sure how you installed that, but it should be < 1.0.0.

3

so that’s interesting, as you’re right, rubygem-rsec-0.4.3-2.el7.noarch is installed, however,

Installed Packages
Name : rubygem-rsec
Arch : noarch
Version : 0.4.3
Release : 2.el7
Size : 30 k
Repo : installed
From repo : foreman
Summary : Extreme Fast Parser Combinator for Ruby
URL : http://rsec.herokuapp.com
Licence : Ruby or BSD
Description : Easy and extreme fast dynamic PEG parser combinator.

it’s come from the foreman repo, so it’s not a different version from elsewhere, and this has been working for a long time, I’m wondering what would have caused that version to get installed and how ? surly there would be an rpm dependency failure ?

4

looking at /usr/local/share/gems the date stamp on most of the files in 2nd June - which is probably around the last time I used the system to build / delete a host so $something has done a gem install/update that has pulled in the 1.0.0 version as a dependency is my guess, but I don’t know and can’t get what, I’ll try to work through this.

Great spot, I had no idea rsec had the version dependency and would never have found that.

5

gem uninstall on the 1.0.0 version fixed it, I used rubygems.org to try to find something with a reverse dependency on rsec 1.0.0 and totally failed, there was nothing in there that depended on it, so I can’t see what would have pulled it in. The only thing that I notice was that puppet-lint got a version upgrade (not sure how as I didn’t issue an upgrade) Great spot thought.