Backup fails inconsistently with Couldn't find DHCP Configuration file

areyus · July 8, 2020, 12:01pm

Problem:
Since upgrading from 1.20 to 2.0, we encounter inconsistent but very frequent failures of “foreman-maintain backup”.
The backup task sometimes runs through without any issue, but most of the time it fails with “Couldn’t find DHCP Configuration file”.
The information from maintain.log can be found below.

Expected outcome:
Backup is successful consistently.

Foreman and Proxy versions:
2.0.1

Foreman and Proxy plugin versions:
Foreman plugin: foreman-tasks, 1.1.1, Ivan Nečas, The goal of this plugin is to unify the way of showing task statuses across the Foreman instance.
Foreman plugin: foreman_bootdisk, 16.0.0, Dominic Cleal, Plugin for Foreman that creates iPXE-based boot disks to provision hosts without the need for PXE infrastructure.
Foreman plugin: foreman_discovery, 16.0.1, Aditi Puntambekar, alongoldboim, Alon Goldboim, amirfefer, Amit Karsale, Amos Benari, Avi Sharvit, Bryan Kearney, bshuster, Daniel Lobato, Daniel Lobato Garcia, Daniel Lobato García, Danny Smit, David Davis, Djebran Lezzoum, Dominic Cleal, Eric D. Helms, Ewoud Kohl van W
ijngaarden, Frank Wall, Greg Sutcliffe, ChairmanTubeAmp, Ido Kanner, imriz, Imri Zvik, Ivan Nečas, Joseph Mitchell Magen, June Zhang, kgaikwad, Lars Berntzon, ldjebran, Lukas Zapletal, Lukáš Zapletal, Marek Hulan, Marek Hulán, Martin Bačovský, Matt Jarvis, Michael Moll, Nick, odovzhenko, Ohad Levy, Ondrej Prazak,
Ondřej Ezr, Ori Rabin, orrabin, Partha Aji, Petr Chalupa, Phirince Philip, Rahul Bajaj, Robert Antoni Buj Gelonch, Scubafloyd, Sean O’Keeffe, Sebastian Gräßl, Shimon Shtein, Shlomi Zadok, Stephen Benjamin, Swapnil Abnave, Thomas Gelf, Timo Goebel, Tomas Strych, Tom Caspy, Tomer Brisker, and Yann Cézard, MaaS D
iscovery Plugin engine for Foreman
Foreman plugin: foreman_hooks, 0.3.16, Dominic Cleal, Plugin engine for Foreman that enables running custom hook scripts on Foreman events
Foreman plugin: foreman_remote_execution, 3.2.2, Foreman Remote Execution team, A plugin bringing remote execution to the Foreman, completing the config management functionality with remote management functionality.
Foreman plugin: foreman_scc_manager, 1.8.3, ATIX AG, Foreman plugin to sync SUSE Customer Center products and repositories into Katello.
Foreman plugin: foreman_snapshot_management, 1.7.1, ATIX AG, Foreman-plugin to manage snapshots in a virtual-hardware environments.
Foreman plugin: foreman_templates, 8.0.0, Greg Sutcliffe, Engine to synchronise provisioning templates from GitHub
Foreman plugin: katello, 3.15.2, N/A, Katello adds Content and Subscription Management to Foreman. For this it relies on Candlepin and Pulp.
Distribution and version:
RHEL7

Other relevant data:
We have updated foreman-maintain manually to 0.6.4 from the 2.1 repos as suggested in another thread because the backup job failed on 0.6.3 with problems regarding sidekiq. I doubt this is the root of the problem, though, since the described error started before we reached 2.0 in the upgrade tests
.
maintain.log:

I, [2020-07-08 13:48:13+0200 #101243]  INFO -- : Proxy features: ["dhcp", "dynflow", "httpboot", "pulp", "pulpcore", "puppet", "puppetca", "ssh", "templates", "tftp"]
I, [2020-07-08 13:48:13+0200 #101243]  INFO -- : Proxy features to backup: ["dhcp", "dynflow", "httpboot", "pulp", "pulpcore", "puppet", "puppetca", "ssh", "templates", "tftp"]
D, [2020-07-08 13:48:13+0200 #101243] DEBUG -- : Running command rpm -q 'foreman' --queryformat="%{VERSION}" with stdin nil
D, [2020-07-08 13:48:13+0200 #101243] DEBUG -- : output of the command:
 2.0.1
D, [2020-07-08 13:48:13+0200 #101243] DEBUG -- : foreman proxy content module detected: certs
D, [2020-07-08 13:48:13+0200 #101243] DEBUG -- : Running command rpm -q 'foreman' --queryformat="%{VERSION}" with stdin nil
D, [2020-07-08 13:48:13+0200 #101243] DEBUG -- : output of the command:
 2.0.1
E, [2020-07-08 13:48:13+0200 #101243] ERROR -- : Couldn't find DHCP Configuration file (RuntimeError)
/usr/share/gems/gems/foreman_maintain-0.6.4/definitions/features/foreman_proxy.rb:207:in `lookup_dhcpd_config_file'
/usr/share/gems/gems/foreman_maintain-0.6.4/definitions/features/foreman_proxy.rb:113:in `dhcpd_config_file'
/usr/share/gems/gems/foreman_maintain-0.6.4/definitions/features/foreman_proxy.rb:70:in `config_files'
/usr/share/gems/gems/foreman_maintain-0.6.4/definitions/procedures/backup/config_files.rb:64:in `config_files'
/usr/share/gems/gems/foreman_maintain-0.6.4/definitions/procedures/backup/config_files.rb:78:in `tar_command'
/usr/share/gems/gems/foreman_maintain-0.6.4/definitions/procedures/backup/config_files.rb:27:in `run'
/usr/share/gems/gems/foreman_maintain-0.6.4/lib/foreman_maintain/executable.rb:139:in `__run__'
/usr/share/gems/gems/foreman_maintain-0.6.4/lib/foreman_maintain/runner/execution.rb:83:in `block (2 levels) in run'
/usr/share/gems/gems/foreman_maintain-0.6.4/lib/foreman_maintain/runner/execution.rb:107:in `capture_errors'
/usr/share/gems/gems/foreman_maintain-0.6.4/lib/foreman_maintain/runner/execution.rb:82:in `block in run'
/usr/share/gems/gems/foreman_maintain-0.6.4/lib/foreman_maintain/runner/execution.rb:101:in `with_metadata_calculation'
/usr/share/gems/gems/foreman_maintain-0.6.4/lib/foreman_maintain/runner/execution.rb:81:in `run'
/usr/share/gems/gems/foreman_maintain-0.6.4/lib/foreman_maintain/runner.rb:122:in `run_step'
/usr/share/gems/gems/foreman_maintain-0.6.4/lib/foreman_maintain/runner.rb:111:in `run_steps'
/usr/share/gems/gems/foreman_maintain-0.6.4/lib/foreman_maintain/runner.rb:104:in `execute_scenario_steps'
/usr/share/gems/gems/foreman_maintain-0.6.4/lib/foreman_maintain/runner.rb:51:in `run_scenario'
/usr/share/gems/gems/foreman_maintain-0.6.4/lib/foreman_maintain/runner.rb:33:in `block in run'
/usr/share/gems/gems/foreman_maintain-0.6.4/lib/foreman_maintain/runner.rb:32:in `each'
/usr/share/gems/gems/foreman_maintain-0.6.4/lib/foreman_maintain/runner.rb:32:in `run'
/usr/share/gems/gems/foreman_maintain-0.6.4/lib/foreman_maintain/cli/base.rb:64:in `run_scenario'
/usr/share/gems/gems/foreman_maintain-0.6.4/lib/foreman_maintain/cli/backup_command.rb:25:in `perform_backup'
/usr/share/gems/gems/foreman_maintain-0.6.4/lib/foreman_maintain/cli/backup_command.rb:125:in `execute'
/usr/share/gems/gems/clamp-1.1.2/lib/clamp/command.rb:63:in `run'
/usr/share/gems/gems/clamp-1.1.2/lib/clamp/subcommand/execution.rb:11:in `execute'
/usr/share/gems/gems/clamp-1.1.2/lib/clamp/command.rb:63:in `run'
/usr/share/gems/gems/clamp-1.1.2/lib/clamp/subcommand/execution.rb:11:in `execute'
/usr/share/gems/gems/clamp-1.1.2/lib/clamp/command.rb:63:in `run'
/usr/share/gems/gems/foreman_maintain-0.6.4/lib/foreman_maintain/cli.rb:34:in `run'
/usr/share/gems/gems/clamp-1.1.2/lib/clamp/command.rb:132:in `run'
/usr/share/gems/gems/foreman_maintain-0.6.4/bin/foreman-maintain:15:in `<top (required)>'
/bin/foreman-maintain:23:in `load'
/bin/foreman-maintain:23:in `<main>'
I, [2020-07-08 13:48:13+0200 #101243]  INFO -- : --- Execution step 'Backup config files' finished ---
I, [2020-07-08 13:48:13+0200 #101243]  INFO -- : === Scenario 'Backup' finished ===
D, [2020-07-08 13:48:13+0200 #101243] DEBUG -- : === Rescue scenario found. Executing ===
E, [2020-07-08 13:48:14+0200 #101243] ERROR -- : The runner is already in quit state (RuntimeError)
/usr/share/gems/gems/foreman_maintain-0.6.4/lib/foreman_maintain/runner.rb:46:in `run_scenario'
/usr/share/gems/gems/foreman_maintain-0.6.4/lib/foreman_maintain/runner.rb:99:in `block in execute_scenario_steps'
/usr/share/gems/gems/foreman_maintain-0.6.4/lib/foreman_maintain/runner.rb:99:in `each'
/usr/share/gems/gems/foreman_maintain-0.6.4/lib/foreman_maintain/runner.rb:99:in `execute_scenario_steps'
/usr/share/gems/gems/foreman_maintain-0.6.4/lib/foreman_maintain/runner.rb:38:in `block in run'
/usr/share/gems/gems/foreman_maintain-0.6.4/lib/foreman_maintain/runner.rb:32:in `each'
/usr/share/gems/gems/foreman_maintain-0.6.4/lib/foreman_maintain/runner.rb:32:in `run'
/usr/share/gems/gems/foreman_maintain-0.6.4/lib/foreman_maintain/cli/base.rb:64:in `run_scenario'
/usr/share/gems/gems/foreman_maintain-0.6.4/lib/foreman_maintain/cli/backup_command.rb:25:in `perform_backup'
/usr/share/gems/gems/foreman_maintain-0.6.4/lib/foreman_maintain/cli/backup_command.rb:125:in `execute'
/usr/share/gems/gems/clamp-1.1.2/lib/clamp/command.rb:63:in `run'
/usr/share/gems/gems/clamp-1.1.2/lib/clamp/subcommand/execution.rb:11:in `execute'
/usr/share/gems/gems/clamp-1.1.2/lib/clamp/command.rb:63:in `run'
/usr/share/gems/gems/clamp-1.1.2/lib/clamp/subcommand/execution.rb:11:in `execute'
/usr/share/gems/gems/clamp-1.1.2/lib/clamp/command.rb:63:in `run'
/usr/share/gems/gems/foreman_maintain-0.6.4/lib/foreman_maintain/cli.rb:34:in `run'
/usr/share/gems/gems/clamp-1.1.2/lib/clamp/command.rb:132:in `run'
/usr/share/gems/gems/foreman_maintain-0.6.4/bin/foreman-maintain:15:in `<top (required)>'
/bin/foreman-maintain:23:in `load'
/bin/foreman-maintain:23:in `<main>'
I, [2020-07-08 13:48:14+0200 #101243]  INFO -- : foreman-maintain command finished with 1

areyus · July 9, 2020, 12:32pm

Upon further digging I fould this line to be the cause of the issue.
We are using the dhcp_infoblox smart proxy plugin, which does not have a config option. From the other sample config files on my system, only the default isc provider seams to have this setting (pointing to the DHCPD config file).
Any suggestions on how to handle/work around this?

What confuses me even more is the fact that the job seems to run fine sometimes, despite that setting not beeing there.

areyus · July 13, 2020, 8:25am

So, I was able to find a “workaround” for this Problem.
The backup task works, as long as there is a config parameter in dhcp_infoblox.yml pointing to any file that exists and can be backupd (I tried the dhcp_infoblox.yml itself). At least for the infoblox provider, it seems like it is ignoring the “config” parameter because it does not know it.
If I find the time (and some ruby know how in me), I might try and come up with a patch.

ekohl · July 13, 2020, 10:30am

This code suggests it should look at :use_provider in dhcp.yml:

github.com

theforeman/foreman_maintain/blob/a55de237a8477a354c4850c2a4680afd934b5297/definitions/features/foreman_proxy.rb#L237


      
            @configs_from_dhcp_yml ||= yaml_load(dhcp_yml_path)
          end
          
          def dhcp_isc_provider?
            configs_from_dhcp_yml[:use_provider] == 'dhcp_isc'
          end
          
          def lookup_using_dhcp_yml
            if configs_from_dhcp_yml.key?(:dhcp_config)
              return configs_from_dhcp_yml[:dhcp_config]
            elsif configs_from_dhcp_yml.key?(:use_provider)
              settings_d_dir = File.dirname(dhcp_yml_path)
              dhcp_provider_fpath = File.join(settings_d_dir, "#{configs_from_dhcp_yml[:use_provider]}.yml")
              dhcp_provider_configs = yaml_load(dhcp_provider_fpath)
              return dhcp_provider_configs[:config] if dhcp_provider_configs.key?(:config)
            else
              raise "Couldn't find DHCP Configurations in #{dhcp_yml_path}"
            end
          end
          
          def lookup_tftp_root_directory

This corresponds to:

github.com

theforeman/smart-proxy/blob/e138e2309d38eabc06b7251f15ad28ad5a264c4f/config/settings.d/dhcp.yml.example#L9


      
          ---
          # Can be true, false, or http/https to enable just one of the protocols
          :enabled: false
          
          # valid providers:
          #   - dhcp_isc (ISC dhcp server)
          #   - dhcp_native_ms (Microsoft native implementation)
          #   - dhcp_libvirt
          #:use_provider: dhcp_isc
          #:server: 127.0.0.1
          # subnets restricts the subnets queried to a subset, to reduce the query time.
          #:subnets: [192.168.205.0/255.255.255.128, 192.168.205.128/255.255.255.128]
          
          # Perform ICMP and TCP ping when searching free IPs from the pool. This makes
          # sure that active IP address is not suggested as free, however in locked down
          # network environments this can cause no free IPs. Enabled by default
          #:ping_free_ip: true

That part is all OK. Then there’s code to finddhcpd.conf

github.com

theforeman/foreman_maintain/blob/a55de237a8477a354c4850c2a4680afd934b5297/definitions/features/foreman_proxy.rb#L212-L217


      
          def lookup_dhcpd_config_file
            dhcpd_config_file = lookup_using_dhcp_yml
            raise "Couldn't find DHCP Configuration file" if dhcpd_config_file.nil?
          
            dhcpd_config_file
          end

That only needs to be backed up for the dhcp_isc provider (dhcp_isc_provider? can return this). Now this code suggests it should do the right thing:

github.com

theforeman/foreman_maintain/blob/a55de237a8477a354c4850c2a4680afd934b5297/definitions/features/foreman_proxy.rb#L72-L74


      
          if backup_features.include?('dhcp') && dhcp_isc_provider?
            configs += ['/var/lib/dhcpd', File.dirname(dhcpd_config_file)]
          end

So in theory this should all work, but somewhere along these lines it doesn’t.

It should be noted that this was introduced in

That was part of 0.6.6 but you’re running 0.6.4 which is the problem.

areyus · July 13, 2020, 12:29pm

Thanks for the heads-up for the already existing patch
In 0.6.4, it does not work because use_provider is set for a lot of providers, but the check for dhcp_isc provider is missing.
But since a fix is already upstream, I’m happy this will work after the next update and I would consider this resolved.