When a host is created with PXEboot the DHCP lease entry contains:
supersede server.next-server = "false";
instead of
supersede server.next-server = 0a:3f:10:13
0a:3f:10:13 being the ip address (hex) of the Foreman/Katello server.
This is causing TFTP Open timeout errors the next time a Host is being created.
Would I need to update
/etc/foreman-proxy/settings.d/tftp.yml
---
# TFTP management
:enabled: https
:tftproot: /var/lib/tftpboot
# Defines the TFTP Servername to use, overrides the name in the subnet declaration
#:tftp_servername:
or
/etc/dhcp/dhcpd.conf ?
Current
# grep tftp /etc/foreman-installer/scenarios.d/katello-answers.yaml
tftp: true
tftp_listen_on: https
tftp_managed: true
tftp_manage_wget: true
tftp_syslinux_filenames:
tftp_root: "/var/lib/tftpboot"
tftp_dirs:
- "/var/lib/tftpboot/pxelinux.cfg"
- "/var/lib/tftpboot/grub"
- "/var/lib/tftpboot/grub2"
- "/var/lib/tftpboot/boot"
- "/var/lib/tftpboot/ztp.cfg"
- "/var/lib/tftpboot/poap.cfg"
tftp_servername:
tftp_replace_grub2_cfg: false
# grep dhcp /etc/foreman-installer/scenarios.d/katello-answers.yaml
dhcp: true
dhcp_listen_on: https
dhcp_managed: true
dhcp_provider: isc
dhcp_subnets: []
dhcp_ping_free_ip: true
dhcp_option_domain:
dhcp_search_domains:
dhcp_interface: eth0
dhcp_additional_interfaces: []
dhcp_gateway:
dhcp_range:
dhcp_pxeserver:
dhcp_pxefilename: pxelinux.0
dhcp_network:
dhcp_netmask:
dhcp_nameservers: default
dhcp_server: 127.0.0.1
dhcp_config: "/etc/dhcp/dhcpd.conf"
dhcp_leases: "/var/lib/dhcpd/dhcpd.leases"
dhcp_key_name:
dhcp_key_secret:
dhcp_omapi_port: 7911
dhcp_peer_address:
dhcp_node_type: standalone
dhcp_failover_address: 10.63.16.19
dhcp_failover_port: 519
dhcp_max_response_delay: 30
dhcp_max_unacked_updates: 10
dhcp_mclt: 300
dhcp_load_split: 255
dhcp_load_balance: 3
dhcp_manage_acls: true
Ok I ran:
# foreman-installer --scenario katello --foreman-proxy-tftp-servername 10.63.16.19
Now it fails trying to access the /dhcp/ end point …
Jun 29 13:37:17 foreman foreman: /usr/share/foreman/lib/proxy_api/resource.rb:72: warning: URI.escape is obsolete
Jun 29 13:37:17 foreman smart-proxy: 10.63.16.19 - - [29/Jun/2021:13:37:17 UTC] "GET /tftp/serverName HTTP/1.1" 200 17
Jun 29 13:37:17 foreman smart-proxy: - -> /tftp/serverName
Jun 29 13:37:17 foreman foreman: /usr/share/foreman/lib/proxy_api/resource.rb:72: warning: URI.escape is obsolete
Jun 29 13:37:17 foreman smart-proxy: 10.63.16.19 - - [29/Jun/2021:13:37:17 UTC] "GET /tftp/serverName HTTP/1.1" 200 17
Jun 29 13:37:17 foreman smart-proxy: - -> /tftp/serverName
Jun 29 13:37:17 foreman smart-proxy: 10.63.16.19 - - [29/Jun/2021:13:37:17 UTC] "POST /dhcp/10.63.16.0 HTTP/1.1" 404 27
Jun 29 13:37:17 foreman smart-proxy: - -> /dhcp/10.63.16.0
https://foreman.fishy.com:9090/dhcp/10.63.16.50 returns 404
Removed the ip 10.63.16.19 from
tftp_servername:
in /etc/foreman-installer/scenarios.d/katello-answers.yaml
Re-ran the installer
and still issue with the /dhcp/ end point…
Jesus…
This is the pink error box that shows up when creating a host
Unable to save
Create DHCP Settings for fred-test1.fishy.com task failed with the following error: ERF12-6899 [ProxyAPI::ProxyException]: Unable to set DHCP entry ([RestClient::NotFound]: 404 Not Found) for proxy https://foreman.fishy.com:9090/dhcp
Ok it looks like the DHCP feature in the smart proxy got messed up…
Disabling all modules in the group ['dhcp'] due to a failure in one of them: comparison of Symbol with String failed
smart proxy logs in DEBUG
2021-06-29T14:17:39 [E] Disabling all modules in the group ['dhcp'] due to a failure in one of them: comparison of Symbol with String failed
Which Symbol? Which String? Where?
I was also hitting this bug today, I have not found how it happens yet, but I think it takes some boolean option and inserts it as string. What was even more confusing it was in a training with 8 attendees but not all have hit the bug with exactly the same setup! I also do not see a change that directly looks to be relevant and as I am not sure who could have a look into this, @tbrisker do you know someone?
I don’t know it that helps
/usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb
It fails on the:
sorted_keys = settings.keys.sort
I’ve added some logging to see what’s there
def log_provider_settings(settings)
default_settings = plugin.plugin_default_settings
to_log = settings.keys.map { |k| "'%s': %s%s" % [k, settings[k], (default_settings.include?(k) && default_settings[k] == settings[k]) ? " (default)" : ""] }.join(", ")
logger.debug "'%s' FRED3_1 settings: %s" % [plugin.plugin_name, to_log]
sorted_keys = settings.keys.sort
to_log = sorted_keys.map { |k| "'%s': %s%s" % [k, settings[k], (default_settings.include?(k) && default_settings[k] == settings[k]) ? " (default)" : ""] }.join(", ")
logger.debug "'%s' settings: %s" % [plugin.plugin_name, to_log]
end
2021-06-29T18:48:43 [D] 'tftp' ports: 'http': false, 'https': true
2021-06-29T18:48:43 [D] 'dhcp' FRED3_1 settings: 'use_provider': dhcp_isc (default), 'server': 127.0.0.1 (default), 'subnets': [] (default), 'ping_free_ip': false, 'enabled': https, 'subnets': ["10.63.4.0/255.255.255.0", "10.63.5.0/255.255.255.0", "10.63.6.0/255.255.255.0", "10.63.7.0/255.255.255.0", "10.63.8.0/255.255.255.0", "10.63.10.0/255.255.255.0", "10.63.14.0/255.255.255.0", "10.63.15.0/255.255.255.0", "10.63.16.0/255.255.255.0", "10.63.17.0/255.255.255.0", "10.63.18.0/255.255.255.0", "10.63.19.0/255.255.255.0", "10.63.20.0/255.255.255.0", "10.63.21.16/255.255.255.240", "10.63.23.0/255.255.255.0", "10.63.28.0/255.255.255.0", "10.63.56.0/255.255.255.0"]
2021-06-29T18:48:43 [E] Disabling all modules in the group ['dhcp'] due to a failure in one of them: comparison of Symbol with String failed
2021-06-29T18:48:43 [D] 'puppetca' FRED3_1 settings: 'use_provider': ["puppetca_hostname_whitelisting", :puppetca_http_api], 'enabled': https, 'puppet_version': 6.22.1
There was one change, removal of deprecated behavior that we appear to log into production.log but we haven’t advertised in the Upgrade warnings unfortunately. This was the original bug:
In short, previously Foreman server generated TFTP servername from smart-proxy hostname by resolving it to IP address and sending this information to TFTP proxy when creating new DHCP record. This was confusing for users with incorrect DNS, also it caused the original bug which I was fixing. So we have decided to remove this behavior and update proxy to either pass hostname instead of IP address (e.g. ISC DHCP accepts this) or DHCP module implementations can choose to do the same DNS resolution.
Up until 2.4, Foreman were only issuing a warning message if DHCP module did not report hostname resolution capability, with 2.5 this has been removed and Foreman no longer converts hostnames to IPs.
And this is where we are today, not sure what is now broken, probably some regression. We need to take a look, any insights appreciated.
So are saying that for the installer this should be given as param:
--foreman-proxy-tftp-servername foreman.fishy.com
instead of this
--foreman-proxy-tftp-servername 10.63.16.19
and is
Disabling all modules in the group ['dhcp'] due to a failure in one of them: comparison of Symbol with String failed
related to the above?
At this very moment I hate myself… 
So first using:
--foreman-proxy-tftp-servername foreman.fishy.com
did fix the tftp
supersede server.next-server
in /var/lib/dhcpd/dhcpd.leases
BUT most importantly and I need to hide in shame for this… The error
Disabling all modules in the group ['dhcp'] due to a failure in one of them: comparison of Symbol with String failed
came from /etc/foreman-proxy/settings.d/dhcp.yml having:
subnets:
instead of
:subnets:
I find this syntax awful and I think it is the time to ditch it:
So what was your problem, Dirk? If there is a scenario we missed it needs to be fixed. Are these upgraded instances? Or new installations?
I am provisioning 2.5 stable at the moment to see myself.
It were fresh installations done like this:
foreman-installer \
--foreman-proxy-tftp=true \
--foreman-proxy-dns=true \
--foreman-proxy-dns-interface=ens3 \
--foreman-proxy-dns-zone=localdomain \
--foreman-proxy-dns-reverse=0.10.in-addr.arpa \
--foreman-proxy-dns-forwarders=8.8.8.8 \
--foreman-proxy-dns-forwarders=8.8.4.4 \
--foreman-proxy-dhcp=true \
--foreman-proxy-dhcp-interface=ens3 \
--foreman-proxy-dhcp-gateway=10.0.0.1 \
--foreman-proxy-dhcp-range="10.0.0.100 10.0.0.200" \
--foreman-proxy-dhcp-nameservers="10.0.0.2"
Than domain and subnet were configured, CentOS or Debian configured as operating system and then a host was created, so nothing special.
When the host then booted it got PXELinux configuration, but with a weird IP address as nextserver. So when debugging I found the supersede server.next-server = "false"; in /var/lib/dhcpd/dhcpd.leases which seemed wrong, so I removed the line and restartet dhcpd and it run fine for the host.
As this fixed it for the one host, we tried to do if removing smart-proxy/omapi_provider.rb at fdeef1dc6febcfae22c8d3273cb18d6bdeb31a23 · theforeman/smart-proxy · GitHub fixes it also for new hosts which it does. This is probably no solution but it really fits my idea of a boolean false interpreted as string.
@Dirk Maybe if you re-run the foreman-installer command adding:
--foreman-proxy-tftp-servername <your foreman server FQDN>
it might solved all this.
It did for me
Damn I just installed my test instance and I do have this option, that’s why it worked for me:
foreman-installer -v --scenario $SCENARIO \
$INITIAL_CMD \
--enable-foreman-plugin-discovery \
--enable-foreman-plugin-bootdisk \
--enable-foreman-plugin-templates \
--enable-foreman-plugin-remote-execution --enable-foreman-proxy-plugin-remote-execution-ssh \
--enable-foreman-plugin-openscap --enable-foreman-proxy-plugin-openscap \
--foreman-proxy-dns true \
--foreman-proxy-dns-interface $NICDEV \
--foreman-proxy-dns-forwarders 192.168.${NATLAN}.1 \
--foreman-proxy-dns-zone nat.lan \
--foreman-proxy-dns-reverse ${NATLAN}.168.192.in-addr.arpa \
--foreman-proxy-dhcp true \
--foreman-proxy-dhcp-interface $NICDEV \
--foreman-proxy-dhcp-gateway=192.168.${NATLAN}.1 \
--foreman-proxy-dhcp-range="192.168.${NATLAN}.10 192.168.${NATLAN}.109" \
--foreman-proxy-dhcp-nameservers="192.168.${NATLAN}.${IP}" \
--foreman-proxy-tftp true \
--foreman-proxy-tftp-servername=192.168.${NATLAN}.${IP} \
--foreman-proxy-puppet true \
--foreman-proxy-puppetca true \
--foreman-proxy-http=true \
--puppet-runmode none \
--foreman-proxy-templates true \
--foreman-proxy-logs true \
--foreman-proxy-register-in-foreman true
It created my record just fine.
host amos-hehir.nat.lan {
dynamic;
hardware ethernet aa:bb:cc:dd:ee:f1;
fixed-address 192.168.199.122;
supersede server.filename = "pxelinux.0";
supersede server.next-server = c0:a8:c7:28;
@ekohl is there a way to “unset” an option to see what Puppet sets when this option was not provided?
@lzap Maybe edit the answer file for your scenario in /etc/foreman-installer/scenarios.d/ and re-run your foreman-installer omitting --foreman-proxy-tftp-servername
Yeah, I can confirm. Reproduced, fixed. It’s a regression in empty string handling:
Thanks a lot.
So as it is a Foreman bug, it needs to be reviewed and merged which should be quite easy here and than be part of the next minor release 2.5.2 which is planned for in two weeks.
So for everyone hitting this bug before this: In this thread are some workarounds and you can apply this fix also manually if needed!
@Dirk or @fred_demarcy, can one of you confirm that the proposed patch fixes the issue for you? I can review the code, but currently don’t have a working tftp environment to test it in. Thanks!