When attempting UEFI HTTPBoot I am seeing TFTP File not found

Problem:

My Hyper-V Gen 2 VM fails to httpboot UEFI. Looks like it’s using TFTP (problem #1?) and HTTP + port 8443 (problem #2?)

Expected outcome:

I can access grubx64.efi at http://:8000/httpboot/grub2/grubx64.efi (via web browser) and I thought my VM would make a request to that URI via HTTP. Instead, it makes a TFTP request to http://:8443/httpboot/grub2/grubx64.efi which results in File not found.

Foreman and Proxy versions:

Foreman 1.20
Smart Proxy: 1.23.1

Foreman and Proxy plugin versions:

n/a

Distribution and version:

Foreman: CentOS Linux release 7.6.1810 (Core)
Smart Proxy: CentOS Linux release 7.7.1908 (Core)

Other relevant data:
My guest’s PXE Loader is set to “Grub2 UEFI HTTP”. When the Gen 2 machine boots, I see requests come in. These are the logs:

Nov 18 15:04:24 smartproxy-hyperv in.tftpd[213643]: RRQ from 10.61.179.153 filename http://smartproxy-hyperv.example.com:8443/httpboot/grub2/grubx64.efi
Nov 18 15:04:24 smartproxy-hyperv in.tftpd[213643]: Client 10.61.179.153 File not found http://smartproxy-hyperv.example.com:8443/httpboot/grub2/grubx64.efi

And these are from a packet trace (confirming the same):

  5 0.000874619 10.61.179.153 -> 10.61.179.254 TFTP 144 Read Request, File: http://smartproxy-hyperv.example.com:8443/httpboot/grub2/grubx64.efi, Transfer type: octet, tsize\000=0\000, blksize\000=1482\000
  6 0.006249638 10.61.179.254 -> 10.61.179.153 TFTP 61 Error Code, Code: File not found, Message: File not found

My Foreman config is here:

[root@smartproxy-hyperv ~]# cat /etc/foreman-installer/scenarios.d/foreman-answers.yaml
# Format:
# <classname>: false - don't include this class
# <classname>: true - include and use the defaults
# <classname>:
#   <param>: <value> - include and override the default(s)
#
# See params.pp in each class for what options are available

---
foreman: false
foreman::cli: false
foreman::cli::ansible: false
foreman::cli::discovery: false
foreman::cli::kubevirt: false
foreman::cli::openscap: false
foreman::cli::remote_execution: false
foreman::cli::tasks: false
foreman::cli::templates: false
foreman_proxy:
  repo: 
  gpgcheck: true
  version: present
  ensure_packages_version: present
  plugin_version: installed
  bind_host:
  - '*'
  http_port: 8000
  ssl_port: 8443
  dir: /usr/share/foreman-proxy
  user: foreman-proxy
  groups: []
  log: /var/log/foreman-proxy/proxy.log
  log_level: DEBUG
  log_buffer: 2000
  log_buffer_errors: 1000
  http: true
  ssl: true
  ssl_ca: /etc/puppetlabs/puppet/ssl/certs/ca.pem
  ssl_cert: /etc/puppetlabs/puppet/ssl/certs/smartproxy-hyperv.example.com.pem
  ssl_key: /etc/puppetlabs/puppet/ssl/private_keys/smartproxy-hyperv.example.com.pem
  foreman_ssl_ca: 
  foreman_ssl_cert: 
  foreman_ssl_key: 
  trusted_hosts:
  - foreman.example.com
  ssl_disabled_ciphers: []
  tls_disabled_versions: []
  manage_sudoersd: true
  use_sudoersd: true
  use_sudoers: true
  puppetca: true
  puppetca_split_configs: true
  puppetca_listen_on: https
  ssldir: /etc/puppetlabs/puppet/ssl
  puppetdir: /etc/puppetlabs/puppet
  puppetca_cmd: /opt/puppetlabs/bin/puppet cert
  puppet_group: puppet
  puppetca_provider: puppetca_hostname_whitelisting
  autosignfile: /etc/puppetlabs/puppet/autosign.conf
  puppetca_sign_all: false
  puppetca_tokens_file: /var/lib/foreman-proxy/tokens.yml
  puppetca_token_ttl: 360
  puppetca_certificate: 
  manage_puppet_group: true
  puppet: false
  puppet_listen_on: https
  puppetrun_provider: 
  customrun_cmd: /bin/false
  customrun_args: -ay -f -s
  mcollective_user: root
  puppetssh_sudo: false
  puppetssh_command: /opt/puppetlabs/bin/puppet agent --onetime --no-usecacheonfailure
  puppetssh_user: root
  puppetssh_keyfile: /etc/foreman-proxy/id_rsa
  puppetssh_wait: false
  salt_puppetrun_cmd: puppet.run
  puppet_user: root
  puppet_url: https://smartproxy-hyperv.example.com:8140
  puppet_ssl_ca: /etc/puppetlabs/puppet/ssl/certs/ca.pem
  puppet_ssl_cert: /etc/puppetlabs/puppet/ssl/certs/smartproxy-hyperv.example.com.pem
  puppet_ssl_key: /etc/puppetlabs/puppet/ssl/private_keys/smartproxy-hyperv.example.com.pem
  puppet_api_timeout: 30
  templates: false
  templates_listen_on: both
  template_url: http://smartproxy-hyperv.example.com:8000
  logs: true
  logs_listen_on: both
  httpboot: true
  httpboot_listen_on: both
  tftp: true
  tftp_listen_on: both
  tftp_managed: true
  tftp_manage_wget: true
  tftp_syslinux_filenames:
  - /usr/share/syslinux/chain.c32
  - /usr/share/syslinux/mboot.c32
  - /usr/share/syslinux/menu.c32
  - /usr/share/syslinux/memdisk
  - /usr/share/syslinux/pxelinux.0
  tftp_root: /var/lib/tftpboot
  tftp_dirs:
  - /var/lib/tftpboot/pxelinux.cfg
  - /var/lib/tftpboot/grub
  - /var/lib/tftpboot/grub2
  - /var/lib/tftpboot/boot
  - /var/lib/tftpboot/ztp.cfg
  - /var/lib/tftpboot/poap.cfg
  tftp_servername: 
  tftp_replace_grub2_cfg: false
  dhcp: true
  dhcp_listen_on: both
  dhcp_managed: true
  dhcp_provider: isc
  dhcp_subnets:
  - 10.61.179.0
  dhcp_option_domain:
  - example.com
  dhcp_search_domains: 
  dhcp_interface: eth0
  dhcp_additional_interfaces: []
  dhcp_gateway: 
  dhcp_range: 10.61.179.20 10.61.179.199
  dhcp_pxeserver: 
  dhcp_pxefilename: pxelinux.0
  dhcp_network: 
  dhcp_netmask: 
  dhcp_nameservers: 10.35.35.35,10.36.36.36
  dhcp_server: 127.0.0.1
  dhcp_config: /etc/dhcp/dhcpd.conf
  dhcp_leases: /var/lib/dhcpd/dhcpd.leases
  dhcp_key_name: 
  dhcp_key_secret: 
  dhcp_omapi_port: 7911
  dhcp_peer_address: 
  dhcp_node_type: standalone
  dhcp_failover_address: 10.61.179.254
  dhcp_failover_port: 519
  dhcp_max_response_delay: 30
  dhcp_max_unacked_updates: 10
  dhcp_mclt: 300
  dhcp_load_split: 255
  dhcp_load_balance: 3
  dhcp_manage_acls: true
  dns: true
  dns_listen_on: both
  dns_managed: true
  dns_provider: infoblox
  dns_interface: eth0
  dns_zone: example.com
  dns_reverse: 
  dns_server: 127.0.0.1
  dns_ttl: 86400
  dns_tsig_keytab: /etc/foreman-proxy/dns.keytab
  dns_tsig_principal: foremanproxy/smartproxy-hyperv.example.com@example.com
  dns_forwarders: []
  libvirt_network: default
  libvirt_connection: qemu:///system
  bmc: false
  bmc_listen_on: https
  bmc_default_provider: ipmitool
  bmc_ssh_user: root
  bmc_ssh_key: /usr/share/foreman/.ssh/id_rsa
  bmc_ssh_powerstatus: 'true'
  bmc_ssh_powercycle: shutdown -r +1
  bmc_ssh_poweroff: shutdown +1
  bmc_ssh_poweron: 'false'
  realm: false
  realm_listen_on: https
  realm_provider: freeipa
  realm_keytab: /etc/foreman-proxy/freeipa.keytab
  realm_principal: realm-proxy@example.com
  freeipa_config: /etc/ipa/default.conf
  freeipa_remove_dns: true
  keyfile: /etc/rndc.key
  register_in_foreman: true
  foreman_base_url: https://foreman.example.com
  registered_name: smartproxy-hyperv.example.com
  registered_proxy_url: 
  oauth_effective_user: admin
  oauth_consumer_key: *********
  oauth_consumer_secret: *********
puppet: false
foreman::plugin::ansible: false
foreman::plugin::azure: false
foreman::plugin::bootdisk: false
foreman::plugin::chef: false
foreman::plugin::cockpit: false
foreman::plugin::default_hostgroup: false
foreman::plugin::dhcp_browser: false
foreman::plugin::digitalocean: false
foreman::plugin::discovery: false
foreman::plugin::expire_hosts: false
foreman::plugin::hooks: false
foreman::plugin::host_extra_validator: false
foreman::plugin::kubevirt: false
foreman::plugin::memcache: false
foreman::plugin::monitoring: false
foreman::plugin::omaha: false
foreman::plugin::openscap: false
foreman::plugin::ovirt_provision: false
foreman::plugin::puppetdb: false
foreman::plugin::remote_execution: false
foreman::plugin::salt: false
foreman::plugin::setup: false
foreman::plugin::snapshot_management: false
foreman::plugin::tasks: false
foreman::plugin::templates: false
foreman::compute::ec2: false
foreman::compute::gce: false
foreman::compute::libvirt: false
foreman::compute::openstack: false
foreman::compute::ovirt: false
foreman::compute::rackspace: false
foreman::compute::vmware: false
foreman_proxy::plugin::ansible: false
foreman_proxy::plugin::chef: false
foreman_proxy::plugin::dhcp::infoblox: false
foreman_proxy::plugin::dhcp::remote_isc: false
foreman_proxy::plugin::discovery: false
foreman_proxy::plugin::dns::infoblox:
  dns_server: infoblox.example.com
  username: svc_foreman
  password: ***********
  dns_view: default
foreman_proxy::plugin::dns::powerdns: false
foreman_proxy::plugin::dynflow: false
foreman_proxy::plugin::monitoring: false
foreman_proxy::plugin::omaha: false
foreman_proxy::plugin::openscap: false
foreman_proxy::plugin::pulp: false
foreman_proxy::plugin::remote_execution::ssh: false
foreman_proxy::plugin::salt: false

My dhcpd.conf looks like this:

[root@smartproxy-hyperv ~]# cat /etc/dhcp/dhcpd.conf 
# dhcpd.conf
omapi-port 7911;

default-lease-time 43200;
max-lease-time 86400;

not authoritative;

ddns-update-style none;

option domain-name "example.com";
option domain-name-servers 10.35.35.35, 10.36.36.36;
option ntp-servers none;

allow booting;
allow bootp;

option fqdn.no-client-update    on;  # set the "O" and "S" flag bits
option fqdn.rcode2            255;
option pxegrub code 150 = text ;

# Bootfile Handoff
next-server 10.61.179.254;
option architecture code 93 = unsigned integer 16 ;
if option architecture = 00:06 {
  filename "grub2/shim.efi";
} elsif option architecture = 00:07 {
  filename "grub2/shim.efi";
} elsif option architecture = 00:09 {
  filename "grub2/shim.efi";
} else {
  filename "pxelinux.0";
}

log-facility local7;

include "/etc/dhcp/dhcpd.hosts";
# example.com
subnet 10.61.179.0 netmask 255.255.255.0 {
  pool
  {
    range 10.61.179.20 10.61.179.199;
  }
  option subnet-mask 255.255.255.0;
}

The good news:

I messed with /etc/dhcpd.conf by hand and restarted dhcpd each time I made a change. Nothing I did was moving the ball forward so I regrouped by deleting the dhcpd.conf file and re-running foreman-installer. After a reboot, I tried my UEFI PXE again and lo and behold, it worked.

Nov 19 22:21:43 smartproxy-hyperv dhcpd: DHCPDISCOVER from 00:1d:d8:b7:1c:06 via eth0
Nov 19 22:21:44 smartproxy-hyperv dhcpd: DHCPOFFER on 10.61.179.23 to 00:1d:d8:b7:1c:06 via eth0
Nov 19 22:21:51 smartproxy-hyperv dhcpd: DHCPREQUEST for 10.61.179.23 (10.61.179.254) from 00:1d:d8:b7:1c:06 via eth0
Nov 19 22:21:51 smartproxy-hyperv dhcpd: DHCPACK on 10.61.179.23 to 00:1d:d8:b7:1c:06 via eth0
Nov 19 22:21:51 smartproxy-hyperv dhcpd: DHCPREQUEST for 10.61.179.23 (10.61.179.254) from 00:1d:d8:b7:1c:06 via 10.61.179.2
Nov 19 22:21:51 smartproxy-hyperv dhcpd: DHCPACK on 10.61.179.23 to 00:1d:d8:b7:1c:06 via 10.61.179.2
Nov 19 22:21:55 smartproxy-hyperv in.tftpd[10293]: RRQ from 10.61.179.23 filename grub2/shim.efi
Nov 19 22:21:55 smartproxy-hyperv in.tftpd[10293]: Error code 8: User aborted the transfer
Nov 19 22:21:55 smartproxy-hyperv in.tftpd[10294]: RRQ from 10.61.179.23 filename grub2/shim.efi
Nov 19 22:21:55 smartproxy-hyperv in.tftpd[10294]: Client 10.61.179.23 finished grub2/shim.efi
Nov 19 22:21:55 smartproxy-hyperv in.tftpd[10295]: RRQ from 10.61.179.23 filename grub2/grubx64.efi
Nov 19 22:21:56 smartproxy-hyperv in.tftpd[10295]: Client 10.61.179.23 finished grub2/grubx64.efi
Nov 19 22:21:56 smartproxy-hyperv in.tftpd[10296]: RRQ from 10.61.179.23 filename /grub2/grub.cfg-01-00-1d-d8-b7-1c-06
Nov 19 22:21:56 smartproxy-hyperv in.tftpd[10296]: Client 10.61.179.23 finished /grub2/grub.cfg-01-00-1d-d8-b7-1c-06
Nov 19 22:21:56 smartproxy-hyperv in.tftpd[10297]: RRQ from 10.61.179.23 filename /EFI/centos/x86_64-efi/command.lst
Nov 19 22:21:56 smartproxy-hyperv in.tftpd[10297]: Client 10.61.179.23 File not found /EFI/centos/x86_64-efi/command.lst
Nov 19 22:21:56 smartproxy-hyperv in.tftpd[10298]: RRQ from 10.61.179.23 filename /EFI/centos/x86_64-efi/fs.lst
Nov 19 22:21:56 smartproxy-hyperv in.tftpd[10298]: Client 10.61.179.23 File not found /EFI/centos/x86_64-efi/fs.lst
Nov 19 22:21:56 smartproxy-hyperv in.tftpd[10299]: RRQ from 10.61.179.23 filename /EFI/centos/x86_64-efi/crypto.lst
Nov 19 22:21:56 smartproxy-hyperv in.tftpd[10299]: Client 10.61.179.23 File not found /EFI/centos/x86_64-efi/crypto.lst
Nov 19 22:21:56 smartproxy-hyperv in.tftpd[10300]: RRQ from 10.61.179.23 filename /EFI/centos/x86_64-efi/terminal.lst
Nov 19 22:21:56 smartproxy-hyperv in.tftpd[10300]: Client 10.61.179.23 File not found /EFI/centos/x86_64-efi/terminal.lst
Nov 19 22:21:56 smartproxy-hyperv in.tftpd[10301]: RRQ from 10.61.179.23 filename /grub2/grub.cfg-01-00-1d-d8-b7-1c-06
Nov 19 22:21:56 smartproxy-hyperv in.tftpd[10301]: Client 10.61.179.23 finished /grub2/grub.cfg-01-00-1d-d8-b7-1c-06

The bad news:

a.) it’s using in.tftpd but I am under the impression that the Grub2 UEFI HTTP PXE Loader would use HTTP. This I do not understand and I’m assuming at this point that it is working as designed.
b.) a diff between the dhcpd.conf file above and the one on my PXE server does not expose what the problem may have been. I’m to believe that it was something somewhere else (not a DHCP config issue) but I don’t know what exactly.

Sorry for such a useless post but maybe it will help somebody somewhere along the way. My stars weren’t aligned until my most recent run of removing /etc/dhcpd.conf, running foreman-installer, and rebooting.

1 Like

Hello,

this is a known bug fixed in 1.24. Earlier versions of Foreman use port that is specified in your HTTPBoot Smart Proxy URL (the field which is stored in Foreman DB in Smart Proxies page). This was incorrect, version 1.24 now correctly detect port which is used for HTTP communication by the proxy and uses that.

No workaround unfortunately other than hardcoding the port into the code. After 1.24 is out (very soon) you should have UEFI HTTP Boot working as expected. Here is the issue I am talking about:

https://projects.theforeman.org/issues/26338

To hotfix your 1.23 just find this line:

and replace it with:

self.class.all_loaders_map(architecture, "#{boot_uri.host}:8000")[host.pxe_loader]

I am assuming you are using the default number 8000. Then it will work for HTTP comm, however if you choose HTTPS PXE loader than it will fail (as that must be 8443 or 9090 depending on your install scenario).

This was actually working and when I came in the next day, it had stopped. I didn’t make any changes, I literally rebooted my VM because I wanted to start the day off knowing that it was working but it failed. It reverted to the same behavior (what I’m referring to as “protocol mismatch” - HTTP over 8443).

I made the change you recommended, here is the full function as it stands now:

  def boot_filename(host = nil)
    return default_boot_filename if host.nil? || host.pxe_loader.nil?
    return host.foreman_url('iPXE') if host.pxe_loader == 'iPXE Embedded'
    architecture = host.arch.nil? ? '' : host.arch.bootfilename_efi
    boot_uri = URI.parse((host.subnet&.httpboot?) ? host.subnet.httpboot.url : Setting[:unattended_url])
    self.class.all_loaders_map(architecture, "#{boot_uri.host}:8000")[host.pxe_loader]
  end

When my VM boots, this is the request that comes in from the client. It is immediately asking for http://<tftp_server>:8443 I have re-ran foreman-installer and rebooted a few times. dhcpd.conf looks pretty standard.

Frame 9: 144 bytes on wire (1152 bits), 144 bytes captured (1152 bits) on interface 0
    Interface id: 0
    Encapsulation type: Ethernet (1)
    <...omitted...>
    Frame Length: 144 bytes (1152 bits)
    Capture Length: 144 bytes (1152 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ip:udp:tftp]
Ethernet II, Src: Microsof_b7:1c:06 (00:1d:d8:b7:1c:06), Dst: Microsof_83:68:64 (00:15:5d:83:68:64)
    <...omitted...>
    Type: IP (0x0800)
Internet Protocol Version 4, Src: 10.61.179.153 (10.61.179.153), Dst: 10.61.179.254 (10.61.179.254)
    Version: 4
    Header length: 20 bytes
    <...omitted...>
    User Datagram Protocol, Src Port: qsm-remote (1166), Dst Port: tftp (69)
    Source port: qsm-remote (1166)
    Destination port: tftp (69)
    Length: 110
    Checksum: 0x6585 [validation disabled]
        [Good Checksum: False]
        [Bad Checksum: False]
Trivial File Transfer Protocol
    Opcode: Read Request (1)
    Source File: http://smartproxy-hyperv.example.com:8443/httpboot/grub2/grubx64.efi
    <...omitted...>

Any ideas?

The PXELoader entry (“filename”) is written only in the moment new host is created or a host enters “build” mode. Then Foreman performs DHCP orchestration creating new reservation in dhcpd.leases file. So you probably need to exit and reenter build mode or there is also a feature called “Rebuild Config” which will delete and create reservation.

After that you should see the correct port in the dhcpd.leases file and it should be booting from the correct (HTTP) port.