Can't PXE boot/kickstart after successful host creation

Support
1

Problem: Newly created VM will not kickstart after successful host creation in Foreman

Expected outcome: VM kickstarts and installs Ubuntu 20.04 LTS

Foreman and Proxy versions: 3.7

Distribution and version: Ubuntu 20.04 LTS

Other relevant data:

I have successfully created a test host in my Foreman environment, with an associated MAC/IP address via DHCP. When I turn on the VM in VMware, it uses its DHCP assigned IP address which matches Foreman, then tries to PXE boot. Yesterday when I had my boot options set to BIOS, it stated the NBP was too large… so, I switched it to EFI. When attempting it this way, it runs through various boot options but ultimately fails and presents a menu to me in which I can select EFI Network boot, and it attempts to PXE boot over IPv4, which is what I think should happen… but it fails again and doesn’t start the PXE boot process.

All templates are properly associated for a Debian/Ubuntu system (i.e. Preseed default, preseed default finish, preseed default PXELinux, preseed default PXEGrub2, etc).

Am I missing something?

2

Output of my dhcpd.conf file:

default-lease-time 43200;
max-lease-time 86400;


not authoritative;


ddns-update-style none;

option domain-name "spectric.dev";
option domain-name-servers dns.spectric.dev,10.48.30.50,10.48.30.253;
option ntp-servers none;

allow booting;
allow bootp;

option fqdn.no-client-update    on;  # set the "O" and "S" flag bits
option fqdn.rcode2            255;
option pxegrub code 150 = text ;




# required for UEFI HTTP boot
if substring(option vendor-class-identifier, 0, 10) = "HTTPClient" {
  option vendor-class-identifier "HTTPClient";
}
# promote vendor in dhcpd.leases
set vendor-string = option vendor-class-identifier;
# next server and filename options
next-server 10.48.30.102;
option architecture code 93 = unsigned integer 16 ;
if option architecture = 00:06 {
  filename "grub2/grubia32.efi";
} elsif option architecture = 00:07 {
  filename "grub2/grubx64.efi";
} elsif option architecture = 00:09 {
  filename "grub2/grubx64.efi";
} else {
  filename "pxelinux.0";
}

log-facility local7;

include "/etc/dhcp/dhcpd.hosts";
# spectric.dev
subnet 10.48.30.0 netmask 255.255.255.0 {
  pool
  {
    range 10.48.30.50 10.48.30.100;
  }

  option subnet-mask 255.255.255.0;
  option routers 10.48.30.255;
  option domain-search "spectric.dev";
}
3

Is there a more specific error on the test host about which step is failing (e.g. getting the DHCP lease, downloading the NBP, running the NBP)? If you look at the log on the DHCP server, is it receiving the right MAC address and responding adequately? If you look at the log on the TFTP server (after adding -v) or HTTP server, is it receiving the request and returning the file successfully?

4

The DHCP lease is fine. There is no file exchange after the DHCPACK goes through.

TFTP, DHCP, etc, are all done through the same server.

The log from /var/log/syslog shows:

Nov 15 16:09:42 puppet dhcpd[95244]: DHCPOFFER on 10.48.30.89 to 00:0c:29:43:12:d5 via ens160
Nov 15 16:09:46 puppet dhcpd[95244]: DHCPREQUEST for 10.48.30.89 (10.48.30.102) from 00:0c:29:43:12:d5 via ens160
Nov 15 16:09:46 puppet dhcpd[95244]: DHCPACK on 10.48.30.89 to 00:0c:29:43:12:d5 via ens160

Then nothing else happens and the VM kicks out to the Boot Manager window.

TFTP is served via /srv/tftp for me. In there, I have all of the necessary files for booting (a config in pxelinux.cfg for the MAC address, the grubx64.efi file etc)…

If there’s anything else I can share/show you, let me know and I’ll gladly provide it.

Side note, sometimes the logs show “Dynamic and static leases present for 10.48.30.89. Remove host declaration or remove 10.48.30.89 from the dynamic address pool.” but then it also finished the request/ack.

5

Are you getting any requests logged by TFTPD or your HTTPD? In my experience TFTPD won’t log individual requests unless you run it -v, so I would make sure if you manually do a request with the tftp command that it actually gets logged, if you don’t already have other examples of TFTP requests getting logged.

6

If I manually run tftp, this is the output:

root:/var/lib/dhcp# tftp 10.48.30.102
tftp> verbose
Verbose mode on.
tftp> get grub2/grubx64.efi
getting from 10.48.30.102:grub2/grubx64.efi to grubx64.efi [netascii]
Received 2285778 bytes in 0.1 seconds [182862240 bits/sec]
tftp>
7

Right but like, is that getting you output in the tftpd log, such that you can have confidence that if your client were making a request, that you would see it in the tftpd log? If not, your client might be making a request, and you’re just not seeing it in the log because tftpd isn’t logging any requests.

8

There are no tftpd logs that I’ve been able to find… even when I was using xinet to run tftp prior to tftpd-hpa… hm.

9

The few times I’ve seen tftp in logs, was in.tftpd in the /var/log/syslog. That is NOT displaying at all during the pxe boot process.

10

So I think a first step would be to make sure when you do a TFTP request manually that you’re getting a corresponding log entry from tftpd, so you can be more sure of the reason you’re not seeing one from your PXE client. That may involve adding -v to its invocation, whether that’s in its systemd or xinetd configuration.

11

E.g. I have

# /etc/systemd/system/tftp.service.d/override.conf
[Service]
ExecStart=
ExecStart=/usr/sbin/in.tftpd -v -s /var/lib/tftpboot

and in the (xinetd) past I’ve had

service tftp
{
        [...]
        server_args     = -v -s /var/lib/tftpboot -m /etc/tftpd.map
12

Yeah, I’ve been trying to figure that bit out with tftpd-hpa. May have to
go back to the one via xinet and see if the logging picks back up. I’ll
report back as soon as I can.

13

Alright, so, I still couldn’t figure out logging for tftpd-hpa … BUT, I started a tcpdump -i ens160 'upd port 69' and I got ZERO traffic during the PXE/EFI boot process. I did try it from another server, running tftp and getting a file… the file transferred successfully and the tcpdump showed the traffic…

It seems as though the test server is never attempting TFTP at all. So, where to next?

14

If you’re into tcpdumping, you could look at what’s happening with the DHCP request and response: do they have the contents you expect them to? Otherwise it sounds like you’re going to need the client to be a bit more specific about what it thinks the problem is. Is it just not saying anything? If it’s scrolling or overwriting too fast, can you have VMware capture the output, or get it as a serial console so it stays in your terminal? Is there some other way in VMware to retrieve what the error was from the VM’s point of view?

15

DHCP logs fine via syslog, showing an offer, request, and ack packet (as
expected). In VMware, it doesn’t show an error message outside of the three
different boot variants not having media.

I can try the tcpdump on the DHCP side for sure and see what happens.

16

I’d say it works according to the last bit of the tcpdump…

root:~# tcpdump -i ens160 port 67 or port 68 -e -n -vv
tcpdump: listening on ens160, link-type EN10MB (Ethernet), capture size 262144 bytes
22:58:42.312622 00:0c:29:43:12:d5 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 389: (tos 0x0, ttl 64, id 50402, offset 0, flags [none], proto UDP (17), length 375)
    0.0.0.0.68 > 255.255.255.255.67: [udp sum ok] BOOTP/DHCP, Request from 00:0c:29:43:12:d5, length 347, xid 0x4a79134f, Flags [Broadcast] (0x8000)
          Client-Ethernet-Address 00:0c:29:43:12:d5
          Vendor-rfc1048 Extensions
            Magic Cookie 0x63825363
            DHCP-Message Option 53, length 1: Discover
            MSZ Option 57, length 2: 1472
            Parameter-Request Option 55, length 35:
              Subnet-Mask, Time-Zone, Default-Gateway, Time-Server
              IEN-Name-Server, Domain-Name-Server, Hostname, BS
              Domain-Name, RP, EP, RSZ
              TTL, BR, YD, YS
              NTP, Vendor-Option, Requested-IP, Lease-Time
              Server-ID, RN, RB, Vendor-Class
              TFTP, BF, GUID, Option 128
              Option 129, Option 130, Option 131, Option 132
              Option 133, Option 134, Option 135
            GUID Option 97, length 17: 0.86.77.151.201.146.26.10.66.6.241.183.233.172.67.18.213
            NDI Option 94, length 3: 1.3.0
            ARCH Option 93, length 2: 7
            Vendor-Class Option 60, length 32: "PXEClient:Arch:00007:UNDI:003000"
22:58:42.312951 00:0c:29:e5:da:db > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328)
    10.48.30.50.67 > 255.255.255.255.68: [udp sum ok] BOOTP/DHCP, Reply, length 300, xid 0x4a79134f, Flags [Broadcast] (0x8000)
          Your-IP 10.48.30.89
          Server-IP 10.48.30.50
          Client-Ethernet-Address 00:0c:29:43:12:d5
          Vendor-rfc1048 Extensions
            Magic Cookie 0x63825363
            DHCP-Message Option 53, length 1: Offer
            Server-ID Option 54, length 4: 10.48.30.50
            Lease-Time Option 51, length 4: 2053495
            Subnet-Mask Option 1, length 4: 255.255.255.0
            Domain-Name-Server Option 6, length 4: 10.48.30.50
            Domain-Name Option 15, length 12: "spectric.dev"
            BR Option 28, length 4: 10.48.30.255
22:58:42.315032 00:0c:29:0c:f1:f9 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328)
    10.48.30.102.67 > 255.255.255.255.68: [udp sum ok] BOOTP/DHCP, Reply, length 300, xid 0x4a79134f, Flags [Broadcast] (0x8000)
          Your-IP 10.48.30.89
          Server-IP 10.48.30.102
          Client-Ethernet-Address 00:0c:29:43:12:d5
          file "grub2/grubx64.efi"
          Vendor-rfc1048 Extensions
            Magic Cookie 0x63825363
            DHCP-Message Option 53, length 1: Offer
            Server-ID Option 54, length 4: 10.48.30.102
            Lease-Time Option 51, length 4: 43200
            Subnet-Mask Option 1, length 4: 255.255.255.0
            Default-Gateway Option 3, length 4: 10.48.30.255
            Domain-Name-Server Option 6, length 4: 10.48.30.50
            Domain-Name Option 15, length 12: "spectric.dev"
22:58:43.361465 00:09:0f:09:01:12 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 377: (tos 0x10, ttl 16, id 0, offset 0, flags [none], proto UDP (17), length 363)
    10.48.30.1.67 > 255.255.255.255.68: [udp sum ok] BOOTP/DHCP, Reply, length 335, xid 0x4a79134f, Flags [Broadcast] (0x8000)
          Your-IP 10.48.30.104
          Client-Ethernet-Address 00:0c:29:43:12:d5
          Vendor-rfc1048 Extensions
            Magic Cookie 0x63825363
            DHCP-Message Option 53, length 1: Offer
            Server-ID Option 54, length 4: 10.48.30.1
            Lease-Time Option 51, length 4: 604800
            Subnet-Mask Option 1, length 4: 255.255.255.0
            Default-Gateway Option 3, length 4: 10.48.30.1
            Domain-Name-Server Option 6, length 12: 10.48.30.50,1.1.1.3,1.0.0.3
            NTP Option 42, length 4: 10.48.30.1
            RN Option 58, length 4: 302400
            RB Option 59, length 4: 529200
            T224 Option 224, length 33: 70.71.49.48.48.70.84.75.50.49.48.48.52.54.56.49.70.71.49.48.48.70.84.75.50.49.48.48.52.57.49.52.0
22:58:46.129783 00:0c:29:43:12:d5 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 401: (tos 0x0, ttl 64, id 50403, offset 0, flags [none], proto UDP (17), length 387)
    0.0.0.0.68 > 255.255.255.255.67: [udp sum ok] BOOTP/DHCP, Request from 00:0c:29:43:12:d5, length 359, xid 0x4a79134f, Flags [Broadcast] (0x8000)
          Client-Ethernet-Address 00:0c:29:43:12:d5
          Vendor-rfc1048 Extensions
            Magic Cookie 0x63825363
            DHCP-Message Option 53, length 1: Request
            Server-ID Option 54, length 4: 10.48.30.102
            Requested-IP Option 50, length 4: 10.48.30.89
            MSZ Option 57, length 2: 65280
            Parameter-Request Option 55, length 35:
              Subnet-Mask, Time-Zone, Default-Gateway, Time-Server
              IEN-Name-Server, Domain-Name-Server, Hostname, BS
              Domain-Name, RP, EP, RSZ
              TTL, BR, YD, YS
              NTP, Vendor-Option, Requested-IP, Lease-Time
              Server-ID, RN, RB, Vendor-Class
              TFTP, BF, GUID, Option 128
              Option 129, Option 130, Option 131, Option 132
              Option 133, Option 134, Option 135
            GUID Option 97, length 17: 0.86.77.151.201.146.26.10.66.6.241.183.233.172.67.18.213
            NDI Option 94, length 3: 1.3.0
            ARCH Option 93, length 2: 7
            Vendor-Class Option 60, length 32: "PXEClient:Arch:00007:UNDI:003000"
22:58:46.130130 00:0c:29:e5:da:db > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328)
    10.48.30.50.67 > 255.255.255.255.68: [udp sum ok] BOOTP/DHCP, Reply, length 300, xid 0x4a79134f, Flags [Broadcast] (0x8000)
          Your-IP 10.48.30.89
          Server-IP 10.48.30.50
          Client-Ethernet-Address 00:0c:29:43:12:d5
          Vendor-rfc1048 Extensions
            Magic Cookie 0x63825363
            DHCP-Message Option 53, length 1: ACK
            Server-ID Option 54, length 4: 10.48.30.50
            Lease-Time Option 51, length 4: 2053491
            Subnet-Mask Option 1, length 4: 255.255.255.0
            Domain-Name-Server Option 6, length 4: 10.48.30.50
            Domain-Name Option 15, length 12: "spectric.dev"
            BR Option 28, length 4: 10.48.30.255
22:58:46.130264 00:0c:29:0c:f1:f9 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328)
    10.48.30.102.67 > 255.255.255.255.68: [udp sum ok] BOOTP/DHCP, Reply, length 300, xid 0x4a79134f, Flags [Broadcast] (0x8000)
          Your-IP 10.48.30.89
          Server-IP 10.48.30.102
          Client-Ethernet-Address 00:0c:29:43:12:d5
          file "grub2/grubx64.efi"
          Vendor-rfc1048 Extensions
            Magic Cookie 0x63825363
            DHCP-Message Option 53, length 1: ACK
            Server-ID Option 54, length 4: 10.48.30.102
            Lease-Time Option 51, length 4: 43200
            Subnet-Mask Option 1, length 4: 255.255.255.0
            Default-Gateway Option 3, length 4: 10.48.30.255
            Domain-Name-Server Option 6, length 4: 10.48.30.50
            Domain-Name Option 15, length 12: "spectric.dev"

but I still don’t get any booting start up… just kicks me to the boot manager menu still

17

If I am reading this correctly… this looks as though the file is being offered or requested by the new/test server… but then TFTP is not initializing for some reason? Is there some special call out I need in the DHCP config that I’m missing?

18

Maybe someone has a dhcpd.conf file they can compare to mine to see if I’m missing a line?