Foreman efi discovery fails on Fujitsu RX2530

Problem:
Foreman efi discovery fails on Fujitsu RX2530 after upgrading foreman and katello to version 2.0.1 and 3.15.3.

I get the following error:
VFS: Cannot open root device “live:/fdi.iso” or unkmnown-vlock(0,0): Error -19
Please append a correct “root=” boot option; here are the available partitions
Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0).

The server has a PDUAL CP200 controller with 2 in raid-1 configured ssd-m2 drives.

This worked well before the upgrade with versions Foreman 1.24.2 and Katello 3.14.x

Expected outcome:
System being discoverd
Foreman and Proxy versions:

Foreman and Proxy plugin versions:
Foreman 2.0.1
Katello 3.15.3

Distribution and version:
CentOS Linux release 7.8.2003 (Core)

Other relevant data:

Are you booting the ISO fro a USB stick or CDROM?

Which version of the FDI ISO?

Can you rule out corrupted media or download? Do a MD5 or something.

Hi Lukas,

Thanks for your reply.

I’m booting the systems from network, PXE.
It fetches the netboot image and I get the Foreman menu where I select to proceed with the “Foreman Discovery Image efi” as te systems only allow UEFI netboot.

This is what I get in the /var/log/messages:

Aug 19 11:31:33 dc1-bootstrap dhcpd: DHCPOFFER on 10.16.16.253 to 4c:52:62:b0:da:7d via br-pxe
Aug 19 11:31:36 dc1-bootstrap dhcpd: DHCPREQUEST for 10.16.16.253 (10.16.16.5) from 4c:52:62:b0:da:7d via br-pxe
Aug 19 11:31:36 dc1-bootstrap dhcpd: DHCPACK on 10.16.16.253 to 4c:52:62:b0:da:7d via br-pxe
Aug 19 11:31:37 dc1-bootstrap dhcpd: DHCPOFFER on 10.16.16.253 to 4c:52:62:b0:da:7d via br-pxe
Aug 19 11:31:41 dc1-bootstrap dhcpd: DHCPREQUEST for 10.16.16.253 (10.16.16.5) from 4c:52:62:b0:da:7d via br-pxe
Aug 19 11:31:41 dc1-bootstrap dhcpd: DHCPACK on 10.16.16.253 to 4c:52:62:b0:da:7d via br-pxe
Aug 19 11:31:41 dc1-bootstrap in.tftpd[19342]: RRQ from 10.16.16.253 filename grub2/shim.efi
Aug 19 11:31:41 dc1-bootstrap in.tftpd[19343]: RRQ from 10.16.16.253 filename grub2/shim.efi
Aug 19 11:31:42 dc1-bootstrap in.tftpd[19343]: Client 10.16.16.253 finished grub2/shim.efi
Aug 19 11:31:42 dc1-bootstrap in.tftpd[19345]: RRQ from 10.16.16.253 filename grub2/grubx64.efi
Aug 19 11:31:44 dc1-bootstrap in.tftpd[19345]: Client 10.16.16.253 finished grub2/grubx64.efi
Aug 19 11:31:44 dc1-bootstrap in.tftpd[19351]: RRQ from 10.16.16.253 filename grub2/grub.cfg-01-4c-52-62-b0-da-7d
Aug 19 11:31:44 dc1-bootstrap in.tftpd[19351]: Client 10.16.16.253 File not found grub2/grub.cfg-01-4c-52-62-b0-da-7d
Aug 19 11:31:44 dc1-bootstrap in.tftpd[19352]: RRQ from 10.16.16.253 filename grub2/grub.cfg-0A1010FD
Aug 19 11:31:44 dc1-bootstrap in.tftpd[19352]: Client 10.16.16.253 File not found grub2/grub.cfg-0A1010FD
Aug 19 11:31:44 dc1-bootstrap in.tftpd[19353]: RRQ from 10.16.16.253 filename grub2/grub.cfg-0A1010F
Aug 19 11:31:44 dc1-bootstrap in.tftpd[19353]: Client 10.16.16.253 File not found grub2/grub.cfg-0A1010F
Aug 19 11:31:44 dc1-bootstrap in.tftpd[19354]: RRQ from 10.16.16.253 filename grub2/grub.cfg-0A1010
Aug 19 11:31:44 dc1-bootstrap in.tftpd[19354]: Client 10.16.16.253 File not found grub2/grub.cfg-0A1010
Aug 19 11:31:44 dc1-bootstrap in.tftpd[19355]: RRQ from 10.16.16.253 filename grub2/grub.cfg-0A101
Aug 19 11:31:44 dc1-bootstrap in.tftpd[19355]: Client 10.16.16.253 File not found grub2/grub.cfg-0A101
Aug 19 11:31:44 dc1-bootstrap in.tftpd[19356]: RRQ from 10.16.16.253 filename grub2/grub.cfg-0A10
Aug 19 11:31:44 dc1-bootstrap in.tftpd[19356]: Client 10.16.16.253 File not found grub2/grub.cfg-0A10
Aug 19 11:31:44 dc1-bootstrap in.tftpd[19357]: RRQ from 10.16.16.253 filename grub2/grub.cfg-0A1
Aug 19 11:31:44 dc1-bootstrap in.tftpd[19357]: Client 10.16.16.253 File not found grub2/grub.cfg-0A1
Aug 19 11:31:44 dc1-bootstrap in.tftpd[19358]: RRQ from 10.16.16.253 filename grub2/grub.cfg-0A
Aug 19 11:31:44 dc1-bootstrap in.tftpd[19358]: Client 10.16.16.253 File not found grub2/grub.cfg-0A
Aug 19 11:31:44 dc1-bootstrap in.tftpd[19359]: RRQ from 10.16.16.253 filename grub2/grub.cfg-0
Aug 19 11:31:44 dc1-bootstrap in.tftpd[19359]: Client 10.16.16.253 File not found grub2/grub.cfg-0
Aug 19 11:31:44 dc1-bootstrap in.tftpd[19360]: RRQ from 10.16.16.253 filename grub2/grub.cfg
Aug 19 11:31:44 dc1-bootstrap in.tftpd[19360]: Client 10.16.16.253 finished grub2/grub.cfg
Aug 19 11:31:44 dc1-bootstrap in.tftpd[19361]: RRQ from 10.16.16.253 filename /EFI/centos/x86_64-efi/command.lst
Aug 19 11:31:44 dc1-bootstrap in.tftpd[19361]: Client 10.16.16.253 File not found /EFI/centos/x86_64-efi/command.lst
Aug 19 11:31:44 dc1-bootstrap in.tftpd[19362]: RRQ from 10.16.16.253 filename /EFI/centos/x86_64-efi/fs.lst
Aug 19 11:31:44 dc1-bootstrap in.tftpd[19362]: Client 10.16.16.253 File not found /EFI/centos/x86_64-efi/fs.lst
Aug 19 11:31:44 dc1-bootstrap in.tftpd[19363]: RRQ from 10.16.16.253 filename /EFI/centos/x86_64-efi/crypto.lst
Aug 19 11:31:44 dc1-bootstrap in.tftpd[19363]: Client 10.16.16.253 File not found /EFI/centos/x86_64-efi/crypto.lst
Aug 19 11:31:44 dc1-bootstrap in.tftpd[19364]: RRQ from 10.16.16.253 filename /EFI/centos/x86_64-efi/terminal.lst
Aug 19 11:31:44 dc1-bootstrap in.tftpd[19364]: Client 10.16.16.253 File not found /EFI/centos/x86_64-efi/terminal.lst
Aug 19 11:31:44 dc1-bootstrap in.tftpd[19365]: RRQ from 10.16.16.253 filename grub2/grub.cfg
Aug 19 11:31:44 dc1-bootstrap in.tftpd[19365]: Client 10.16.16.253 finished grub2/grub.cfg
Aug 19 11:31:44 dc1-bootstrap in.tftpd[19366]: RRQ from 10.16.16.253 filename /httpboot/grub2/grub.cfg-4c:52:62:b0:da:7d
Aug 19 11:31:44 dc1-bootstrap in.tftpd[19366]: Client 10.16.16.253 File not found /httpboot/grub2/grub.cfg-4c:52:62:b0:da:7d
Aug 19 11:31:44 dc1-bootstrap in.tftpd[19367]: RRQ from 10.16.16.253 filename /grub2/grub.cfg-4c:52:62:b0:da:7d
Aug 19 11:31:44 dc1-bootstrap in.tftpd[19367]: Client 10.16.16.253 File not found /grub2/grub.cfg-4c:52:62:b0:da:7d
Aug 19 11:31:44 dc1-bootstrap in.tftpd[19368]: RRQ from 10.16.16.253 filename grub2/grub.cfg-4c:52:62:b0:da:7d
Aug 19 11:31:44 dc1-bootstrap in.tftpd[19368]: Client 10.16.16.253 File not found grub2/grub.cfg-4c:52:62:b0:da:7d
Aug 19 11:31:44 dc1-bootstrap in.tftpd[19369]: RRQ from 10.16.16.253 filename grub.cfg-4c:52:62:b0:da:7d
Aug 19 11:31:44 dc1-bootstrap in.tftpd[19369]: Client 10.16.16.253 File not found grub.cfg-4c:52:62:b0:da:7d
Aug 19 11:33:53 dc1-bootstrap in.tftpd[19719]: RRQ from 10.16.16.253 filename boot/fdi-image/vmlinuz0
Aug 19 11:34:00 dc1-bootstrap in.tftpd[19719]: Client 10.16.16.253 finished boot/fdi-image/vmlinuz0
Aug 19 11:34:00 dc1-bootstrap in.tftpd[19739]: RRQ from 10.16.16.253 filename boot/fdi-image/initrd0.img
Aug 19 11:35:40 dc1-bootstrap in.tftpd[19739]: Client 10.16.16.253 finished boot/fdi-image/initrd0.img

However, during the upload of the fdi-image, I get on the remote console the error "timeout reading ‘boot/fdi/initrd0.img’.
But after a few seconds it goes further starting the image, and then after a few seconds I get this Kernel Panic.

More details:

We are using the latest version of the FDI, more specific https://downloads.theforeman.org/discovery/releases/latest/fdi-image-3.5.7.tar.
I followed the prcedure as described in GitHub - theforeman/foreman-discovery-image: Foreman discovery image live distro.

I was able to perform a PXE installation on another similar Fujitsu RX2530M5 without any issue, so the image should be fine.
However, on 2 other same systems it fails with this error.

What else can I try or which logs should/can I check to get more details for this issue?

Lukas,

And here in addition the details of the Foreman Discovery Image efi:

image

Can you download grub2-efi from Fedora Rawhide, replace the grubx64.efi on the TFTP server and try again?

Hi Lukas,

Thanks for your feedback.
Meanwhile I have collected a tcpdump, and it results in a TFTP error code 5.

But I will try to apply the Fedora Rawhide an dreplace the grubx64.efi on the TFTP-server

Note: before I upgraded Foreman and Katello, I have performed a system CentOS 7 upgrade. So I would have expected that all would work fine out of the box.

Lukas,

After the CentOS 7 upgrade We have following grub2-efi packages on our system:

grub2-efi-x64-2.02-0.86.el7.centos.x86_64
grub2-efi-x64-modules-2.02-0.86.el7.centos.noarch

Sure I have to try the Fedora Rawhide version?

Absolutely, I have been working closely with grub2 Red Hat team in the past months fixing some serious UEFI issues. Most of the patches are not even in Fedora 32.

OK, will download all dependency packages as well in order to install it.
Will keep you informed about the status.
Thnx

Just download this file and extract grubx64.efi from the RPM file. No need to install, I’d actually would NOT suggest that on production install.

Hi Lukas,

Downloaded the grub2-efi-x64-2.04-27.fc33.x86_64.rpm, but extracting it with CentOS 7.8.2003 results in lots of “cpio: Malformed number”.

rpm -q -l -p grub2-efi-x64-2.04-27.fc33.x86_64.rpm shows me all available files.

However, rpm2cpio grub2-efi-x64-2.04-27.fc33.x86_64.rpm | cpio -t or extracting the grubx64.efi in it with rpm2cpio grub2-efi-x64-2.04-27.fc33.x86_64.rpm | cpio -idm /boot/efi/EFI/fedora/grubx64.efi results in “Malformed number”.

Any sugestions?

Is it corrupt?

You can use Midnight Commander to extract the file. Anyway:

https://koji.fedoraproject.org/koji/buildinfo?buildID=1589892

[lzap@box foreman]$ cd /tmp
[lzap@box tmp]$ wget https://kojipkgs.fedoraproject.org//packages/grub2/2.04/28.fc34/x86_64/grub2-efi-x64-2.04-28.fc34.x86_64.rpm

[lzap@box tmp]$ rpm2cpio grub2-efi-x64-2.04-28.fc34.x86_64.rpm | cpio -idmv
./boot/efi/EFI/fedora/fonts
./boot/efi/EFI/fedora/grubx64.efi
./boot/grub2/grubenv
./boot/loader/entries
./etc/grub2-efi.cfg
4941 blocks

[lzap@box tmp]$ md5sum ./boot/efi/EFI/fedora/grubx64.efi
0a81dbc8182debf246673c70b8102d88  ./boot/efi/EFI/fedora/grubx64.efi

For the record, Red Hat QA have identified a regression in (likely) a grub2 update in RHEL7. We are currently investigating the problem. Suggestion is to either downgrade or upgrade to Fedora Rawhide grub, until we find out what patch caused this.

https://bugzilla.redhat.com/show_bug.cgi?id=1870035

Hi Lukas,

Thnx for the link, it is a more recent version then the one I used (28 vs 27).

In Centos 7 it fails with these errors, but a colleague who is using Fedora on his laptop was able to extract it.
The Rawhide file is now copied in /var/lib/tftpboot/grub2, and I’m going to test it right away.
Will keep you informed if that solved the problem.

Hi Lukas,

Now I get this in /var/log/messages:

Aug 21 10:12:12 dc1-bootstrap in.tftpd[12554]: RRQ from 10.16.16.250 filename /grub2/grub.cfg-4c:52:62:b0:e4:3d
Aug 21 10:12:12 dc1-bootstrap in.tftpd[12554]: Client 10.16.16.250 File not found /grub2/grub.cfg-4c:52:62:b0:e4:3d
Aug 21 10:12:12 dc1-bootstrap in.tftpd[12555]: RRQ from 10.16.16.250 filename grub2/grub.cfg-4c:52:62:b0:e4:3d
Aug 21 10:12:12 dc1-bootstrap in.tftpd[12555]: Client 10.16.16.250 File not found grub2/grub.cfg-4c:52:62:b0:e4:3d
Aug 21 10:12:12 dc1-bootstrap in.tftpd[12556]: RRQ from 10.16.16.250 filename grub.cfg-4c:52:62:b0:e4:3d
Aug 21 10:12:12 dc1-bootstrap in.tftpd[12556]: Client 10.16.16.250 File not found grub.cfg-4c:52:62:b0:e4:3d
Aug 21 10:12:27 dc1-bootstrap in.tftpd[12591]: RRQ from 10.16.16.250 filename boot/fdi-image/vmlinuz0
Aug 21 10:12:34 dc1-bootstrap in.tftpd[12591]: Client 10.16.16.250 finished boot/fdi-image/vmlinuz0
Aug 21 10:12:34 dc1-bootstrap in.tftpd[12613]: RRQ from 10.16.16.250 filename boot/fdi-image/initrd0.img
Aug 21 10:14:22 dc1-bootstrap in.tftpd[12613]: Client 10.16.16.250 finished boot/fdi-image/initrd0.img
Aug 21 10:14:22 dc1-bootstrap in.tftpd[12613]: Client 10.16.16.250 timed out

But on the iRMC video console of the host, the screen remains black!

Well Foreman version has unlikely to do anything with this, looks like a network error. When you create a host for the particular MAC address, will the host boot into Anaconda? That’s only 50 MB initramdisk while discovery is about 300 MB.

Update EFI firmware on that host, grub uses PXE stack for networking, if it times out it can be something in the fireware too.

If you think it’s a regression, try downgrading grubx64.efi and find a version which worked, send me version which does work and does not work so I can get to grub developers to investigate this.

Also try a host in BIOS mode on the same network to rule out it’s faulty firewall or too much traffic.

Hi Lukas,

Removing the grub2 packages v 0.86, which were upgraded during the OS upgrade performed before the Foreman/katello upgrade, and installing the older v0.81 solved the problem.

See also Cannot boot anymore Discovery image

1 Like

Hi Lukas,

Some extra feedback.

As said above, the Foreman Discovery Image efi works when using grubx64.efi from the grub2-efi v0.81.
However, once I have finished the host-config en submit it for final server installation, it ends up with a grub-prompt.

In /var/log/messages I see following for the configured host:

And then it ends with the grub-prompt.

But, once I use the Rawhide version of grbx64.efi in /var/lib/tftboot/grub2, it will go further with loading the grub-cgf file for the host and the PXE-installation goes well.

So, in order to let the Foreman Discovery image efi work as expected, I need to use the grubx64.efi in /var/lib/tftpboot/grub2, and for the final PXE installation I need to change it back to the Rawhide version of it.

Hope this info is helpful for the development team and others having the same issue.