Problem with CentOS8 UEFI provisioning

Blas · February 9, 2021, 11:45am

Problem:
I have a problem provisioning CentOS8 via UEFI. Provisioning of legacy CentOS8 or CentOS7 UEFI is working fine.

Screenshots:

Partition table:

zerombr
clearpart --all --initlabel

part /boot/efi --fstype="efi" --ondisk=sda --size=200 --fsoptions="umask=0077,shortname=efi"
part /boot --fstype="xfs" --ondisk=sda --size=1024

part swap --size=8000 --ondisk=sda
part / --fstype=xfs --ondisk=sda --size=1024 --grow

Expected outcome:
Foreman successful provision of CentOS8

Foreman and Proxy versions:
foreman.noarch 2.1.4-1.el8

Foreman and Proxy plugin versions:
rubygem-foreman_discovery.noarch 16.1.2-1.fm2_1.el8

Distribution and version:

Other relevant data:

lzap · February 9, 2021, 12:49pm

Hello, this is now the 2nd time someone reported it. Which CentOS 8 version are you installing? Is this Stream? Or CentOS 8 legacy?

lzap · February 9, 2021, 12:55pm

For the record, the current CentOS 8 Stream PXE init ram disk looks okay:

$ shasum initrd.img
e146124966bd73f422814e8fccc7708b74a5e06d  initrd.img

$ lsinitrd initrd.img
(no errors)

Can you share your shasum? Can you try to delete the file and try again? There is a known bug in TFTP smart proxy - if a file gets replaced on the upstream site it simply appends the rest instead redownloading. This corrupts it. We need to fix this and now with CentOS 8 Stream which rebases the PXE images on a regular basis this should be our priority now.

lzap · February 9, 2021, 2:05pm

This patch should resolve the problem with continue wget argument:

github.com/theforeman/smart-proxy

Fixes #2412 - don't continue wget downloads

theforeman:develop ← lzap:no-continue-wget-2412

opened 02:05PM - 09 Feb 21 UTC

lzap

+35 -10

With CentOS 8 Stream this is a more painful issue, I was planning rewriting this… code but there is no time - let's change -c to --timestamping --no-if-modified-since. This will keep the current behavior - if a file is present and the same size and its timestamp matches what server reports, do not redownload it. If it's different, redownload it from the beginning - no longer attempt to continue and append. The second option `--no-if-modified-since` makes `wget` to request the timestamp via HEAD HTTP call instead of If-Modified-Since because not all mirrors support this option: ``` $ LC_ALL=C wget --timestamping --no-if-modified-since http://mirror.centos.org/centos/8-stream/BaseOS/x86_64/os/images/pxeboot/vmlinuz --2021-02-09 15:02:49-- http://mirror.centos.org/centos/8-stream/BaseOS/x86_64/os/images/pxeboot/vmlinuz Resolving mirror.centos.org (mirror.centos.org)... 95.216.19.148, 2a00:1dc0:3061::10 Connecting to mirror.centos.org (mirror.centos.org)|95.216.19.148|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 9936008 (9.5M) Server file no newer than local file 'vmlinuz' -- not retrieving. $ LC_ALL=C wget --timestamping http://mirror.centos.org/centos/8-stream/BaseOS/x86_64/os/images/pxeboot/vmlinuz --2021-02-09 15:02:55-- http://mirror.centos.org/centos/8-stream/BaseOS/x86_64/os/images/pxeboot/vmlinuz Resolving mirror.centos.org (mirror.centos.org)... 95.216.19.148, 2a00:1dc0:3061::10 Connecting to mirror.centos.org (mirror.centos.org)|95.216.19.148|:80... connected. HTTP request sent, awaiting response... 304 Not Modified File 'vmlinuz' not modified on server. Omitting download. $ LC_ALL=C wget --timestamping http://ftp.sh.cvut.cz/centos/8-stream/BaseOS/x86_64/os/images/pxeboot/initrd.img --2021-02-09 15:03:20-- http://ftp.sh.cvut.cz/centos/8-stream/BaseOS/x86_64/os/images/pxeboot/initrd.img Resolving ftp.sh.cvut.cz (ftp.sh.cvut.cz)... 147.32.127.196, 2001:718:2::196 Connecting to ftp.sh.cvut.cz (ftp.sh.cvut.cz)|147.32.127.196|:80... connected. HTTP request sent, awaiting response... 200 OK Server ignored If-Modified-Since header for file 'initrd.img'. You might want to add --no-if-modified-since option. ```

viwon · February 9, 2021, 3:56pm

I think there is a bug currently in RHEL8 that is causing this I believe.
https://bugzilla.redhat.com/show_bug.cgi?id=1916168

Not sure the real issue, because its being tracked in an internal only bug, maybe Lukas has access.

My kickstarts do not work with 8.3+ media, and the same issue occurs in Centos/Cento 8 Streams.

Try using the 8.2 boot media to get your initial kickstart boot kernel/initrd to use.

Blas · February 10, 2021, 7:57am

Hello. Sorry for not being clear, but I’m having problems installing CentOS8 legacy on UEFI system. The installation via legacy BIOS is working fine.

Which CentOS 8 version are you installing? Is this Stream? Or CentOS 8 legacy?

The CentOS version is legacy 8.3.2011

$ sha1sum centos-8-FrlYBgzfk21z-initrd.img
21d62130a85b3b4c87a38a3d94c4680b2b548be2

$ lsinitrd initrd.img
(no errors)

@viwon If I use 8.2, the installation goes through smootly.

Thank you both for your help.

lzap · February 10, 2021, 10:58am

It appears that the RHEL team tracks nightly build bugs as private, I don’t know why perhaps to prevent them from being indexed by google because it was reported on unfinished release. I guess things must be lined up because of CentOS Stream, I asked them about this.

Anyway, the referenced BZ is I think a different problem. I don’t see any kernel modules failed to be loaded there, that looks like some NetworkManager issue.

If you are on SecureBoot, mind the Boothole CVE: Boot Hole Vulnerability - GRUB 2 boot loader - CVE-2020-10713 - Red Hat Customer Portal - old kernels will no longer load.

I am gonna try this now in a libvirt EFI environment.

lzap · February 10, 2021, 12:52pm

So I tested this on Foreman 2.3 running on CentOS 7.x (latest stable). I made sure that grub2 was updated to the latest EL7 version, I set up a libvirt VM EFI on an isolated network. And - downloading of initramdisk failed in all times with “Timeout error” leading to kernel panic.

I noticed something, the initramdisk has grown substationally over the last couple of years:

7.0: 33 MB
7.5: 50 MB
7.8: 53 MB
8.0: 58 MB
8.1: 59 MB
8.2: 62 MB
8.3: 70 MB

I wonder if this is simply a bug in grub2 EFI stack or in libvirt PXE stack, we have seen couple of them in the past. This can lead to timeout errors or corrupted init ramdisk which can then lead to very random errors (like the one you experience).

Now, I copied grubx64.efi from my Fedora 33 and then my VM immediately booted and provisioned just fine. Can you try that?