I have just noticed something odd: I was trying to kickstart a CentOS 8.4 host from the synced content. It didn’t work. The problem: the anaconda is running on a kernel 4.18.0-240 although it expects modules for 4.1.0-305.3.1.el8. 4.18.0-240 is the kernel for CentOS 8.3.
So I have checked the content of /var/lib/tftpboot/boot:
[root@foreman boot]# ls -la centos-8-base-2744-* -rw-r--r--. 1 foreman-proxy foreman-proxy 75378504 Jun 6 13:31 centos-8-base-2744-initrd.img -rw-r--r--. 1 foreman-proxy foreman-proxy 10026120 Jun 6 13:31 centos-8-base-2744-vmlinuz [root@foreman boot]# file centos-8-base-2744-vmlinuz centos-8-base-2744-vmlinuz: Linux kernel x86 boot executable bzImage, version 4.18.0-240.el8.x86_64 (email@example.com) #1 S, RO-rootFS, swap_dev 0x9, Normal VGA
I can see in the logs that this is the vmlinuz the host is loading.
In /var/log/foreman-proxy/proxy.log I can see it gets the vmlinuz with wget:
2021-08-23T19:10:24 4e404324 [I]  Started task ["/usr/bin/wget", "--connect-timeout=10", "--dns-timeout=10", "--read-timeout=60", "--tries=3", "--no-check-certificate", "-nv", "-c", "http://foreman.example.com/pulp/content/ORG/Production/centos8/custom/centos8/BaseOS_x86_64//images/pxeboot/vmlinuz", "-O", "/var/lib/tftpboot/boot/centos-8-base-2744-vmlinuz"] 2021-08-23T19:10:24 4e404324 [I] Started POST /tftp/fetch_boot_file
I have verified that the URL actually loads the correct vmlinuz, e.g. if I save it in a new file:
[root@foreman boot]# /usr/bin/wget --connect-timeout=10 --dns-timeout=10 --read-timeout=60 --tries=3 --no-check-certificate -c http://foreman.example.com/pulp/content/ORG/Production/centos8/custom/centos8/BaseOS_x86_64//images/pxeboot/vmlinuz -O vmlinuz ... HTTP request sent, awaiting response... 200 OK Length: 10026120 (9.6M) [application/octet-stream] Saving to: ‘vmlinuz’ ... 2021-08-23 19:54:44 (359 MB/s) - ‘vmlinuz’ saved [10026120/10026120] [root@foreman boot]# file vmlinuz vmlinuz: Linux kernel x86 boot executable bzImage, version 4.18.0-305.3.1.el8.x86_64 (firstname.lastname@example.org) , RO-rootFS, swap_dev 0x9, Normal VGA
The problem is the
wget -c option when the file already exists: it looks at the size of the current file and if that size is smaller than the remote file it downloads the remaining bytes. This works fine if the local and remote files are really the same and the local file is really a partial download of the remote file.
But if it’s a different file the resulting file is bogus: it’s the original file with something added from a different file added at the end.
I guess what lead to this issue is that I have set up my “CentOS 8” product with the URL for the latest version, i.e.
http://mirror.centos.org/centos/8/BaseOS/x86_64/os/. When 8.3 was the latest version it loaded the centos-8-base-2744-vmlinuz file with the 8.3 kernel.
When 8.4 came with the new vmlinuz the wget does not replace the file centos-8-base-2744-vmlinuz but just adds some bytes at the end to reach the size of the new vmlinuz.
So the quick workaround would be to delete the files in tftpboot and wget will retrieve the complete latest version. Of course, this is not really a solution. I guess I am not the only one who syncs using only the major version repository URLs.
I think the provisioning of files /var/lib/tftpboot needs to be handle better or at least remove the
-c option from that call to make sure it downloads the complete new file.
I guess of CentOS 8 Stream it is even worse, because you wouldn’t necessary notice if vmlinuz or initrd has changed…
Foreman and Proxy versions:
Foreman 2.5.2, Katello 126.96.36.199
Distribution and version: