Loading tftpboot with wget

Problem:

I have just noticed something odd: I was trying to kickstart a CentOS 8.4 host from the synced content. It didn’t work. The problem: the anaconda is running on a kernel 4.18.0-240 although it expects modules for 4.1.0-305.3.1.el8. 4.18.0-240 is the kernel for CentOS 8.3.

So I have checked the content of /var/lib/tftpboot/boot:

[root@foreman boot]# ls -la centos-8-base-2744-*
-rw-r--r--. 1 foreman-proxy foreman-proxy 75378504 Jun  6 13:31 centos-8-base-2744-initrd.img
-rw-r--r--. 1 foreman-proxy foreman-proxy 10026120 Jun  6 13:31 centos-8-base-2744-vmlinuz
[root@foreman boot]# file centos-8-base-2744-vmlinuz
centos-8-base-2744-vmlinuz: Linux kernel x86 boot executable bzImage, version 4.18.0-240.el8.x86_64 (mockbuild@kbuilder.bsys.centos.org) #1 S, RO-rootFS, swap_dev 0x9, Normal VGA

I can see in the logs that this is the vmlinuz the host is loading.

In /var/log/foreman-proxy/proxy.log I can see it gets the vmlinuz with wget:

2021-08-23T19:10:24 4e404324 [I] [55918] Started task ["/usr/bin/wget", "--connect-timeout=10", "--dns-timeout=10", "--read-timeout=60", "--tries=3", "--no-check-certificate", "-nv", "-c", "http://foreman.example.com/pulp/content/ORG/Production/centos8/custom/centos8/BaseOS_x86_64//images/pxeboot/vmlinuz", "-O", "/var/lib/tftpboot/boot/centos-8-base-2744-vmlinuz"]
2021-08-23T19:10:24 4e404324 [I] Started POST /tftp/fetch_boot_file 

I have verified that the URL actually loads the correct vmlinuz, e.g. if I save it in a new file:

[root@foreman boot]# /usr/bin/wget --connect-timeout=10 --dns-timeout=10 --read-timeout=60 --tries=3 --no-check-certificate -c http://foreman.example.com/pulp/content/ORG/Production/centos8/custom/centos8/BaseOS_x86_64//images/pxeboot/vmlinuz -O vmlinuz
...
HTTP request sent, awaiting response... 200 OK
Length: 10026120 (9.6M) [application/octet-stream]
Saving to: ‘vmlinuz’
...
2021-08-23 19:54:44 (359 MB/s) - ‘vmlinuz’ saved [10026120/10026120]

[root@foreman boot]# file vmlinuz 
vmlinuz: Linux kernel x86 boot executable bzImage, version 4.18.0-305.3.1.el8.x86_64 (mockbuild@kbuilder.bsys.centos.org) , RO-rootFS, swap_dev 0x9, Normal VGA

The problem is the wget -c option when the file already exists: it looks at the size of the current file and if that size is smaller than the remote file it downloads the remaining bytes. This works fine if the local and remote files are really the same and the local file is really a partial download of the remote file.

But if it’s a different file the resulting file is bogus: it’s the original file with something added from a different file added at the end.

I guess what lead to this issue is that I have set up my “CentOS 8” product with the URL for the latest version, i.e. http://mirror.centos.org/centos/8/BaseOS/x86_64/os/. When 8.3 was the latest version it loaded the centos-8-base-2744-vmlinuz file with the 8.3 kernel.

When 8.4 came with the new vmlinuz the wget does not replace the file centos-8-base-2744-vmlinuz but just adds some bytes at the end to reach the size of the new vmlinuz.

So the quick workaround would be to delete the files in tftpboot and wget will retrieve the complete latest version. Of course, this is not really a solution. I guess I am not the only one who syncs using only the major version repository URLs.

I think the provisioning of files /var/lib/tftpboot needs to be handle better or at least remove the -c option from that call to make sure it downloads the complete new file.

I guess of CentOS 8 Stream it is even worse, because you wouldn’t necessary notice if vmlinuz or initrd has changed…

Foreman and Proxy versions:
Foreman 2.5.2, Katello 4.1.2.1

Distribution and version:
CentOS 7.9

Just opened redmine issue: Bug #33328: Files in tftpboot/boot don't pick up changed files - Foreman

1 Like

This is neverending story and we need to solve this and we need to do this now.

There were several patches trying to solve this, I believe we need to rearchitect how kernel/initeramdisk are downloaded. They should not be downloaded when new host is created (there are also concurrent issues like when download is slower than server is booting), but it will be lot of work.

Latest patch that aims at least to remove the -c argument and provide some more relevant ones is here: Fixes #2412 - don't continue wget downloads by lzap · Pull Request #785 · theforeman/smart-proxy · GitHub</

If you could test it with CentOS Stream (which is indeed a lot worse) I would appreciate if you could get back to us and confirm it works.

1 Like

I am still on 2.5, thus your patch doesn’t directly apply. But just staring at the code change it seems to do what it’s supposed to do: remove the ‘-c’ option.

Yes it can be simplified even to that. If there are no objections I think this is super important one to fix also in older releases and I could prepare cherry pick for 2.5 and possibly 2.4 (if we plan any). Really hard to troubleshoot this one (Anaconda randomly fails, various weird errors). @tbrisker @upadhyeammit

It is literally one-liner, make sure you replace -c with -timestamping --no-if-modified-since so existing unmodified files aren’t overwritten over and over again.

The patch has been merged today and its on its way into 3.0, thanks all!

1 Like

As mentioned in github, --no-if-modified-since does not work with EL7 wget and -timestamping is meaningless with -O.

That would break it work EL7…

Hello Lukas,

Thanks for proactively tagging us.

We are close to 3.0 GA, its planned on 7th of September. As per this if I see

We have found a problem in the patch, it made it into nightly unfortunately. One of the arguments is not available in EL7 version of wget. I am working on a solution.

1 Like