Proxy 404 when provisioning new build results in 0 byte vmlinuz and initrd

hillonious · August 5, 2021, 4:06pm

Problem:
When trying to provision a new host 0 byte vmlinuz and initrd files are created within tftpboot

Expected outcome:
Host can pxeboot as expected and tftp files are not 0 byte files

FYI: Our existing foreman/katello 2.2/3.17 hosts are working as expected

Foreman and Proxy versions:
Foreman 2.5

Foreman and Proxy plugin versions:
Katello 4.1

Distribution and version:
CentOS 7 latest packages applied
vCPUs 8
20GB RAM

Other relevant data:
Proxy log during build (replaced real mac with all X’s here):

2021-08-05T15:49:40 040ee20b [I] Started GET /v2/features
2021-08-05T15:49:40 040ee20b [I] Finished GET /v2/features with 200 (228.32 ms)
2021-08-05T15:49:43 c68ca273 [I] Started GET /tftp/serverName
2021-08-05T15:49:43 c68ca273 [I] Finished GET /tftp/serverName with 200 (0.8 ms)
2021-08-05T15:49:43 c68ca273 [I] Started GET /dhcp/10.242.56.0/mac/XX:XX:XX:XX:XX:XX
2021-08-05T15:49:43 c68ca273 [I] Finished GET /dhcp/10.242.56.0/mac/XX:XX:XX:XX:XX:XX with 200 (0.67 ms)
2021-08-05T15:49:43 c68ca273 [I] Started GET /dhcp/10.242.56.0/ip/10.242.56.5
2021-08-05T15:49:43 c68ca273 [I] Finished GET /dhcp/10.242.56.0/ip/10.242.56.5 with 200 (0.75 ms)
2021-08-05T15:49:43 c68ca273 [I] Started GET /dhcp/10.242.56.0/mac/XX:XX:XX:XX:XX:XX
2021-08-05T15:49:43 c68ca273 [I] Finished GET /dhcp/10.242.56.0/mac/XX:XX:XX:XX:XX:XX with 200 (0.76 ms)
2021-08-05T15:49:43 c68ca273 [I] Started POST /tftp/PXELinux/XX:XX:XX:XX:XX:XX
2021-08-05T15:49:43 c68ca273 [I] Finished POST /tftp/PXELinux/XX:XX:XX:XX:XX:XX with 200 (0.85 ms)
2021-08-05T15:49:43 c68ca273 [I] Started POST /tftp/PXEGrub2/XX:XX:XX:XX:XX:XX
2021-08-05T15:49:43 c68ca273 [I] Finished POST /tftp/PXEGrub2/XX:XX:XX:XX:XX:XX with 200 (1.0 ms)
2021-08-05T15:49:43 c68ca273 [I] Started POST /tftp/PXEGrub/XX:XX:XX:XX:XX:XX
2021-08-05T15:49:43 c68ca273 [I] Finished POST /tftp/PXEGrub/XX:XX:XX:XX:XX:XX with 200 (1.02 ms)
2021-08-05T15:49:43 c68ca273 [I] Started POST /tftp/iPXE/XX:XX:XX:XX:XX:XX
2021-08-05T15:49:43 c68ca273 [I] Finished POST /tftp/iPXE/XX:XX:XX:XX:XX:XX with 200 (0.95 ms)
2021-08-05T15:49:43 c68ca273 [I] Started POST /tftp/fetch_boot_file
2021-08-05T15:49:43 c68ca273 [I] Finished POST /tftp/fetch_boot_file with 200 (0.88 ms)
2021-08-05T15:49:43 c68ca273 [I] [29701] Started task ["/usr/bin/wget", "--connect-timeout=10", "--dns-timeout=10", "--read-timeout=60", "--tries=3", "--no-check-certificate", "-nv", "-c", "http://pv-sat001.ourcompany.com/pulp/content/Sky/Library/custom/CentOS_7_x86_64/CentOS_7_base_x86_64//images/pxeboot/vmlinuz", "-O", "/var/lib/tftpboot/boot/centos_7_base_x86_64-1-vmlinuz"]
2021-08-05T15:49:43 c68ca273 [I] Started POST /tftp/fetch_boot_file
2021-08-05T15:49:43 c68ca273 [I] Finished POST /tftp/fetch_boot_file with 200 (0.81 ms)
2021-08-05T15:49:43 c68ca273 [I] [29704] Started task ["/usr/bin/wget", "--connect-timeout=10", "--dns-timeout=10", "--read-timeout=60", "--tries=3", "--no-check-certificate", "-nv", "-c", "http://pv-sat001.ourcompany.com/pulp/content/Sky/Library/custom/CentOS_7_x86_64/CentOS_7_base_x86_64//images/pxeboot/initrd.img", "-O", "/var/lib/tftpboot/boot/centos_7_base_x86_64-1-initrd.img"]
2021-08-05T15:49:44 c68ca273 [W] [29701] http://pv-sat001.ourcompany.com/pulp/content/OurCompany/Library/custom/CentOS_7_x86_64/CentOS_7_base_x86_64//images/pxeboot/vmlinuz:

2021-08-05T15:49:44 c68ca273 [W] [29701] 2021-08-05 15:49:44 ERROR 404: Not Found.

2021-08-05T15:49:44 c68ca273 [W] [29704] http://pv-sat001.ourcompany.com/pulp/content/OurCompany/Library/custom/CentOS_7_x86_64/CentOS_7_base_x86_64//images/pxeboot/initrd.img:

2021-08-05T15:49:44 c68ca273 [W] [29704] 2021-08-05 15:49:44 ERROR 404: Not Found.

2021-08-05T15:51:19  [E] <OpenSSL::SSL::SSLError> SSL_accept SYSCALL returned=5 errno=0 state=SSLv2/v3 read client hello A
	/opt/rh/rh-ruby27/root/usr/share/ruby/webrick/server.rb:299:in `accept'
	/opt/rh/rh-ruby27/root/usr/share/ruby/webrick/server.rb:299:in `block (2 levels) in start_thread'
	/opt/rh/rh-ruby27/root/usr/share/ruby/webrick/utils.rb:263:in `timeout'
	/opt/rh/rh-ruby27/root/usr/share/ruby/webrick/server.rb:297:in `block in start_thread'
	/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2021-08-05T15:56:19  [E] <OpenSSL::SSL::SSLError> SSL_accept SYSCALL returned=5 errno=0 state=SSLv2/v3 read client hello A
	/opt/rh/rh-ruby27/root/usr/share/ruby/webrick/server.rb:299:in `accept'
	/opt/rh/rh-ruby27/root/usr/share/ruby/webrick/server.rb:299:in `block (2 levels) in start_thread'
	/opt/rh/rh-ruby27/root/usr/share/ruby/webrick/utils.rb:263:in `timeout'
	/opt/rh/rh-ruby27/root/usr/share/ruby/webrick/server.rb:297:in `block in start_thread'
	/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'

Attempted fixes (none working):

Remove tftpboot files, cancel build and hit build again => zero byte files are recreated
Restart smart proxy systemd service, cancel build and hit build again => zero byte files are created
Use foreman-maintain to restart all services, cancel build and hit build again => zero byte files are created

hillonious · August 5, 2021, 4:14pm

Smart proxy Features

Ansible
DHCP
Dynflow
Logs
Pulpcore
Registration
SSH
TFTP

gvde · August 5, 2021, 4:18pm

Are those two URLs shown for the wget commands working?

Try

$ curl -v -o /dev/null http://pv-sat001.ourcompany.com/pulp/content/Sky/Library/custom/CentOS_7_x86_64/CentOS_7_base_x86_64//images/pxeboot/vmlinuz

You can also try those two wget commands from the logs in a shell. Change the -O output to someplace else for tests. Check if you can download it.

hillonious · August 5, 2021, 4:28pm

Hi @gvde, thanks for your input again

Looks like this is probably a problem too:

$ curl --cert /etc/foreman/client_cert.pem --key /etc/foreman/client_key.pem 'https://pv-sat001.ourcompany.com:9090/v2/features'
curl: (58) unable to load client key: -8178 (SEC_ERROR_BAD_KEY)

Attempt to grab vmlinuz:

$ curl -v http://pv-sat001.ourcompany.com/pulp/content/Sky/Library/custom/CentOS_7_x86_64/CentOS_7_base_x86_64//images/pxeboot/vmlinuz
* About to connect() to pv-sat001.ourcompany.bskyb.com port 80 (#0)
*   Trying 10.242.56.14...
* Connected to pv-sat001.ourcompany.com (10.242.56.14) port 80 (#0)
> GET /pulp/content/Sky/Library/custom/CentOS_7_x86_64/CentOS_7_base_x86_64//images/pxeboot/vmlinuz HTTP/1.1
> User-Agent: curl/7.29.0
> Host: pv-sat001.ourcompany.com
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Date: Thu, 05 Aug 2021 16:27:04 GMT
< Server: Python/3.6 aiohttp/3.7.4
< Content-Type: text/plain; charset=utf-8
< Content-Length: 14
< Via: 1.1 pv-sat001.ourcompany.com
<
* Connection #0 to host pv-sat001.ourcompany.com left intact
404: Not Found

gvde · August 5, 2021, 4:39pm

hillonious:

Looks like this is probably a problem too:

$ curl --cert /etc/foreman/client_cert.pem --key /etc/foreman/client_key.pem 'https://pv-sat001.ourcompany.com:9090/v2/features'
curl: (58) unable to load client key: -8178 (SEC_ERROR_BAD_KEY)

/etc/foreman/client_key.pem should only be readable by user foreman. Thus, you need to run this as root…

Is pv-sat001 your main katello server or a content proxy?

You can check at http://pv-sat001.example.com/pulp/content/ to see which repositories are actually available. It should list Sky/Library/custom/CentOS_7_x86_64/CentOS_7_base_x86_64.

Sidenote: if you want to obfuscate your domain name, don’t just use a random other name from somebody else. That’s what example.com, example.org, example.net are specifically reserved for in RFC 2606.

hillonious · August 5, 2021, 4:46pm

Sorry RE obfuscation, will do that in future

Thanks, key and cert curl working as expected with root; I should have spotted that

Yes pv-sat001 is/will be the main site katello server

404 when trying to hit
http://pv-sat001.example.com/pulp/content
http://pv-sat001.example.com/pulp/

Perhaps not very helpful but Foreman UI | Content | Products | CentOS_7_x86_64 shows packages for base, extras, updates etc

foreman-maintain health check isn’t reporting any issues

hillonious · August 5, 2021, 5:00pm

I get 404 for http://pv-sat001.example.com/pulp/content/ on a working foreman/katello 2.2/3.17 server if that helps

hillonious · August 5, 2021, 5:03pm

Apologies, trailing slash matters (which is what you suggested) and yes I get a listing with Sky/Library/custom/CentOS_7_x86_64/CentOS_7_base_x86_64 within:

Sky/Library/custom/Bacula_EL7_x86_64/Bacula_Enterprise_EL7_x86_64/
Sky/Library/custom/Bacula_EL7_x86_64/Bacula__mysql_x86_64/
Sky/Library/custom/Bacula_EL7_x86_64/Bacula_bweb_EL7_x86_64/
Sky/Library/custom/Bacula_EL7_x86_64/Bacula_dedup_EL7_x86_64/
Sky/Library/custom/Bacula_EL7_x86_64/Bacula_singleitemrestore_EL7_x86_64/
Sky/Library/custom/Bacula_EL7_x86_64/Bacula_snapshot_EL7_x86_64/
Sky/Library/custom/Bacula_EL7_x86_64/Bacula_vsphere_EL7_x86_64/
Sky/Library/custom/CentOS_7_x86_64/CentOS_7_base_x86_64/
Sky/Library/custom/CentOS_7_x86_64/CentOS_7_extras_x86_64/
Sky/Library/custom/CentOS_7_x86_64/CentOS_7_scl_rh_x86_64/
Sky/Library/custom/CentOS_7_x86_64/CentOS_7_scl_x86_64/
Sky/Library/custom/CentOS_7_x86_64/CentOS_7_updates_x86_64/
Sky/Library/custom/CentOS_7_x86_64/Docker-ce_EL7_x86_64/
Sky/Library/custom/CentOS_7_x86_64/EPEL_EL7_x86_64/
Sky/Library/custom/CentOS_7_x86_64/HP-SPP_EL7_x86_64/
Sky/Library/custom/CentOS_7_x86_64/HPE-STK_EL7_x86_64/
Sky/Library/custom/CentOS_7_x86_64/HPE-ilorest_EL7_x86_64/
Sky/Library/custom/CentOS_7_x86_64/Jenkins_EL7_x86_64/
Sky/Library/custom/CentOS_7_x86_64/Kubernetes_EL7_x86_64/
Sky/Library/custom/CentOS_7_x86_64/Mongodb_EL7_x86_64/
Sky/Library/custom/CentOS_7_x86_64/Mono_EL7_x86_64/
Sky/Library/custom/CentOS_7_x86_64/Nagios_EL7_x86_64/
Sky/Library/custom/CentOS_7_x86_64/influxdb_EL7_x86_64/
Sky/Library/custom/Fossid_EL7_x86_64/Anku-7_x86_64/
Sky/Library/custom/Gitlab_EL7_x86_64/Gitlab_CE_EL7_x86_64/
Sky/Library/custom/Gitlab_EL7_x86_64/Gitlab_EE_EL7_x86_64/
Sky/Library/custom/Gitlab_EL7_x86_64/Gitlab_runner_EL7_x86_64/
Sky/Library/custom/GrafanaEnterprise_EL7_x86_64/GrafanaEnterprise_EL7_x86_64/
Sky/Library/custom/HarbottleTomcat_EL7_x86_64/Harbottle-tomcat_EL7_x86_64/
Sky/Library/custom/MySQL_EL7_x86_64/MySQL_EL7_x86_64/
Sky/Library/custom/Nvidia_EL7_x86_64/Nvidia_container_runtime_EL7_x86_64/
Sky/Library/custom/Nvidia_EL7_x86_64/Nvidia_cudacompute_EL7_x86_64/
Sky/Library/custom/Nvidia_EL7_x86_64/Nvidia_libcontainer_EL7_x86_64/
Sky/Library/custom/PowerDNS_EL7_x86_64/Power_DNS_AuthoritativeServer_x86_64/
Sky/Library/custom/PowerDNS_EL7_x86_64/Power_DNS_Recursor_x86_64/
Sky/Library/custom/Zoneminder_EL7_x86_64/Zoneminder_EL7_x86_64/```

hillonious · August 5, 2021, 5:09pm

I’m just walking the URL now, assuming images is missing

hillonious · August 5, 2021, 5:13pm

http://pv-sat001.example.com/pulp/content/Sky/Library/custom/CentOS_7_x86_64/CentOS_7_base_x86_64/ is missing images as expected only the following present:

.treeinfo
Packages/
config.repo
repodata/

Is there something I can kick off to recreate those?

I can try a rebuild, however my other development instance xv-sat001 has the same structure after multiple rebuilds

hillonious · August 5, 2021, 5:27pm

@gvde images now created after triggering an Advanced Sync | Complete Sync on the base repo

Odd that the initial sync didn’t do that, I’m confident a provision will now work but will test and report back, I’ll trawl the foreman sync ansible module and see if I can do the same when building the instance (not seen this issue before)

gvde · August 5, 2021, 5:34pm

I guess it depends when and with what katello version you did the original sync.

hillonious · August 5, 2021, 5:38pm

Fresh install foreman/katello 2.5/4.1

pxeboot now working as expected

Thanks again for your support, very much appreciated

hillonious · August 5, 2021, 6:19pm

Just to clarify this was seen on a fresh install for foreman/katello 2.5/4.1

A sync of the base repo had been triggered and completed, on our other xv-sat001 development instance (same exact configuration) with nightly sync plans (2 days now, so 2 sync cycles completed), it still hasn’t synced the images path

It does feel strange, but I’m happy an advanced sync appears to be our solution. Thanks