Failed to fetch kickstart from https:.....:9090/unattended/provision?token=*****

sandeepkumar86k · July 3, 2019, 5:08pm

Problem:

I think I have ruined some configuration, my unattended provisioning was working fine earlier. After I tried installing third party SSL it stopped working.
I can’t find the exact reason or pinpoint the exact, requesting your expertise to help solve this issue.
I am attaching the production logs, please do help me.

Expected outcome:
Successful provisioning of the server

Foreman and Proxy versions:

Foreman and Proxy plugin versions:

Other relevant data:

logs

/usr/share/passenger/phusion_passenger/request_handler.rb:455:in block (3 levels) in start_threads' /opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:inblock in create_with_logging_context’
2019-07-04T00:44:56 [W|tem|9b6977e8] unable to obtain template url set by proxy https://repo.mbi.nus.edu.sg:9090/unattended. falling back on proxy url.
2019-07-04T00:44:56 [I|blo|9b6977e8] Unattended render of ‘Kickstart default PXEGrub’ = ‘b778dc97cf7aa326eee1a1e1ff33f09ffe370ee730971643786aa7320763e1e8’

This file was deployed via ‘Kickstart default PXEGrub’ template

default=0
timeout=10

title Kickstart default PXEGrub
root (nd)
kernel (nd)/…/boot/centos7_dvd_ftp-6s10R3DkW0Bm-vmlinuz ks=https://repo.mbi.nus.edu.sg:9090/unattended/provision?token=ef141144-ce61-4a7e-84bc-53aa33cc2fd3 network ksdevice=bootif ks.device=bootif BOOTIF=00-56-6f-d2-17-00-02 kssendmac ks.sendmac inst.ks.sendmac
initrd (nd)/…/boot/centos7_dvd_ftp-6s10R3DkW0Bm-initrd.img

2019-07-04T00:44:56 [I|app|9b6977e8] Deploying TFTP PXEGrub configuration for ipa.mbi.nus.edu.sg
2019-07-04T00:44:56 [W|app|9b6977e8] Failed to obtain template server from smart-proxy https://repo.mbi.nus.edu.sg:9090/unattended
RestClient::NotFound: 404 Not Found
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/abstract_response.rb:223:in exception_with_response' /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/abstract_response.rb:103:inreturn!’
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:809:in process_result' /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:725:inblock in transmit’
/opt/rh/rh-ruby25/root/usr/share/ruby/net/http.rb:910:in `start’

opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/executor.rb:14:in call' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:incall’
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/sendfile.rb:111:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/secure_headers-6.0.0/lib/secure_headers/middleware.rb:13:incall’
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/engine.rb:524:in call' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:inpublic_send’
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in method_missing' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/urlmap.rb:68:inblock in call’
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/urlmap.rb:53:in each' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/urlmap.rb:53:incall’
/usr/share/passenger/phusion_passenger/rack/thread_handler_extension.rb:74:in process_request' /usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:141:inaccept_and_process_next_request’
/usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:109:in main_loop' /usr/share/passenger/phusion_passenger/request_handler.rb:455:inblock (3 levels) in start_threads’
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context’
2019-07-04T00:44:56 [W|tem|9b6977e8] unable to obtain template url set by proxy https://repo.mbi.nus.edu.sg:9090/unattended. falling back on proxy url.
2019-07-04T00:44:56 [I|blo|9b6977e8] Unattended render of ‘Kickstart default iPXE’ = ‘9f7327e3a5ed002dfabc38fc8d8b1fd6834d6de0dd91d19c5bb79717150a361c’
#!gpxe

kernel ftp://repo.mbi.nus.edu.sg/pub/CentOS_7_x86_64//images/pxeboot/vmlinuz initrd=initrd.img ks=https://repo.mbi.nus.edu.sg:9090/unattended/provision?token=ef141144-ce61-4a7e-84bc-53aa33cc2fd3 inst.stage2=ftp://repo.mbi.nus.edu.sg/pub/CentOS_7_x86_64/ ksdevice=56:6f:d2:17:00:02 network kssendmac ks.sendmac inst.ks.sendmac ip=${netX/ip} netmask=${netX/netmask} gateway=${netX/gateway} dns=${dns}
initrd ftp://repo.mbi.nus.edu.sg/pub/CentOS_7_x86_64//images/pxeboot/initrd.img

boot

2019-07-04T00:44:56 [I|app|9b6977e8] Deploying TFTP iPXE configuration for ipa.mbi.nus.edu.sg
2019-07-04T00:44:56 [I|app|9b6977e8] Fetching required TFTP boot files for ipa.mbi.nus.edu.sg
2019-07-04T00:44:57 [I|app|9b6977e8] Processed 5 tasks from queue ‘Host::Managed Main’, completed 5/5
2019-07-04T00:44:57 [I|aud|9b6977e8] Nic::Managed (45) create event on mac 56:6f:d2:17:00:02
2019-07-04T00:44:57 [I|aud|9b6977e8] Nic::Managed (45) create event on ip

wings · July 4, 2019, 1:36am

Dumb first question, but I have to ask it -

Are you using a firewall on your Foreman host? You may need to open up TCP 9090 if you are.

sandeepkumar86k · July 4, 2019, 2:14am

I have enabled all the internal traffic in firewall. I even tested by disabling firewalld, still same issue.

Attaching the screenshot of error on the VM.

wings · July 4, 2019, 4:03am

Okay, the next thing to check - can you ping that hostname (repo.mbi.nus.edu.sg) from the VM, or from a VM on the same network as your test VM?

sandeepkumar86k · July 4, 2019, 4:19am

Yes, it’s reachable.
I suspect it’s related to SSL:
sharing proxy logs:

2019-07-04T10:12:21 7c71cdfc [I] Finished POST /fetch_boot_file with 200 (4.34 ms)
2019-07-04T10:12:21 7c71cdfc [I] Started POST /fetch_boot_file
2019-07-04T10:12:21 7c71cdfc [E] [21671] 2019-07-04 10:12:21 URL: ftp://repo.mbi.nus.edu.sg/pub/CentOS_7_x86_64//images/pxeboot/vmlinuz [6639904] -> “/var/lib/tftpboot/boot/centos7_dvd_ftp-6s10R3DkW0Bm-vmlinuz” [1]

2019-07-04T10:12:21 7c71cdfc [I] Finished POST /fetch_boot_file with 200 (1.09 ms)
2019-07-04T10:12:21 7c71cdfc [E] [21679] 2019-07-04 10:12:21 URL: ftp://repo.mbi.nus.edu.sg/pub/CentOS_7_x86_64//images/pxeboot/initrd.img [52584760] -> “/var/lib/tftpboot/boot/centos7_dvd_ftp-6s10R3DkW0Bm-initrd.img” [1]

2019-07-04T10:12:21 7c71cdfc [I] Started DELETE /autosign/ipa.mbi.nus.edu.sg
2019-07-04T10:12:21 7c71cdfc [E] Attempt to remove nonexistent client autosign for ipa.mbi.nus.edu.sg
2019-07-04T10:12:21 7c71cdfc [I] Finished DELETE /autosign/ipa.mbi.nus.edu.sg with 404 (0.79 ms)
2019-07-04T10:12:21 7c71cdfc [I] Started DELETE /ipa.mbi.nus.edu.sg
2019-07-04T10:12:23 7c71cdfc [E] Attempt to remove nonexistent client certificate for ipa.mbi.nus.edu.sg
2019-07-04T10:12:23 7c71cdfc [I] Finished DELETE /ipa.mbi.nus.edu.sg with 404 (1635.97 ms)
2019-07-04T10:12:23 7c71cdfc [I] Started POST /autosign/ipa.mbi.nus.edu.sg
2019-07-04T10:12:23 7c71cdfc [I] Finished POST /autosign/ipa.mbi.nus.edu.sg with 200 (0.56 ms)
2019-07-04T10:12:46 [E] OpenSSL::SSL::SSLError: SSL_accept returned=1 errno=0 state=SSLv3 read client certificate A: tlsv1 alert unknown ca
/usr/share/ruby/openssl/ssl.rb:280:in `accept’
2019-07-04T12:05:30 73dfd6d2 [I] Started GET /
2019-07-04T12:05:30 73dfd6d2 [I] Finished GET / with 404 (0.26 ms)
~

sandeepkumar86k · July 4, 2019, 4:24am

Btw is there any way I can reset SSL/CA to default/self-signed certs to verify if it works?

Thanks.

areyus · July 4, 2019, 7:23am

Hi,

if I remember correctly, you should be able to revert to self-signed certificates by using foreman-installer --certs-regenerate --certs-regenerate-ca.
This should generate new certificates and configure all componentents on your Foreman server to use those. I would recommend making a backup beforehand and running the above command with an additional -v --noop for a dry-run first to check what would be changed beforehand.

Regards

areyus · July 4, 2019, 10:11am

I just checked the installer options once more.
According to the docks, it should be this option:

--certs-reset                 This option will reset any custom certificates and use the self-signed CA instead. Note that any clients will need to be updated with the latest katello-ca-consumer RPM, and any external proxies will need to have the certs updated by generating a new certs tarball. (default: false)

Regards