Curl for unattended kickstart file takes a long time and fails in dracut

Support
1

Problem:
The curl download of the unattended kickstart file fails and I believe that it has timed out waiting for the kickstart file to be generated.

Foreman and Proxy versions:
Foreman 2.1.1

Foreman and Proxy plugin versions:
foreman-tasks 2.0.2
foreman_ansible 5.1.1
foreman_remote_execution 3.3.5

Other relevant data:
I can run the curl http://foreman.demo.com/unattended/provision?token=76f6f775-ee77-48f9-9faf-4b2dca24f81b in the browser and it (eventually) returns with the kickstart contents
I ran the curl from the command line on the foreman server where I assume the first response is similar to what is received in the dracut-initqueue phase of boot:

# curl http://foreman.demo.com/unattended/provision?token=76f6f775-ee77-48f9-9faf-4b2dca24f81b
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>502 Proxy Error</title>
</head><body>
<h1>Proxy Error</h1>
<p>The proxy server received an invalid
response from an upstream server.<br />
The proxy server could not handle the request <em><a href="/unattended/provision">GET&nbsp;/unattended/provision</a></em>.<p>
Reason: <strong>Error reading from remote server</strong></p></p>
</body></html>

# curl http://foreman.demo.com/unattended/provision?token=76f6f775-ee77-48f9-9faf-4b2dca24f81b
 <kickstart file>
2

In addition from a different server I get the following output from a curl with verbose mode:

# curl -v http://foreman.demo.com/unattended/provision?token=76f6f775-ee77-48f9-9faf-4b2dca24f81b
* About to connect() to foreman.demo.com port 80 (#0)
*   Trying 192.168.33.183...
* Connected to foreman.demo.com (192.168.33.183) port 80 (#0)
> GET /unattended/provision?token=76f6f775-ee77-48f9-9faf-4b2dca24f81b HTTP/1.1
> User-Agent: curl/7.29.0
> Host: foreman.demo.com
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Date: Mon, 11 Jan 2021 08:35:44 GMT
< Server: Apache
< Content-Type: text/plain; charset=UTF-8
< Cache-Control: no-cache
< X-Request-Id: 5104fde0-674a-44a8-bb25-7211365f8a31
< X-Runtime: 0.039116
< X-Frame-Options: sameorigin
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< X-Download-Options: noopen
< X-Permitted-Cross-Domain-Policies: none
< Content-Security-Policy: default-src 'self'; child-src 'self'; connect-src 'self' ws: wss:; img-src 'self' data:; script-src 'unsafe-eval' 'unsafe-inline' 'self'; style-src 'unsafe-inline' 'self'
< Via: 1.1 foreman.demo.com
< Transfer-Encoding: chunked
<
unattended: unable to find a host that matches the request from 127.0.0.1
* Connection #0 to host foreman.demo.com left intact
3

If you get 404 it means the token has expired and is no longer valid.

Timeouts - check network. The ERB can contain some macros that performs queries (e.g. DNS query, talking to remote proxy etc).

4

I saw your response to a previous query of mine but I don’t think that timeouts are the issue. The token duration is 360 (default) and ping is very responsive (0.243 ms).
The error occurs when I am creating a new host and the dracut is trying to get the kickstart file for the first time (about 30 seconds in user time has passed).
I think the boot curl error is similar to the error when I run the curl from another server and get the 404 immediately. Why does it work in the browser and not from the command line or in dracut?

5

Preview in browser is entirely a different thing, in that case Foreman just renders the output, but you need to be authorized user.

When connecting to the endpoint via curl without authorization, you can only see the content if you do this from the very same IP address as the provisioning interface has or if you specify MAC address of the provisioning interface via HTTP header (that’s what Anaconda installer does automatically).

6

Thanks for the prompt reply Izap.
So my originally question still stands - why am I getting a curl 404 error immediately after the new host is created?

7

This indicates there must be HTTP proxy configured for something. Have you configured for Anaconda installer? For Foreman server itself? For the proxy? Or all of them?

This looks like some network misconfiguration and you need to dig it out yourself, I can hardly help you based on the observations you shared with us so far.

Also, please format your posts appropriately. I will fix what you have posted so far for you.

8

For the 404 error (when you proxy does work actually), you can check tokens table if the token is still valid:

foreman-rake console

> Token.all.pluck(:value, :expires)
=> [["835adc0e-9739-419c-86dc-f70ed0b42364", Fri, 04 Dec 2020 13:35:52 UTC +00:00]]

You will also see it in the host detail in UI/CLI.

9

I believe the solution was opening port 8000 on the firewall on the foreman provisioning server