Failed to Retrieve the Preconfiguration file at /unattended/provision

Hi all! I'm kinda new to the IT scene but immediately gravitated to
Foreman. I'm putting together a proof of concept cluster that can be scaled
up to a datacenter all managed by Foreman. Thus far, I've been able to get
through any problem areas but I'm stuck on this issue.

During the PXE boot, of either host, it is unable to retrieve the preconfig
file located at http://node/unattended/provision.

Here are my configuration details:

I'm running Foreman 1.3 on Ubuntu 12.04, and the foreman-proxy is managing
dhcp, tftp, puppet, and puppet ca. I have a DNS server with an entry for
"node" (the foreman box).

I have two OS definitions; one with CentOS and another with Ubuntu. They
both have associations with a partition table, installation media,
PXElinux, and a provision template. Both the PXElinux and provision are
unmodified from the default for their respective distro.

I have three defined hosts, one of which, of course, is running Foreman
with the foreman-proxy.

The other two are CentOS and Ubuntu with all the important parameters
filled out except a Host Group. They are allocated IPs automatically by the
DHCP server foreman-proxy and associated with a subnet I have also defined.

I verified in /var/lib/tftpboot/boot that the initrd files for Ubuntu and
CentOS exist as do the pxe files in and /var/lib/tftpboot/pxelinux.cfg. I
can further verify the PXELinux and provision templates are associated
correctly with each host by reviewing them via the Templates tab of each
respective Host's details page.

I've enabled debug mode and have reviewed the logs without much insight.
The errors that the foreman production log generates are:

Fog is not installed - unable to manage compute resources

which is sometimes followed up with:

Encryption_Key is not defined, so encryption is turned off for
ComputeResource

What might be a clue is, if I try to manually run, on the Foreman box, wget
–no-check-certificate http://node/, a file is saved. If, after a failed
pxe boot of another host, but still in the boot menu, the file can not be
saved running the same wget statement. The error is that the url is not
HTTP or FTP.

Maybe you guys can help? Thanks guys!

Hey,

what you need to check is if the box which is booting is really able to
reach the "node". In this case you need to make sure:

  • "node" hostname does resolve
  • IP addres of "node" is reachable
  • port 80 is opened

What foreman uses as the hostname can be set in the global settings
under the foreman_url settings. So if foreman itself is available under
different hostname in the network you are booting in, change this
setting.

··· On Thu, Oct 17, 2013 at 06:44:45PM -0700, charles clark wrote: > During the PXE boot, of either host, it is unable to retrieve the preconfig > file located at http://node/unattended/provision.


Later,

Lukas “lzap” Zapletal
irc: lzap #theforeman

Thanks for the reply Lukas!

I tested and double checked my DNS. If I attempt to resolve 'node' from any
other box on the subnet using a browser, I'm successfully able to login to
Foreman. From the same box, if I attempt wget --no-check-certificate
http://node/, I'm successfully able to download the index.html file.

However, if I open a shell right after a failed pxeboot, and attempt:

wget http://node/ … wget responds: bad address 'node'

If I try:

wget http://10.10.1.2/ … wget responds: not an http or ftp url:
https://10.10.1.2/

Any thoughts?

··· On Friday, October 18, 2013 12:23:50 AM UTC-7, Lukas Zapletal wrote: > > On Thu, Oct 17, 2013 at 06:44:45PM -0700, charles clark wrote: > > During the PXE boot, of either host, it is unable to retrieve the > preconfig > > file located at http://node/unattended/provision. > > Hey, > > what you need to check is if the box which is booting is really able to > reach the "node". In this case you need to make sure: > > - "node" hostname does resolve > - IP addres of "node" is reachable > - port 80 is opened > > What foreman uses as the hostname can be set in the global settings > under the foreman_url settings. So if foreman itself is available under > different hostname in the network you are booting in, change this > setting. > > -- > Later, > > Lukas "lzap" Zapletal > irc: lzap #theforeman >

I forgot to mention earlier that the foreman_url was already set to 'node'
in the Foreman Settings.

··· On Friday, October 18, 2013 1:20:18 PM UTC-7, charles clark wrote: > > Thanks for the reply Lukas! > > I tested and double checked my DNS. If I attempt to resolve 'node' from > any other box on the subnet using a browser, I'm successfully able to login > to Foreman. From the same box, if I attempt wget --no-check-certificate > http://node/, I'm successfully able to download the index.html file. > > However, if I open a shell right after a failed pxeboot, and attempt: > > wget http://node/ ... wget responds: bad address 'node' > > If I try: > > wget http://10.10.1.2/ ... wget responds: not an http or ftp url: > https://10.10.1.2/ > > Any thoughts? > > On Friday, October 18, 2013 12:23:50 AM UTC-7, Lukas Zapletal wrote: >> >> On Thu, Oct 17, 2013 at 06:44:45PM -0700, charles clark wrote: >> > During the PXE boot, of either host, it is unable to retrieve the >> preconfig >> > file located at http://node/unattended/provision. >> >> Hey, >> >> what you need to check is if the box which is booting is really able to >> reach the "node". In this case you need to make sure: >> >> - "node" hostname does resolve >> - IP addres of "node" is reachable >> - port 80 is opened >> >> What foreman uses as the hostname can be set in the global settings >> under the foreman_url settings. So if foreman itself is available under >> different hostname in the network you are booting in, change this >> setting. >> >> -- >> Later, >> >> Lukas "lzap" Zapletal >> irc: lzap #theforeman >> >

> Thanks for the reply Lukas!
>
> I tested and double checked my DNS. If I attempt to resolve 'node' from
> any other box on the subnet using a browser, I'm successfully able to login
> to Foreman. From the same box, if I attempt wget --no-check-certificate
> http://node/, I'm successfully able to download the index.html file.
>
> However, if I open a shell right after a failed pxeboot, and attempt:
>
> wget http://node/ … wget responds: bad address 'node'
>
> If I try:
>
> wget http://10.10.1.2/ … wget responds: not an http or ftp url:
> https://10.10.1.2/
>
> Any thoughts?
>
within anaconda/preseed, usually the wget implementation is from busybox
which does not support https urls… normally /unattended/* urls are
reachable via http (because of the above limitation)…

how are you generating the url? manually?

Ohad

··· On Fri, Oct 18, 2013 at 11:20 PM, charles clark wrote:

On Friday, October 18, 2013 12:23:50 AM UTC-7, Lukas Zapletal wrote:

On Thu, Oct 17, 2013 at 06:44:45PM -0700, charles clark wrote:

During the PXE boot, of either host, it is unable to retrieve the
preconfig
file located at http://node/unattended/**provisionhttp://node/unattended/provision.

Hey,

what you need to check is if the box which is booting is really able to
reach the “node”. In this case you need to make sure:

  • “node” hostname does resolve
  • IP addres of “node” is reachable
  • port 80 is opened

What foreman uses as the hostname can be set in the global settings
under the foreman_url settings. So if foreman itself is available under
different hostname in the network you are booting in, change this
setting.


Later,

Lukas “lzap” Zapletal
irc: lzap #theforeman


You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at http://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/groups/opt_out.

Hi ohad, I can confirm that busybox is the environment.

During the pxboot attempt, the URL is generated successfully by foreman.

Once the boost process errors, I manually launch a shell and attempted the wget.

If wget handles https okay under busybox, then maybe the problem is how the new host is trying to resolve the URL?

Check your DHCP setting, more specifically if you have proper DNS1 and
DNS2 server entries set. It must resolve, otherwise you are not
kickstarting anything.

LZ

··· On Fri, Oct 18, 2013 at 02:39:46PM -0700, charles clark wrote: > I forgot to mention earlier that the foreman_url was already set to 'node' > in the Foreman Settings. > > On Friday, October 18, 2013 1:20:18 PM UTC-7, charles clark wrote: > > > > Thanks for the reply Lukas! > > > > I tested and double checked my DNS. If I attempt to resolve 'node' from > > any other box on the subnet using a browser, I'm successfully able to login > > to Foreman. From the same box, if I attempt wget --no-check-certificate > > http://node/, I'm successfully able to download the index.html file. > > > > However, if I open a shell right after a failed pxeboot, and attempt: > > > > wget http://node/ ... wget responds: bad address 'node' > > > > If I try: > > > > wget http://10.10.1.2/ ... wget responds: not an http or ftp url: > > https://10.10.1.2/ > > > > Any thoughts? > > > > On Friday, October 18, 2013 12:23:50 AM UTC-7, Lukas Zapletal wrote: > >> > >> On Thu, Oct 17, 2013 at 06:44:45PM -0700, charles clark wrote: > >> > During the PXE boot, of either host, it is unable to retrieve the > >> preconfig > >> > file located at http://node/unattended/provision. > >> > >> Hey, > >> > >> what you need to check is if the box which is booting is really able to > >> reach the "node". In this case you need to make sure: > >> > >> - "node" hostname does resolve > >> - IP addres of "node" is reachable > >> - port 80 is opened > >> > >> What foreman uses as the hostname can be set in the global settings > >> under the foreman_url settings. So if foreman itself is available under > >> different hostname in the network you are booting in, change this > >> setting. > >> > >> -- > >> Later, > >> > >> Lukas "lzap" Zapletal > >> irc: lzap #theforeman > >> > >


Later,

Lukas “lzap” Zapletal
irc: lzap #theforeman

Just throw my last comment away since I just reread your post and noticed
you said wget does "not" support https! That's in line with the results I'm
seeing too.

The URL that's generated by foreman is an http, not https. But, when I
manually attempt to wget the http url in the shell, wget gets feedback that
the box requires an https request. I'm just thinking out load, but maybe
this is an apache config issue? Why else would an http request to the box
only respond with https?

··· On Sunday, October 20, 2013 8:31:32 AM UTC-7, charles clark wrote: > > Hi ohad, I can confirm that busybox is the environment. > > During the pxboot attempt, the URL is generated successfully by foreman. > > Once the boost process errors, I manually launch a shell and attempted the > wget. > > If wget handles https okay under busybox, then maybe the problem is how > the new host is trying to resolve the URL? > >

Can you provide the logs from Foreman when you (or ideally, the
installer) make the request? /var/log/foreman/production.log

Foreman will use the source IP address of the machine to identify it, so
if you run the wget from another host then the logs might indicate that
it can't identify the IP and it may redirect you to the (HTTPS) login page.

··· On 20/10/13 18:07, charles clark wrote: > Just throw my last comment away since I just reread your post and > noticed you said wget does "not" support https! That's in line with the > results I'm seeing too. > > The URL that's generated by foreman is an http, not https. But, when I > manually attempt to wget the http url in the shell, wget gets feedback > that the box requires an https request. I'm just thinking out load, but > maybe this is an apache config issue? Why else would an http request to > the box only respond with https?


Dominic Cleal
Red Hat Engineering

Hi, Dominic.

Is there a way to set the production.log to debug mode like foreman-proxy?
In the production.log, I only see multiple entries of "Connecting to
database specified by database.yml" but nothing further.

Thanks for helping out guys!

··· On Sunday, October 20, 2013 11:12:32 AM UTC-7, Dominic Cleal wrote: > > On 20/10/13 18:07, charles clark wrote: > > Just throw my last comment away since I just reread your post and > > noticed you said wget does "not" support https! That's in line with the > > results I'm seeing too. > > > > The URL that's generated by foreman is an http, not https. But, when I > > manually attempt to wget the http url in the shell, wget gets feedback > > that the box requires an https request. I'm just thinking out load, but > > maybe this is an apache config issue? Why else would an http request to > > the box only respond with https? > > Can you provide the logs from Foreman when you (or ideally, the > installer) make the request? /var/log/foreman/production.log > > Foreman will use the source IP address of the machine to identify it, so > if you run the wget from another host then the logs might indicate that > it can't identify the IP and it may redirect you to the (HTTPS) login > page. > > > -- > Dominic Cleal > Red Hat Engineering >

production.log should log every single request by default. The message
you quote usually happens at application startup, or when a cronjob fires.

Does it log requests if you visit the web interface? If so, and no
request is being logged when you provision a host, then it suggests the
request is failing on a more basic network issue before it reaches
Foreman as Lukas suggested. Try capturing packets from the host you're
provisioning.

Debug mode is available, but I'm not sure it'll help if requests aren't
logged:
http://projects.theforeman.org/projects/foreman/wiki/Troubleshooting#How-do-I-enable-debugging

··· -- Dominic Cleal Red Hat Engineering

On 20/10/13 20:14, charles clark wrote:

Hi, Dominic.

Is there a way to set the production.log to debug mode like
foreman-proxy? In the production.log, I only see multiple entries of
"Connecting to database specified by database.yml" but nothing further.

Thanks for helping out guys!

On Sunday, October 20, 2013 11:12:32 AM UTC-7, Dominic Cleal wrote:

On 20/10/13 18:07, charles clark wrote:
> Just throw my last comment away since I just reread your post and
> noticed you said wget does "not" support https! That's in line
with the
> results I'm seeing too.
>
> The URL that's generated by foreman is an http, not https. But,
when I
> manually attempt to wget the http url in the shell, wget gets
feedback
> that the box requires an https request. I'm just thinking out
load, but
> maybe this is an apache config issue? Why else would an http
request to
> the box only respond with https?

Can you provide the logs from Foreman when you (or ideally, the
installer) make the request?  /var/log/foreman/production.log

Foreman will use the source IP address of the machine to identify
it, so
if you run the wget from another host then the logs might indicate that
it can't identify the IP and it may redirect you to the (HTTPS)
login page.


-- 
Dominic Cleal
Red Hat Engineering


You received this message because you are subscribed to the Google
Groups “Foreman users” group.
To unsubscribe from this group and stop receiving emails from it, send
an email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at http://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/groups/opt_out.

You're right Dominic; the production.log is populated when navigating the
web interface but not one entry during the provision attempt.

Here's a tcpdump of the foreman/dhcp server while attempting the provision:

13:08:45.374365 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP,
Request from 8c:89:a5:11:fd:71 (oui Unknown), length 548
13:08:45.374733 IP node.local.bootps > 255.255.255.255.bootpc: BOOTP/DHCP,
Reply, length 322
13:08:45.375034 IP node.local.33966 >
cns.sanjose.ca.sanfran.comcast.net.domain: 12478+ PTR?
255.255.255.255.in-addr.arpa. (46)
13:08:45.376660 ARP, Request who-has 10.10.1.20 tell 10.10.1.1, length 46
13:08:45.401515 IP cns.sanjose.ca.sanfran.comcast.net.domain >
node.local.33966: 12478 NXDomain 0/1/0 (114)
13:08:45.502438 IP6 fe80::6e62:6dff:fe8f:6ba2.mdns > ff02::fb.mdns: 0 PTR
(QM)? 255.255.255.255.in-addr.arpa. (46)
13:08:45.502532 IP node.local.mdns > 224.0.0.251.mdns: 0 PTR (QM)?
255.255.255.255.in-addr.arpa. (46)
13:08:46.376647 IP 10.10.1.1.bootps > 255.255.255.255.bootpc: BOOTP/DHCP,
Reply, length 548
13:08:46.504230 IP6 fe80::6e62:6dff:fe8f:6ba2.mdns > ff02::fb.mdns: 0 PTR
(QM)? 255.255.255.255.in-addr.arpa. (46)
13:08:46.504320 IP node.local.mdns > 224.0.0.251.mdns: 0 PTR (QM)?
255.255.255.255.in-addr.arpa. (46)
13:08:47.516467 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP,
Request from 8c:89:a5:11:fd:71 (oui Unknown), length 548
13:08:47.516818 IP node.local.bootps > 255.255.255.255.bootpc: BOOTP/DHCP,
Reply, length 322
13:08:47.517149 ARP, Request who-has node.local tell 10.10.1.12, length 46
13:08:47.517174 ARP, Reply node.local is-at 6c:62:6d:8f:6b:a2 (oui
Unknown), length 28
13:08:47.517225 IP 10.10.1.12.2070 > node.local.tftp: 27 RRQ "pxelinux.0"
octet tsize 0
13:08:47.518607 IP node.local.44894 > 10.10.1.12.2070: UDP, length 14
13:08:47.518672 IP 10.10.1.12.2070 > node.local.44894: UDP, length 17
13:08:47.518748 IP 10.10.1.12.2071 > node.local.tftp: 32 RRQ "pxelinux.0"
octet blksize 1456
13:08:47.520174 IP node.local.35186 > 10.10.1.12.2071: UDP, length 15
13:08:47.520244 IP 10.10.1.12.2071 > node.local.35186: UDP, length 4
13:08:47.520412 IP node.local.35186 > 10.10.1.12.2071: UDP, length 1460
13:08:47.520510 IP 10.10.1.12.2071 > node.local.35186: UDP, length 4
13:08:47.520534 IP node.local.35186 > 10.10.1.12.2071: UDP, length 1460
13:08:47.520651 IP 10.10.1.12.2071 > node.local.35186: UDP, length 4
13:08:47.520714 IP node.local.35186 > 10.10.1.12.2071: UDP, length 1460
13:08:47.520805 IP 10.10.1.12.2071 > node.local.35186: UDP, length 4
13:08:47.520827 IP node.local.35186 > 10.10.1.12.2071: UDP, length 1460
13:08:47.520928 IP 10.10.1.12.2071 > node.local.35186: UDP, length 4
13:08:47.520986 IP node.local.35186 > 10.10.1.12.2071: UDP, length 1460
13:08:47.521098 IP 10.10.1.12.2071 > node.local.35186: UDP, length 4
13:08:47.521154 IP node.local.35186 > 10.10.1.12.2071: UDP, length 1460
13:08:55.410520 IP node.local.57457 > 172.16.1.20.domain: 54889+ PTR?
20.1.10.10.in-addr.arpa. (41)
13:08:55.411859 IP 172.16.1.20.domain > node.local.57457: 54889 NXDomain*
0/1/0 (83)
13:08:55.512775 IP6 fe80::6e62:6dff:fe8f:6ba2.mdns > ff02::fb.mdns: 0 PTR
(QM)? 20.1.10.10.in-addr.arpa. (41)
13:09:00.415652 IP node.local.60915 > 172.16.1.20.domain: 8373+ PTR?
12.1.10.10.in-addr.arpa. (41)
13:09:00.417431 IP 172.16.1.20.domain > node.local.60915: 8373 NXDomain*
0/1/0 (83)
13:09:00.418958 ARP, Request who-has 10.10.1.1 tell node.local, length 28
13:09:00.419140 ARP, Reply 10.10.1.1 is-at c0:c1:c0:92:9c:cf (oui Unknown),
length 46
13:09:00.518310 IP6 fe80::6e62:6dff:fe8f:6ba2.mdns > ff02::fb.mdns: 0 PTR
(QM)? 12.1.10.10.in-addr.arpa. (41)
13:09:00.518405 IP node.local.mdns > 224.0.0.251.mdns: 0 PTR (QM)?
12.1.10.10.in-addr.arpa. (41)
13:09:00.602311 IP6 fe80::8e89:a5ff:fe11:fd71 > ip6-allrouters: ICMP6,
router solicitation, length 8
13:09:01.104055 IP6 fe80::8e89:a5ff:fe11:fd71 > ip6-allrouters: ICMP6,
router solicitation, length 8
13:09:01.519996 IP6 fe80::6e62:6dff:fe8f:6ba2.mdns > ff02::fb.mdns: 0 PTR
(QM)? 12.1.10.10.in-addr.arpa. (41)
13:09:05.421519 IP node.local.40451 > 172.16.1.20.domain: 32826+ PTR?
1.7.d.f.1.1.e.f.f.f.5.a.9.8.e.8.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa.
(90)
13:09:05.423313 IP 172.16.1.20.domain > node.local.40451: 32826 NXDomain*
0/1/0 (125)
13:09:06.091670 IP6 fe80::8e89:a5ff:fe11:fd71 > ip6-allrouters: ICMP6,
router solicitation, length 16
13:09:08.247980 ARP, Request who-has 10.10.1.1 tell 10.10.1.12, length 46
13:09:10.418611 ARP, Request who-has node.local tell 10.10.1.1, length 46
13:09:10.418648 ARP, Reply node.local is-at 6c:62:6d:8f:6b:a2 (oui
Unknown), length 28

As expected, the host communicating with the DHCP server, is allocated an
IP, downloads the pxelinux.0 file and the the provisioning url. At this
point no more messages are logged by tcpdump and the provisioning fails.

You guys have to be right that it's a networking issue. Just to be thorough
and the likeliness that I'm missing something, here is my dhcpd.conf and
subnet declaration (within foreman):

Subnet
Network Address: 10.10.1.0
Netmask: 255.255.255.0
Gateway Address: 10.10.1.1
Primary DNS server: 172.16.1.20
Secondary DNS server: 68.87.76.182
Start of IP range: 10.10.1.10
End of IP range: 10.10.1.20
VLAN ID: empty
Domain: checkmark on the domain name
DHCP Proxy: node
TFTP Proxy: node
DNS Proxy: empty

··· ----

dhcpd.conf

omapi-port 7911;

default-lease-time 600;
max-lease-time 7200;

ddns-update-style none;

option domain-name “cluster”;
option domain-name-servers 172.16.1.20, 68.87.76.182;

allow booting;
allow bootp;

option fqdn.no-client-update on; # set the “O” and “S” flag bits
option fqdn.rcode2 255;
option pxegrub code 150 = text ;

PXE Handoff.

next-server 10.10.1.2;
filename “pxelinux.0”;

log-facility local7;

include “/etc/dhcp/dhcpd.hosts”;
#################################

cluster

#################################
subnet 10.10.1.0 netmask 255.255.255.0 {

option subnet-mask 255.255.255.0;
option routers 10.10.1.1;
}

Thanks guys!

On Monday, October 21, 2013 2:20:09 AM UTC-7, Dominic Cleal wrote:

production.log should log every single request by default. The message
you quote usually happens at application startup, or when a cronjob fires.

Does it log requests if you visit the web interface? If so, and no
request is being logged when you provision a host, then it suggests the
request is failing on a more basic network issue before it reaches
Foreman as Lukas suggested. Try capturing packets from the host you’re
provisioning.

Debug mode is available, but I’m not sure it’ll help if requests aren’t
logged:

Troubleshooting - Foreman


Dominic Cleal
Red Hat Engineering

On 20/10/13 20:14, charles clark wrote:

Hi, Dominic.

Is there a way to set the production.log to debug mode like
foreman-proxy? In the production.log, I only see multiple entries of
"Connecting to database specified by database.yml" but nothing further.

Thanks for helping out guys!

On Sunday, October 20, 2013 11:12:32 AM UTC-7, Dominic Cleal wrote:

On 20/10/13 18:07, charles clark wrote: 
> Just throw my last comment away since I just reread your post and 
> noticed you said wget does "not" support https! That's in line 
with the 
> results I'm seeing too. 
> 
> The URL that's generated by foreman is an http, not https. But, 
when I 
> manually attempt to wget the http url in the shell, wget gets 
feedback 
> that the box requires an https request. I'm just thinking out 
load, but 
> maybe this is an apache config issue? Why else would an http 
request to 
> the box only respond with https? 

Can you provide the logs from Foreman when you (or ideally, the 
installer) make the request?  /var/log/foreman/production.log 

Foreman will use the source IP address of the machine to identify 
it, so 
if you run the wget from another host then the logs might indicate 

that

it can't identify the IP and it may redirect you to the (HTTPS) 
login page. 


-- 
Dominic Cleal 
Red Hat Engineering 


You received this message because you are subscribed to the Google
Groups “Foreman users” group.
To unsubscribe from this group and stop receiving emails from it, send
an email to foreman-user...@googlegroups.com <javascript:>.
To post to this group, send email to forema...@googlegroups.com<javascript:>.

Visit this group at http://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/groups/opt_out.

Good news everyone!

I rebooted the isc-dhcp service and viola, it works :slight_smile:

Thanks for the help in pushing me in the right direction. Now I can get
into the fun stuff with puppet modules.