Internal server errors on /unattented/provision

I dont know what I broke, but suddenly, I cant do unattended
provisioning in Foreman 1.3.

I have a host, which I provisioned with Foreman, that recently got
updated to RHEL 6.5. This is machine that I use for testing, and i
was testing some new kickstart stuff and re-provisioned it. I had
issued with RHEL 6.5 not yet being setup as an OS in foreman, so i
took care of that, and found that builds would fail. anaconda gets to
the point where it's trying to get
http://my.foreman.url/unattended/provision, and then it reports that
it couldn't get the kickstart script. I check foreman's
/var/log/foreman/production.log, and i see this:

==> foreman/production.log <==
Rendered inline template (243.2ms)
Rendered text template (0.0ms)
Completed 500 Internal Server Error in 2277ms (Views: 0.6ms |
ActiveRecord: 7.3ms)

I booted the host off of a rhel 6 iso, and got it to a shell. It has
the same macaddr/ip address from dhcp as the build was expecting, so i
should be able to wget or curl the dynamic /unattended/provision url.
I did a wget -wO - http://my.foreman.url/unattended/provision and
watched the logs on the foreman server. Same errorin the logs. And
the output of wget is nothing.

I undid (i think anyway) all the changes i made to my kickstart
template, and the issue still persists. I even gave my foreman a full
reboot, just to make sure this wasnt some sort of caching problem or
something.

Any suggestions?

Can you provide more of the log file, this looks to only be a few
lines of the request.


Dominic Cleal
Red Hat Engineering

··· On 11/12/13 21:17, Nathan wrote: > I dont know what I broke, but suddenly, I cant do unattended > provisioning in Foreman 1.3. > > I have a host, which I provisioned with Foreman, that recently got > updated to RHEL 6.5. This is machine that I use for testing, and > i was testing some new kickstart stuff and re-provisioned it. I > had issued with RHEL 6.5 not yet being setup as an OS in foreman, > so i took care of that, and found that builds would fail. anaconda > gets to the point where it's trying to get > http://my.foreman.url/unattended/provision, and then it reports > that it couldn't get the kickstart script. I check foreman's > /var/log/foreman/production.log, and i see this: > > ==> foreman/production.log <== Rendered inline template (243.2ms) > Rendered text template (0.0ms) Completed 500 Internal Server Error > in 2277ms (Views: 0.6ms | ActiveRecord: 7.3ms)

That it is. I wasn't sure what log data was pertinent.

Here are the log's I'm tailing.

[root@masterofpuppet PROD log S:E]# pwd
/var/log
[root@masterofpuppet PROD log S:E]# tail -f foreman/production.log
foreman-proxy/access.log foreman-proxy/proxy.log httpd/access_log
httpd/error_log

And here are all of the log entries that occur from the moment I make
the request via wget, until wget returns the 500 error…

==> foreman/production.log <==

Started GET "/unattended/provision" for 139.147.165.244 at 2013-12-12
08:51:17 -0500
Processing by UnattendedController#provision as /
Found testprov2.dev.lafayette.edu
Remove puppet certificate for testprov2.dev.lafayette.edu

==> foreman-proxy/proxy.log <==
D, [2013-12-12T08:51:17.092663 #22983] DEBUG – : Found puppetca at
/usr/bin/puppet
D, [2013-12-12T08:51:17.092851 #22983] DEBUG – : Found sudo at
/usr/bin/sudo
D, [2013-12-12T08:51:17.092930 #22983] DEBUG – : Executing
/usr/bin/sudo -S /usr/bin/puppet cert --ssldir /var/lib/puppet/ssl

==> foreman/production.log <==
Adding autosign entry for testprov2.dev.lafayette.edu

==> foreman-proxy/proxy.log <==
I, [2013-12-12T08:51:18.701100 #22983] INFO – : Added
testprov2.dev.lafayette.edu to autosign

==> foreman/production.log <==
Rendered inline template (250.8ms)
Rendered text template (0.0ms)
Completed 500 Internal Server Error in 1956ms (Views: 1.1ms |
ActiveRecord: 7.9ms)

==> httpd/access_log <==
139.147.165.244 - - [12/Dec/2013:08:51:17 -0500] "GET
/unattended/provision HTTP/1.0" 500 127 "-" "Wget/1.12 (linux-gnu)"
^C
[root@masterofpuppet PROD log S:E]#

··· On 12/12/2013 03:38 AM, Dominic Cleal wrote: > On 11/12/13 21:17, Nathan wrote: >> I dont know what I broke, but suddenly, I cant do unattended >> provisioning in Foreman 1.3. > >> I have a host, which I provisioned with Foreman, that recently >> got updated to RHEL 6.5. This is machine that I use for testing, >> and i was testing some new kickstart stuff and re-provisioned it. >> I had issued with RHEL 6.5 not yet being setup as an OS in >> foreman, so i took care of that, and found that builds would >> fail. anaconda gets to the point where it's trying to get >> http://my.foreman.url/unattended/provision, and then it reports >> that it couldn't get the kickstart script. I check foreman's >> /var/log/foreman/production.log, and i see this: > >> ==> foreman/production.log <== Rendered inline template (243.2ms) >> Rendered text template (0.0ms) Completed 500 Internal Server >> Error in 2277ms (Views: 0.6ms | ActiveRecord: 7.3ms) > > Can you provide more of the log file, this looks to only be a few > lines of the request. > >

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> That it is. I wasn't sure what log data was pertinent.
>
> Here are the log's I'm tailing.
>

looks like a template error, can you have a look at the template audit log
and see if there were changes to it?

Ohad

··· On Thu, Dec 12, 2013 at 3:54 PM, Nathan wrote:

[root@masterofpuppet PROD log S:E]# pwd
/var/log
[root@masterofpuppet PROD log S:E]# tail -f foreman/production.log
foreman-proxy/access.log foreman-proxy/proxy.log httpd/access_log
httpd/error_log

And here are all of the log entries that occur from the moment I make
the request via wget, until wget returns the 500 error…

==> foreman/production.log <==

Started GET “/unattended/provision” for 139.147.165.244 at 2013-12-12
08:51:17 -0500
Processing by UnattendedController#provision as /
Found testprov2.dev.lafayette.edu
Remove puppet certificate for testprov2.dev.lafayette.edu

==> foreman-proxy/proxy.log <==
D, [2013-12-12T08:51:17.092663 #22983] DEBUG – : Found puppetca at
/usr/bin/puppet
D, [2013-12-12T08:51:17.092851 #22983] DEBUG – : Found sudo at
/usr/bin/sudo
D, [2013-12-12T08:51:17.092930 #22983] DEBUG – : Executing
/usr/bin/sudo -S /usr/bin/puppet cert --ssldir /var/lib/puppet/ssl

==> foreman/production.log <==
Adding autosign entry for testprov2.dev.lafayette.edu

==> foreman-proxy/proxy.log <==
I, [2013-12-12T08:51:18.701100 #22983] INFO – : Added
testprov2.dev.lafayette.edu to autosign

==> foreman/production.log <==
Rendered inline template (250.8ms)
Rendered text template (0.0ms)
Completed 500 Internal Server Error in 1956ms (Views: 1.1ms |
ActiveRecord: 7.9ms)

==> httpd/access_log <==
139.147.165.244 - - [12/Dec/2013:08:51:17 -0500] “GET
/unattended/provision HTTP/1.0” 500 127 “-” “Wget/1.12 (linux-gnu)”
^C
[root@masterofpuppet PROD log S:E]#

On 12/12/2013 03:38 AM, Dominic Cleal wrote:

On 11/12/13 21:17, Nathan wrote:

I dont know what I broke, but suddenly, I cant do unattended
provisioning in Foreman 1.3.

I have a host, which I provisioned with Foreman, that recently
got updated to RHEL 6.5. This is machine that I use for testing,
and i was testing some new kickstart stuff and re-provisioned it.
I had issued with RHEL 6.5 not yet being setup as an OS in
foreman, so i took care of that, and found that builds would
fail. anaconda gets to the point where it’s trying to get
http://my.foreman.url/unattended/provision, and then it reports
that it couldn’t get the kickstart script. I check foreman’s
/var/log/foreman/production.log, and i see this:

==> foreman/production.log <== Rendered inline template (243.2ms)
Rendered text template (0.0ms) Completed 500 Internal Server
Error in 2277ms (Views: 0.6ms | ActiveRecord: 7.3ms)

Can you provide more of the log file, this looks to only be a few
lines of the request.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.15 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlKpwAEACgkQsZqG4IN3sukRZACgnrCXf7Kbdenb1SOO7vDw+vR6
scUAn3noukp8Nf3bLZi2ugEnzyr8IROa
=2e1N
-----END PGP SIGNATURE-----


You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at http://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/groups/opt_out.

>
>
>
>
> That it is. I wasn't sure what log data was pertinent.
>
> Here are the log's I'm tailing.
>
>
>> looks like a template error, can you have a look at the template
>> audit log and see if there were changes to it?
>
I thought the same thing. However, i reverted all of the templates
and snippets called in this provisioning task to known-good versions.
In other words, I had made changes, i've reverted them.

Is there some log that I can watch to see where the error is
happening? Some manner of debug or verbose log option?

Thanks.

··· On 12/12/2013 08:58 AM, Ohad Levy wrote: > On Thu, Dec 12, 2013 at 3:54 PM, Nathan > wrote:

Ohad

[root@masterofpuppet PROD log S:E]# pwd /var/log
[root@masterofpuppet PROD log S:E]# tail -f foreman/production.log
foreman-proxy/access.log foreman-proxy/proxy.log httpd/access_log
httpd/error_log

And here are all of the log entries that occur from the moment I
make the request via wget, until wget returns the 500 error…

==> foreman/production.log <==

Started GET “/unattended/provision” for 139.147.165.244 at
2013-12-12 08:51:17 -0500 Processing by
UnattendedController#provision as / Found
testprov2.dev.lafayette.edu http://testprov2.dev.lafayette.edu
Remove puppet certificate for testprov2.dev.lafayette.edu
http://testprov2.dev.lafayette.edu

==> foreman-proxy/proxy.log <== D, [2013-12-12T08:51:17.092663
#22983] DEBUG – : Found puppetca at /usr/bin/puppet D,
[2013-12-12T08:51:17.092851 #22983] DEBUG – : Found sudo at
/usr/bin/sudo D, [2013-12-12T08:51:17.092930 #22983] DEBUG – :
Executing /usr/bin/sudo -S /usr/bin/puppet cert --ssldir
/var/lib/puppet/ssl --clean testprov2.dev.lafayette.edu
http://testprov2.dev.lafayette.edu I, [2013-12-12T08:51:18.624152
#22983] INFO – : cleaned puppet certificate for
testprov2.dev.lafayette.edu http://testprov2.dev.lafayette.edu

==> foreman/production.log <== Adding autosign entry for
testprov2.dev.lafayette.edu http://testprov2.dev.lafayette.edu

==> foreman-proxy/proxy.log <== I, [2013-12-12T08:51:18.701100
#22983] INFO – : Added testprov2.dev.lafayette.edu
http://testprov2.dev.lafayette.edu to autosign

==> foreman/production.log <== Rendered inline template (250.8ms)
Rendered text template (0.0ms) Completed 500 Internal Server Error
in 1956ms (Views: 1.1ms | ActiveRecord: 7.9ms)

==> httpd/access_log <== 139.147.165.244 - - [12/Dec/2013:08:51:17
-0500] “GET /unattended/provision HTTP/1.0” 500 127 “-” “Wget/1.12
(linux-gnu)” ^C [root@masterofpuppet PROD log S:E]#

On 12/12/2013 03:38 AM, Dominic Cleal wrote:

On 11/12/13 21:17, Nathan wrote:

I dont know what I broke, but suddenly, I cant do unattended
provisioning in Foreman 1.3.

I have a host, which I provisioned with Foreman, that recently
got updated to RHEL 6.5. This is machine that I use for
testing, and i was testing some new kickstart stuff and
re-provisioned it. I had issued with RHEL 6.5 not yet being
setup as an OS in foreman, so i took care of that, and found
that builds would fail. anaconda gets to the point where it’s
trying to get http://my.foreman.url/unattended/provision, and
then it reports that it couldn’t get the kickstart script. I
check foreman’s /var/log/foreman/production.log, and i see
this:

==> foreman/production.log <== Rendered inline template
(243.2ms) Rendered text template (0.0ms) Completed 500 Internal
Server Error in 2277ms (Views: 0.6ms | ActiveRecord: 7.3ms)

Can you provide more of the log file, this looks to only be a
few lines of the request.

– You received this message because you are subscribed to the
Google Groups “Foreman users” group. To unsubscribe from this group
and stop receiving emails from it, send an email to
foreman-users+unsubscribe@googlegroups.com
mailto:foreman-users%2Bunsubscribe@googlegroups.com. To post to
this group, send email to foreman-users@googlegroups.com
mailto:foreman-users@googlegroups.com. Visit this group at
http://groups.google.com/group/foreman-users. For more options,
visit https://groups.google.com/groups/opt_out.

– You received this message because you are subscribed to the
Google Groups “Foreman users” group. To unsubscribe from this group
and stop receiving emails from it, send an email to
foreman-users+unsubscribe@googlegroups.com. To post to this group,
send email to foreman-users@googlegroups.com. Visit this group at
http://groups.google.com/group/foreman-users. For more options,
visit https://groups.google.com/groups/opt_out.