Node.rb Net::HTTPNotFound Error

I have installed foreman (v1.9.0) on a RHEL 7.1 VM as per the official
documentation.

My environment is not connected to the internet and required downloading
all of the RPMs from the

Foreman site and building my own locally hosted repo

My current environment consists of:

  • 1 x Puppet master/foreman
  • 2 x agents (RHEL 6.5 & 7)

Foreman is configured as the ENC which will service 9 environments (inc
production). A smart proxy has been configured in foreman for the
foreman/puppetmaster.

There were no issues on the agents in generating the CSR and getting it
signed by the puppet master.

When I run the puppet agent on a remote machine with the command

puppet agent --no-daemonize --server <FQDN> --trace

I get the following errors

Warning: Unable to fetch my node definition, but the agent will continue:

Warning: Error 400 on SERVER: Failed to find < agent FQDN> via exec:
Execution of '/etc/puppet/node.rb < agent FQDN>' returned 1:

Running the referenced command on the puppet master

sudo -u puppet /etc/puppet/node.rb <agent FQDN>

Returns the error:

Error retrieving node < agent FQDN> Net::HTTPNotFound

Check Foreman's /var/log/foreman/production.log for more information.

Researching this error on the web "Foreman with puppet node.rb error 404
Not Found
<http://stackoverflow.com/questions/22649563/foreman-with-puppet-node-rb-error-404-not-found>"
provides the standard response of the puppet master is not aware (DNS,
/etc/hosts) of the agent. In my case this is not the issue, as running the
same node.rb command against the puppetmaster returns the same error. A
simple ping test confirms that DNS is correctly setup and the agents are
listed under hosts in Foreman

.

There are no logs generated in /var/log/foreman/production.log (inc debug)
when this fails,

although there are http 404 errors generated in
/var/log/httpd/foreman-ssl_access_ssl.log

< PUPPET MASTER IP> - - [01/Oct/2015:09:53:28 +1000] "POST
/api/hosts/facts /HTTP/1.1" 404 27 "-" "Ruby"

< PUPPET MASTER IP> - - [01/Oct/2015:09:53:28 +1000] "GET /node/< PUPPET
>?format=yml HTTP/1.1" 404 52 "-" "Ruby"

node.rb references foreman.yaml which is correct in relation to the listed
entries I can confirm like:

  • :url:
  • :ssl_ca:
  • :ssl_cert:
  • :ssl_key:
  • etc

It is also the default file generated by the install process with no
changes.

The agents are in foreman, but to achieve that once I configured
puppet.conf and generated/signed the agent certs I ran the command

puppet agent -t <puppet master FQDN>

I don't know if this is related but I cannot get agents imported into
foreman until I manually ran the command

foreman-rake puppet:import:hosts_and_facts

calling one of the hosts in foreman and then clicking on 'YAML' shows
correct information for each agent.

I have even pulled the information out of the certificates using openssl to
confirm that they are correct.

I have run the command

curl -k https://< Puppet Master FQDN>/foreman

This returns the output

<html><body>You are being <a href="https://< Puppet Master FQDN>/foreman/users/login">redirected</a>.</body></html>

This generates the log entry in ~/foreman/production.log

2015-10-01 10:04:57 [app] [I] | | Staged GET "/foreman/" for < PUPPET
MASTER IP> at 2015-10-01 10:04:57 +1000 2015-10-01 10:04:57 [app] [I]
Processing by DashboardController#index as / 2015-10-01 10:04:57 [app]
[I]Redirected to https://< PUPPET MASTER>/foreman/users/login 2015-10-01
10:04:57 [app] [I] Filter chain halted as :require_login rendered or
redirected 2015-10-01 10:04:57 [app] [I]Completed 302 Found in 7ms (Active
Record: 0.6ms)

While in ~/httpd/foreman-ssl_access_ssl.log I get this

< PUPPET MASTER IP> - - [01/Oct/2015:10:05:26 +1000] "GET /foreman/
HTTP/1.1" 302 129 "-" "curl/7.29.0"

I still get the 404 http errors in the same log after the foreman redirect.

So the question to ask is what can I do to the Puppet/Foreman HTTP config
to resolve the 404 errors

>
> There are no logs generated in /var/log/foreman/production.log (inc debug)
> when this fails, although there are http 404 errors generated in
> /var/log/httpd/foreman-ssl_access_ssl.log
>
> < PUPPET MASTER IP> - - [01/Oct/2015:09:53:28 +1000] "POST /api/hosts/facts
> /HTTP/1.1" 404 27 "-" "Ruby"
>
> < PUPPET MASTER IP> - - [01/Oct/2015:09:53:28 +1000] "GET /node/< PUPPET
> >?format=yml HTTP/1.1" 404 52 "-" "Ruby"

That's odd. So it seems node.rb is correctly looking up the host,
making a connection, being given a 404 and thus returning HTTPNotFound
to you becuase of the 404. This would be the place to troubleshoot
further. Is the :url: setting the same as the one in the Apache vhost?
If not, Apache won't route the request to Rails, I don't think.

> I don't know if this is related but I cannot get agents imported into
> foreman until I manually ran the command
>
> foreman-rake puppet:import:hosts_and_facts

It's probably related - the report processor uses a very similar
method to upload report data to foreman (and references the same
foreman.yml file). New reports are normally how Foreman detects new
hosts for creation, which is consistent with the issue you're seeing.

> I have run the command
>
> curl -k https://< Puppet Master FQDN>/foreman
>
> This returns the output
>
> <html><body>You are being <a href="https://< Puppet Master
> >/foreman/users/login">redirected</a>.</body></html>
>
> This generates the log entry in ~/foreman/production.log

So this FQDN is being correctly routed by Apache - is this the same
fqdn as the :url: field in foreman.yml?

Final question; it's hard to be sure as you're redacted a lot of your
data, but there are some suggestions that you're running Foreman in a
sub-directory (http://myserver.foo.com/foreman) rather than at the
root (http://myserver.foo.com) - is that the case?

··· On 6 October 2015 at 11:25, barry wrote:


Greg

This turned out to be the problem, so the foreman.yaml config file
needed updating with the /foreman suffix.

See

for details.

··· On 09/10/15 10:44, Greg Sutcliffe wrote: > On 6 October 2015 at 11:25, barry wrote: >> I have run the command >> >> curl -k https://< Puppet Master FQDN>/foreman >> >> This returns the output >> >> You are being redirected. >> >> This generates the log entry in ~/foreman/production.log > > So this FQDN is being correctly routed by Apache - is this the *same* > fqdn as the :url: field in foreman.yml? > > Final question; it's hard to be sure as you're redacted a lot of your > data, but there are some suggestions that you're running Foreman in a > sub-directory (http://myserver.foo.com/foreman) rather than at the > root (http://myserver.foo.com) - is that the case?


Dominic Cleal
dominic@cleal.org