Foreman-proxy help

Dominic_Cleal1 · February 8, 2013, 9:43am

> I could use some help trying to get foreman-proxy working.
>
> I've got a puppet-3.0.1-1 running on a CentOS 6.3 server. I used Mark
> Stanislavï¿½s excellent post to get puppet running
>
> http://www.uncompiled.com/centos-6-puppet-27-mcollective-foreman-rabbit
>
> # rpm -qa | grep puppet
> puppet-dashboard-1.2.10-1.el6.noarch
> puppet-server-3.0.1-1.el6.noarch
> puppet-3.0.1-1.el6.noarch
> #
>
> I'm using passengerï¿½
>
> # rpm -qa | grep passenger
> rubygem-passenger-native-libs-3.0.12-1.el6_1.8.7.352.x86_64
> passenger-release-3-6.el6.noarch
> rubygem-passenger-native-3.0.12-1.el6.x86_64
> mod_passenger-3.0.12-1.el6.x86_64
> rubygem-passenger-3.0.12-1.el6.x86_64
> #
>
> Iï¿½ve installed foreman from rpm as well
>
> # rpm -qa | grep foreman
> foreman-1.1stable-1.el6.noarch
> foreman-proxy-1.1stable-1.el6.noarch
> foreman-vmware-1.1stable-1.el6.noarch
> foreman-ec2-1.1stable-1.el6.noarch
> #
>
> Here are my gems:
[snip]
>
> My problem is foreman-proxy doesnï¿½t seem to actually do anything. It
> wonï¿½t even write to the log file specified in settings.yml. If I put a
> typo in settings.yml, foreman-proxy complained when I restarted, so I
> know the file is being read. I put a large number of warning statements
> in my code to figure out what was going on.
[snip]
> Now when I run foreman-proxy, at least it creates the log file specified
> in settings.yml. It still wonï¿½t do anything else meaningful ï¿½ Iï¿½m
> trying to get DHCP working first.

It doesn't do anything by itself - it'll just listen on port 8443 until
instructed to do something. How are you trying to use it?

Typically you'd start up the proxy, then go into the Foreman UI, More,
Smart Proxies and then add it by URL. You should then see a number of
features listed, such as Puppet, PuppetCA, DHCP, depending on what's
configured in the proxy's settings.yml.

You can also query it directly yourself, e.g.

curl -H "Accept: application/json" http://localhost:8443/features
["dhcp","dns","puppet","puppetca","tftp"]

Once you've added the proxy to Foreman's UI, you can select it in your
configuration to handle certain tasks. e.g. under Domains, you could
select the proxy for DNS if it provides the DNS feature and manages that
domain. For DHCP, you'd be able to select it under the Subnet if it
supported the DHCP feature.

> I donï¿½t know ruby so I have no idea whatï¿½s going on.
> Related dumb question - does foreman-proxy need to be run as passenger?
> If so, how do I do this? I see a config.ru, but I donï¿½t see any
> example configurations for /etc/httpd/conf.d

That's not necessary, but some people might decide to deploy it that way
if they're comfortable with Rack applications. Most users should deploy
it as a standalone service using the package, which runs a WEBrick
daemon with no httpd or passenger required.

We use passenger for the Foreman UI itself, as it scales better,
Apache's mod_ssl works better etc.

> Iï¿½ve tried running the foreman installer on this server, it didnï¿½t seem
> to do anything.
> The server is a VM with a snapshot so I can easily back out changes like
> that.
> Iï¿½ve tried various versions of foreman and the proxy, nothing has helped.

The installer should just set up the package and configuration
(settings.yml) if you apply the foreman_proxy class, then you're able to
use the proxy from Foreman.

···

On 08/02/13 00:26, John Smith wrote:

–
Dominic Cleal
Red Hat Engineering

John_Smith · February 8, 2013, 10:30pm

I get this error:

curl -H "Accept: application/json" http://localhost:8443/features

curl: (56) Failure when receiving data from the peer

···

On Friday, February 8, 2013 1:43:37 AM UTC-8, Dominic Cleal wrote: > > On 08/02/13 00:26, John Smith wrote: > > I could use some help trying to get foreman-proxy working. > > > > I've got a puppet-3.0.1-1 running on a CentOS 6.3 server. I used Mark > > Stanislavï¿½s excellent post to get puppet running > > > > http://www.uncompiled.com/centos-6-puppet-27-mcollective-foreman-rabbit > > > > # rpm -qa | grep puppet > > puppet-dashboard-1.2.10-1.el6.noarch > > puppet-server-3.0.1-1.el6.noarch > > puppet-3.0.1-1.el6.noarch > > # > > > > I'm using passengerï¿½ > > > > # rpm -qa | grep passenger > > rubygem-passenger-native-libs-3.0.12-1.el6_1.8.7.352.x86_64 > > passenger-release-3-6.el6.noarch > > rubygem-passenger-native-3.0.12-1.el6.x86_64 > > mod_passenger-3.0.12-1.el6.x86_64 > > rubygem-passenger-3.0.12-1.el6.x86_64 > > # > > > > Iï¿½ve installed foreman from rpm as well > > > > # rpm -qa | grep foreman > > foreman-1.1stable-1.el6.noarch > > foreman-proxy-1.1stable-1.el6.noarch > > foreman-vmware-1.1stable-1.el6.noarch > > foreman-ec2-1.1stable-1.el6.noarch > > # > > > > Here are my gems: > [snip] > > > > My problem is foreman-proxy doesnï¿½t seem to actually do anything. It > > wonï¿½t even write to the log file specified in settings.yml. If I put > a > > typo in settings.yml, foreman-proxy complained when I restarted, so I > > know the file is being read. I put a large number of warning statements > > in my code to figure out what was going on. > [snip] > > Now when I run foreman-proxy, at least it creates the log file specified > > in settings.yml. It still wonï¿½t do anything else meaningful ï¿½ Iï¿½m > > trying to get DHCP working first. > > It doesn't do anything by itself - it'll just listen on port 8443 until > instructed to do something. How are you trying to use it? > > Typically you'd start up the proxy, then go into the Foreman UI, More, > Smart Proxies and then add it by URL. You should then see a number of > features listed, such as Puppet, PuppetCA, DHCP, depending on what's > configured in the proxy's settings.yml. > > You can also query it directly yourself, e.g. > > curl -H "Accept: application/json" http://localhost:8443/features > #

[“dhcp”,“dns”,“puppet”,“puppetca”,“tftp”]

Once you’ve added the proxy to Foreman’s UI, you can select it in your
configuration to handle certain tasks. e.g. under Domains, you could
select the proxy for DNS if it provides the DNS feature and manages that
domain. For DHCP, you’d be able to select it under the Subnet if it
supported the DHCP feature.

if I start foreman-proxy and use netstat -a I can see that it is listening
on post 8443. However, that’s the extent of my ‘success’. If I goto
http://servername:8443/features, I get this response: “Error 101
(net::ERR_CONNECTION_RESET): The connection was reset.”

Because of the 101 response, I also can’t add a new proxy within foreman -
it complains that it can’t talk to the proxy either (since it trying to to
do same thing I’m doing with a web browser)

How do I enable more debugging?
I get zero useful logs anywhere on the server for this to tell me what I’ve
done wrong.
I am using passenger for both puppet and foreman.
I assume I am using webrick for foreman-proxy

Thanks,
JS

I donï¿½t know ruby so I have no idea whatï¿½s going on.
Related dumb question - does foreman-proxy need to be run as passenger?
If so, how do I do this? I see a config.ru, but I donï¿½t see any
example configurations for /etc/httpd/conf.d

That’s not necessary, but some people might decide to deploy it that way
if they’re comfortable with Rack applications. Most users should deploy
it as a standalone service using the package, which runs a WEBrick
daemon with no httpd or passenger required.

We use passenger for the Foreman UI itself, as it scales better,
Apache’s mod_ssl works better etc.

Iï¿½ve tried running the foreman installer on this server, it didnï¿½t
seem
to do anything.
The server is a VM with a snapshot so I can easily back out changes like
that.
Iï¿½ve tried various versions of foreman and the proxy, nothing has
helped.

The installer should just set up the package and configuration
(settings.yml) if you apply the foreman_proxy class, then you’re able to
use the proxy from Foreman.

–
Dominic Cleal
Red Hat Engineering

Dominic_Cleal1 · February 8, 2013, 10:45pm

Both these errors look like the proxy's configured for HTTPS and the
three ssl* options in /etc/foreman-proxy/settings.yml have been set. In
that case, use https:// URLS throughout. When testing with curl, you
may need to add -k to ignore the certificate.

···

On 08/02/13 22:30, John Smith wrote: > > On Friday, February 8, 2013 1:43:37 AM UTC-8, Dominic Cleal wrote: > > On 08/02/13 00:26, John Smith wrote: > > I could use some help trying to get foreman-proxy working. > > > > I've got a puppet-3.0.1-1 running on a CentOS 6.3 server. I used > Mark > > Stanislavï¿½s excellent post to get puppet running > > > > > http://www.uncompiled.com/centos-6-puppet-27-mcollective-foreman-rabbit > > > > > > # rpm -qa | grep puppet > > puppet-dashboard-1.2.10-1.el6.noarch > > puppet-server-3.0.1-1.el6.noarch > > puppet-3.0.1-1.el6.noarch > > # > > > > I'm using passengerï¿½ > > > > # rpm -qa | grep passenger > > rubygem-passenger-native-libs-3.0.12-1.el6_1.8.7.352.x86_64 > > passenger-release-3-6.el6.noarch > > rubygem-passenger-native-3.0.12-1.el6.x86_64 > > mod_passenger-3.0.12-1.el6.x86_64 > > rubygem-passenger-3.0.12-1.el6.x86_64 > > # > > > > Iï¿½ve installed foreman from rpm as well > > > > # rpm -qa | grep foreman > > foreman-1.1stable-1.el6.noarch > > foreman-proxy-1.1stable-1.el6.noarch > > foreman-vmware-1.1stable-1.el6.noarch > > foreman-ec2-1.1stable-1.el6.noarch > > # > > > > Here are my gems: > [snip] > > > > My problem is foreman-proxy doesnï¿½t seem to actually do > anything. It > > wonï¿½t even write to the log file specified in settings.yml. If > I put a > > typo in settings.yml, foreman-proxy complained when I restarted, so I > > know the file is being read. I put a large number of warning > statements > > in my code to figure out what was going on. > [snip] > > Now when I run foreman-proxy, at least it creates the log file > specified > > in settings.yml. It still wonï¿½t do anything else meaningful ï¿½ > Iï¿½m > > trying to get DHCP working first. > > It doesn't do anything by itself - it'll just listen on port 8443 until > instructed to do something. How are you trying to use it? > > Typically you'd start up the proxy, then go into the Foreman UI, More, > Smart Proxies and then add it by URL. You should then see a number of > features listed, such as Puppet, PuppetCA, DHCP, depending on what's > configured in the proxy's settings.yml. > > You can also query it directly yourself, e.g. > > curl -H "Accept: application/json" http://localhost:8443/features > > I get this error: > > # curl -H "Accept: application/json" http://localhost:8443/features > curl: (56) Failure when receiving data from the peer > # > > ["dhcp","dns","puppet","puppetca","tftp"] > > Once you've added the proxy to Foreman's UI, you can select it in your > configuration to handle certain tasks. e.g. under Domains, you could > select the proxy for DNS if it provides the DNS feature and manages > that > domain. For DHCP, you'd be able to select it under the Subnet if it > supported the DHCP feature. > > if I start foreman-proxy and use netstat -a I can see that it is > listening on post 8443. However, that's the extent of my 'success'. If > I goto http://servername:8443/features, I get this response: "Error 101 > (net::ERR_CONNECTION_RESET): The connection was reset."

–
Dominic Cleal
Red Hat Engineering

John_Smith · February 8, 2013, 11:54pm

>
> Both these errors look like the proxy's configured for HTTPS and the
> three ssl* options in /etc/foreman-proxy/settings.yml have been set. In
> that case, use https:// URLS throughout. When testing with curl, you
> may need to add -k to ignore the certificate.
>
> –
> Dominic Cleal
> Red Hat Engineering
>

I am even more confused, and I didn't think that was possible …

If I run
curl -k https://localhost:8443/features
I get a reply with 2957 lines… here are a few relevant lines:

/usr/share/foreman-proxy/bin/…/lib/helpers.rb in log_halt
logger.error message
/usr/share/foreman-proxy/bin/…/lib/smart_proxy.rb in nil
log_halt 403, "Untrusted client
#{request.env["REMOTE_HOST"].downcase} attempted to access
#{request.path_info}. Check :trusted_hosts: in settings.yml"
/usr/lib/ruby/1.8/webrick/httpserver.rb in service
si.service(req, res)
/usr/lib/ruby/1.8/webrick/httpserver.rb in run
server.service(req, res)
/usr/lib/ruby/1.8/webrick/server.rb in start_thread
block ? block.call(sock) : run(sock)
/usr/lib/ruby/1.8/webrick/server.rb in start
Thread.start{
/usr/lib/ruby/1.8/webrick/server.rb in start_thread
Thread.start{
/usr/lib/ruby/1.8/webrick/server.rb in start
th = start_thread(sock, &block)
/usr/lib/ruby/1.8/webrick/server.rb in each
svrs[0].each{|svr|
/usr/lib/ruby/1.8/webrick/server.rb in start
svrs[0].each{|svr|
/usr/lib/ruby/1.8/webrick/server.rb in start
yield
/usr/lib/ruby/1.8/webrick/server.rb in start
server_type.start{
/usr/share/foreman-proxy/bin/smart-proxy in nil
SmartProxy.run!()

I've already added my host to trust_hosts - here is my settings.yml
foreman, puppet, and hq-puppet-01 are all different DNS names for my server
that is running puppet, foreman, and foreman-proxy

···

--- :ssl_certificate: /var/lib/puppet/ssl/certs/hq-puppet-01.domain.com.pem :ssl_ca_file: /var/lib/puppet/ssl/certs/ca.pem :ssl_private_key: /var/lib/puppet/ssl/private_keys/hq-puppet-01.domain.com.pem :trusted_hosts: - foreman.domain.com - puppet.domain.com - hq-puppet-01.domain.com - localhost - my PC's local IP address :daemon: true :daemon_pid: /var/run/foreman-proxy/foreman-proxy.pid :port: 8443 :tftp: false :dns: false :dhcp: true :dhcp_vendor: isc :dhcp_config: /etc/dhcp/dhcpd.conf :dhcp_leases: /var/lib/dhcpd/dhcpd.leases :dhcp_key_name: omapi-key :dhcp_key_secret: "key" :puppetca: true :ssldir: /var/lib/puppet/ssl :puppetdir: /etc/puppet :puppet: true :puppet_conf: /etc/puppet/puppet.conf :bmc: false :log_file: /var/log/foreman-proxy/proxy.log :log_level: DEBUG

If I remove the 3 SSL lines from settings.yml and restart foreman-proxy, I
get a proper curl response:

curl http://localhost:8443/features

DHCP Web interface

DHCP Browser

Home

Features

Suported features

<li>dhcp</li>

<li>puppet</li>

<li>puppetca</li>

  </div>
  <div id='footer'>
    <p> &copy; 2010 <a href='mailto:ohadlevy@gmail.com'> Ohad Levy </a>

  </div>
</div>

However, even with the ssl lines removed from settings.yml, I get the same
bad response from a browser when I try to go to http://server:8443/features
from my PC.
I’ve tried multiple browsers, same problem

log_halt 403, “Untrusted client #{request.env[“REMOTE_HOST”].downcase}
attempted to access #{request.path_info}. Check :trusted_hosts: in
settings.yml”

So I don’t understand why my specified trusted hosts aren’t working in SSL
mode, nor do I uinderstand why I can curl successfully from the localhost
without SSL but can’t access the same port from my PC…

Thanks,
JS

John_Smith · February 11, 2013, 12:27am

> However, even with the ssl lines removed from settings.yml, I get the same
> bad response from a browser when I try to go to
> http://server:8443/features from my PC.
> I've tried multiple browsers, same problem
>
> log_halt 403, "Untrusted client #{request.env["REMOTE_HOST"].downcase}
> attempted to access #{request.path_info}. Check :trusted_hosts: in
> settings.yml"
>
> So I don't understand why my specified trusted hosts aren't working in SSL
> mode, nor do I uinderstand why I can curl successfully from the localhost
> without SSL but can't access the same port from my PC…
>
> Thanks,
> JS
>

I've made some progress. I modified my /etc/puppet/puppet.conf file to add
these lines per the very very recent documentation update here

https://github.com/theforeman/theforeman.org/pull/12

[main]
privatekeydir = $ssldir/private_keys { group = service }
hostprivkey = $privatekeydir/$certname.pem { mode = 640 }

I've added the SSL lines back in to /etc/foreman-proxy/settings.yml
I've also remove all trusted_hosts from /etc/foreman-proxy/settings.yml

I can now successfully access
https://servername.domain.com:8443/features

from any PC. It even tells me about the DHCP server I have configured
locally. Finally some progress

However, I still can't add the proxy to foreman. When I attempt to add my
dhcp proxy, I get this:

Unable to save
Unable to communicate with the proxy: Connection refused - connect(2)
Please check the proxy is configured and running on the host before saving.

Apparently all the error messages are canned and templated, so I can't even
determine what code is breaking and why.
Adding my hosts back into trusted_hosts doesn't help

About the only useful documentation for this error is here:

this doesn't seem to apply to me, I can su to both foreman and
foreman-proxy and read the private keys.
both my foreman and foreman-proxy users are members of the puppet group

Dominic_Cleal1 · February 11, 2013, 8:57am

It is a generic TCP error message, but there's not much more info to
provide. Can you access that URL from the Foreman host?

What URL exactly are you entering in Foreman? It should be:
https://servername.domain.com:8443

My first guess would be that either the port is incorrect or there's a
firewall blocking access.

···

On 11/02/13 00:27, John Smith wrote: > I can now successfully access > https://servername.domain.com:8443/features > > from any PC. It even tells me about the DHCP server I have configured > locally. Finally some progress :) > > However, I still can't add the proxy to foreman. When I attempt to add > my dhcp proxy, I get this: > > Unable to save > Unable to communicate with the proxy: Connection refused - connect(2) > Please check the proxy is configured and running on the host before saving. > > Apparently all the error messages are canned and templated, so I can't > even determine what code is breaking and why. > Adding my hosts back into trusted_hosts doesn't help

–
Dominic Cleal
Red Hat Engineering

John_Smith · February 11, 2013, 5:09pm

I finally figured it out. I had an old incorrect entry in /etc/hosts for
this server, that was causing the above error message. I had to use
tcpdump to figure out what was really happening, tcpdump pointed me to the
old IP address of this server, which lead me to my out of date /etc/hosts
entry.

After that, i was still getting a 403 forbidden error along with
corresponding entries in /var/log/foreman-proxy/proxy.log telling me about
an untrusted client. That client was the current IP address of the foreman
/ puppet server. I had to re-add the entry back into /etc/hosts with the
current IP address of the puppet server before I was able to successfully
create a DHCP proxy. I checked all the DNS settings for this server, they
were correct but if hardcoding things in /etc/hosts is what it takes to
make things work I can live with that

thanks Dominic for all the assistance!!!

JS

···

On Monday, February 11, 2013 12:57:05 AM UTC-8, Dominic Cleal wrote:

It is a generic TCP error message, but there’s not much more info to
provide. Can you access that URL from the Foreman host?

What URL exactly are you entering in Foreman? It should be:
https://servername.domain.com:8443

My first guess would be that either the port is incorrect or there’s a
firewall blocking access.

–
Dominic Cleal
Red Hat Engineering

Dominic_Cleal1 · February 11, 2013, 5:16pm

>
>
> It is a generic TCP error message, but there's not much more info to
> provide. Can you access that URL from the Foreman host?
>
> What URL exactly are you entering in Foreman? It should be:
> https://servername.domain.com:8443
> <https://servername.domain.com:8443>
>
> My first guess would be that either the port is incorrect or there's a
> firewall blocking access.
>
> –
> Dominic Cleal
> Red Hat Engineering
>
>
> I finally figured it out. I had an old incorrect entry in /etc/hosts
> for this server, that was causing the above error message. I had to use
> tcpdump to figure out what was really happening, tcpdump pointed me to
> the old IP address of this server, which lead me to my out of date
> /etc/hosts entry.
>
> After that, i was still getting a 403 forbidden error along with
> corresponding entries in /var/log/foreman-proxy/proxy.log telling me
> about an untrusted client. That client was the current IP address of
> the foreman / puppet server. I had to re-add the entry back into
> /etc/hosts with the current IP address of the puppet server before I was
> able to successfully create a DHCP proxy. I checked all the DNS
> settings for this server, they were correct but if hardcoding things in
> /etc/hosts is what it takes to make things work I can live with that

I have a "trusted hosts" list feature I need to finish off which might
help you in the future, which would allow you to trust a particular
hostname without needing to have rDNS exactly match the proxy URLs.

> thanks Dominic for all the assistance!!!

You're welcome!

···

On 11/02/13 17:09, John Smith wrote: > On Monday, February 11, 2013 12:57:05 AM UTC-8, Dominic Cleal wrote:

–
Dominic Cleal
Red Hat Engineering