Enabling LDAP in settings.yaml breaks reports?

Hi,

We have a functional foreman 1.1 server running on CentOS 6.4
If I change
:ldap: true
in /etc/foreman/settings.yaml, I can make foreman authenticate with our
Active Directory servers, which is really cool, but puppet reports no
longer get processed properly, which is not so cool.

This appears to be tied somehow to More / Settings / Auth / require_ssl_puppetmasters,
which is currently true.

We just stumbled into this:
https://groups.google.com/forum/#!msg/foreman-users/ZWdBGe3s37k/bAADUap2qe0J

Do we need to add SSL to both
/etc/foreman/settings.yaml
as well as
/etc/foreman-proxy/settings.yaml

If so, why is simply enabling LDAP triggering this?

Thanks in advance!

It might be in the process of enabling LDAP, you also enabled
authentication? If auth was previously disabled, then any data (i.e.
reports) can be uploaded into Foreman by anyone, which will now be in a
secure mode.

How did you install Foreman? Is it using HTTPS or plain HTTP?

The first thing to do is check /var/log/foreman/production.log when a
report is uploaded at the end of a Puppet run. You should see a block
of logging starting:

Started POST "/reports/create?format=yml" for 192.168.101.10 at Thu Apr
18 18:05:33 +0000 2013

In the middle should be a reason why the request is being denied,
related to the link you found.

What happens by default is Foreman checks the puppetmaster uploading the
report is a known host with a smart proxy on it. It sounds like you
have this already, so it might be the hostname of the registered smart
proxy doesn't match the hostname in the log file (from rDNS).

For completeness, it's also possible to disable this security measure by
disabling the "restrict_registered_puppetmasters" setting in the UI,
however it will make your server remotely vulnerable.

··· On 26/04/13 01:14, John Smith wrote: > Hi, > > We have a functional foreman 1.1 server running on CentOS 6.4 > If I change > :ldap: true > in /etc/foreman/settings.yaml, I can make foreman authenticate with our > Active Directory servers, which is really cool, but puppet reports no > longer get processed properly, which is not so cool. > > This appears to be tied somehow to More / Settings / Auth > / require_ssl_puppetmasters, which is currently true. > > We just stumbled into this: > https://groups.google.com/forum/#!msg/foreman-users/ZWdBGe3s37k/bAADUap2qe0J > > Do we need to add SSL to both > /etc/foreman/settings.yaml > as well as > /etc/foreman-proxy/settings.yaml > > If so, why is simply enabling LDAP triggering this? > > Thanks in advance!


Dominic Cleal
Red Hat Engineering

>
> How did you install Foreman? Is it using HTTPS or plain HTTP?
>
> The first thing to do is check /var/log/foreman/production.log when a
> report is uploaded at the end of a Puppet run. You should see a block
> of logging starting:
>
> Started POST "/reports/create?format=yml" for 192.168.101.10 at Thu Apr
> 18 18:05:33 +0000 2013
>
> In the middle should be a reason why the request is being denied,
> related to the link you found.
>
> What happens by default is Foreman checks the puppetmaster uploading the
> report is a known host with a smart proxy on it. It sounds like you
> have this already, so it might be the hostname of the registered smart
> proxy doesn't match the hostname in the log file (from rDNS).
>
> For completeness, it's also possible to disable this security measure by
> disabling the "restrict_registered_puppetmasters" setting in the UI,
> however it will make your server remotely vulnerable.
>
> –
> Dominic Cleal
> Red Hat Engineering
>

We're using SSL.
We appear to have the exact same problem as Antonio Xanxess in this post

https://groups.google.com/forum/?fromgroups=#!topic/foreman-users/eRkKDkaX8RA

We did not have a reverse DNS entry for this server.
However, adding a reverse DNS entry for this server has not fixed the
problem.
Setting restrict_registered_puppetmasters to false allows reports to
process, but we don't want to do that.
The only thing I can think of… the FQDN for the server is
hq-puppet-01.domain.com
The URL for the proxy is
https://hq-puppet-01.domain.com:8443
We have cnames for both foreman and puppet pointing to the FQDN
hq-puppet-01.domain.com
It is possible that somewhere foreman or the proxy is upset that the FQDN
does not match the cname.
Any suggestions where I should try debugging the code to see what's wrong?
Is there a higher level of debugging output available from foreman like
there is with foreman-proxy?

here's the output of our /var/log/foreman/production.log file:
Started POST "/reports/create?format=yml" for 10.224.98.30 at Wed May 01
07:47:30 -0700 2013
Processing by ReportsController#create as YML
Parameters: {"report"=>"[FILTERED]"}
No SSL cert with CN supplied - request from 10.224.98.30,
Redirected to https://foreman.domain.com/users/login
Completed 403 Forbidden in 14ms

dig -x 10.224.98.30

[snip]
;; ANSWER SECTION:
30.98.224.10.in-addr.arpa. 3600 IN PTR hq-puppet-01.domain.com.
[snip]

here's my /etc/foreman/settings.yaml
[snip]

··· --- :modulepath: /etc/puppet/modules/ :tftppath: /opt/anaconda/tftpboot :ldap: true :puppet_server: puppet :unattended: true :document_root: /usr/share/foreman/public :foreman_url: foreman.domain.com :puppetconfdir: /etc/puppet/puppet.conf :login: true :require_ssl: true [snip]

here’s my /etc/foreman-proxy/settings.yaml
[snip]

:ssl_certificate: /var/lib/puppet/ssl/certs/hq-puppet-01.domain.com.pem
:ssl_ca_file: /var/lib/puppet/ssl/ca/ca_crt.pem
:ssl_private_key:
/var/lib/puppet/ssl/private_keys/hq-puppet-01.domain.com.pem
:trusted_hosts:

  • hq-puppet-01.domain.com
  • foreman.domain.com
    :daemon: true
    :daemon_pid: /var/run/foreman-proxy/foreman-proxy.pid
    :port: 8443
    :tftp: true
    :tftproot: /opt/anaconda/tftpboot
    :tftp_servername: hq-puppet-01.domain.com
    :dns: false
    :dhcp: true
    :dhcp_vendor: isc
    :dhcp_config: /etc/dhcp/dhcpd.conf
    :dhcp_leases: /var/lib/dhcpd/dhcpd.leases
    :dhcp_key_name: omapi_key
    :dhcp_key_secret: “secret”
    :puppetca: true
    :ssldir: /var/lib/puppet/ssl
    :puppetdir: /etc/puppet
    :puppet: true
    :puppet_conf: /etc/puppet/puppet.conf
    :bmc: false
    :log_file: /var/log/foreman-proxy/proxy.log
    :log_level: DEBUG
    [snip]

Thanks for all your help!

This error pointed me to smart_proxy_auth.rb
Unless I'm missing something, the end of smart_proxy_auth.rb is not quite
ideal.
I see this line
return false unless request_hosts
but after it, we've got debugging statements which won't get executed if
request_hosts is false.
This may be why we (and others) are failing silently when reverse DNS is
not set up for the host.

Regardless… now that my reverse DNS problem is corrected, the error
message I'm getting now:

No SSL cert with CN supplied - request from 10.224.98.30,

leads me to look up at this line:
dn = request.env[Setting[:ssl_client_dn_env]]

We don't have a value for ssl_client_dn_env, which is why we get the error
above
In foreman, I see in More / Settings / Auth that ssl_client_dn_env is set
to SSL_CLIENT_S_DN
However, we apparently don't have a value for SSL_CLIENT_S_DN set anywhere.

I do have SSL_CLIENT_S_DN in my /etc/puppet/puppet.conf:

[snip]
[master]
reports = store,log,foreman
#reporturl = http://hq-puppet-01.domain.com:3001/reports/upload
ssl_client_header = SSL_CLIENT_S_DN
ssl_client_verify_header = SSL_CLIENT_VERIFY
[snip]

my /etc/httpd/conf.d/puppet.conf also has SSL_CLIENT_S_DN

[snip]
PassengerHighPerformance on
PassengerMaxPoolSize 12
PassengerPoolIdleTime 1500
PassengerStatThrottleRate 120
RackAutoDetect Off
RailsAutoDetect Off

Listen 8140
<VirtualHost *:8140>
SSLEngine on
SSLProtocol -ALL +SSLv3 +TLSv1
SSLCipherSuite ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP
SSLCertificateFile /var/lib/puppet/ssl/certs/hq-puppet-01.domain.com.pem
SSLCertificateKeyFile
/var/lib/puppet/ssl/private_keys/hq-puppet-01.domain.com.pem
SSLCertificateChainFile /var/lib/puppet/ssl/ca/ca_crt.pem
SSLCACertificateFile /var/lib/puppet/ssl/ca/ca_crt.pem
SSLCARevocationFile /var/lib/puppet/ssl/ca/ca_crl.pem
SSLVerifyClient optional
SSLVerifyDepth 1
SSLOptions +StdEnvVars
RequestHeader set X-SSL-Subject %{SSL_CLIENT_S_DN}e
RequestHeader set X-Client-DN %{SSL_CLIENT_S_DN}e
RequestHeader set X-Client-Verify %{SSL_CLIENT_VERIFY}e
RackAutoDetect On
DocumentRoot /usr/share/puppet/rack/puppetmasterd/public/
<Directory /usr/share/puppet/rack/puppetmasterd/>
Options None
AllowOverride None
Order allow,deny
allow from all
</Directory>
</VirtualHost>
[snip]

So exactly how should we set up SSL_CLIENT_S_DN?

Thanks!!!

> We're using SSL.
> We appear to have the exact same problem as Antonio Xanxess in this post
>
> https://groups.google.com/forum/?fromgroups=#!topic/foreman-users/eRkKDkaX8RA
>
> We did not have a reverse DNS entry for this server.
> However, adding a reverse DNS entry for this server has not fixed the
> problem.
> Setting restrict_registered_puppetmasters to false allows reports to
> process, but we don't want to do that.
> The only thing I can think of… the FQDN for the server is
> hq-puppet-01.domain.com
> The URL for the proxy is
> https://hq-puppet-01.domain.com:8443
> We have cnames for both foreman and puppet pointing to the FQDN
> hq-puppet-01.domain.com
> It is possible that somewhere foreman or the proxy is upset that the
> FQDN does not match the cname.

It's failing before that point, as it's trying to get the DN from the
certificate to identify the host, but it isn't available. Once it's got
the DN, it'll be able to get the hostname and that's where the
URL/hostname comparisons happen.

> Any suggestions where I should try debugging the code to see what's wrong?
> Is there a higher level of debugging output available from foreman like
> there is with foreman-proxy?

Not at that stage, you'd have to add some. Perhaps something like:

logger.info request.env.inspect

Then you could look for any SSL* environment variables that would
provide hints.

> No SSL cert with CN supplied - request from 10.224.98.30,
>
> leads me to look up at this line:
> dn = request.env[Setting[:ssl_client_dn_env]]
>
> We don't have a value for ssl_client_dn_env, which is why we get the
> error above
> In foreman, I see in More / Settings / Auth that ssl_client_dn_env is
> set to SSL_CLIENT_S_DN
> However, we apparently don't have a value for SSL_CLIENT_S_DN set
> anywhere.

So this is meant to be added by Apache with the following statement:

SSLOptions +StdEnvVars

This adds a series of environment variables to the request, which we're
picking up with the request.env lookup in the code you found. The
variable names are standardised, see
http://httpd.apache.org/docs/2.2/mod/mod_ssl.html#envvars. The
puppet.conf setup is slightly different as it converts the environment
variables to headers, but we don't require that setup.

There should only be a couple of reasons for this:

  1. the script doing the upload isn't using certificate and keys, I can't
    see the config for that in your e-mails (this isn't the proxy config,
    it's the foreman.rb for reporting, or node.rb for ENC).

  2. maybe SSL verification is failing, check Apache error logs? The
    debug I suggested at the top may also help, there could be other hints
    in the environment variables.

Cheers,

··· On 01/05/13 16:45, John Smith wrote:


Dominic Cleal
Red Hat Engineering

>
> 1. the script doing the upload isn't using certificate and keys, I can't
> see the config for that in your e-mails (this isn't the proxy config,
> it's the foreman.rb for reporting, or node.rb for ENC).
>
> 2. maybe SSL verification is failing, check Apache error logs? The
> debug I suggested at the top may also help, there could be other hints
> in the environment variables.
>
> Cheers,
>
> –
> Dominic Cleal
> Red Hat Engineering
>

Dominic, you have once again saved the day.

Over a year ago I hacked our puppet reports foreman.rb in /usr/lib/ruby/site_ruby/1.8/puppet/reports/foreman.rb
to make it work with ssl - at the time this was not easy, at least for me.
I never updated it until today follow the fine manual here

http://theforeman.org/manuals/1.1/index.html#3.5.4PuppetReports
https://raw.github.com/theforeman/puppet-foreman/master/templates/foreman-report.rb.erb

Updating foreman and puppet via RPMs never updated the foreman reports
processor.

I also updated my /etc/httpd/conf.d/foreman.conf, I didn't have SSLOptions
+StdEnvVars properly defined there like I did in our
/etc/httpd/conf.d/puppet.conf

I am amazed that our SSL configuration worked at all, and that somehow
setting LDAP to true in /etc/foreman/settings.yaml broke things seemingly
unrelated so badly.

Thanks again Dominic!