Generating SSL keys for foreman/ access to Foreman proxies in an "HA" setup

A_Non_E_Mouse6996 · October 25, 2016, 5:39pm

Hi,

I'm working on moving from a standalone Foreman host to a clustered setup
similar to Foreman :: Journey to High Availability
and with the help of the docs have got most things working with clustered
Postgres, Foreman hosts communicating with memached, Foreman machines
behind haproxy, common keys for database and cookies and so on.

Foreman-installer command used below for info.

I'm having trouble working out how to generate SSL certificates for the
Foreman servers to communicate with the proxies without running into "alert
unknown CA" and similar problems.

I've tried generating certs on the first foreman server for the other hosts
with
puppet cert generate [fqdn]
and pushing the ca.pem and the generated certs and private keys to the
other hosts.

I seem to be missing a step - the machine where the certs were generated
can talk to the locally install smart proxy but communication with the
smart proxy installed on the other hosts fails with unknown CA errors.

Testing with
openssl s_client

-connect $(hostname -f):443 \

-cert /var/lib/puppet/ssl/certs/$(hostname -f).pem \

-key /var/lib/puppet/ssl/private_keys/$(hostname -f).pem \

-CAfile /var/lib/puppet/ssl/certs/ca.pem

to try to get some more info tells gives "Verify return code: 19 (self
signed cert in certificate chain)" - but I see this on the machine which
can talk to its smart proxy too

Is the best way to generate the certs documented anywhere (whether with
puppet, certtool or openssl)?

Is it best practice to use the same cert for Apache for all of the machines
behind haproxy and to use separate keys for foreman-proxy and to specify
them with --foreman-client-ssl-* and --foreman-proxy-ssl-*?

Many thanks for any pointers,
Tim

** Current foreman-installer command

foreman-installer -v \

--enable-foreman \

--enable-foreman-cli \

--enable-foreman-proxy \

--foreman-proxy-trusted-hosts=&quot;foreman.example.com&quot; \

--foreman-proxy-trusted-hosts=&quot;foreman-01.example.com&quot; \

--foreman-proxy-trusted-hosts=&quot;foreman-02.example.com&quot; \

--foreman-proxy-trusted-hosts=&quot;foreman-03.example.com&quot; \

--foreman-proxy-trusted-hosts=&quot;foreman-04.example.com&quot; \

--foreman-admin-password=&quot;foo&quot; \

--foreman-oauth-consumer-key=&quot;foo&quot; \

--foreman-oauth-consumer-secret=&quot;foo&quot; \

--enable-foreman-plugin-bootdisk \

--enable-foreman-plugin-chef \

--enable-foreman-plugin-discovery \

--enable-foreman-plugin-hooks \

--enable-foreman-plugin-host-extra-validator \

--enable-foreman-plugin-memcache \

--foreman-plugin-memcache-hosts=&quot;memcached-01.example.com&quot; \

--foreman-plugin-memcache-hosts=&quot;memcached-02.example.com&quot; \

--foreman-plugin-memcache-hosts=&quot;memcached-03.example.com&quot; \

--foreman-plugin-memcache-hosts=&quot;memcached-04.example.com&quot; \

--enable-foreman-plugin-remote-execution \

--enable-foreman-plugin-salt \

--enable-foreman-plugin-setup \

--enable-foreman-plugin-tasks \

--enable-foreman-plugin-templates \

--enable-foreman-compute-openstack \

--enable-foreman-compute-vmware \

--foreman-db-database=&quot;foo&quot; \

--foreman-db-host=&quot;foo&quot; \

--foreman-db-manage=&quot;false&quot; \

--foreman-db-password=&quot;foo&quot; \

--foreman-db-port=&quot;4344&quot; \

--foreman-db-type=&quot;postgresql&quot; \

--foreman-db-username=&quot;foo&quot;

discr33t · October 25, 2016, 11:56pm

Tim,

Are your smart-proxies clustered as well? If so are you putting the same
certificate on each smart-proxy and referencing a url such as
puppetserver.domain.com or puppetca.domain.com? Also how are you handling
your SSL? Are you offloading at the load balancer or doing passthrough to
the backend server?

-Chris
IRC: discr33t

···

On Tuesday, October 25, 2016 at 3:00:42 PM UTC-4, a.non.e....@gmail.com wrote: > > Hi, > > I'm working on moving from a standalone Foreman host to a clustered setup > similar to > https://theforeman.org/2015/12/journey_to_high_availability.html and with > the help of the docs have got most things working with clustered Postgres, > Foreman hosts communicating with memached, Foreman machines behind haproxy, > common keys for database and cookies and so on. > > Foreman-installer command used below for info. > > I'm having trouble working out how to generate SSL certificates for the > Foreman servers to communicate with the proxies without running into "alert > unknown CA" and similar problems. > > I've tried generating certs on the first foreman server for the other > hosts with > puppet cert generate [fqdn] > and pushing the ca.pem and the generated certs and private keys to the > other hosts. > > I seem to be missing a step - the machine where the certs were generated > can talk to the locally install smart proxy but communication with the > smart proxy installed on the other hosts fails with unknown CA errors. > > Testing with > openssl s_client > > -connect $(hostname -f):443 \ > > -cert /var/lib/puppet/ssl/certs/$(hostname -f).pem \ > > -key /var/lib/puppet/ssl/private_keys/$(hostname -f).pem \ > > -CAfile /var/lib/puppet/ssl/certs/ca.pem > to try to get some more info tells gives "Verify return code: 19 (self > signed cert in certificate chain)" - but I see this on the machine which > can talk to its smart proxy too :( > > Is the best way to generate the certs documented anywhere (whether with > puppet, certtool or openssl)? > > Is it best practice to use the same cert for Apache for all of the > machines behind haproxy and to use separate keys for foreman-proxy and to > specify them with --foreman-client-ssl-* and --foreman-proxy-ssl-*? > > Many thanks for any pointers, > Tim > > ** Current foreman-installer command > > foreman-installer -v \ > > --enable-foreman \ > > --enable-foreman-cli \ > > --enable-foreman-proxy \ > > --foreman-proxy-trusted-hosts="foreman.example.com" \ > > --foreman-proxy-trusted-hosts="foreman-01.example.com" \ > > --foreman-proxy-trusted-hosts="foreman-02.example.com" \ > > --foreman-proxy-trusted-hosts="foreman-03.example.com" \ > > --foreman-proxy-trusted-hosts="foreman-04.example.com" \ > > --foreman-admin-password="foo" \ > > --foreman-oauth-consumer-key="foo" \ > > --foreman-oauth-consumer-secret="foo" \ > > --enable-foreman-plugin-bootdisk \ > > --enable-foreman-plugin-chef \ > > --enable-foreman-plugin-discovery \ > > --enable-foreman-plugin-hooks \ > > --enable-foreman-plugin-host-extra-validator \ > > --enable-foreman-plugin-memcache \ > > --foreman-plugin-memcache-hosts="memcached-01.example.com" \ > > --foreman-plugin-memcache-hosts="memcached-02.example.com" \ > > --foreman-plugin-memcache-hosts="memcached-03.example.com" \ > > --foreman-plugin-memcache-hosts="memcached-04.example.com" \ > > --enable-foreman-plugin-remote-execution \ > > --enable-foreman-plugin-salt \ > > --enable-foreman-plugin-setup \ > > --enable-foreman-plugin-tasks \ > > --enable-foreman-plugin-templates \ > > --enable-foreman-compute-openstack \ > > --enable-foreman-compute-vmware \ > > --foreman-db-database="foo" \ > > --foreman-db-host="foo" \ > > --foreman-db-manage="false" \ > > --foreman-db-password="foo" \ > > --foreman-db-port="4344" \ > > --foreman-db-type="postgresql" \ > > --foreman-db-username="foo" >

A_Non_E_Mouse6996 · October 26, 2016, 12:12pm

Thanks for the reply Chris.

The smart-proxies are not clustered / presented behind a common URL at the
moment. They're in foreman as individually named and addressed hosts and
each currently has certs with the machine name.

Foreman itself has SSL passed through from haproxy.

I think I need to setup
–foreman-proxy-foreman-ssl-ca
to point to a ca chain or similar that covers the machines?

Busily relearning a lot of SSL and I'm wondering if I'm making silly errors
generating certificates and making assumptions about operations.

Tim

···

On Wednesday, 26 October 2016 00:56:12 UTC+1, Christopher Pisano wrote: > > Tim, > > Are your smart-proxies clustered as well? If so are you putting the same > certificate on each smart-proxy and referencing a url such as > puppetserver.domain.com or puppetca.domain.com? Also how are you handling > your SSL? Are you offloading at the load balancer or doing passthrough to > the backend server? > > -Chris > IRC: discr33t > > On Tuesday, October 25, 2016 at 3:00:42 PM UTC-4, a.non.e....@gmail.com > wrote: >> >> Hi, >> >> I'm working on moving from a standalone Foreman host to a clustered setup >> similar to >> https://theforeman.org/2015/12/journey_to_high_availability.html and >> with the help of the docs have got most things working with clustered >> Postgres, Foreman hosts communicating with memached, Foreman machines >> behind haproxy, common keys for database and cookies and so on. >> >> Foreman-installer command used below for info. >> >> I'm having trouble working out how to generate SSL certificates for the >> Foreman servers to communicate with the proxies without running into "alert >> unknown CA" and similar problems. >> >> I've tried generating certs on the first foreman server for the other >> hosts with >> puppet cert generate [fqdn] >> and pushing the ca.pem and the generated certs and private keys to the >> other hosts. >> >> I seem to be missing a step - the machine where the certs were generated >> can talk to the locally install smart proxy but communication with the >> smart proxy installed on the other hosts fails with unknown CA errors. >> >> Testing with >> openssl s_client >> >> -connect $(hostname -f):443 \ >> >> -cert /var/lib/puppet/ssl/certs/$(hostname -f).pem \ >> >> -key /var/lib/puppet/ssl/private_keys/$(hostname -f).pem \ >> >> -CAfile /var/lib/puppet/ssl/certs/ca.pem >> to try to get some more info tells gives "Verify return code: 19 (self >> signed cert in certificate chain)" - but I see this on the machine which >> can talk to its smart proxy too :( >> >> Is the best way to generate the certs documented anywhere (whether with >> puppet, certtool or openssl)? >> >> Is it best practice to use the same cert for Apache for all of the >> machines behind haproxy and to use separate keys for foreman-proxy and to >> specify them with --foreman-client-ssl-* and --foreman-proxy-ssl-*? >> >> Many thanks for any pointers, >> Tim >> >> ** Current foreman-installer command >> >> foreman-installer -v \ >> >> --enable-foreman \ >> >> --enable-foreman-cli \ >> >> --enable-foreman-proxy \ >> >> --foreman-proxy-trusted-hosts="foreman.example.com" \ >> >> --foreman-proxy-trusted-hosts="foreman-01.example.com" \ >> >> --foreman-proxy-trusted-hosts="foreman-02.example.com" \ >> >> --foreman-proxy-trusted-hosts="foreman-03.example.com" \ >> >> --foreman-proxy-trusted-hosts="foreman-04.example.com" \ >> >> --foreman-admin-password="foo" \ >> >> --foreman-oauth-consumer-key="foo" \ >> >> --foreman-oauth-consumer-secret="foo" \ >> >> --enable-foreman-plugin-bootdisk \ >> >> --enable-foreman-plugin-chef \ >> >> --enable-foreman-plugin-discovery \ >> >> --enable-foreman-plugin-hooks \ >> >> --enable-foreman-plugin-host-extra-validator \ >> >> --enable-foreman-plugin-memcache \ >> >> --foreman-plugin-memcache-hosts="memcached-01.example.com" \ >> >> --foreman-plugin-memcache-hosts="memcached-02.example.com" \ >> >> --foreman-plugin-memcache-hosts="memcached-03.example.com" \ >> >> --foreman-plugin-memcache-hosts="memcached-04.example.com" \ >> >> --enable-foreman-plugin-remote-execution \ >> >> --enable-foreman-plugin-salt \ >> >> --enable-foreman-plugin-setup \ >> >> --enable-foreman-plugin-tasks \ >> >> --enable-foreman-plugin-templates \ >> >> --enable-foreman-compute-openstack \ >> >> --enable-foreman-compute-vmware \ >> >> --foreman-db-database="foo" \ >> >> --foreman-db-host="foo" \ >> >> --foreman-db-manage="false" \ >> >> --foreman-db-password="foo" \ >> >> --foreman-db-port="4344" \ >> >> --foreman-db-type="postgresql" \ >> >> --foreman-db-username="foo" >> >