Foreman Proxy Installation Failure with "tlsv1 alert unknown ca"

Support
1

Hi all,

I’m having an issue with Foreman/Katello Smart Proxy Installation on CentOS 7 (Katello 3.4).

Upon completing the installer (gets to about 90%) I get:

 /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[server.example.com]/ensure: change from absent to present failed: Proxy server.example.com cannot be registered: Unable to communicate with the proxy: ERF12-2530 [ProxyAPI::ProxyException]: Unable to detect features ([RestClient::SSLCertificateNotVerified]: SSL_connect returned=1 errno=0 state=error: certificate verify failed) for proxy https://server.example.com:9090/features Please check the proxy is configured and running on the host.

Same returned In the /var/log/foreman-installer/foreman-proxy-content.log

When I go over to /var/log/foreman-proxy/proxy.log I get the following:

ERROR -- : OpenSSL::SSL::SSLError: SSL_accept returned=1 errno=0 state=SSLv3 read client certificate A: tlsv1 alert unknown ca

Apparently, something is wrong with the Certificates generated from the Master?

The expected behavior, of course, is to have the proxy come up and be installed properly.

The thing is, I’m not using any weird CAs or external certs…this is all coming from the CA generated when Katello was first installed so I don’t even know where to begin to debug things. I understand where things might go south when trying to use my own certs, but this is all Katello’s stuff and with Katello doing the packaging and installing of all the certificates for me, I have no idea where to begin debugging this.

There are a few threads/tickets out there talking about running kattelo-certs-check (which I have not done) but these all seems to indicate being used for custom ssl certs, which we are not using.

Further, there’s information out there about modifying apache configs and issues with Cert locations (SSLCACertificatePath) being built into the apache configs, but that appears to have been corrected in 3.4 as that entry does not appear in any of the apache config files in /etc/httpd/conf.d.

I’d be grateful just for a place to begin debugging this.

I followed the procedure documented at Foreman :: Plugin Manuals

Thanks!
Doug Wagner

2

So this is mostly for the archives at this point.

I tried everything I could find to get this working and finally gave up. As the installation was not yet fully fleshed out (we only had about 150 of our 1000 servers in it) I decided to throw in the towel on my Katello 3.4 installation and just stand up a new 3.5 installation and merge to it.

The 3.5 install seems to be working fine with regards to smart proxy stuff, so I don’t know what happened to my 3.4 install.

1 Like
3

Sorry you didn’t get an answer sooner @Illydth, but happy to hear an upgrade got you movie g forward :wink:

4

Lessons Learned in case anyone comes across this:

By Default, 9090 (the port used by the Katello Smart Proxy) is used both for Cockpit AND Puppet (PuppetDB I believe uses 9090 as a default port…it’s been a bit since I set this up). If EITHER of these is installed and/or configured at their defaults it will cause your Smart Proxy installations to blow up.

Basically: Be VERY VERY VERY Careful what you’re installing outside of the basic Katello Installation. EVERYTHING out there likes to use identical ports to everything else and as such it’s almost guaranteed that anything else you install WILL port conflict with something Katello already has installed and is using.

As a warning: To get Puppet, Puppet DB, Katello and Cockpit all working together you actually have to go in and modify Katello/Foreman Plugin CODE to get things to not conflict and to actually work.

If you are getting ANY weird issues with the smart proxy installation, it’s likely that one of the half-million ports Katello wants open and available is in use.

And don’t expect Katello to tell you that either.

5

Thanks for coming back and updating the issue!
This feedback sounds like something @katello team would be interested in.

6

For some further information and discussion around this topic especially around Cockpit check out 1. I generally don’t recommend changing any ports on a Katello installation. Our puppet modules, and thus installer, present a vast array of configuration “options” to a user that are not necessarily guaranteed to be configurable outside of the defaults. This is a bit of a by-product of having attempted to build fully configurable puppet modules without much thought to the consequences early on.

7

@ehelms

Thanks for the response!

Yea, I am intimately familiar with the link you just sent, I hit that a few months back when I set my mind to solving the actual problem.

Yes, I guess one solution would be to force Foreman/Katello to use a different smart proxy port, but that seems like…Goliath moving out of David’s way I guess.

Foreman/Katello is a complex/complicated piece of software with so many moving parts it’s easy to get lost when trying to manage them. For a complex Foreman/Katello infrastructure (I’m up to 8 smart proxies in 4 global locations right now) and with a team of 4+ people…you just KNOW that someone’s going to get something wrong the next time we go to install or upgrade, forget a flag, and be back at the original problem of “Why isn’t my smart proxy connecting and why is it throwing cert errors at us after a year+ of use…”

IMHO (and this is COMPLETELY my opinion as a former software architect for a large development shop who used to write software for Sprint PCS…by this I mean I feel qualified to give an opinion, NOT qualified to tell you you’re doing it wrong! :slight_smile: ) you have a master system with master ports. PuppetDB and Cockpit are NOT major parts of your software. You don’t fix the parent software to support a plugin, you change the plugin to support the parent software.

PuppetDB is easy to get around, it’s config file allows me to change the conflicting 8080 port that it’s HTTP communications naturally goes over to another port just by modifying the config file. This is simple and not something Foreman/Katello should be trying to support working around. (Documentation about this conflict might be nice, but other than that just because a piece of non-critical software conflicts with your master system doesn’t mean change the master system…)

Cockit is ALSO Easy to get around. A SystemD Dropin File changes the port under which cockpit runs with little to no effort…Cockipt’s own installation documentation describes in full detail how to make this change…again, not something Katello/Foreman needs to work around.

So here’s the REAL PROBLEM…the foreman plugin for Cockpit has NO CONFIGURABLE OPTIONS…meaning you CANNOT switch the port that FOREMAN/KATELLO is looking for to FIND cockpit. While the cockpit developers understood that ports are ephemeral on a system and that there will always be conflicts (and thus the need to support running on arbitrary ports), the Cockpit Pulg-In development team did NOT seem to take this approach…change cockpit all you want, you will NEVER get Katello to talk with it on anything but the default 9000 port.

Not to be too obnoxious about this, but I would have assumed that whoever developed this plugin would have immediately realized the port conflict (developer/API documentation for Foreman HAS TO include a listing of used ports right?) and thrown a red flag. Instead that plugin looks ONLY for a single port, hard coded into the code itself, and has no configurable option to change that. There is almost NO situation where hard coding a port into the code is the right solution.

This problem exists as the foreman cockpit plugin’s problem. The cockpit plugin owns the issue (because the rest of the pieces involved are all configurable) and thus the cockpit plugin should be solving the issue.

I’m again not sure why “we need to change Foreman’s Smart Proxy to Reside on 8443” is the right solution to make the cockpit plugin integration work. I appreciate that the opinion is that 8443 is the right port for the proxy to run on, but that has NOTHING to do with the currently broken

Don’t boil the ocean to solve a small problem. Allow the port in the Foreman Cockpit Plugin to be configured within the GUI and/or other config files.

Till then, IMHO the solution is not to manipulate the Foreman architecture to work around Cockpit, the solution is to change the cockpit plugin to work around the Foreman Architecture.

https://github.com/theforeman/foreman_cockpit/issues/14