Foreman/Katello Smart Proxy Custom certificate

Support
, ,
1

Problem:

Trying to follow the instructions at Installing an External Smart Proxy Server 3.7. However, the Smart Proxy machine, a standalone DHCP server, is Ubuntu and so doesn’t have foreman-installer-katello available. Can I ignore this advice, as this Smart Proxy won’t be used for hosting content? Or is the certificate for the Smart Proxy only needed for Smart Proxies which host content?

Expected outcome:

No need for Katello on a non-content Smart Proxy.

Foreman and Proxy versions:

Foreman: 3.7.0
Katello: 4.9
Smart Proxy: 3.7.0

Foreman and Proxy plugin versions:

n/a

Distribution and version:

Foreman server: CentOS stream 8
Smart Proxy: Ubuntu 20.04

Other relevant data:

2

There is this article (Installing a puppet smart proxy against a katello main server), though I’m not using puppet (AFAIK), it does provide helpful insight in how to add custom certs to a smart-proxy deployment.

The following however doesn’t work and changes the group ownership of the key file from foreman-proxy to puppet:

sudo foreman-installer \
  --foreman-proxy-ssl-cert "/etc/ssl/certs/$(hostname).crt" \
  --foreman-proxy-ssl-key "/etc/ssl/private/$(hostname).key" \
  --foreman-proxy-ssl-ca "/etc/ipa/ca.crt" \
  --foreman-proxy-register-in-foreman "true" \
  --foreman-proxy-trusted-hosts "foreman.site.domain.com" \
  --foreman-proxy-trusted-hosts "$(hostname)" \
  --foreman-proxy-oauth-consumer-key "********" \
  --foreman-proxy-oauth-consumer-secret "********"

The output complains about the permissions on the key file:

2023-08-09 01:22:09 [ERROR ] [configure] Aug 09 01:22:08 dhcp.site.domain.com systemd[1]: Starting Foreman Proxy...
2023-08-09 01:22:09 [ERROR ] [configure] Aug 09 01:22:09 dhcp.site.domain.com smart-proxy[1089286]: /usr/lib/ruby/vendor_ruby/mustermann/pattern.rb:59: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
2023-08-09 01:22:09 [ERROR ] [configure] Aug 09 01:22:09 dhcp.site.domain.com smart-proxy[1089286]: /usr/lib/ruby/vendor_ruby/mustermann/regular.rb:22: warning: The called method `initialize' is defined here
2023-08-09 01:22:09 [ERROR ] [configure] Aug 09 01:22:09 dhcp.site.domain.com smart-proxy[1089286]: /usr/lib/ruby/vendor_ruby/mustermann/pattern.rb:59: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
2023-08-09 01:22:09 [ERROR ] [configure] Aug 09 01:22:09 dhcp.site.domain.com smart-proxy[1089286]: /usr/lib/ruby/vendor_ruby/mustermann/regexp_based.rb:17: warning: The called method `initialize' is defined here
2023-08-09 01:22:09 [ERROR ] [configure] Aug 09 01:22:09 dhcp.site.domain.com smart-proxy[1089286]: /usr/lib/ruby/vendor_ruby/mustermann/ast/compiler.rb:43: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
2023-08-09 01:22:09 [ERROR ] [configure] Aug 09 01:22:09 dhcp.site.domain.com smart-proxy[1089286]: /usr/lib/ruby/vendor_ruby/mustermann/ast/compiler.rb:49: warning: The called method `pattern' is defined here
2023-08-09 01:22:09 [ERROR ] [configure] Aug 09 01:22:09 dhcp.site.domain.com smart-proxy[1089286]: /usr/lib/ruby/vendor_ruby/rsec/helpers.rb:90: warning: constant ::Fixnum is deprecated
2023-08-09 01:22:09 [ERROR ] [configure] Aug 09 01:22:09 dhcp.site.domain.com smart-proxy[1089286]: Errors detected on startup, see log for details. Exiting: Permission denied @  rb_sysopen - /etc/ssl/private/dhcp.site.domain.com.key
2023-08-09 01:22:09 [ERROR ] [configure] Aug 09 01:22:09 dhcp.site.domain.com systemd[1]: foreman-proxy.service: Main process exited, code=exited, status=1/FAILURE
2023-08-09 01:22:09 [ERROR ] [configure] Aug 09 01:22:09 dhcp.site.domain.com systemd[1]: foreman-proxy.service: Failed with result 'exit-code'.
2023-08-09 01:22:09 [ERROR ] [configure] Aug 09 01:22:09 dhcp.site.domain.com systemd[1]: Failed to start Foreman Proxy.
2023-08-09 01:22:09 [NOTICE] [configure] System configuration has finished.

Before running foreman-installer:

user@dhcp:~$ sudo ls -al /etc/ssl/private/dhcp.site.domain.com.key
-rw-r----- 1 root foreman-proxy 1704 Aug  8 18:41 /etc/ssl/private/dhcp.site.domain.com.key

And after:

user@dhcp:~$ sudo ls -al /etc/ssl/private/dhcp.site.domain.com.key
-rw-r----- 1 root puppet 1704 Aug  8 18:41 /etc/ssl/private/dhcp.site.domain.com.key

The puppet group exists and contains the foreman-proxy user:

user@dhcp:~$ sudo grep foreman-proxy /etc/group
foreman-proxy:x:997:
puppet:x:1001:foreman-proxy

How do I appease foreman-installer so that I can run foreman-proxy for ISC DHCP on this Ubuntu host? The Foreman server is a CentOS 8 server with Katello.

3

Manually copying the smart-proxy cert and key files to /etc/foreman-proxy/, changing the group ownership to foreman-proxy and setting permissions to 440 appears to appease foreman-proxy. Symlinks didn’t work, but with a port-renew script this requirement can be met.

In many examples, I see this value configured with the certificate chain of the Foreman server. However, this means that one would need to manually update this certificate regularly when a client should trust the server certificate if it trusts the root CA certificate. If a Smart Proxy must be hard coded with the server cert then this is a major pita when it comes to automating certificate renewals.

2023-08-09 12:47:21 [ERROR ] [configure] Error making POST request to Foreman at https://foreman.site.domain.com/api/v2/smart_proxies: Unable to communicate with the proxy: ERF12-2530 [ProxyAPI::ProxyException]: Unable to detect features ([OpenSSL::SSL::SSLError]: SSL_read: tlsv1 alert unknown ca) for proxy https://dhcp.site.domain.com:8443/v2/features Please check the proxy is configured and running on the host.
2023-08-09 12:47:21 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[dhcp.site.domain.com]/ensure: change from 'absent' to 'present' failed: Error making POST request to Foreman at https://foreman.site.domain/api/v2/smart_proxies: Unable to communicate with the proxy: ERF12-2530 [ProxyAPI::ProxyException]: Unable to detect features ([OpenSSL::SSL::SSLError]: SSL_read: tlsv1 alert unknown ca) for proxy https://dhcp.site.domain.com:8443/v2/features Please check the proxy is configured and running on the host.
4

This looks somewhat relevant, but I can’t work out what is regarded as server config and proxy config when running foreman-installer on a smart proxy that isn’t a Foreman Katello content server. How to install with own CA/Certs? - #3 by Christoph

5

Some logs from /var/log/foreman-installer/foreman.log:

2023-08-09 14:06:44 [DEBUG ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[dhcp.site.domain.com]: Starting to evaluate the resource (293 of 309)
2023-08-09 14:06:44 [DEBUG ] [configure] Foreman_smartproxy[dhcp.site.domain.com](provider=rest_v3): Making get request to https://foreman.site.domain.com/api/v2/smart_proxies?search=name%3D%22dhcp.site.domain.com%22
2023-08-09 14:06:44 [DEBUG ] [configure] Foreman_smartproxy[dhcp.site.domain.com](provider=rest_v3): Received response 200 from request to https://foreman.site.domain.com/api/v2/smart_proxies?search=name%3D%22dhcp.site.domain.com%22
2023-08-09 14:06:44 [DEBUG ] [configure] Foreman_smartproxy[dhcp.site.domain.com](provider=rest_v3): Making post request to https://foreman.site.domain.com/api/v2/smart_proxies
2023-08-09 14:06:44 [DEBUG ] [configure] Foreman_smartproxy[dhcp.site.domain.com](provider=rest_v3): Received response 422 from request to https://foreman.site.domain.com/api/v2/smart_proxies
2023-08-09 14:06:44 [ERROR ] [configure] Error making POST request to Foreman at https://foreman.site.domain.com/api/v2/smart_proxies: Unable to communicate with the proxy: ERF12-2530 [ProxyAPI::ProxyException]: Unable to detect features ([OpenSSL::SSL::SSLError]: SSL_read: tlsv1 alert unknown ca) for proxy https://dhcp.site.domain.com:8443/v2/features Please check the proxy is configured and running on the host.
2023-08-09 14:06:44 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[dhcp.site.domain.com]/ensure: change from 'absent' to 'present' failed: Error making POST request to Foreman at https://foreman.site.domain.com/api/v2/smart_proxies: Unable to communicate with the proxy: ERF12-2530 [ProxyAPI::ProxyException]: Unable to detect features ([OpenSSL::SSL::SSLError]: SSL_read: tlsv1 alert unknown ca) for proxy https://dhcp.site.domain.com:8443/v2/features Please check the proxy is configured and running on the host.
2023-08-09 14:06:44 [DEBUG ] [configure] Foreman_smartproxy[dhcp.site.domain.com](provider=rest_v3): Making get request to https://foreman.site.domain.com/api/v2/smart_proxies?search=name%3D%22dhcp.site.domain.com%22
2023-08-09 14:06:44 [DEBUG ] [configure] Foreman_smartproxy[dhcp.site.domain.com](provider=rest_v3): Received response 200 from request to https://foreman.site.domain.com/api/v2/smart_proxies?search=name%3D%22dhcp.site.domain.com%22
2023-08-09 14:06:44 [DEBUG ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[dhcp.site.domain.com]: Skipping refresh; smart proxy is not registered
6

Logs from the Foreman server:

/var/log/foreman/production.log:2023-08-09T14:06:44 [I|app|a164136e] Started POST "/api/v2/smart_proxies" for 10.2.0.5 at 2023-08-09 14:06:44 +0200
/var/log/foreman/production.log:2023-08-09T14:06:44 [I|app|a164136e] Processing by Api::V2::SmartProxiesController#create as JSON
/var/log/foreman/production.log:2023-08-09T14:06:44 [I|app|a164136e]   Parameters: {"smart_proxy"=>{"name"=>"dhcp.site.domain.com", "url"=>"https://dhcp.site.domain.com:8443"}, "apiv"=>"v2"}
/var/log/foreman/production.log:2023-08-09T14:06:44 [I|app|a164136e] Authorized user foreman_api_admin(API Admin)
/var/log/foreman/production.log:2023-08-09T14:06:44 [E|app|a164136e] Unprocessable entity SmartProxy (id: new):
/var/log/foreman/production.log:2023-08-09T14:06:44 [I|app|a164136e]   Rendered api/v2/errors/unprocessable_entity.json.rabl within api/v2/layouts/error_layout (Duration: 4.1ms | Allocations: 5254)
/var/log/foreman/production.log:2023-08-09T14:06:44 [I|app|a164136e]   Rendered layout api/v2/layouts/error_layout.json.erb (Duration: 7.3ms | Allocations: 10477)
/var/log/foreman/production.log:2023-08-09T14:06:44 [I|app|a164136e] Completed 422 Unprocessable Entity in 67ms (Views: 7.8ms | ActiveRecord: 7.5ms | Allocations: 23360)
7

Transferring the generated tar file to the Smart Proxy and handling them manually hasn’t helped either. In case anyone is wondering the two hosts are in the same subnet.

sudo tar xvf /root/$(hostname)-certs.tar -C /etc/foreman-proxy/ --exclude=*.rpm
sudo cp /etc/ssl/private/$(hostname).key /etc/foreman-proxy/
sudo cp /etc/foreman-proxy/ssl-build/katello-server-ca.crt /etc/foreman-proxy/foreman_ssl_ca.pem
sudo cp /etc/foreman-proxy/ssl-build/katello-default-ca.crt /etc/foreman-proxy/ssl_ca.pem
sudo cp /etc/foreman-proxy/ssl-build/$(hostname)/$(hostname)-foreman-proxy-client.crt /etc/foreman-proxy/foreman_ssl.crt
sudo cp /etc/foreman-proxy/ssl-build/$(hostname)/$(hostname)-foreman-proxy-client.key /etc/foreman-proxy/foreman_ssl.key
sudo chown :foreman-proxy /etc/foreman-proxy/*.key
sudo chmod g+r /etc/foreman-proxy/*.key

Trying to configure Smart Proxy and register it:

sudo foreman-installer \ --foreman-proxy-ssl-ca "/etc/foreman-proxy/ssl_ca.crt" \ --foreman-proxy-ssl-cert "/etc/foreman-proxy/$(hostname).crt" \ --foreman-proxy-ssl-key "/etc/foreman-proxy/$(hostname).key" \ --foreman-proxy-foreman-ssl-ca "/etc/foreman-proxy/foreman_ssl_ca.pem" \ --foreman-proxy-foreman-ssl-cert "/etc/foreman-proxy/foreman_ssl.crt" \ --foreman-proxy-foreman-ssl-key "/etc/foreman-proxy/foreman_ssl.key" \ --foreman-proxy-register-in-foreman "true" \ --foreman-proxy-trusted-hosts "foreman.site.domain.com" \ --foreman-proxy-trusted-hosts "$(hostname)"

I also tried using the IPA root ca, but the result is the same. I’d prefer to use the IPA CA as the validity of the local CA is much longer than a host pem certificate.
--foreman-proxy-foreman-ssl-ca "/etc/ipa.ca.crt"

The errors are still the same. What am I doing wrong here?!

8

How can I test the smart-poxy connection using API v2 with a valid client certificate? I get the same error when I try to use the Foreman generated certificates

user@dhcp:~$ sudo curl --cert /etc/foreman-proxy/foreman_ssl.crt --key /etc/foreman-proxy/foreman_ssl.key --cacert /etc/ipa/ca.crt https://dhcp.site.domain.com:8443/v2/features
curl: (56) OpenSSL SSL_read: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca, errno 0

user@dhcp:~$ sudo curl --cert /etc/foreman-proxy/foreman_ssl.crt --key /etc/foreman-proxy/foreman_ssl.key --cacert /etc/foreman-proxy/foreman_ssl_ca.pem https://dhcp.site.domain.com:8443/v2/features
curl: (56) OpenSSL SSL_read: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca, errno 0

user@dhcp:~$ sudo curl --cert /etc/foreman-proxy/foreman_ssl.crt --key /etc/foreman-proxy/foreman_ssl.key --cacert /etc/foreman-proxy/ssl_ca.pem https://dhcp.site.domain.com:8443/v2/features
curl: (56) OpenSSL SSL_read: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca, errno 0

The certificates are from the generated tar file:
cp ~/ssl-build/katello-default-ca.crt ssl_ca.pem
cp ~/ssl-build/katello-server-ca.crt foreman_ssl_ca.pem
cp ~/ssl-build/$(hostname)/$(hostname)-foreman-proxy-client.crt foreman_ssl_cert.pem
cp ~/ssl-build/$(hostname)/$(hostname)-foreman-proxy-client.key foreman_ssl_key.pem

If I try to connect to the foreman server proxy URL on the foreman server I get:

user@foreman:~/temp/ssl-build/dhcp.site.domain.com$ curl https://foreman.site.domain.com/api/v2/smart_proxies
{
  "error": {"message":"Unable to authenticate user "}
}
user@foreman:~/temp/ssl-build/dhcp.site.domain.com$ curl --cert dhcp.site.domain.com-foreman-proxy-client.crt --key dhcp.site.domain.com-foreman-proxy-client.key  https://foreman.site.domain.com/api/v2/smart_proxies
{
  "error": {"message":"Unable to authenticate user "}
}
9

Could the issue be that there’s no Puppet server on a default Foreman/Katello? This post mentions smart proxy registration using oauth to authenticate new smart proxies against puppet, but puppet is no longer included by default when deploying Foreman with Katello.

10

I think the instructions from my tutorial post should work for you as well. As I set up a smart proxy with puppet master, I have added everything to get the puppet master working. But if you omit all the puppet things it should work as well. Without the puppet parts all what’s left is the smart proxy configuration. After that you have to manually set up the proxy configuration for the services you want to use.

So quickly going through the tutorial, I’d say you do step 1 through 4, omit 5, do 6 (without puppet repo), step 7 (which does the certs for the smart proxy), skip 8 as that’s for puppet, do 9 without puppetmaster.

And then finally step 10, without --enable-puppet and the --puppet-* options, but with all other options. Possibly, some may be superfluous, but be sure to check. Comparing it with out foreman-installer command you are definitively missing a few, including the --enable-foreman-proxy and a couple of --foreman-proxy-foreman-ssl-*.

And the certificates you are using are incorrect. I highly recommend to extract the certs from the cert tar and also place them into the directories and names I have used. That’s where the proxy expects them and that’s where it should have the correct permissions. But I don’t know if it would look different on debian.

Either way, for instance --foreman-proxy-ssl-ca is not the CA for the server, but the client. The server CA would be in --foreman-proxy-foreman-ssl-ca. Thus, you are configuring the client cert on the server and the server cert for the client…

Of course, always remember my tutorial start with a clean system without any previous proxy installation or configuration. Each foreman-installer run keeps the previous configurations and only changes what you pass on command line.

As you are using debian for the proxy, it may be necessary to install a content proxy in the normal way on a test server to learn, where it places the certs. That may be the best idea how to do it correctly and then strip it down as I did for the tutorial…

1 Like
11

Thank you, for your input. I think your advice to build a test server using CentOS is the way to go. As then I can see where things end up, and hopefully find where I’m going wrong.

12

I actually meant to install a katello content proxy on debian because I don’t know the placement on debian. My tutorial is for EL8 and the paths where I have put the certs are the paths where the content proxy would place them. I suspect for the paths are identical for debian as it’s all foreman paths but you’ll never know…

1 Like