Foreman-installer --scenario foreman-proxy-content fails

We are trying to add and new smart proxy to the existing foreman server.
After following this guide the last step is to run foreman-installer --scenario foreman-proxy-content command and it fails with below error.

[ERROR 2021-01-29T18:35:57 main] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[smartproxy-hostname]/ensure: change from 'absent' to 'present' failed: Proxy smartproxy-hostname cannot be registered: Unable to communicate with the proxy: ERF12-2530 [ProxyAPI::ProxyException]: Unable to detect features ([Errno::ECONNRESET]: Connection reset by peer - SSL_connect) for proxy https://smartproxy-hostname:9090/v2/features Please check the proxy is configured and running on the host.

Expected outcome:
Configured Smart Proxy

Foreman and Proxy versions:

Foreman and Proxy plugin versions:

Distribution and version:
Oracle Linux 7.8

Other relevant data:
Command used to install the smart proxy:

foreman-installer \
–scenario foreman-proxy-content \
–certs-tar-file “/root/smart-proxy-hostname-certs.tar”\
–foreman-proxy-content-parent-fqdn “foreman-hostname”\
–foreman-proxy-register-in-foreman “true”\
–foreman-proxy-foreman-base-url “https://foreman-hostname”\
–foreman-proxy-trusted-hosts “foreman-hostname”\
–foreman-proxy-trusted-hosts “smart-proxy-hostname”\
–foreman-proxy-oauth-consumer-key “< key >”\
–foreman-proxy-oauth-consumer-secret “< secret >”\
–puppet-server-foreman-url “https://foreman-hostname”

foreman-proxy-content.log (9.5 KB)

Hi @bhatsu, this does look like a network issue but can’t be sure yet, can you try to ping the katello server from the server where you are installing the foreman-proxy? If that works, also make sure you can reach https://smartproxy-hostname:9090/v2/features using the certificates you are providing using curl or something similar. If you can do that, let us know and we can debug further.

We cannot try the ping test because icmp is blocked. But we did try this:

On foreman server (katello) when we run echo | openssl s_client -connect new-smartproxy-host:9090 and we get this output


no peer certificate available

No client certificate CA names sent

SSL handshake has read 0 bytes and written 289 bytes

New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
    Protocol  : TLSv1.2
    Cipher    : 0000
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    Start Time: 1612290162
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)

But when we try the same command locally on the new smartproxy-host it works. This tells me that the certificates are in place ?

Note: It is worth mentioning that this new proxy server has a different domain name than the exiting foreman/katello server. Do you think that can cause an issue ?

So you do have a network issues. Sounds like you block all ICMP. That is a serious network issue. You must not block all ICMP. ICMP is an essential network protocol. Blocking all ICMP messages causes all kinds of weird issues and problems. Some ICMP messages are mandatory and must not be blocked. If you feel like filtering ICMP then you need to read the RFCs and make sure to allow the important messages.

The s_client output also suggests that something is interfering with the connection. If you break IP by blocking all ICMP who knows what else you break… And as it’s working locally it suggests that it’s the filtering in between causing the problem because I guess locally it’s not filtered…

On the master run

curl -v --cert /etc/foreman-proxy/foreman_ssl_cert-key /etc/foreman-proxy/foreman_ssl_key.pem -H "Accept: application/json" ''

That must work. If that doesn’t work I would highly recommend to fix your firewall/filter before continuing.

And either way: fix your network. It’s broken.

1 Like

I can connect from master to the proxy host on 9090 just fine. So I doubt network/firewall is an issue.
In my previous comment you can see that the output of openssl does show CONNECTED

I tried your command and it fails for us.

* <url> malformed
* Closing connection -1
curl: (3) <url> malformed
* About to connect() to proxy-name port 9090 (#0)
*   Trying 10.x.x.x...
* Connected to proxy-name (10.x.x.x) port 9090 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TCP connection reset by peer
* Closing connection 0

So this tells me we have an issue with certificates.

That’s the thing: if you break your network by filtering all ICMP it can have all kinds of effects, e.g. MTU issues. Possibly you may be able to connect fine (i.e. establish the TCP connection) because it only requires smaller packets but the moment something larger is transferred weird things happen, because your broke IP fragmentation.

In particular if you are in an internal network as it seems… Your network is broken. You cannot rule out anything based on that…

The URL is malformed. As you don’t post the exact command you have tried, it’s hard to tell what you really did…

I think this message is a red herring. I get the same message for an existing working proxy as well.

As I wrote before: you don’t post the exact command. I don’t know what you do. I don’t know if you do it right or not…

Something is resetting the TLS connection. So far, all we know that you have a broken network due to a overzealous firewall. Check the firewall. Check with tcpdump on both ends and capture the traffic to find out what device is actually sending that reset.

1 Like

Thank you for your input. I will check the network again today and report back.

So it was the firewall causing the issue.
We had to create an Application Overridge Policy as it seems only allowing the ports wasn’t good enough.

Thank you for all your help!

BTW, ICMP is still blocked :wink:

1 Like