Remote execution plugin fails on smart-proxy

ahoiroman · February 24, 2020, 12:50pm

Problem:

The remote execution plugin fails to execute.

This is how I installed the foreman-smart-proxy:

foreman-installer \
  --no-enable-foreman \
  --no-enable-foreman-cli \
  --enable-foreman-proxy \
  --enable-foreman-proxy-plugin-remote-execution-ssh \
  --enable-foreman-proxy-plugin-discovery \
  --foreman-proxy-puppet=true \
  --foreman-proxy-plugin-discovery-install-images=true \
  --foreman-proxy-templates=true \
  --foreman-proxy-template-url=http://foreman-proxy.example.com:8000 \
  --foreman-proxy-templates-listen-on=both \
  --foreman-proxy-puppetca=true \
  --foreman-proxy-tftp=true \
  --foreman-proxy-http=true \
  --foreman-proxy-foreman-ssl-ca=/etc/foreman-proxy/ca.pem \
  --foreman-proxy-foreman-ssl-cert=/etc/foreman-proxy/cert.pem \
  --foreman-proxy-foreman-ssl-key=/etc/foreman-proxy/key.pem \
  --foreman-proxy-foreman-base-url=https://foreman.example.com \
  --foreman-proxy-trusted-hosts=foreman.example.com \
  --foreman-proxy-oauth-consumer-key=... \
  --foreman-proxy-oauth-consumer-secret=...

If I am running a task on a host that is using the smart-proxy, it fails:

tailf /var/log/foreman-proxy/proxy.log shows:

/usr/share/ruby/webrick/server.rb:295:in `block in start_thread'
/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `call'
/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2020-02-24T13:42:36 94d4b09f [I] Finished GET /dynflow/tasks/count with 500 (43.48 ms)

/var/log/foreman-proxy/smart_proxy_dynflow_core.log shows:

/opt/rh/rh-ruby25/root/usr/share/ruby/webrick/server.rb:297:in `block in start_thread'SSL_accept returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain) (OpenSSL::SSL::SSLError)
/opt/rh/rh-ruby25/root/usr/share/ruby/webrick/server.rb:299:in `accept'
/opt/rh/rh-ruby25/root/usr/share/ruby/webrick/server.rb:299:in `block (2 levels) in start_thread'
/opt/rh/rh-ruby25/root/usr/share/ruby/webrick/utils.rb:263:in `timeout'
/opt/rh/rh-ruby25/root/usr/share/ruby/webrick/server.rb:297:in `block in start_thread'^C

The config seems fine to me:

vim /etc/smart_proxy_dynflow_core/settings.yml


---
# Path to dynflow database, leave blank for in-memory non-persistent database
:database:
:console_auth: true

# URL of the foreman, used for reporting back
:foreman_url: https://foreman.example.com

# SSL settings for client authentication against foreman.
:foreman_ssl_ca: /etc/foreman-proxy/ca.pem
:foreman_ssl_cert: /etc/foreman-proxy/cert.pem
:foreman_ssl_key: /etc/foreman-proxy/key.pem

# Listen on address
:listen: 0.0.0.0

# Listen on port
:port: 8008

:use_https: true
:ssl_ca_file: /etc/foreman-proxy/ca.pem
:ssl_certificate: /etc/foreman-proxy/cert.pem
:ssl_private_key: /etc/foreman-proxy/key.pem
# :ssl_ca_file: ssl/ca.pem
# :ssl_private_key: ssl/localhost.pem
# :ssl_certificate: ssl/certs/localhost.pem

# Use this option only if you need to disable certain cipher suites.
# Note: we use the OpenSSL suite name, take a look at:
# https://www.openssl.org/docs/manmaster/apps/ciphers.html#CIPHER-SUITE-NAMES
# for more information.
#:ssl_disabled_ciphers: [CIPHER-SUITE-1, CIPHER-SUITE-2]

# Use this option only if you need to strictly specify TLS versions to be
# disabled. SSLv3 and TLS v1.0 are always disabled and cannot be configured.
# Specify versions like: '1.1', or '1.2'
#:tls_disabled_versions: []

# File to log to, leave empty for logging to STDOUT
# :log_file: /var/log/foreman-proxy/smart_proxy_dynflow_core.log

# Log level, one of UNKNOWN, FATAL, ERROR, WARN, INFO, DEBUG
# :log_level: ERROR

Expected outcome:
Working task execution

Foreman and Proxy versions:

Foreman (main)

Discovery

Version

1.0.5

Dynflow

false

Templates

Version

1.24.2

aruzicka · February 24, 2020, 12:58pm

Hi,
could you also post contents of /etc/foreman-proxy/settings.yml? Smart proxy dynflow core and smart proxy need to use the exact same certificate, which is probably not happening here.

ahoiroman · February 24, 2020, 1:01pm

Thanks for your reply

On the smart-proxy (foreman-proxy.example.com):

---
### File managed with puppet ###
## Module:           'foreman_proxy'

:settings_directory: /etc/foreman-proxy/settings.d

# SSL Setup

# if enabled, all communication would be verified via SSL
# NOTE that both certificates need to be signed by the same CA in order for this to work
# see http://theforeman.org/projects/smart-proxy/wiki/SSL for more information
:ssl_ca_file: /etc/foreman-proxy/ca.pem
:ssl_certificate: /etc/foreman-proxy/cert.pem
:ssl_private_key: /etc/foreman-proxy/key.pem

# Use this option only if you need to disable certain cipher suites.
# Note: we use the OpenSSL suite name, such as "RC4-MD5".
# The complete list of cipher suite names can be found at:
# https://www.openssl.org/docs/manmaster/man1/ciphers.html#CIPHER-SUITE-NAMES
#:ssl_disabled_ciphers: [CIPHER-SUITE-1, CIPHER-SUITE-2]

# Use this option only if you need to strictly specify TLS versions to be
# disabled. SSLv3 and TLS v1.0 are always disabled and cannot be configured.
# Specify versions like: '1.1', or '1.2'
#:tls_disabled_versions: []

# the hosts which the proxy accepts connections from
# commenting the following lines would mean every verified SSL connection allowed
:trusted_hosts:
  - foreman.example.com

# Endpoint for reverse communication
:foreman_url: https://foreman.example.com

# SSL settings for client authentication against Foreman. If undefined, the values
# from general SSL options are used instead. Mainly useful when Foreman uses
# different certificates for its web UI and for smart-proxy requests.
:foreman_ssl_ca: /etc/foreman-proxy/ca.pem
:foreman_ssl_cert: /etc/foreman-proxy/cert.pem
:foreman_ssl_key: /etc/foreman-proxy/key.pem

# by default smart_proxy runs in the foreground. To enable running as a daemon, uncomment 'daemon' setting
:daemon: true
# Only used when 'daemon' is set to true.
# Uncomment and modify if you want to change the default pid file '/var/run/foreman-proxy/foreman-proxy.pid'
#:daemon_pid: /var/run/foreman-proxy/foreman-proxy.pid

# host and ports configuration
# Host or IP to bind ports to (e.g. *, localhost, 0.0.0.0, ::, 192.168.1.20)
:bind_host: '::'
# http is disabled by default. To enable, uncomment 'http_port' setting
# https is enabled if certificate, CA certificate, and private key are present in locations specifed by
# ssl_certificate, ssl_ca_file, and ssl_private_key correspondingly
# default values for https_port is 8443
:https_port: 8443
:http_port: 8000
# Log configuration
# Uncomment and modify if you want to change the location of the log file or use STDOUT or SYSLOG values
:log_file: /var/log/foreman-proxy/proxy.log
# Uncomment and modify if you want to change the log level
# WARN, DEBUG, ERROR, FATAL, INFO, UNKNOWN
:log_level: INFO
# The maximum size of a log file before it's rolled (in MiB)
#:file_rolling_size: 100
# The maximum age of a log file before it's rolled (in seconds). Also accepts 'daily', 'weekly', or 'monthly'.
#:file_rolling_age: weekly
# Number of log files to keep
#:file_rolling_keep: 6
# Logging pattern for file-based loging
#:file_logging_pattern: '%d %.8X{request} [%.1l] %m'
# Logging pattern for syslog or journal loging
#:system_logging_pattern: '%.8X{request} [%.1l] %m'


# Log buffer size and extra buffer size (for errors). Defaults to 3000 messages in total,
# which is about 500 kB request.
:log_buffer: 2000
:log_buffer_errors: 1000

aruzicka · February 24, 2020, 2:33pm

Hm, I don’t see anything suspicious there.

You had a foreman installation with a proxy, and then on another machine you ran the installer command you mentioned in the first post?

ahoiroman · February 24, 2020, 3:05pm

Correct - The first foreman-instance is using a default config - the second instance is a foreman-proxy I built with the config I mentioned above

aruzicka · February 24, 2020, 3:11pm

Where do the certificates used on the proxy machine come from?

ahoiroman · February 24, 2020, 3:14pm

I created them on the foreman master using puppetserver ca generate --certname foreman-proxy.example.com. After that I copied them to the foreman-proxy:

[root@hts-sys-forep1 foreman-proxy]# ll
insgesamt 20
-rw-r--r--. 1 puppet puppet        3870 20. Feb 14:08 ca.pem
-rw-r--r--. 1 puppet puppet        1972 20. Feb 14:08 cert.pem
-rw-r-----. 1 puppet puppet        3243 20. Feb 14:09 key.pem
-rw-r--r--. 1 root   root             0 21. Jan 22:17 migration_state
drwxr-xr-x. 2 root   root          4096 24. Feb 11:19 settings.d
-rw-r-----. 1 root   foreman-proxy 3372 24. Feb 08:47 settings.yml

ahoiroman · February 24, 2020, 4:27pm

I also checked the cert manually:

$ openssl x509 -in /etc/foreman-proxy/cert.pem -text

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 6 (0x6)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=Puppet CA: foreman.example.com
        Validity
            Not Before: Feb 19 11:57:05 2020 GMT
            Not After : Feb 18 11:57:05 2025 GMT
        Subject: CN=foreman-proxy.example.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (4096 bit)
                Modulus:
                    00:b9:22:5f:b5:8f:eb:89:61:5b:1c:b5:5d:bc:8a:
                    5e:6f:d9:95:30:05:2f:84:ba:9f:e0:2a:40:05:97:
                    [...]
                    76:52:54:65:dd:2e:04:72:5a:76:77:e4:8e:f0:4e:
                    64:b1:2b
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            Netscape Comment:
                Puppet Server Internal Certificate
            X509v3 Authority Key Identifier:
                keyid:99:3E:3E:EB:45:BE:A4:E2:B6:7B:33:54:BA:0A:96:44:06:83:53:F6

            X509v3 Subject Key Identifier:
                04:02:72:5E:1F:4A:00:C7:90:A2:2C:19:90:73:A1:B8:07:65:5B:74
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Extended Key Usage: critical
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
    Signature Algorithm: sha256WithRSAEncryption
         20:7c:38:00:2e:85:34:fa:84:d9:3b:dd:eb:11:4e:5b:0c:e0:
         [...]
         b5:eb:c6:d2:77:bf:8d:52
-----BEGIN CERTIFICATE-----
MIIFhDCCA2ygAwIBAgIBBjANBgkqhkiG9w0BAQsFADAuMSwwKgYDVQQDDCNQdXBw
[...]
h5odXN1p+MDYJ1+7s3fCZLXrxtJ3v41S
-----END CERTIFICATE-----

CA is foreman.example.com and CN is foreman-proxy.example.com

infomatico · February 25, 2020, 12:31am

@ahoiroman
What operating system do you use?

@aruzicka
Maybe this error is because the ruby concurrent is not fully compatible with 1.24 version?

aruzicka · February 25, 2020, 7:45am

Judging by the paths he is on EL*-derivative, which is fine.

That only happens on debian/ubuntu and the error looks differently.

ahoiroman · February 25, 2020, 11:24am

I solved the problem recreating the certificates and restarting the dynflow-service on the foreman-proxy.

Thank you all so much for your help, I really appreciate this

infomatico · February 25, 2020, 12:17pm

Ok. Any news about remote execution fully work without error on Debian/Ubuntu OS?

agutierr · August 15, 2020, 12:07pm

Hey! Same problem here, Foreman 2.1 under Debian 10 with remote plugin execution. If I use the foreman-installer “as-is” works fine. But when I replace Foreman UI’s certificate with my OWN certificate (signed by Geant) Remote execution fails after 10 minutes…

infomatico · August 15, 2020, 12:23pm

An old bug that has not yet been fixed. I’ve tried everything and I
couldn’t.

agutierr · August 15, 2020, 12:32pm

I was so happy with my Foreman UI with green tick in the browser… Arrggggggg

agutierr · August 15, 2020, 7:17pm

@informatico ! I think I got it to work!!!