Remote Execution Plugin troubleshooting

Good morning foremans,

I’m iterating my Foreman installation since 2016 and i tried a few times to use remote exec and Ansible but haven’t been able. The websocket is running with official ssl certificates and somehow i get now feedback from the remote exec task which is definitely running on the remote machine.

Problem:
No feedback of Remote Execution Task from remote host, UI doesn’t get feedback

Expected outcome:
Remote Execution Task ends successful and Foreman UI gets feedback changes status to successful ended

Foreman and Proxy versions:
Foreman 1.23.1 installed using rpms
foreman_remote_execution 1.8.2

Distribution and version:
CentOS Linux release 7.7.1908 (Core)

Other relevant data:

/etc/smart_proxy_dynflow_core/settings.yml:

# Path to dynflow database, leave blank for in-memory non-persistent database
:database:

# URL of the foreman, used for reporting back
:foreman_url: 'https://urzlxdeploy.rz.uni-leipzig.de:443'

# SSL settings for client authentication against Foreman
# :foreman_ssl_ca: ssl/foreman_ca.pem
:foreman_ssl_ca: /etc/puppetlabs/puppet/ssl/certs/ca.pem
# :foreman_ssl_key: ssl/foreman_key.pem
:foreman_ssl_key: /etc/pki/tls/private/urzlxdeploy.rz.uni-leipzig.de-pkey-nopw.pem
# :foreman_ssl_cert: ssl/foreman_cert.pem
:foreman_ssl_cert: /etc/ssl/certs/cert-10263428829321932413835964919.pem

:console_auth: false

# Set to true to make the core fork to background after start
# :daemo           nize: false
# :pid_file: /var/run/foreman-proxy/smart_proxy_dynflow_core.pid

# Listen on address
:listen: 127.0.0.1

# Listen on port
:port: 8008

# SSL settings for running core as https service
# :use_https: false
# :ssl_ca_file: ssl/ca.pem
# :ssl_private_key: ssl/localhost.pem
# :ssl_certificate: ssl/certs/localhost.pem

# Use this option only if you need to disable certain cipher suites.
# Note: we use the OpenSSL suite name, take a look at:
# https://www.openssl.org/docs/manmaster/apps/ciphers.html#CIPHER-SUITE-NAMES
# for more information.
#:ssl_disabled_ciphers: [CIPHER-SUITE-1, CIPHER-SUITE-2]

# Use this option only if you need to strictly specify TLS versions to be
# disabled. SSLv3 and TLS v1.0 are always disabled and cannot be configured.
# Specify versions like: '1.1', or '1.2'
#:tls_disabled_versions: []
# File to log to, leave empty for logging to STDOUT
:log_file: /var/log/foreman-proxy/smart_proxy_dynflow_core.log

# Log level, one of UNKNOWN, FATAL, ERROR, WARN, INFO, DEBUG
:log_level: WARN 
       
# Maximum age of execution plans to keep before having them cleaned
# by the execution plan cleaner (in seconds), defaults to 24 hours
# :execution_plan_cleaner_age: 86400

/etc/foreman-proxy/settings.d/remote_execution_ssh.yml:

---
:enabled: true
:ssh_identity_key_file: '~/.ssh/id_rsa_foreman_proxy'
:local_working_dir: '/var/tmp'
:remote_working_dir: '/var/tmp'
# :kerberos_auth: false
# :async_ssh: false

# Defines how often (in seconds) should the runner check
# for new data leave empty to use the runner's default
# (1 second for regular, 60 seconds with async_ssh enabled)
# :runner_refresh_interval:

# Defines the verbosity of logging coming from Net::SSH
# one of :debug, :info, :warn, :error, :fatal
# must be lower than general log level
# :ssh_log_level: fatal

/etc/foreman/settings.yaml:

---
### File managed with puppet ###
## Module:           'foreman'


:unattended: true
:login: true
:require_ssl: true
:locations_enabled: false
:organizations_enabled: false
:puppetrun: false
:puppetssldir: /etc/puppetlabs/puppet/ssl

# The following values are used for providing default settings during db migrate
:oauth_active: true
:oauth_map_users: false
:oauth_consumer_key: *
:oauth_consumer_secret: *

# Websockets
:websockets_encrypt: on
#:websockets_ssl_key: /etc/puppetlabs/puppet/ssl/private_keys/urzlxdeploy.rz.uni-leipzig.de.pem
:websockets_ssl_key: /etc/pki/tls/private/urzlxdeploy.rz.uni-leipzig.de-pkey-nopw.pem
#:websockets_ssl_cert: /etc/puppetlabs/puppet/ssl/certs/urzlxdeploy.rz.uni-leipzig.de.pem
:websockets_ssl_cert: /etc/ssl/certs/cert-10263428829321932413835964919.pem

# SSL-settings
:ssl_certificate: /etc/puppetlabs/puppet/ssl/certs/urzlxdeploy.rz.uni-leipzig.de.pem
:ssl_ca_file: /etc/puppetlabs/puppet/ssl/certs/ca.pem
:ssl_priv_key: /etc/puppetlabs/puppet/ssl/private_keys/urzlxdeploy.rz.uni-leipzig.de.pem

# Log settings for the current environment can be adjusted by adding them
# here. For example, if you want to increase the log level.
:logging:
  :level: error

# Individual logging types can be toggled on/off here
:loggers:

/etc/httpd/conf.d/05-foreman-ssl.conf:

# ************************************                                                                                                                                                                                                                        
# Vhost template in module puppetlabs-apache                                                                                                                                                                                                                  
# Managed by Puppet                                                                                                                                                                                                                                           
# ************************************                                                                                                                                                                                                                        
                                                                                                                                                                                                                                                              
<VirtualHost *:443>                                                                                                                                                                                                                                           
  ServerName urzlxdeploy.rz.uni-leipzig.de                                                                                                                                                                                                                    
                                                                                                                                                                                                                                                              
  ## Vhost docroot                                                                                                                                                                                                                                            
  DocumentRoot "/usr/share/foreman/public"                                                                                                                                                                                                                    
                                                                                                                                                                                                                                                              
  ## Directories, there should at least be a declaration for /usr/share/foreman/public                                                                                                                                                                        
                                                                                                                                                                                                                                                              
  <Directory "/usr/share/foreman/public">                                                                                                                                                                                                                     
    Options SymLinksIfOwnerMatch                                                                                                                                                                                                                              
    AllowOverride None                                                                                                                                                                                                                                        
    Require all granted                                                                                                                                                                                                                                       
  </Directory>                                                                                                                                                                                                                                                
                                                                                                                                                                                                                                                              
  ## Logging                                                                                                                                                                                                                                                  
  ErrorLog "/var/log/httpd/foreman-ssl_error_ssl.log"                                                                                                                                                                                                         
  ServerSignature Off                                                                                                                                                                                                                                         
  CustomLog "/var/log/httpd/foreman-ssl_access_ssl.log" combined                                                                                                                                                                                              
                                                                                                                                                                                                                                                              
  ## Server aliases                                                                                                                                                                                                                                           
  ServerAlias foreman                                                                                                                                                                                                                                         
                                                                                                                                                                                                                                                              
  ## SSL directives
  SSLEngine on
  #SSLCertificateFile      "/etc/puppetlabs/puppet/ssl/certs/urzlxdeploy.rz.uni-leipzig.de.pem"
  SSLCertificateFile      "/etc/ssl/certs/cert-10263428829321932413835964919.pem"
  #SSLCertificateKeyFile   "/etc/puppetlabs/puppet/ssl/private_keys/urzlxdeploy.rz.uni-leipzig.de.pem"
  SSLCertificateKeyFile   "/etc/pki/tls/private/urzlxdeploy.rz.uni-leipzig.de-pkey-nopw.pem"
  SSLCertificateChainFile "/etc/ssl/certs/cachain.crt"
  SSLVerifyClient         optional
  SSLCACertificateFile    "/etc/puppetlabs/puppet/ssl/certs/ca.pem"
  SSLCARevocationFile     "/etc/puppetlabs/puppet/ssl/crl.pem"
  SSLVerifyDepth          3
  SSLCARevocationCheck    "chain"
  SSLOptions +StdEnvVars +ExportCertData
  ## Custom fragment                                                                                                                                                                                                                                 [20/1680]
  # Static public dir serving

<Directory /usr/share/foreman/public>

  <IfVersion < 2.4>
    Allow from all
  </IfVersion>
  <IfVersion >= 2.4>
    Require all granted
  </IfVersion>

</Directory>

<Directory ~ /usr/share/foreman/public/(assets|webpack)>

  # Use standard http expire header for assets instead of ETag
  <IfModule mod_expires.c>
    Header unset ETag
    FileETag None
    ExpiresActive On
    ExpiresDefault "access plus 1 year"
  </IfModule>

  # Return compressed assets if they are precompiled
  <IfModule mod_rewrite.c>
    RewriteEngine on
    # Make sure the browser supports gzip encoding and file with .gz added
    # does exist on disc before we rewrite with the extension
    RewriteCond %{HTTP:Accept-Encoding} \b(x-)?gzip\b
    RewriteCond %{REQUEST_FILENAME} \.(css|js|svg)$
    RewriteCond %{REQUEST_FILENAME}.gz -s
    RewriteRule ^(.+) $1.gz [L]
    # Set headers for all possible assets which are compressed
    <FilesMatch \.css\.gz$>
      ForceType text/css
      Header set Content-Encoding gzip
      SetEnv no-gzip
    </FilesMatch>
    <FilesMatch \.js\.gz$>
      ForceType text/javascript
      Header set Content-Encoding gzip
      SetEnv no-gzip
    </FilesMatch>
    <FilesMatch \.svg\.gz$>
      ForceType image/svg+xml
      Header set Content-Encoding gzip
      SetEnv no-gzip
    </FilesMatch>
  </IfModule>

</Directory>

<IfVersion < 2.4>
  Include /etc/httpd/conf.d/05-foreman-ssl.d/*.conf
</IfVersion>
<IfVersion >= 2.4>
  IncludeOptional /etc/httpd/conf.d/05-foreman-ssl.d/*.conf
</IfVersion>


  PassengerAppRoot /usr/share/foreman
  PassengerRuby /usr/bin/tfm-ruby
  PassengerMinInstances 1
  PassengerStartTimeout 600
  PassengerPreStart https://urzlxdeploy.rz.uni-leipzig.de:443

  AddDefaultCharset UTF-8
  KeepAlive on
  KeepAliveTimeout 5
  MaxKeepAliveRequests 100
</VirtualHost>

Thanks in advance,

Vadim

Hi, could you also post contents of /etc/foreman-proxy/settings.d/dynflow.yml?

Sure, sorry that I misted it :wink:

:enabled: true
:database: /var/lib/foreman-proxy/dynflow/dynflow.sqlite
:core_url: 'http://127.0.0.1:8008'

# If true, external core will be used even if the core gem is available
# If false, the feature will be disabled if the core gem is not available
# If unset, the process will fallback to external core if the core gem is not available
# :external_core: true

Cheers,

Vadim

What is most likely going on is:

  1. You kick off a job
  2. Foreman delegates the job to foreman-proxy
  3. Foreman-proxy delegates it to smart_proxy_dynflow_core
  4. Job runs
  5. smart_proxy_dynflow_core tries to call back to Foreman and fails
  6. Foreman waits around 10 minutes, then checks on the foreman proxy and turns the job over to failed

Does this description fit? If that is the case, contents of /etc/foreman-proxy/settings.yml could be helpful. Also try checking /var/log/httpd/* and /var/log/foreman/* for any POST request going to /api/tasks/.*/callback

Thanks for making it clear for me. Yes - I would think this is this what is going on.

---
### File managed with puppet ###
## Module:           'foreman_proxy'

:settings_directory: /etc/foreman-proxy/settings.d

# SSL Setup

# if enabled, all communication would be verified via SSL
# NOTE that both certificates need to be signed by the same CA in order for this to work
# see http://theforeman.org/projects/smart-proxy/wiki/SSL for more information
:ssl_ca_file: /etc/puppetlabs/puppet/ssl/certs/ca.pem
:ssl_certificate: /etc/puppetlabs/puppet/ssl/certs/urzlxdeploy.rz.uni-leipzig.de.pem
:ssl_private_key: /etc/puppetlabs/puppet/ssl/private_keys/urzlxdeploy.rz.uni-leipzig.de.pem

# Use this option only if you need to disable certain cipher suites.
# Note: we use the OpenSSL suite name, take a look at:
# https://www.openssl.org/docs/manmaster/apps/ciphers.html#CIPHER-SUITE-NAMES
# for more information.
#:ssl_disabled_ciphers: [CIPHER-SUITE-1, CIPHER-SUITE-2]

# the hosts which the proxy accepts connections from
# commenting the following lines would mean every verified SSL connection allowed
:trusted_hosts:
  - urzlxdeploy.rz.uni-leipzig.de

# Endpoint for reverse communication
:foreman_url: https://urzlxdeploy.rz.uni-leipzig.de

# SSL settings for client authentication against Foreman. If undefined, the values
# from general SSL options are used instead. Mainly useful when Foreman uses
# different certificates for its web UI and for smart-proxy requests.
#:foreman_ssl_ca: ssl/certs/ca.pem
#:foreman_ssl_cert: ssl/certs/fqdn.pem
#:foreman_ssl_key: ssl/private_keys/fqdn.pem

# by default smart_proxy runs in the foreground. To enable running as a daemon, uncomment 'daemon' setting
:daemon: true
# Only used when 'daemon' is set to true.
# Uncomment and modify if you want to change the default pid file '/var/run/foreman-proxy/foreman-proxy.pid'
#:daemon_pid: /var/run/foreman-proxy/foreman-proxy.pid

# host and ports configuration
# Host or IP to bind ports to (e.g. *, localhost, 0.0.0.0, ::, 192.168.1.20)
:bind_host: '*'
# http is disabled by default. To enable, uncomment 'http_port' setting
# https is enabled if certificate, CA certificate, and private key are present in locations specifed by
# ssl_certificate, ssl_ca_file, and ssl_private_key correspondingly
# default values for https_port is 8443
:https_port: 8443
:http_port: 8000
# Log configuration
# Uncomment and modify if you want to change the location of the log file or use STDOUT or SYSLOG values
:log_file: /var/log/foreman-proxy/proxy.log
# Uncomment and modify if you want to change the log level
# WARN, DEBUG, ERROR, FATAL, INFO, UNKNOWN
:log_level: WARN

# Log buffer size and extra buffer size (for errors). Defaults to 3000 messages in total,
# which is about 500 kB request.
:log_buffer: 2000
:log_buffer_errors: 1000

Logs:

/var/log/httpd/foreman-ssl_access_ssl.log:

139.18.16.90 - - [30/Jan/2020:11:01:45 +0100] "POST /foreman_tasks/api/tasks/callback HTTP/1.1" 403 97 "-" "rest-client/2.0.2 (linux-gnu x86_64) ruby/2.5.5p157"

No relevant hits under /var/log/foreman/

This probably means httpd rejects the incoming request without even passing it to Foreman. That makes me believe there is something wrong with the way how foreman_ssl_* settings are configured in /etc/smart_proxy_dynflow_core/settings.yml.

I’d try checking if you can take the certs configured there and try to make a request with them (using curl or something) to Foreman.

Is the client cert signed by the same CA as Foreman’s or is the client cert’s CA trusted by Foreman?

Who is talking to Foreman, the remote client or dynflow daemon?

So - this is a local communication

When i start a discovery via pxe and discovery image all facts are sent to Foreman.
Can you give me an example how to write this command using curl?

Cheers,
Vadim

Shell i use the certs from the Puppet-ca for client auth? That would make sense.

If both foreman and smart_proxy_dynflow_core run on the same machine.

Something like this should work. If you get 302, then you have the right certs.

curl --cert /etc/ssl/certs/cert-10263428829321932413835964919.pem \
  --key /etc/pki/tls/private/urzlxdeploy.rz.uni-leipzig.de-pkey-nopw.pem \
  --/etc/puppetlabs/puppet/ssl/certs/ca.pem \
  https://$(hostname -f) -ivv >/dev/null

I think that’s what the installer does by default.

So - i tried it:

this is my first try by using my official ssl cert:

[root@urzlxdeploy settings.d]# curl --cert /etc/ssl/certs/cert-10263428829321932413835964919.pem   --key /etc/pki/tls/private/urzlxdeploy.rz.uni-leipzig.de-pkey-nopw.pem   --cacert /etc/ssl/certs/cachain.crt   https://$(hostname -f) -ivv >/dev/null  
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* About to connect() to urzlxdeploy.rz.uni-leipzig.de port 443 (#0)
*   Trying 139.18.16.90...
* Connected to urzlxdeploy.rz.uni-leipzig.de (139.18.16.90) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/ssl/certs/cachain.crt
  CApath: none
* NSS: client certificate from file
*       subject: CN=urzlxdeploy.rz.uni-leipzig.de,OU=URZ,O=Universitaet Leipzig,L=Leipzig,ST=Sachsen,C=DE
*       start date: Jun 19 13:26:36 2019 GMT
*       expire date: Sep 20 13:26:36 2021 GMT
*       common name: urzlxdeploy.rz.uni-leipzig.de
*       issuer: CN=DFN-Verein Global Issuing CA,OU=DFN-PKI,O=Verein zur Foerderung eines Deutschen Forschungsnetzes e. V.,C=DE
* NSS error -12195 (SSL_ERROR_UNKNOWN_CA_ALERT)
* Peer does not recognize and trust the CA that issued your certificate.
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Closing connection 0
curl: (35) Peer does not recognize and trust the CA that issued your certificate.

and this my second try by using certififace of the puppet-ca:

[root@urzlxdeploy settings.d]# curl --cert /etc/puppetlabs/puppet/ssl/certs/urzlxdeploy.rz.uni-leipzig.de.pem --key /etc/puppetlabs/puppet/ssl/private_keys/urzlxdeploy.rz.uni-leipzig.de.pem --cacert /etc/puppetlabs/puppet/ssl/certs/ca.pem https://$(hostname -f) -ivv >/dev/null 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* About to connect() to urzlxdeploy.rz.uni-leipzig.de port 443 (#0)
*   Trying 139.18.16.90...
* Connected to urzlxdeploy.rz.uni-leipzig.de (139.18.16.90) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/puppetlabs/puppet/ssl/certs/ca.pem
  CApath: none
* Server certificate:
*       subject: CN=urzlxdeploy.rz.uni-leipzig.de,OU=URZ,O=Universitaet Leipzig,L=Leipzig,ST=Sachsen,C=DE
*       start date: Jun 19 13:26:36 2019 GMT
*       expire date: Sep 20 13:26:36 2021 GMT
*       common name: urzlxdeploy.rz.uni-leipzig.de
*       issuer: CN=DFN-Verein Global Issuing CA,OU=DFN-PKI,O=Verein zur Foerderung eines Deutschen Forschungsnetzes e. V.,C=DE
* NSS error -8172 (SEC_ERROR_UNTRUSTED_ISSUER)
* Peer's certificate issuer has been marked as not trusted by the user.
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Closing connection 0
curl: (60) Peer's certificate issuer has been marked as not trusted by the user.
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

This is what i get, if i don’t use client auth:

[root@urzlxdeploy settings.d]# curl -ivv https://$(hostname -f) -ivv >/dev/null                                                                                                                                     
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* About to connect() to urzlxdeploy.rz.uni-leipzig.de port 443 (#0)
*   Trying 139.18.16.90...
* Connected to urzlxdeploy.rz.uni-leipzig.de (139.18.16.90) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* NSS: client certificate not found (nickname not specified)
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
*       subject: CN=urzlxdeploy.rz.uni-leipzig.de,OU=URZ,O=Universitaet Leipzig,L=Leipzig,ST=Sachsen,C=DE
*       start date: Jun 19 13:26:36 2019 GMT
*       expire date: Sep 20 13:26:36 2021 GMT
*       common name: urzlxdeploy.rz.uni-leipzig.de
*       issuer: CN=DFN-Verein Global Issuing CA,OU=DFN-PKI,O=Verein zur Foerderung eines Deutschen Forschungsnetzes e. V.,C=DE
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: urzlxdeploy.rz.uni-leipzig.de
> Accept: */*
>
< HTTP/1.1 302 Found
< Date: Thu, 30 Jan 2020 13:15:02 GMT
< Server: Apache/2.4.6 (CentOS)
< Cache-Control: no-cache
< X-Request-Id: 31339aeb-a66d-45d7-a7ac-2ab8eaa4e57d
< X-Runtime: 0.011926
< Strict-Transport-Security: max-age=631139040; includeSubdomains
< X-Frame-Options: sameorigin
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< X-Download-Options: noopen
< X-Permitted-Cross-Domain-Policies: none
< Content-Security-Policy: default-src 'self'; child-src 'self'; connect-src 'self' ws: wss:; img-src 'self' data: *.gravatar.com; script-src 'unsafe-eval' 'unsafe-inline' 'self'; style-src 'unsafe-inline' 'self'
< X-Powered-By: Phusion Passenger 4.0.53
< Set-Cookie: _session_id=09483f10255445983d2493511da29630; path=/; secure; HttpOnly; SameSite=Lax
< Location: https://urzlxdeploy.rz.uni-leipzig.de/users/login
< Status: 302 Found
< Transfer-Encoding: chunked
< Content-Type: text/html; charset=utf-8
<
{ [data not shown]
100   115    0   115    0     0    850      0 --:--:-- --:--:-- --:--:--   851
* Connection #0 to host urzlxdeploy.rz.uni-leipzig.de left intact

What if you use cert and key from puppet, but /etc/ssl/certs/cachain.crt for CA cert?

Wow - this looks much better or did i misinterpret what? :

[root@urzlxdeploy settings.d]# curl --cert /etc/puppetlabs/puppet/ssl/certs/urzlxdeploy.rz.uni-leipzig.de.pem --key /etc/puppetlabs/puppet/ssl/private_keys/urzlxdeploy.rz.uni-leipzig.de.pem --cacert /etc/ssl/certs/cachain.crt https://$(hostname -f) -ivv >/dev/null 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                                                                                                                                                                                          
                                 Dload  Upload   Total   Spent    Left  Speed                                                                                                                                                                                            
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* About to connect() to urzlxdeploy.rz.uni-leipzig.de port 443 (#0)                                                                                                                        
*   Trying 139.18.16.90...                                                                                                                                                                                                                                               
* Connected to urzlxdeploy.rz.uni-leipzig.de (139.18.16.90) port 443 (#0)                                                                                                                                                                                                
* Initializing NSS with certpath: sql:/etc/pki/nssdb                                                                                                                                                                                                                     
*   CAfile: /etc/ssl/certs/cachain.crt                                                                                                                                                                                                                                   
  CApath: none                                                                                                                                                                                                                                                           
* NSS: client certificate from file                                                                                                                                                                                                                                      
*       subject: CN=urzlxdeploy.rz.uni-leipzig.de                                                                                                                                                                                                                        
*       start date: Oct 30 16:47:52 2017 GMT                                                                                                                                                                                                                             
*       expire date: Oct 30 16:47:52 2022 GMT                                                                                                                                                                                                                            
*       common name: urzlxdeploy.rz.uni-leipzig.de                                                                                                                                                                                                                       
*       issuer: CN=Puppet CA: urzlxdeploy.rz.uni-leipzig.de                                                                                                                                                                                                              
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256                                                                                                                                                                                                             
* Server certificate:                                                                                                                                                                                                                                                    
*       subject: CN=urzlxdeploy.rz.uni-leipzig.de,OU=URZ,O=Universitaet Leipzig,L=Leipzig,ST=Sachsen,C=DE                                                                                                                                                                
*       start date: Jun 19 13:26:36 2019 GMT                                                                                                                                                                                                                             
*       expire date: Sep 20 13:26:36 2021 GMT                                                                                                                                                                                                                            
*       common name: urzlxdeploy.rz.uni-leipzig.de                                                                                                                                                                                                                       
*       issuer: CN=DFN-Verein Global Issuing CA,OU=DFN-PKI,O=Verein zur Foerderung eines Deutschen Forschungsnetzes e. V.,C=DE                                                                                                                                           
> GET / HTTP/1.1                                                                                                                                                                                                                                                         
> User-Agent: curl/7.29.0                                                                                                                                                                                                                                                
> Host: urzlxdeploy.rz.uni-leipzig.de                                                                                                                                                                                                                                    
> Accept: */*                                                                                                                                                                                                                                                            
>                                                                                                                                                                                                                                                                        
< HTTP/1.1 302 Found                                                                                                                                                                                                                                                     
< Date: Thu, 30 Jan 2020 13:30:14 GMT                                                                                                                                                                                                                                    
< Server: Apache/2.4.6 (CentOS)                                                                                                                                                                                                                                          
< Cache-Control: no-cache                                                                                                                                                                                                                                                
< X-Request-Id: 54ed4e7f-2111-4075-ace8-56e5e0104f4e                                                                                                                                                                                                                     
< X-Runtime: 0.012448                                                                                                                                                                                                                                                    
< Strict-Transport-Security: max-age=631139040; includeSubdomains                                                                                                                                                                                                        
< X-Frame-Options: sameorigin                                                                                                                                                                                                                                            
< X-Content-Type-Options: nosniff                                                                                                                                                                                                                                        
< X-XSS-Protection: 1; mode=block                                                                                                                                                                                                                                        
< X-Download-Options: noopen                                                                                                                                                                                                                                             
< X-Permitted-Cross-Domain-Policies: none                                                                                                                                                                                                                                
< Content-Security-Policy: default-src 'self'; child-src 'self'; connect-src 'self' ws: wss:; img-src 'self' data: *.gravatar.com; script-src 'unsafe-eval' 'unsafe-inline' 'self'; style-src 'unsafe-inline' 'self'                                                     
< X-Powered-By: Phusion Passenger 4.0.53                                                                                                                                                                                                                                 
< Set-Cookie: _session_id=f4ffcc98dc7c2241ec402cd2e9327b0c; path=/; secure; HttpOnly; SameSite=Lax                                                                                                                                                                       
< Location: https://urzlxdeploy.rz.uni-leipzig.de/users/login                                                                                                                                                                                                            
< Status: 302 Found                                                                                                                                                                                                                                                      
< Transfer-Encoding: chunked                                                                                                                                                                                                                                             
< Content-Type: text/html; charset=utf-8                                                                                                                                                                                                                                 
<                                                                                                                                                                                                                                                                        
{ [data not shown]                                                                                                                                                                                                                                                       
100   115    0   115    0     0    788      0 --:--:-- --:--:-- --:--:--   793                                                                                                                                                                                           
* Connection #0 to host urzlxdeploy.rz.uni-leipzig.de left intact

I changed /etc/smart_proxy_dynflow_core/settings.yml and restarted dynflow daemon - no success.


Path to dynflow database, leave blank for in-memory non-persistent database
database:

URL of the foreman, used for reporting back
foreman_url: ‘https://urzlxdeploy.rz.uni-leipzig.de:443

SSL settings for client authentication against Foreman
:foreman_ssl_ca: ssl/foreman_ca.pem
:foreman_ssl_ca: /etc/puppetlabs/puppet/ssl/certs/ca.pem
foreman_ssl_ca: /etc/ssl/certs/cachain.crt
:foreman_ssl_ca: /etc/puppetlabs/puppet/ssl/certs/urzlxdeploy.rz.uni-leipzig.de.pem
:foreman_ssl_key: ssl/foreman_key.pem
:foreman_ssl_key: /etc/pki/tls/private/urzlxdeploy.rz.uni-leipzig.de-pkey-nopw.pem
foreman_ssl_key: /etc/puppetlabs/puppet/ssl/private_keys/urzlxdeploy.rz.uni-leipzig.de.pem
:foreman_ssl_cert: ssl/foreman_cert.pem
:foreman_ssl_cert: /etc/ssl/certs/cert-10263428829321932413835964919.pem
foreman_ssl_cert: /etc/puppetlabs/puppet/ssl/private_keys/urzlxdeploy.rz.uni-leipzig.de.pem

console_auth: false

Set to true to make the core fork to background after start
:daemonize: false
:pid_file: /var/run/foreman-proxy/smart_proxy_dynflow_core.pid

Listen on address
listen: 127.0.0.1

Listen on port
port: 8008

SSL settings for running core as https service
:use_https: false
:ssl_ca_file: ssl/ca.pem
:ssl_private_key: ssl/localhost.pem
:ssl_certificate: ssl/certs/localhost.pem

Use this option only if you need to disable certain cipher suites.
Note: we use the OpenSSL suite name, take a look at:
https://www.openssl.org/docs/manmaster/apps/ciphers.html#CIPHER-SUITE-NAMES
for more information.
:ssl_disabled_ciphers: [CIPHER-SUITE-1, CIPHER-SUITE-2]

Use this option only if you need to strictly specify TLS versions to be
disabled. SSLv3 and TLS v1.0 are always disabled and cannot be configured.
Specify versions like: ‘1.1’, or ‘1.2’
:tls_disabled_versions:

File to log to, leave empty for logging to STDOUT
log_file: /var/log/foreman-proxy/smart_proxy_dynflow_core.log

Log level, one of UNKNOWN, FATAL, ERROR, WARN, INFO, DEBUG
log_level: WARN

Maximum age of execution plans to keep before having them cleaned
by the execution plan cleaner (in seconds), defaults to 24 hours
:execution_plan_cleaner_age: 86400

As far as I can tell, that looks good.

I’m not sure if the file got mangled in-flight, but it looks wrong. You have duplicate keys there (last one wins) and you’re mixing strings and symbols as keys.

Sorry - my fault. This is my config again:

--
# Path to dynflow database, leave blank for in-memory non-persistent database
:database:

# URL of the foreman, used for reporting back
:foreman_url: 'https://urzlxdeploy.rz.uni-leipzig.de:443'

# SSL settings for client authentication against Foreman
# :foreman_ssl_ca: ssl/foreman_ca.pem
#:foreman_ssl_ca: /etc/puppetlabs/puppet/ssl/certs/ca.pem
:foreman_ssl_ca: /etc/ssl/certs/cachain.crt
#:foreman_ssl_ca: /etc/puppetlabs/puppet/ssl/certs/urzlxdeploy.rz.uni-leipzig.de.pem
# :foreman_ssl_key: ssl/foreman_key.pem
# :foreman_ssl_key: /etc/pki/tls/private/urzlxdeploy.rz.uni-leipzig.de-pkey-nopw.pem
:foreman_ssl_key: /etc/puppetlabs/puppet/ssl/private_keys/urzlxdeploy.rz.uni-leipzig.de.pem
# :foreman_ssl_cert: ssl/foreman_cert.pem
# :foreman_ssl_cert: /etc/ssl/certs/cert-10263428829321932413835964919.pem
:foreman_ssl_cert: /etc/puppetlabs/puppet/ssl/private_keys/urzlxdeploy.rz.uni-leipzig.de.pem

:console_auth: false

# Set to true to make the core fork to background after start
# :daemonize: false
# :pid_file: /var/run/foreman-proxy/smart_proxy_dynflow_core.pid

# Listen on address
:listen: 127.0.0.1

# Listen on port
:port: 8008

# SSL settings for running core as https service
# :use_https: false
# :ssl_ca_file: ssl/ca.pem
# :ssl_private_key: ssl/localhost.pem
# :ssl_certificate: ssl/certs/localhost.pem

# Use this option only if you need to disable certain cipher suites.
# Note: we use the OpenSSL suite name, take a look at:
# https://www.openssl.org/docs/manmaster/apps/ciphers.html#CIPHER-SUITE-NAMES
# for more information.
#:ssl_disabled_ciphers: [CIPHER-SUITE-1, CIPHER-SUITE-2]

# Use this option only if you need to strictly specify TLS versions to be
# disabled. SSLv3 and TLS v1.0 are always disabled and cannot be configured.
# Specify versions like: '1.1', or '1.2'
#:tls_disabled_versions: []

# File to log to, leave empty for logging to STDOUT
:log_file: /var/log/foreman-proxy/smart_proxy_dynflow_core.log

# Log level, one of UNKNOWN, FATAL, ERROR, WARN, INFO, DEBUG
:log_level: WARN

# Maximum age of execution plans to keep before having them cleaned
# by the execution plan cleaner (in seconds), defaults to 24 hours
# :execution_plan_cleaner_age: 86400

I guess we found a solution!

I still had an error in my configuration. The foreman_ssl_cert: pointed to the key instead. I changed that and now it runs.

Thanks Adam, great support!

It would be nice if it gets documented somehow . It is not clear, if you rely on official certificate for the websocket that you have to use the chain of the websocket certificate and the rest from the private puppet ca .

Anyway thanks again!

Cheers, Vadim