Remote Execution jobs run then are marked failed after 10 minutes

Support
1

Problem: After Katello upgrade to 3.12, Remote Execution jobs still run, but eventually report as failed. This occurs running jobs as Ansible Commands or SSH commands.

Expected outcome: Remote Execution jobs should run and report as completed successfully.

Foreman and Proxy versions:
katello-3.12.3-1.el7.noarch
foreman-1.22.1-1.el7.noarch
foreman-proxy-1.22.1-1.el7.noarch

Foreman and Proxy plugin versions:
tfm-rubygem-foreman_ansible-3.0.3-1.fm1_22.el7.noarch
tfm-rubygem-foreman_remote_execution_core-1.2.0-1.el7.noarch
tfm-rubygem-foreman-tasks-0.15.11-1.fm1_22.el7.noarch
tfm-rubygem-foreman_ansible_core-3.0.1-1.fm1_22.el7.noarch
tfm-rubygem-hammer_cli_foreman_tasks-0.0.13-1.fm1_22.el7.noarch
tfm-rubygem-foreman_remote_execution-1.8.2-1.fm1_22.el7.noarch
rubygem-smart_proxy_dynflow-0.2.3-1.el7.noarch
rubygem-smart_proxy_remote_execution_ssh-0.2.1-1.el7.noarch
rubygem-smart_proxy_pulp-1.4.1-1.el7.noarch
tfm-rubygem-smart_proxy_dynflow_core-0.2.2-1.fm1_22.el7.noarch
rubygem-smart_proxy_ansible-3.0.1-1.fm1_22.el7.noarch

Distribution and version:
Red Hat Enterprise Linux 7.7
Uname: 3.10.0-1062.4.1.el7.x86_64 #1 SMP Wed Sep 25 09:42:57 EDT 2019 x86_64 x86_64 x86_64 GNU/Linux

Other relevant data:
Server originally installed with documentation Foreman :: Plugin Manuals
Upgraded to 3.11, all functionality working.
Upgraded to 3.12, issues began.

**Processes and ports they are running on:
port 8000 and 9090
foreman+ 1974 ruby /usr/share/foreman-proxy/bin/smart-proxy --no-daemonize

port 8008
foreman+ 3287 ruby /usr/bin/smart_proxy_dynflow_core -d -p /var/run/foreman-proxy/smart_proxy_dynflow_core.pid

port 8443 and 8005
tomcat 1984 /usr/lib/jvm/jre/bin/java -Xms1024m -Xmx4096m -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar -Dcatalina.base=/usr/share/tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/cache/tomcat/temp -Djava.util.logging.config.file=/usr/share/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager org.apache.catalina.startup.Bootstrap start

port 80
root 2070 /usr/sbin/httpd -DFOREGROUND

port 8140
puppet 2082 /usr/bin/java -Xms2G -Xmx2G -Djruby.logger.class=com.puppetlabs.jruby_utils.jruby.Slf4jLogger -Djava.security.egd=file:/dev/urandom -XX:OnOutOfMemoryError=kill -9 %p -cp /opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar:/opt/puppetlabs/server/apps/puppetserver/jruby-1_7.jar:/opt/puppetlabs/server/data/puppetserver/jars/* clojure.main -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetserver/conf.d --bootstrap-config /etc/puppetlabs/puppetserver/services.d/,/opt/puppetlabs/server/apps/puppetserver/config/services.d/ --restart-file /opt/puppetlabs/server/data/puppetserver/restartcounter

Settings
/etc/foreman-proxy/settings.yml
:ssl_ca_file: /etc/foreman-proxy/ssl_ca.pem
:ssl_certificate: /etc/foreman-proxy/ssl_cert.pem
:ssl_private_key: /etc/foreman-proxy/ssl_key.pem
:trusted_hosts:

Endpoint for reverse communication

:foreman_url: http://kat7.xxx.xxx.com
:bind_host: ‘*’
:https_port: 9090
:http_port: 8000

/etc/smart_proxy_dynflow_core/settings.yml
:foreman_url: http://kat7.xxx.xxx.com
:listen: 170.xxx.xxx.xxx

Listen on port

:port: 8008

/etc/foreman-proxy/settings.d/dynflow.yml
:enabled: true
#:database: /var/lib/foreman-proxy/dynflow/dynflow.sqlite
:database:
:core_url: ‘http://kat7.xxx.xxx.com:8008’

/etc/foreman-proxy/settings.d/remote_execution_ssh.yml
:enabled: true
:ssh_identity_key_file: /var/lib/foreman-proxy/ssh/id_rsa_foreman_proxy
:local_working_dir: /var/tmp
:remote_working_dir: /var/tmp
:kerberos_auth: false
:async_ssh: false

Do I have any settings configured incorrectly?

2

Hi, it is hard to say, but having :foreman_url: http://kat7.xxx.xxx.com in /etc/smart_proxy_dynflow_core/settings.yml is suspicious. Shouldn’t that use https?

Try checking httpd logs and production.log if the callback from the smart proxy ever reaches foreman.

3

Thanks for the quick response. I changed that to https, and am still getting the same result. I see this in the Dynflow console for the task.
Foreman::Exception

ERF42-1682 [Foreman::Exception]: The smart proxy task 026f1dbc-71f2-4bdd-bab4-5d91f8f4545a failed.

4

I also found these in foreman-ssl_error_ssl.log

[Tue Nov 19 13:34:17.485856 2019] [ssl:error] [pid 2506] [client 170.4.83.147:47174] AH02039: Certificate Verification: Error (26): unsupported certificate purpose
[Tue Nov 19 13:34:38.425112 2019] [ssl:warn] [pid 2510] [client 170.4.83.147:47252] AH02227: Failed to set r->user to ‘SSL_CLIENT_S_DN_CN’
[Tue Nov 19 13:34:38.451156 2019] [ssl:warn] [pid 15980] [client 170.4.83.147:47254] AH02227: Failed to set r->user to ‘SSL_CLIENT_S_DN_CN’
[Tue Nov 19 13:56:40.216421 2019] [ssl:error] [pid 7805] [client 170.4.83.147:49080] AH02039: Certificate Verification: Error (26): unsupported certificate purpose

5

[19/Nov/2019:13:34:13 EST] “POST /tasks/launch? HTTP/1.1” 200 110
SSL_connect returned=1 errno=0 state=error: sslv3 alert unsupported certificate (OpenSSL::SSL::SSLError)

/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/concurrent-ruby-1.1.4/lib/concurrent/executor/ruby_thread_po
ol_executor.rb:319:in block in create_worker'A sub task failed (RuntimeError) /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/action/with_sub_plans.rb:230:incheck_f
or_errors!’

SSL_connect returned=1 errno=0 state=error: sslv3 alert unsupported certificate (OpenSSL::SSL::SSLError)

OK, so maybe there is a problem with my certs? I do not use puppet. Can someone point me to documentation on how to recreate the appropriate certs, if you feel that is necessary?

thanks