Build Procedure for testing TheForeman-2.1rc2 Remote Execution ============================================================== This is a clean build of two systems from scratch. One is CentOS 7.6 x86_64 and one is CentOS 8.1 x86_64. I was able to reproduce the problem with Remote Execution on the new minimal CentOS 8 foreman 2.1rc2 system. The original problem was discovered on another CentOS 8.1 VM, with a lot more provisioning features configured and running. The problem was also present in foreman 2.1rc1. Note: I have replaced all hostnames, doamin names etc with generic replacements of exactly the same length :-) Software: Foreman-2.1rc2 OS: Clean ISO image based VMs + yum/dnf update CentOS 7.6 - virtualcentos7.sc.mydomain.net.au CentOS 8.1 - virtualcentos8.sc.mydomain.net.au Platform: VMware vSphere / ESXi 6.7U3 VMs: 2 CPUs / 4GB RAM / 16GB Disk (thin) Manual: https://theforeman.org/manuals/2.1/index.html Note: Using account 'admin' with sudo privileges created by the OS installer Note: Common commands will NOT be prefixed with OS version (easier copy/paste) CentOS7: echo PS1="'"'[\u@\h \W] CentOS 7 \$ '"'" >> .bashrc CentOS8: echo PS1="'"'[\u@\h \W] CentOS 8 \$ '"'" >> .bashrc echo "set enable-bracketed-paste on" >> ~/.inputrc exec bash -l echo 'Defaults timestamp_timeout = 120' | sudo tee -a /etc/sudoers sudo -l # Verify the basics hostnamectl timedatectl # Note: /usr/bin/yum on CentOS 8 is a symlink to dnf-3, # so I'm going to use yum(8) on the command line, # to reduce the need for unique CentOS 7 / 8 command lines sudo yum makecache sudo yum upgrade sudo systemctl reboot CentOS7: sudo yum -y install https://yum.puppet.com/puppet6-release-el-7.noarch.rpm sudo yum -y install http://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm sudo yum -y install https://yum.theforeman.org/releases/2.1/el7/x86_64/foreman-release.rpm sudo yum -y install foreman-release-scl CentOS8: sudo yum -y install https://yum.puppet.com/puppet-release-el-8.noarch.rpm # IMPORTANT NOTE: From 2.1rc1 experience don't install the EPEL package, as # version conflicts exist for rubygems between EPEL and Foreman Repos, # seen after fully installing Foreman and running 'sudo dnf yum upgrade' sudo dnf -y install https://yum.theforeman.org/releases/2.1/el8/x86_64/foreman-release.rpm # Back to common commands sudo yum makecache sudo yum repolist # White space reduced Result on CentOS7: repo id repo name status base/7/x86_64 CentOS-7 - Base 10,070 centos-sclo-rh/x86_64 CentOS-7 - SCLo rh 6,509 epel/x86_64 Extra Packages ... Linux 7 - x86_64 13,311 extras/7/x86_64 CentOS-7 - Extras 397 foreman/x86_64 Foreman 2.1 626 foreman-plugins/x86_64 Foreman plugins 2.1 357 puppet6/x86_64 Puppet 6 Repository el 7 - x86_64 223 updates/7/x86_64 CentOS-7 - Updates 737 Result on CentOS8: repo id repo name status AppStream CentOS-8 - AppStream 5,318 BaseOS CentOS-8 - Base 1,661 extras CentOS-8 - Extras 20 foreman Foreman 2.1 606 foreman-plugins Foreman plugins 2.1 351 puppet Puppet Repository el 8 - x86_64 170 # Some convenience tools sudo yum install bash-completion sudo yum install strace lsof iputils tcpdump exec bash -l # Checks for Foreman Requirements # Ensure that ping $(hostname -f) shows the real IP address, not # 127.0.0.1. Change or remove this entry from /etc/hosts if present. ping $(hostname -f) # Check that the umask is 0022 sudo umask # The follow software will be used on this foreman instance, # so lets install it ahead of time. # - DHCP/PXE Client boot support => DHCP Server # - TFTP boot support => TFTP Server (and client) # - omshell(1) passwordless use => BIND Tools dnssec-keygen(8) # - Email Notifications => Postfix Mail Server CentOS7: sudo yum -y install dhcp tftp tftp-server bind-utils postfix CentOS8: sudo yum -y install dhcp-server tftp tftp-server bind-utils postfix sudo systemctl enable --now tftp sudo systemctl enable --now postfix sudo systemctl status tftp postfix # For later kickstart template development sudo yum -y install pykickstart # Install the Foreman Installer sudo yum -y install foreman-installer # Verify everything is up-to-date (both systems required no changes) sudo yum upgrade # Shutdown the VMs; Snapshot them (Named: Basics Completed); Start VMs sudo systemctl poweroff # # Configuration : Foreman All-In-One # (i.e. Foreman + Smart-Proxy & tools on the same host) # Purpose : System Provisioning # # Smart Proxy Features : DHCP, TFTP, HTTPBoot, ... (in phase two of deployment) # # Smart-Proxy DISABLED Features: DNS (i.e. DNS Updating via nsupdate(8)) # # hammer(1) CLI: Enabling for all install plugins # # Phase 1 - Basic Foreman # Note: The first two blocks of options are the defaults, and they # are included on the command line to show the basic configuration, # and that third block shows we are not configuring any smart # proxies related to provisioning (as they need a lot more # options to configure correctly) sudo foreman-installer \ \ --enable-foreman \ --enable-foreman-cli \ --enable-foreman-proxy \ --enable-puppet \ \ --foreman-proxy-puppet true \ --foreman-proxy-puppetca true \ --foreman-proxy-realm false \ --foreman-proxy-ssl true \ --foreman-proxy-templates false \ \ --foreman-proxy-bmc false \ --foreman-proxy-dhcp false \ --foreman-proxy-dns false \ --foreman-proxy-http false \ --foreman-proxy-httpboot false \ --foreman-proxy-tftp false \ \ --foreman-initial-organization 'My Organisation.' \ --foreman-initial-location 'My Location....' \ --foreman-email-smtp-domain 'mydomain.net.au' \ \ --foreman-initial-admin-username 'admin' \ --foreman-initial-admin-password 'ChangeMe!' \ --foreman-initial-admin-email 'System-Administrator@mydomain.net.au' \ --foreman-initial-admin-first-name 'System' \ --foreman-initial-admin-last-name 'Administrator' \ # Expected Result Preparing installation Done foreman-rake upgrade:run finished successfully! Success! * Foreman is running at https://<>.sc.mydomain.net.au Initial credentials are admin / ChangeMe! * Foreman Proxy is running at https://<>.sc.mydomain.net.au:8443 The full log is at /var/log/foreman-installer/foreman.log # Success on both CentOS 7 & 8 as expected. # Configure the system's firewall to allow remote WebUI access sudo firewall-cmd --add-service=http --permanent sudo firewall-cmd --add-service=https --permanent sudo firewall-cmd --reload sudo firewall-cmd --list-all | egrep -v ': $' # Log in to both WebUI and change the admin password and check Smart Proxies # Now instead of configuring all the provisioning infrastructure, lets # do the Remote Execution facility that is causing the problem. # Manual: https://theforeman.org/plugins/foreman_remote_execution/1.7/index.html sudo foreman-installer \ --enable-foreman-plugin-remote-execution \ --enable-foreman-proxy-plugin-remote-execution-ssh CentOS 7: sudo yum list installed | egrep -i -A 1 'remote|dyn' # Reformatted foreman-dynflow-sidekiq.noarch 2.1.0-0.20.rc2.el7 @foreman foreman-installer.noarch 1:2.1.0-0.3.rc2.el7 @foreman tfm-rubygem-dynflow.noarch 1.4.4-1.fm2_1.el7 @foreman tfm-rubygem-erubi.noarch 1.9.0-1.el7 @foreman tfm-rubygem-foreman_remote_execution.noarch 3.2.1-1.fm2_1.el7 @foreman-plugins tfm-rubygem-foreman_remote_execution_core.noarch 1.3.0-1.el7 @foreman-plugins tfm-rubygem-smart_proxy_dynflow.noarch 0.2.4-5.fm2_1.el7 @foreman-plugins tfm-rubygem-smart_proxy_dynflow_core.noarch 0.2.5-1.fm2_1.el7 @foreman-plugins tfm-rubygem-smart_proxy_remote_execution_ssh.noarch 0.3.0-3.fm2_1.el7 @foreman-plugins CentOS 8: sudo yum list installed | egrep -i 'remote|dyn' foreman-dynflow-sidekiq.noarch 2.1.0-0.20.rc2.el8 @foreman rubygem-dynflow.noarch 1.4.4-1.fm2_1.el8 @foreman rubygem-foreman_remote_execution.noarch 3.2.1-1.fm2_1.el8 @foreman-plugins rubygem-foreman_remote_execution_core.noarch 1.3.0-1.el8 @foreman-plugins rubygem-smart_proxy_dynflow.noarch 0.2.4-5.fm2_1.el8 @foreman-plugins rubygem-smart_proxy_dynflow_core.noarch 0.2.5-1.fm2_1.el8 @foreman-plugins rubygem-smart_proxy_remote_execution_ssh.noarch 0.3.0-3.fm2_1.el8 @foreman-plugins # Check SSH setup and ensure the server has access to the target # (which for this first test will be itself) sudo ls -alL ~foreman-proxy/.ssh TEST_TARGET=$(hostname -f) sudo \ ssh-copy-id -i ~foreman-proxy/.ssh/id_rsa_foreman_proxy.pub \ root@${TEST_TARGET}: sudo -u foreman-proxy \ ssh -i /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy \ root@${TEST_TARGET} ls -als # Check the server is publishing its SSH public key curl -k https://$(hostname -f):8443/ssh/pubkey # Check the WebUI -> Infrastructure -> Smart Proxies, that your # server is listing 'DynFlow' and 'SSH' in the 'Active features' # list on the 'Overview' tab, and the 'Services' tab should list # these features' version number. Currently (2.1rc2) they is # DynFlow 0.2.4, and SSH 0.3.0. # Check the WebUI -> Monitor -> Tasks and WebUI -> Monitor -> # 'Recurring logics' pages can both be displayed. # Goto WebUI -> Hosts -> All Hosts, select the Foreman server, # and use the 'Select Action' menu to select 'Schedule Remote Job'. # Select a Job to run: # Try 1: Packages -> Check Update - SSH Default; then Submit. # When the task succeeds or fails, click on the host name to # see the command output and/or error messages. CentOS 7: 1: Loaded plugins: fastestmirror 2: Loading mirror speeds from cached hostfile 3: * base: mirror.ventraip.net.au 4: * centos-sclo-rh: mirror.ventraip.net.au 5: * epel: mirror.realcompute.io 6: * extras: mirror.ventraip.net.au 7: * updates: mirror.ventraip.net.au 8: Exit status: 0 CentOS 8: 1: Initialization error: RestClient::NotFound - 404 Not Found 2: Failed to initialize: TypeError - Value (ActiveSupport::TimeWithZone) '2020-06-05 15:02:46 +1000' is not any of: (Time | NilClass). # Try 2: Commands -> Run Command - SSH Default; Enter a # command (say: echo "Hello World"), then Submit. # When the task succeeds or fails, click on the host name to # see the command output and/or error messages. CentOS 7: 1: Hello World 2: Exit status: 0 CentOS 8: 1: Initialization error: RestClient::NotFound - 404 Not Found 2: Failed to initialize: TypeError - Value (ActiveSupport::TimeWithZone) '2020-06-05 15:10:46 +1000' is not any of: (Time | NilClass). # Looking for clues: # I have used '***' at the end of problematic log file lines sudo find /var/log/httpd /var/log/foreman* -type f -ls # This log shows that CentOS 8 is having issues with /tasks/launch sudo less /var/log/foreman-proxy/proxy.log # CentOS 7: (Last Few Entries Log) 2020-06-05T15:04:12 f13c7994 [I] Started GET /version 2020-06-05T15:04:12 f13c7994 [I] Finished GET /version with 200 (0.3 ms) 2020-06-05T15:04:12 de0efefa [I] Started POST /dynflow/tasks/launch 2020-06-05T15:04:12 de0efefa [I] Finished POST /dynflow/tasks/launch with 200 (31.93 ms) 2020-06-05T15:06:38 3ed9700b [I] Started GET /dynflow/tasks/count state=running 2020-06-05T15:06:38 3ed9700b [I] Finished GET /dynflow/tasks/count with 200 (16.39 ms) 2020-06-05T15:06:38 32468c7e [I] Started POST /dynflow/tasks/launch 2020-06-05T15:06:38 32468c7e [I] Finished POST /dynflow/tasks/launch with 200 (44.25 ms) # CentOS 8: (Last Few Entries Log) 2020-06-05T15:10:30 7c80c898 [I] Started GET /version 2020-06-05T15:10:30 7c80c898 [I] Finished GET /version with 200 (0.35 ms) 2020-06-05T15:10:31 428cc829 [I] Started POST /dynflow/tasks/launch 2020-06-05T15:10:31 428cc829 [I] Finished POST /dynflow/tasks/launch with 404 (16.86 ms) *** 2020-06-05T15:10:31 c7f52c44 [I] Started POST /dynflow/tasks/ 2020-06-05T15:10:31 c7f52c44 [I] Finished POST /dynflow/tasks/ with 500 (34.28 ms) *** 2020-06-05T15:10:31 18990529 [I] Started GET /dynflow/tasks/status 2020-06-05T15:10:31 18990529 [I] Finished GET /dynflow/tasks/status with 404 (21.94 ms) *** 2020-06-05T15:10:31 41c3b647 [I] Started POST /dynflow/tasks/launch 2020-06-05T15:10:31 41c3b647 [I] Finished POST /dynflow/tasks/launch with 404 (13.92 ms) *** 2020-06-05T15:10:31 0ddb9aeb [I] Started POST /dynflow/tasks/ 2020-06-05T15:10:31 0ddb9aeb [I] Finished POST /dynflow/tasks/ with 500 (33.12 ms) *** # This log shows that CentOS 8 is having issues with /tasks/launch? sudo less /var/log/foreman-proxy/smart_proxy_dynflow_core.log # CentOS 7: (Entire Log) 172.11.55.123 - - [05/Jun/2020:15:04:12 AEST] "GET /tasks/count?state=running HTTP/1.1" 200 29 172.11.55.123 - - [05/Jun/2020:15:04:12 AEST] "POST /tasks/launch? HTTP/1.1" 200 110 172.11.55.123 - - [05/Jun/2020:15:06:38 AEST] "GET /tasks/count?state=running HTTP/1.1" 200 29 172.11.55.123 - - [05/Jun/2020:15:06:38 AEST] "POST /tasks/launch? HTTP/1.1" 200 110 # CentOS 8: (Entire Log) 172.11.55.124 - - [05/Jun/2020:15:02:30 AEST] "GET /tasks/count?state=running HTTP/1.1" 200 29 172.11.55.124 - - [05/Jun/2020:15:02:31 AEST] "POST /tasks/launch? HTTP/1.1" 404 46 *** 172.11.55.124 - - [05/Jun/2020:15:02:31 AEST] "POST /tasks/? HTTP/1.1" 500 153486 *** 172.11.55.124 - - [05/Jun/2020:15:02:31 AEST] "GET /tasks/status? HTTP/1.1" 404 555 *** 172.11.55.124 - - [05/Jun/2020:15:02:31 AEST] "POST /tasks/launch? HTTP/1.1" 404 46 *** 172.11.55.124 - - [05/Jun/2020:15:02:31 AEST] "POST /tasks/? HTTP/1.1" 500 153525 *** 172.11.55.124 - - [05/Jun/2020:15:10:30 AEST] "GET /tasks/count?state=running HTTP/1.1" 200 29 172.11.55.124 - - [05/Jun/2020:15:10:31 AEST] "POST /tasks/launch? HTTP/1.1" 404 46 *** 172.11.55.124 - - [05/Jun/2020:15:10:31 AEST] "POST /tasks/? HTTP/1.1" 500 153524 *** 172.11.55.124 - - [05/Jun/2020:15:10:31 AEST] "GET /tasks/status? HTTP/1.1" 404 555 *** 172.11.55.124 - - [05/Jun/2020:15:10:31 AEST] "POST /tasks/launch? HTTP/1.1" 404 46 *** 172.11.55.124 - - [05/Jun/2020:15:10:31 AEST] "POST /tasks/? HTTP/1.1" 500 153524 *** # Lets try to compare the /etc trees on each system and see what can be seen after weeding CentOS 8: mkdir /var/tmp/CentOS7 CentOS 7: sudo rsync -av /etc admin@virtualcentos8:/var/tmp/CentOS7/ CentOS 8: sudo diff -r /var/tmp/CentOS7/etc/ /etc/ 2>&1 | tee CentOS-7and8-etc-differences.diff cp CentOS-7and8-etc-differences.diff CentOS-7and8-etc-differences-manually-filtered.diff vim CentOS-7and8-etc-differences-manually-filtered.diff # Increase the logging levels on the smart-proxy and dynflow processes echo ':log_level: DEBUG' | sudo tee -a /etc/smart_proxy_dynflow_core/settings.yml sudo grep log_level /etc/foreman-proxy/settings.yml sudo sed -i -e 's/:log_level: INFO/:log_level: DEBUG/' /etc/foreman-proxy/settings.yml sudo grep log_level /etc/foreman-proxy/settings.yml # Restart Foreman sudo foreman-maintain service restart # Try 3: Commands -> Run Command - SSH Default; Enter a # command (say: echo "Debugging On"), then Submit. # When the task succeeds or fails, click on the host name to # see the command output and/or error messages. CentOS 7: 1: Debugging On 2: Exit status: 0 CentOS 8: 1: Initialization error: RestClient::NotFound - 404 Not Found 2: Failed to initialize: TypeError - Value (ActiveSupport::TimeWithZone) '2020-06-05 16:30:39 +1000' is not any of: (Time | NilClass). sudo less /var/log/foreman-proxy/proxy.log CentOS 7: 2020-06-05T16:28:48 [D] accept: 172.11.55.123:34456 2020-06-05T16:28:48 [D] Rack::Handler::WEBrick is invoked. 2020-06-05T16:28:48 e87c3ed7 [I] Started GET /dynflow/tasks/count state=running 2020-06-05T16:28:48 e87c3ed7 [D] verifying remote client virtualcentos7.sc.mydomain.net.au (based on SSL_CLIENT_CERT) against trusted_hosts ["virtualcentos7.sc.mydomain.net.au"] 2020-06-05T16:28:48 e87c3ed7 [D] Proxy request from virtualcentos7.sc.mydomain.net.au:8443/dynflow/tasks/count to https://virtualcentos7.sc.mydomain.net.au:8008/tasks/count 2020-06-05T16:28:49 e87c3ed7 [D] Proxy request status 200 - # 2020-06-05T16:28:49 e87c3ed7 [I] Finished GET /dynflow/tasks/count with 200 (310.73 ms) 2020-06-05T16:28:49 [D] close: 172.11.55.123:34456 2020-06-05T16:28:49 [D] accept: 172.11.55.123:34460 2020-06-05T16:28:49 [D] Rack::Handler::WEBrick is invoked. 2020-06-05T16:28:49 e95e64ef [I] Started POST /dynflow/tasks/launch 2020-06-05T16:28:49 e95e64ef [D] verifying remote client virtualcentos7.sc.mydomain.net.au (based on SSL_CLIENT_CERT) against trusted_hosts ["virtualcentos7.sc.mydomain.net.au"] 2020-06-05T16:28:49 e95e64ef [D] Proxy request from virtualcentos7.sc.mydomain.net.au:8443/dynflow/tasks/launch to https://virtualcentos7.sc.mydomain.net.au:8008/tasks/launch 2020-06-05T16:28:49 e95e64ef [D] Proxy request status 200 - # 2020-06-05T16:28:49 e95e64ef [I] Finished POST /dynflow/tasks/launch with 200 (38.98 ms) 2020-06-05T16:28:49 [D] close: 172.11.55.123:34460 CentOS 8: 2020-06-05T16:30:23 [D] accept: 172.11.55.124:38252 2020-06-05T16:30:23 [D] Rack::Handler::WEBrick is invoked. 2020-06-05T16:30:23 5bb40802 [I] Started GET /dynflow/tasks/count state=running 2020-06-05T16:30:23 5bb40802 [D] verifying remote client virtualcentos8.sc.mydomain.net.au (based on SSL_CLIENT_CERT) against trusted_hosts ["virtualcentos8.sc.mydomain.net.au"] 2020-06-05T16:30:23 5bb40802 [D] Proxy request from virtualcentos8.sc.mydomain.net.au:8443/dynflow/tasks/count to https://virtualcentos8.sc.mydomain.net.au:8008/tasks/count 2020-06-05T16:30:24 5bb40802 [D] Proxy request status 200 - # 2020-06-05T16:30:24 5bb40802 [I] Finished GET /dynflow/tasks/count with 200 (258.07 ms) 2020-06-05T16:30:24 [D] close: 172.11.55.124:38252 2020-06-05T16:30:24 [D] accept: 172.11.55.124:38256 2020-06-05T16:30:24 [D] Rack::Handler::WEBrick is invoked. 2020-06-05T16:30:24 0d32852b [I] Started GET /version 2020-06-05T16:30:24 0d32852b [I] Finished GET /version with 200 (0.29 ms) 2020-06-05T16:30:24 [D] close: 172.11.55.124:38256 2020-06-05T16:30:24 [D] accept: 172.11.55.124:38258 2020-06-05T16:30:24 [D] Rack::Handler::WEBrick is invoked. 2020-06-05T16:30:24 a3bffafe [I] Started POST /dynflow/tasks/launch 2020-06-05T16:30:24 a3bffafe [D] verifying remote client virtualcentos8.sc.mydomain.net.au (based on SSL_CLIENT_CERT) against trusted_hosts ["virtualcentos8.sc.mydomain.net.au"] 2020-06-05T16:30:24 a3bffafe [D] Proxy request from virtualcentos8.sc.mydomain.net.au:8443/dynflow/tasks/launch to https://virtualcentos8.sc.mydomain.net.au:8008/tasks/launch 2020-06-05T16:30:24 a3bffafe [D] Proxy request status 404 - # *** 2020-06-05T16:30:24 a3bffafe [I] Finished POST /dynflow/tasks/launch with 404 (14.84 ms) *** 2020-06-05T16:30:24 [D] close: 172.11.55.124:38258 2020-06-05T16:30:24 [D] accept: 172.11.55.124:38262 2020-06-05T16:30:24 [D] Rack::Handler::WEBrick is invoked. 2020-06-05T16:30:24 9b769d7d [I] Started POST /dynflow/tasks/ 2020-06-05T16:30:24 9b769d7d [D] verifying remote client virtualcentos8.sc.mydomain.net.au (based on SSL_CLIENT_CERT) against trusted_hosts ["virtualcentos8.sc.mydomain.net.au"] 2020-06-05T16:30:24 9b769d7d [D] Proxy request from virtualcentos8.sc.mydomain.net.au:8443/dynflow/tasks/ to https://virtualcentos8.sc.mydomain.net.au:8008/tasks/ 2020-06-05T16:30:24 9b769d7d [D] Proxy request status 500 - # *** 2020-06-05T16:30:24 9b769d7d [I] Finished POST /dynflow/tasks/ with 500 (51.18 ms) *** 2020-06-05T16:30:24 [D] close: 172.11.55.124:38262 2020-06-05T16:30:24 [D] accept: 172.11.55.124:38266 2020-06-05T16:30:24 [D] Rack::Handler::WEBrick is invoked. 2020-06-05T16:30:24 d07acd82 [I] Started GET /dynflow/tasks/status 2020-06-05T16:30:24 d07acd82 [D] verifying remote client virtualcentos8.sc.mydomain.net.au (based on SSL_CLIENT_CERT) against trusted_hosts ["virtualcentos8.sc.mydomain.net.au"] 2020-06-05T16:30:24 d07acd82 [D] Proxy request from virtualcentos8.sc.mydomain.net.au:8443/dynflow/tasks/status to https://virtualcentos8.sc.mydomain.net.au:8008/tasks/status 2020-06-05T16:30:24 d07acd82 [D] Proxy request status 404 - # *** 2020-06-05T16:30:24 d07acd82 [I] Finished GET /dynflow/tasks/status with 404 (15.55 ms) *** 2020-06-05T16:30:24 [D] accept: 172.11.55.124:38272 2020-06-05T16:30:24 [D] close: 172.11.55.124:38266 2020-06-05T16:30:24 [D] Rack::Handler::WEBrick is invoked. 2020-06-05T16:30:24 7ffd2eb8 [I] Started POST /dynflow/tasks/launch 2020-06-05T16:30:24 7ffd2eb8 [D] verifying remote client virtualcentos8.sc.mydomain.net.au (based on SSL_CLIENT_CERT) against trusted_hosts ["virtualcentos8.sc.mydomain.net.au"] 2020-06-05T16:30:24 7ffd2eb8 [D] Proxy request from virtualcentos8.sc.mydomain.net.au:8443/dynflow/tasks/launch to https://virtualcentos8.sc.mydomain.net.au:8008/tasks/launch 2020-06-05T16:30:24 7ffd2eb8 [D] Proxy request status 404 - # *** 2020-06-05T16:30:24 7ffd2eb8 [I] Finished POST /dynflow/tasks/launch with 404 (20.03 ms) *** 2020-06-05T16:30:24 [D] close: 172.11.55.124:38272 2020-06-05T16:30:24 [D] accept: 172.11.55.124:38276 2020-06-05T16:30:24 [D] Rack::Handler::WEBrick is invoked. 2020-06-05T16:30:24 2871119b [I] Started POST /dynflow/tasks/ 2020-06-05T16:30:24 2871119b [D] verifying remote client virtualcentos8.sc.mydomain.net.au (based on SSL_CLIENT_CERT) against trusted_hosts ["virtualcentos8.sc.mydomain.net.au"] 2020-06-05T16:30:24 2871119b [D] Proxy request from virtualcentos8.sc.mydomain.net.au:8443/dynflow/tasks/ to https://virtualcentos8.sc.mydomain.net.au:8008/tasks/ 2020-06-05T16:30:24 2871119b [D] Proxy request status 500 - # *** 2020-06-05T16:30:24 2871119b [I] Finished POST /dynflow/tasks/ with 500 (31.23 ms) *** 2020-06-05T16:30:24 [D] close: 172.11.55.124:38276 sudo less /var/log/foreman-proxy/smart_proxy_dynflow_core.log CentOS 7: D, [2020-06-05T16:25:34.815058 #16077] DEBUG -- : Rack::Handler::WEBrick is mounted on /. Could not open DB for dynflow at '', will keep data in memory. Restart will drop all dynflow data.Execution plan cleaner removing 0 execution plans.WEBrick::HTTPServer#start: pid=16077 port=8008Executor heartbeatExecutor heartbeatExecutor heartbeatExecutor heartbeatExecutor heartbeatExecutor heartbeatExecutor heartbeatExecutor heartbeatExecutor heartbeatExecutor heartbeatExecutor heartbeatExecutor heartbeataccept: 172.11.55.123:46948Rack::Handler::WEBrick is invoked.172.11.55.123 - - [05/Jun/2020:16:28:48 AEST] "GET /tasks/count?state=running HTTP/1.1" 200 29 close: 172.11.55.123:46948accept: 172.11.55.123:46952Rack::Handler::WEBrick is invoked.ExecutionPlan ... [[ The ... means the output got a little chaotic after that, but it debug and that system works]]] CentOS 8: D, [2020-06-05T16:26:07.284746 #11678] DEBUG -- : Rack::Handler::WEBrick is mounted on /. Could not open DB for dynflow at '', will keep data in memory. Restart will drop all dynflow data.Execution plan cleaner removing 0 execution plans.WEBrick::HTTPServer#start: pid=11678 port=8008Executor heartbeatExecutor heartbeatExecutor heartbeatExecutor heartbeatExecutor heartbeatExecutor heartbeatExecutor heartbeatExecutor heartbeatExecutor heartbeatExecutor heartbeatExecutor heartbeatExecutor heartbeatExecutor heartbeatExecutor heartbeatExecutor heartbeatExecutor heartbeatExecutor heartbeataccept: 172.11.55.124:42592Rack::Handler::WEBrick is invoked.172.11.55.124 - - [05/Jun/2020:16:30:23 AEST] "GET /tasks/count?state=running HTTP/1.1" 200 29 close: 172.11.55.124:42592accept: 172.11.55.124:42598Rack::Handler::WEBrick is invoked.172.11.55.124 - - [05/Jun/2020:16:30:24 AEST] "POST /tasks/launch? HTTP/1.1" 404 46 *** close: 172.11.55.124:42598accept: 172.11.55.124:42602Rack::Handler::WEBrick is invoked.172.11.55.124 - - [05/Jun/2020:16:30:24 AEST] "POST /tasks/? HTTP/1.1" 500 153485 *** close: 172.11.55.124:42602accept: 172.11.55.124:42606Rack::Handler::WEBrick is invoked.172.11.55.124 - - [05/Jun/2020:16:30:24 AEST] "GET /tasks/status? HTTP/1.1" 404 555 *** close: 172.11.55.124:42606accept: 172.11.55.124:42612Rack::Handler::WEBrick is invoked.172.11.55.124 - - [05/Jun/2020:16:30:24 AEST] "POST /tasks/launch? HTTP/1.1" 404 46 *** close: 172.11.55.124:42612accept: 172.11.55.124:42616Rack::Handler::WEBrick is invoked.172.11.55.124 - - [05/Jun/2020:16:30:24 AEST] "POST /tasks/? HTTP/1.1" 500 153524 *** close: 172.11.55.124:42616Executor heartbeatExecutor heartbeatExecutor heartbeatExecutor heartbeatExecutor heartbeatExecutor heartbeatExecutor heartbeatExecutor heartbeatExecutor heartbeatExecutor heartbeatExecutor heartbeatExecutor heartbeatExecutor heartbeatExecutor heartbeatExecutor heartbeatExecutor heartbeatExecutor heartbeatExecutor heartbeatExecutor heartbeatExecutor heartbeatExecutor heartbeatExecutor heartbeatExecutor heartbeatExecutor heartbeatExecutor heartbeatExecutor heartbeatExecutor heartbeatExecutor heartbeatExecutor heartbeatExecutor heartbeatExecutor heartbeatExecutor heartbeatExecutor heartbeatExecutor heartbeatExecutor heartbeatExecutor heartbeatExecutor heartbeatExecutor heartbeatExecutor heartbeatExecutor heartbeatExecutor heartbeatExecutor heartbeat CentOS 8: journalctl | egrep -i 'dynflow|proxy|foreman' | less journalctl | egrep -i 'dynflow|proxy|foreman' > journalctl.CentOS8 sed \ -e 's/XX.XXXXXXXX.XXX.XX/sc.mydomain.net.au/g' \ -e 's/virtualXXXXXXX/virtualcentos8/g' \ journalctl.CentOS8 > journalctl.CentOS8-redacted.txt