Problem:
Remote Execution - Job - Pending Status:
Whenever I try to execute Job using Remote Execute Plugin, I get Pending Status. proxy.log and production logs (attached) do not show anything significant. Am I missing something?
I installed foreman on Ubuntu 16 using following steps:
sudo apt-get -y install ca-certificates
cd /tmp && wget https://apt.puppetlabs.com/puppet5-release-xenial.deb
sudo dpkg -i /tmp/puppet5-release-xenial.deb
#Enable the Foreman repo:
echo "deb http://deb.theforeman.org/ xenial 1.20" | sudo tee /etc/apt/sources.list.d/foreman.list
echo "deb http://deb.theforeman.org/ plugins 1.20" | sudo tee -a /etc/apt/sources.list.d/foreman.list
sudo apt-get -y install ca-certificates
wget -q https://deb.theforeman.org/pubkey.gpg -O- | sudo apt-key add -
sudo apt-get update && sudo apt-get -y install foreman-installer
root@foreman:~# foreman-installer --enable-foreman-plugin-remote-execution --enable-foreman-proxy-plugin-remote-execution-ssh --enable-foreman-proxy-plugin-dynflow --enable-foreman-plugin-tasks
Adding SSH Keys:
root@foreman:~/.ssh# curl -k https://localhost:8443/ssh/pubkey >> authorized_keys
Expected outcome:
The Job should run a successful ping attempt
Foreman and Proxy versions:
Foreman 1.21
Foreman-Proxy
Foreman and Proxy plugin versions:
Dynflow: 0.2.1
Puppet 1.20.2
Other relevant data:
[e.g. logs from Foreman and/or the Proxy, modified templates, commands issued, etc]
(for logs, surround with three back-tic
2019-05-23T13:41:14 [I] Successfully initialized 'foreman_proxy'
2019-05-23T13:41:14 [W] Missing SSL setup, https is disabled.
2019-05-23T13:41:14 [I] Smart proxy has launched on 1 socket(s), waiting for requests
2019-05-23T13:45:25 [I] Successfully initialized 'foreman_proxy'
2019-05-23T13:45:25 [I] Successfully initialized 'tftp'
2019-05-23T13:45:25 [I] Successfully initialized 'puppetca_hostname_whitelisting'
2019-05-23T13:45:25 [I] Successfully initialized 'puppetca'
2019-05-23T13:45:25 [I] Started puppet class cache initialization
2019-05-23T13:45:25 [I] Successfully initialized 'puppet_proxy_puppet_api'
2019-05-23T13:45:25 [I] Successfully initialized 'puppet'
2019-05-23T13:45:25 [I] Successfully initialized 'logs'
2019-05-23T13:45:25 [I] WEBrick 1.3.1
2019-05-23T13:45:25 [I] ruby 2.3.1 (2016-04-26) [x86_64-linux-gnu]
2019-05-23T13:45:25 [I]
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 2 (0x2)
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN=Puppet CA: foreman.macrocom.de
Validity
Not Before: May 22 13:42:36 2019 GMT
Not After : May 21 13:42:36 2024 GMT
Subject: CN=foreman.macrocom.de
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (4096 bit)
Modulus:
00:d5:8e:5d:79:fe:0c:e6:b7:ff:56:be:cc:43:ed:
7c:c5:8d:a0:f2:68:77:cd:11:dc:fb:d8:39:29:ed:
d9:bd:fd:ca:9a:5a:44:ce:de:f5:2c:e8:9c:c5:fe:
f8:72:2d:96:91:2a:da:7f:2a:55:c3:be:5a:39:89:
ca:8d:c5:9b:3f:87:84:df:72:42:8f:06:29:75:52:
5d:71:91:fc:26:7b:77:4c:28:2f:fc:f8:1f:57:be:
fd:ae:96:bb:52:31:98:49:5d:a0:b6:66:2f:ef:a4:
ca:48:a0:8a:14:33:ce:58:f8:b4:da:b6:c6:6e:25:
98:1e:87:8e:f7:7b:1e:5e:8b:c7:b3:66:c9:0f:57:
aa:fe:38:2e:e8:7b:e9:1b:09:3d:2e:a7:be:d8:e6:
88:3c:7b:b3:09:6a:18:37:42:5b:6e:fd:4b:c1:46:
b4:cc:ce:c9:28:7e:cc:c2:79:8a:21:89:5c:1f:de:
05:02:bf:4e:d7:3d:c0:63:4b:c0:58:da:e0:b7:64:
d1:ed:23:21:fd:f2:5b:10:49:65:d6:64:21:b4:e6:
16:a1:fe:a1:3e:72:52:9c:50:6e:81:b9:d3:47:7d:
6d:e3:64:81:22:5c:d2:18:0c:2f:4e:7c:0d:f9:3b:
51:65:63:8d:78:7e:55:90:a0:5b:eb:df:4f:88:29:
50:fc:cd:80:00:9e:83:6f:4c:a9:af:f4:3a:46:20:
6e:9b:95:b6:01:b3:eb:81:f3:4b:31:c0:83:a6:75:
b4:91:86:ab:b4:21:58:27:ed:41:1a:35:77:f4:28:
97:8d:d3:91:58:0c:97:7b:ce:f7:7d:44:99:0a:7b:
39:53:a8:98:a6:f5:41:3d:09:f1:5f:9e:41:7e:a7:
62:81:65:01:72:0b:1f:c1:d4:e8:b4:27:98:98:ee:
18:c1:aa:8a:54:c8:54:15:69:83:aa:0f:12:0c:a9:
cd:90:a2:48:b8:6f:0d:57:93:07:99:f6:b1:e4:dc:
f9:b3:96:98:05:f1:58:39:73:29:b2:ca:e9:3a:1a:
c1:33:2f:d0:7c:3f:94:49:2f:b2:ef:88:53:91:df:
ef:0f:0b:24:40:cd:a6:c0:87:9f:a1:37:4d:56:20:
92:8c:7f:db:ef:52:24:69:8d:d2:0b:d1:33:58:2d:
cf:56:75:4d:46:62:98:b5:02:64:7b:7a:1f:3e:82:
f8:c3:db:17:bb:f5:4c:d8:8c:4a:a6:1b:8b:f8:24:
bc:ab:a3:77:cb:f8:7d:f7:06:dc:a3:f8:a3:54:94:
97:54:e7:d9:14:4e:f0:b1:31:79:e3:0d:09:32:ca:
69:e5:27:1e:9f:0d:1c:af:63:13:e7:b1:14:f6:75:
69:90:93
Exponent: 65537 (0x10001)
X509v3 extensions:
Netscape Comment:
.(Puppet Ruby/OpenSSL Internal Certificate
X509v3 Subject Alternative Name:
DNS:foreman.macrocom.de, DNS:puppet, DNS:puppet.macrocom.de
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage: critical
TLS Web Server Authentication, TLS Web Client Authentication
X509v3 Basic Constraints: critical
CA:FALSE
X509v3 Subject Key Identifier:
2D:58:E3:20:AC:A4:F3:6E:F0:B8:5F:BD:CA:5A:57:06:E6:95:CA:55
X509v3 Authority Key Identifier:
keyid:1F:33:0B:B2:33:C5:5D:E4:5A:AE:A6:01:07:88:CC:80:32:41:55:4A
Signature Algorithm: sha256WithRSAEncryption
4f:cb:72:36:96:53:68:13:1c:6a:b3:bd:2c:1c:ee:1a:be:f8:
07:40:5b:d8:3b:af:8e:77:d5:bf:f8:d5:2c:9b:1c:26:69:ac:
83:54:2d:89:3c:9f:7d:cd:e6:4d:20:55:8b:e6:99:d8:08:14:
2d:7f:37:75:5f:2f:71:94:94:ae:44:f5:75:f0:18:51:7c:ec:
68:24:5b:26:76:f8:be:8d:2c:bf:e6:58:c7:db:e0:93:f0:75:
af:ee:17:0a:77:8e:73:d7:58:08:b9:72:4b:7e:4c:a2:f2:3f:
a1:3e:0a:0e:2a:8a:40:7b:25:7c:89:06:04:d9:95:32:50:c5:
f1:bf:ba:81:1a:91:e7:46:f3:af:bc:db:dd:8d:17:b0:da:cc:
dd:9d:2d:79:cc:8e:d2:97:6f:7c:f6:39:0a:ff:39:eb:db:45:
60:8f:b0:bd:19:9a:5b:2e:4f:cf:6b:de:26:6e:1b:a4:42:c0:
de:92:b3:6a:3c:69:38:13:9e:44:0d:b1:60:49:0d:7d:b9:11:
b1:5f:e3:1d:48:66:cb:2f:84:e5:af:b0:cb:f8:e9:f1:ad:c1:
4c:c9:f8:55:8b:6b:c0:83:d9:aa:51:a5:9a:64:08:80:bb:3c:
45:7c:7c:dc:a0:d5:93:a3:47:94:54:43:36:85:fd:70:25:07:
a9:ef:bd:1f:11:e0:24:3a:5e:8c:e6:94:c3:6b:a8:b4:34:80:
19:c0:94:82:1d:00:b8:f3:48:b2:2c:d8:c8:06:7f:19:1c:10:
de:99:8a:67:2c:d9:5f:4c:74:bb:6a:84:ce:00:b2:a2:ec:cd:
c2:58:8e:3f:07:bf:01:5d:55:04:55:28:ea:dd:5d:be:af:59:
31:e7:b1:d9:25:1a:72:3f:ae:33:a0:e9:5e:a4:85:36:32:f7:
2d:bb:9c:af:da:b9:05:e4:38:6d:58:4b:d7:a1:ff:dc:7a:b9:
ad:7a:36:62:4e:76:c2:a5:0e:02:af:7a:a3:c1:71:06:c8:3c:
37:49:ae:e6:4e:18:3e:8b:9c:72:a7:c2:3b:b2:d3:c1:3c:07:
55:7a:b4:49:e3:df:4a:a8:89:5c:9e:3b:9d:f5:fe:64:ab:94:
27:96:9d:80:40:37:8e:ab:c3:52:5c:37:36:bb:72:07:88:ea:
2e:18:ea:f6:21:ae:a9:53:24:37:3e:58:26:8f:01:d9:47:f5:
14:89:87:7e:fe:4f:83:c6:28:44:22:f5:38:7c:09:30:d1:82:
92:fb:ed:c2:7b:9d:60:49:db:de:ca:5e:94:f7:df:4c:55:9d:
69:8b:db:a7:e1:6f:66:b0:11:f7:ac:fe:b7:2c:50:f0:a8:0e:
d4:c6:4b:a2:b1:52:70:eb
2019-05-23T13:45:25 [I] WEBrick::HTTPServer#start: pid=12927 port=8443
2019-05-23T13:45:25 [I] Smart proxy has launched on 1 socket(s), waiting for requests
2019-05-23T13:45:27 [I] Finished puppet class cache initialization
2019-05-23T13:45:27 3930edfa [I] Started GET /features
2019-05-23T13:45:27 3930edfa [I] Finished GET /features with 200 (4.26 ms)
2019-05-23T13:45:27 4a17dbdf [I] Started GET /features
2019-05-23T13:45:27 4a17dbdf [I] Finished GET /features with 200 (0.2 ms)
2019-05-23T13:45:27 4a17dbdf [I] Started GET /features
2019-05-23T13:45:27 4a17dbdf [I] Finished GET /features with 200 (0.19 ms)
2019-05-23T13:56:15 [I] going to shutdown ...
2019-05-23T13:56:15 [I] WEBrick::HTTPServer#start done.
2019-05-23T13:56:16 [I] Successfully initialized 'dynflow'
2019-05-23T13:56:16 [E] Couldn't enable 'ssh': Ssh public key file /var/lib/foreman-proxy/ssh/id_rsa_foreman_proxy doesn't exist.
You can generate one with `ssh-keygen -t rsa -b 4096 -f /var/lib/foreman-proxy/ssh/id_rsa_foreman_proxy -N ''`
2019-05-23T13:56:16 [I] Successfully initialized 'foreman_proxy'
2019-05-23T13:56:16 [I] Successfully initialized 'tftp'
2019-05-23T13:56:16 [I] Successfully initialized 'puppetca_hostname_whitelisting'
2019-05-23T13:56:16 [I] Successfully initialized 'puppetca'
2019-05-23T13:56:16 [I] Started puppet class cache initialization
2019-05-23T13:56:16 [I] Successfully initialized 'puppet_proxy_puppet_api'
2019-05-23T13:56:16 [I] Successfully initialized 'puppet'
2019-05-23T13:56:16 [I] Successfully initialized 'logs'
I, [2019-05-23T13:56:16.707305 #15624] INFO -- : Settings loaded, reloading logger
2019-05-23T13:56:16 [I] WEBrick 1.3.1
2019-05-23T13:56:16 [I] ruby 2.3.1 (2016-04-26) [x86_64-linux-gnu]
2019-05-23T13:56:16 [I]
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 2 (0x2)
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN=Puppet CA: foreman.macrocom.de
Validity
Not Before: May 22 13:42:36 2019 GMT
Not After : May 21 13:42:36 2024 GMT
Subject: CN=foreman.macrocom.de
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (4096 bit)
Modulus:
00:d5:8e:5d:79:fe:0c:e6:b7:ff:56:be:cc:43:ed:
7c:c5:8d:a0:f2:68:77:cd:11:dc:fb:d8:39:29:ed:
d9:bd:fd:ca:9a:5a:44:ce:de:f5:2c:e8:9c:c5:fe:
f8:72:2d:96:91:2a:da:7f:2a:55:c3:be:5a:39:89:
ca:8d:c5:9b:3f:87:84:df:72:42:8f:06:29:75:52:
5d:71:91:fc:26:7b:77:4c:28:2f:fc:f8:1f:57:be:
fd:ae:96:bb:52:31:98:49:5d:a0:b6:66:2f:ef:a4:
ca:48:a0:8a:14:33:ce:58:f8:b4:da:b6:c6:6e:25:
98:1e:87:8e:f7:7b:1e:5e:8b:c7:b3:66:c9:0f:57:
aa:fe:38:2e:e8:7b:e9:1b:09:3d:2e:a7:be:d8:e6:
88:3c:7b:b3:09:6a:18:37:42:5b:6e:fd:4b:c1:46:
b4:cc:ce:c9:28:7e:cc:c2:79:8a:21:89:5c:1f:de:
05:02:bf:4e:d7:3d:c0:63:4b:c0:58:da:e0:b7:64:
d1:ed:23:21:fd:f2:5b:10:49:65:d6:64:21:b4:e6:
16:a1:fe:a1:3e:72:52:9c:50:6e:81:b9:d3:47:7d:
6d:e3:64:81:22:5c:d2:18:0c:2f:4e:7c:0d:f9:3b:
51:65:63:8d:78:7e:55:90:a0:5b:eb:df:4f:88:29:
50:fc:cd:80:00:9e:83:6f:4c:a9:af:f4:3a:46:20:
6e:9b:95:b6:01:b3:eb:81:f3:4b:31:c0:83:a6:75:
b4:91:86:ab:b4:21:58:27:ed:41:1a:35:77:f4:28:
97:8d:d3:91:58:0c:97:7b:ce:f7:7d:44:99:0a:7b:
39:53:a8:98:a6:f5:41:3d:09:f1:5f:9e:41:7e:a7:
62:81:65:01:72:0b:1f:c1:d4:e8:b4:27:98:98:ee:
18:c1:aa:8a:54:c8:54:15:69:83:aa:0f:12:0c:a9:
cd:90:a2:48:b8:6f:0d:57:93:07:99:f6:b1:e4:dc:
f9:b3:96:98:05:f1:58:39:73:29:b2:ca:e9:3a:1a:
c1:33:2f:d0:7c:3f:94:49:2f:b2:ef:88:53:91:df:
ef:0f:0b:24:40:cd:a6:c0:87:9f:a1:37:4d:56:20:
92:8c:7f:db:ef:52:24:69:8d:d2:0b:d1:33:58:2d:
cf:56:75:4d:46:62:98:b5:02:64:7b:7a:1f:3e:82:
f8:c3:db:17:bb:f5:4c:d8:8c:4a:a6:1b:8b:f8:24:
bc:ab:a3:77:cb:f8:7d:f7:06:dc:a3:f8:a3:54:94:
97:54:e7:d9:14:4e:f0:b1:31:79:e3:0d:09:32:ca:
69:e5:27:1e:9f:0d:1c:af:63:13:e7:b1:14:f6:75:
69:90:93
Exponent: 65537 (0x10001)
X509v3 extensions:
Netscape Comment:
.(Puppet Ruby/OpenSSL Internal Certificate
X509v3 Subject Alternative Name:
DNS:foreman.macrocom.de, DNS:puppet, DNS:puppet.macrocom.de
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage: critical
TLS Web Server Authentication, TLS Web Client Authentication
X509v3 Basic Constraints: critical
CA:FALSE
X509v3 Subject Key Identifier:
2D:58:E3:20:AC:A4:F3:6E:F0:B8:5F:BD:CA:5A:57:06:E6:95:CA:55
X509v3 Authority Key Identifier:
keyid:1F:33:0B:B2:33:C5:5D:E4:5A:AE:A6:01:07:88:CC:80:32:41:55:4A
Signature Algorithm: sha256WithRSAEncryption
4f:cb:72:36:96:53:68:13:1c:6a:b3:bd:2c:1c:ee:1a:be:f8:
07:40:5b:d8:3b:af:8e:77:d5:bf:f8:d5:2c:9b:1c:26:69:ac:
83:54:2d:89:3c:9f:7d:cd:e6:4d:20:55:8b:e6:99:d8:08:14:
2d:7f:37:75:5f:2f:71:94:94:ae:44:f5:75:f0:18:51:7c:ec:
68:24:5b:26:76:f8:be:8d:2c:bf:e6:58:c7:db:e0:93:f0:75:
af:ee:17:0a:77:8e:73:d7:58:08:b9:72:4b:7e:4c:a2:f2:3f:
a1:3e:0a:0e:2a:8a:40:7b:25:7c:89:06:04:d9:95:32:50:c5:
f1:bf:ba:81:1a:91:e7:46:f3:af:bc:db:dd:8d:17:b0:da:cc:
dd:9d:2d:79:cc:8e:d2:97:6f:7c:f6:39:0a:ff:39:eb:db:45:
60:8f:b0:bd:19:9a:5b:2e:4f:cf:6b:de:26:6e:1b:a4:42:c0:
de:92:b3:6a:3c:69:38:13:9e:44:0d:b1:60:49:0d:7d:b9:11:
b1:5f:e3:1d:48:66:cb:2f:84:e5:af:b0:cb:f8:e9:f1:ad:c1:
4c:c9:f8:55:8b:6b:c0:83:d9:aa:51:a5:9a:64:08:80:bb:3c:
45:7c:7c:dc:a0:d5:93:a3:47:94:54:43:36:85:fd:70:25:07:
a9:ef:bd:1f:11:e0:24:3a:5e:8c:e6:94:c3:6b:a8:b4:34:80:
19:c0:94:82:1d:00:b8:f3:48:b2:2c:d8:c8:06:7f:19:1c:10:
de:99:8a:67:2c:d9:5f:4c:74:bb:6a:84:ce:00:b2:a2:ec:cd:
c2:58:8e:3f:07:bf:01:5d:55:04:55:28:ea:dd:5d:be:af:59:
31:e7:b1:d9:25:1a:72:3f:ae:33:a0:e9:5e:a4:85:36:32:f7:
2d:bb:9c:af:da:b9:05:e4:38:6d:58:4b:d7:a1:ff:dc:7a:b9:
ad:7a:36:62:4e:76:c2:a5:0e:02:af:7a:a3:c1:71:06:c8:3c:
37:49:ae:e6:4e:18:3e:8b:9c:72:a7:c2:3b:b2:d3:c1:3c:07:
55:7a:b4:49:e3:df:4a:a8:89:5c:9e:3b:9d:f5:fe:64:ab:94:
27:96:9d:80:40:37:8e:ab:c3:52:5c:37:36:bb:72:07:88:ea:
2e:18:ea:f6:21:ae:a9:53:24:37:3e:58:26:8f:01:d9:47:f5:
14:89:87:7e:fe:4f:83:c6:28:44:22:f5:38:7c:09:30:d1:82:
92:fb:ed:c2:7b:9d:60:49:db:de:ca:5e:94:f7:df:4c:55:9d:
69:8b:db:a7:e1:6f:66:b0:11:f7:ac:fe:b7:2c:50:f0:a8:0e:
d4:c6:4b:a2:b1:52:70:eb
2019-05-23T13:56:16 [I] WEBrick::HTTPServer#start: pid=15624 port=8443
2019-05-23T13:56:16 [I] Smart proxy has launched on 1 socket(s), waiting for requests
2019-05-23T13:56:17 [I] Finished puppet class cache initialization
W, [2019-05-23T13:56:38.047463 #15624] WARN -- : Could not open DB for dynflow at '', will keep data in memory. Restart will drop all dynflow data.
2019-05-23T13:56:38 [I] [2019-05-23 13:56:38.119 #15624] 1 -- dynflow: Execution plan cleaner removing 0 execution plans.
2019-05-23T13:56:38 b0162129 [I] [2019-05-23 13:56:38.130 #15624] 1 -- Started GET /features
2019-05-23T13:56:38 b0162129 [I] [2019-05-23 13:56:38.133 #15624] 1 -- Finished GET /features with 200 (3.68 ms)
2019-05-23T13:56:38 b0162129 [I] [2019-05-23 13:56:38.209 #15624] 1 -- Started GET /pubkey
2019-05-23T13:56:38 b0162129 [W] [2019-05-23 13:56:38.214 #15624] 2 -- Error processing request 'b0162129-939e-4046-bfb6-e7874f1f8a03
Errno::ENOENT: No such file or directory @ rb_sysopen - /var/lib/foreman-proxy/ssh/id_rsa_foreman_proxy.pub
/usr/lib/ruby/vendor_ruby/smart_proxy_remote_execution_ssh/api.rb:5:in `read'
/usr/lib/ruby/vendor_ruby/smart_proxy_remote_execution_ssh/api.rb:5:in `block in <class:Api>'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1611:in `call'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1611:in `block in compile!'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:975:in `block (3 levels) in route!'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:994:in `route_eval'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:975:in `block (2 levels) in route!'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1015:in `block in process_route'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1013:in `catch'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1013:in `process_route'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:973:in `block in route!'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:972:in `each'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:972:in `route!'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1085:in `block in dispatch!'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1067:in `block in invoke'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1067:in `catch'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1067:in `invoke'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1082:in `dispatch!'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:907:in `block in call!'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1067:in `block in invoke'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1067:in `catch'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1067:in `invoke'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:907:in `call!'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:895:in `call'
/usr/share/foreman-proxy/lib/proxy/log.rb:86:in `call'
/usr/share/foreman-proxy/lib/proxy/request_id_middleware.rb:14:in `call'
/usr/lib/ruby/vendor_ruby/rack/protection/xss_header.rb:18:in `call'
/usr/lib/ruby/vendor_ruby/rack/protection/path_traversal.rb:16:in `call'
/usr/lib/ruby/vendor_ruby/rack/protection/json_csrf.rb:18:in `call'
/usr/lib/ruby/vendor_ruby/rack/protection/base.rb:50:in `call'
/usr/lib/ruby/vendor_ruby/rack/protection/base.rb:50:in `call'
/usr/lib/ruby/vendor_ruby/rack/protection/frame_options.rb:31:in `call'
/usr/lib/ruby/vendor_ruby/rack/nulllogger.rb:9:in `call'
/usr/lib/ruby/vendor_ruby/rack/head.rb:13:in `call'
/usr/lib/ruby/vendor_ruby/sinatra/show_exceptions.rb:25:in `call'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:182:in `call'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:2013:in `call'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1487:in `block in call'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1787:in `synchronize'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1487:in `call'
/usr/lib/ruby/vendor_ruby/rack/urlmap.rb:66:in `block in call'
/usr/lib/ruby/vendor_ruby/rack/urlmap.rb:50:in `each'
/usr/lib/ruby/vendor_ruby/rack/urlmap.rb:50:in `call'
/usr/lib/ruby/vendor_ruby/rack/builder.rb:153:in `call'
/usr/lib/ruby/vendor_ruby/rack/handler/webrick.rb:88:in `service'
/usr/lib/ruby/2.3.0/webrick/httpserver.rb:140:in `service'
/usr/lib/ruby/2.3.0/webrick/httpserver.rb:96:in `run'
/usr/lib/ruby/2.3.0/webrick/server.rb:296:in `block in start_thread'
/usr/lib/ruby/vendor_ruby/logging/diagnostic_context.rb:448:in `block in create_with_logging_context'
2019-05-23T13:56:38 b0162129 [I] [2019-05-23 13:56:38.214 #15624] 1 -- Finished GET /pubkey with 500 (5.08 ms)
2019-05-23T13:56:38 a7ee1d56 [I] [2019-05-23 13:56:38.386 #15624] 1 -- Started GET /features
2019-05-23T13:56:38 a7ee1d56 [I] [2019-05-23 13:56:38.387 #15624] 1 -- Finished GET /features with 200 (0.25 ms)
2019-05-23T13:56:38 a7ee1d56 [I] [2019-05-23 13:56:38.455 #15624] 1 -- Started GET /pubkey
2019-05-23T13:56:38 a7ee1d56 [W] [2019-05-23 13:56:38.461 #15624] 2 -- Error processing request 'a7ee1d56-7564-49d6-940a-844aed699168
Errno::ENOENT: No such file or directory @ rb_sysopen - /var/lib/foreman-proxy/ssh/id_rsa_foreman_proxy.pub
/usr/lib/ruby/vendor_ruby/smart_proxy_remote_execution_ssh/api.rb:5:in `read'
/usr/lib/ruby/vendor_ruby/smart_proxy_remote_execution_ssh/api.rb:5:in `block in <class:Api>'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1611:in `call'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1611:in `block in compile!'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:975:in `block (3 levels) in route!'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:994:in `route_eval'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:975:in `block (2 levels) in route!'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1015:in `block in process_route'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1013:in `catch'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1013:in `process_route'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:973:in `block in route!'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:972:in `each'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:972:in `route!'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1085:in `block in dispatch!'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1067:in `block in invoke'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1067:in `catch'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1067:in `invoke'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1082:in `dispatch!'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:907:in `block in call!'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1067:in `block in invoke'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1067:in `catch'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1067:in `invoke'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:907:in `call!'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:895:in `call'
/usr/share/foreman-proxy/lib/proxy/log.rb:86:in `call'
/usr/share/foreman-proxy/lib/proxy/request_id_middleware.rb:14:in `call'
/usr/lib/ruby/vendor_ruby/rack/protection/xss_header.rb:18:in `call'
/usr/lib/ruby/vendor_ruby/rack/protection/path_traversal.rb:16:in `call'
/usr/lib/ruby/vendor_ruby/rack/protection/json_csrf.rb:18:in `call'
/usr/lib/ruby/vendor_ruby/rack/protection/base.rb:50:in `call'
/usr/lib/ruby/vendor_ruby/rack/protection/base.rb:50:in `call'
/usr/lib/ruby/vendor_ruby/rack/protection/frame_options.rb:31:in `call'
/usr/lib/ruby/vendor_ruby/rack/nulllogger.rb:9:in `call'
/usr/lib/ruby/vendor_ruby/rack/head.rb:13:in `call'
/usr/lib/ruby/vendor_ruby/sinatra/show_exceptions.rb:25:in `call'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:182:in `call'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:2013:in `call'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1487:in `block in call'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1787:in `synchronize'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1487:in `call'
/usr/lib/ruby/vendor_ruby/rack/urlmap.rb:66:in `block in call'
/usr/lib/ruby/vendor_ruby/rack/urlmap.rb:50:in `each'
/usr/lib/ruby/vendor_ruby/rack/urlmap.rb:50:in `call'
/usr/lib/ruby/vendor_ruby/rack/builder.rb:153:in `call'
/usr/lib/ruby/vendor_ruby/rack/handler/webrick.rb:88:in `service'
/usr/lib/ruby/2.3.0/webrick/httpserver.rb:140:in `service'
/usr/lib/ruby/2.3.0/webrick/httpserver.rb:96:in `run'
/usr/lib/ruby/2.3.0/webrick/server.rb:296:in `block in start_thread'
/usr/lib/ruby/vendor_ruby/logging/diagnostic_context.rb:448:in `block in create_with_logging_context'
2019-05-23T13:56:38 a7ee1d56 [I] [2019-05-23 13:56:38.462 #15624] 1 -- Finished GET /pubkey with 500 (6.43 ms)
2019-05-23T13:57:37 73250ab0 [I] [2019-05-23 13:57:37.094 #15624] 1 -- Started GET /pubkey
2019-05-23T13:57:37 73250ab0 [I] [2019-05-23 13:57:37.095 #15624] 1 -- Finished GET /pubkey with 200 (0.24 ms)
2019-05-23T13:57:37 73250ab0 [I] [2019-05-23 13:57:37.167 #15624] 1 -- Started GET /features
2019-05-23T13:57:37 73250ab0 [I] [2019-05-23 13:57:37.167 #15624] 1 -- Finished GET /features with 200 (0.24 ms)
2019-05-23T14:08:15 [I] [2019-05-23 14:08:15.613 #15624] 1 -- dynflow: start terminating delayed_executor...
2019-05-23T14:08:15 [I] [2019-05-23 14:08:15.613 #15624] 1 -- dynflow: start terminating throttle_limiter...
2019-05-23T14:08:15 [I] [2019-05-23 14:08:15.616 #15624] 1 -- dynflow: start terminating executor...
2019-05-23T14:08:15 [I] [2019-05-23 14:08:15.616 #15624] 1 -- dynflow: shutting down Core ...
2019-05-23T14:08:15 [I] [2019-05-23 14:08:15.618 #15624] 1 -- dynflow: ... Dynflow core terminated.
2019-05-23T14:08:15 [I] [2019-05-23 14:08:15.619 #15624] 1 -- dynflow: start terminating executor dispatcher...
2019-05-23T14:08:15 [I] [2019-05-23 14:08:15.620 #15624] 1 -- dynflow: start terminating client dispatcher...
2019-05-23T14:08:15 [I] [2019-05-23 14:08:15.621 #15624] 1 -- dynflow: stop listening for new events...
2019-05-23T14:08:15 [I] [2019-05-23 14:08:15.622 #15624] 1 -- dynflow: start terminating clock...
2019-05-23T14:08:15 [I] [2019-05-23 14:08:15.624 #15624] 1 -- going to shutdown ...: going to shutdown ...
2019-05-23T14:08:15 [I] [2019-05-23 14:08:15.624 #15624] 1 -- WEBrick::HTTPServer#start done.: WEBrick::HTTPServer#start done.
2019-05-23T14:08:16 [I] Successfully initialized 'dynflow'
2019-05-23T14:08:16 [I] Successfully initialized 'ssh'
2019-05-23T14:08:16 [I] Successfully initialized 'foreman_proxy'
2019-05-23T14:08:16 [I] Successfully initialized 'tftp'
2019-05-23T14:08:16 [I] Successfully initialized 'puppetca_hostname_whitelisting'
2019-05-23T14:08:16 [I] Successfully initialized 'puppetca'
2019-05-23T14:08:16 [I] Started puppet class cache initialization
2019-05-23T14:08:16 [I] Successfully initialized 'puppet_proxy_puppet_api'
2019-05-23T14:08:16 [I] Successfully initialized 'puppet'
2019-05-23T14:08:16 [I] Successfully initialized 'logs'
I, [2019-05-23T14:08:16.567272 #22965] INFO -- : Settings loaded, reloading logger
2019-05-23T14:08:16 [I] WEBrick 1.3.1
2019-05-23T14:08:16 [I] ruby 2.3.1 (2016-04-26) [x86_64-linux-gnu]
2019-05-23T14:08:16 [I]
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 2 (0x2)
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN=Puppet CA: foreman.macrocom.de
Validity
Not Before: May 22 13:42:36 2019 GMT
Not After : May 21 13:42:36 2024 GMT
Subject: CN=foreman.macrocom.de
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (4096 bit)
Modulus:
00:d5:8e:5d:79:fe:0c:e6:b7:ff:56:be:cc:43:ed:
7c:c5:8d:a0:f2:68:77:cd:11:dc:fb:d8:39:29:ed:
d9:bd:fd:ca:9a:5a:44:ce:de:f5:2c:e8:9c:c5:fe:
f8:72:2d:96:91:2a:da:7f:2a:55:c3:be:5a:39:89:
ca:8d:c5:9b:3f:87:84:df:72:42:8f:06:29:75:52:
5d:71:91:fc:26:7b:77:4c:28:2f:fc:f8:1f:57:be:
fd:ae:96:bb:52:31:98:49:5d:a0:b6:66:2f:ef:a4:
ca:48:a0:8a:14:33:ce:58:f8:b4:da:b6:c6:6e:25:
98:1e:87:8e:f7:7b:1e:5e:8b:c7:b3:66:c9:0f:57:
aa:fe:38:2e:e8:7b:e9:1b:09:3d:2e:a7:be:d8:e6:
88:3c:7b:b3:09:6a:18:37:42:5b:6e:fd:4b:c1:46:
b4:cc:ce:c9:28:7e:cc:c2:79:8a:21:89:5c:1f:de:
05:02:bf:4e:d7:3d:c0:63:4b:c0:58:da:e0:b7:64:
d1:ed:23:21:fd:f2:5b:10:49:65:d6:64:21:b4:e6:
16:a1:fe:a1:3e:72:52:9c:50:6e:81:b9:d3:47:7d:
6d:e3:64:81:22:5c:d2:18:0c:2f:4e:7c:0d:f9:3b:
51:65:63:8d:78:7e:55:90:a0:5b:eb:df:4f:88:29:
50:fc:cd:80:00:9e:83:6f:4c:a9:af:f4:3a:46:20:
6e:9b:95:b6:01:b3:eb:81:f3:4b:31:c0:83:a6:75:
b4:91:86:ab:b4:21:58:27:ed:41:1a:35:77:f4:28:
97:8d:d3:91:58:0c:97:7b:ce:f7:7d:44:99:0a:7b:
39:53:a8:98:a6:f5:41:3d:09:f1:5f:9e:41:7e:a7:
62:81:65:01:72:0b:1f:c1:d4:e8:b4:27:98:98:ee:
18:c1:aa:8a:54:c8:54:15:69:83:aa:0f:12:0c:a9:
cd:90:a2:48:b8:6f:0d:57:93:07:99:f6:b1:e4:dc:
f9:b3:96:98:05:f1:58:39:73:29:b2:ca:e9:3a:1a:
c1:33:2f:d0:7c:3f:94:49:2f:b2:ef:88:53:91:df:
ef:0f:0b:24:40:cd:a6:c0:87:9f:a1:37:4d:56:20:
92:8c:7f:db:ef:52:24:69:8d:d2:0b:d1:33:58:2d:
cf:56:75:4d:46:62:98:b5:02:64:7b:7a:1f:3e:82:
f8:c3:db:17:bb:f5:4c:d8:8c:4a:a6:1b:8b:f8:24:
bc:ab:a3:77:cb:f8:7d:f7:06:dc:a3:f8:a3:54:94:
97:54:e7:d9:14:4e:f0:b1:31:79:e3:0d:09:32:ca:
69:e5:27:1e:9f:0d:1c:af:63:13:e7:b1:14:f6:75:
69:90:93
Exponent: 65537 (0x10001)
X509v3 extensions:
Netscape Comment:
.(Puppet Ruby/OpenSSL Internal Certificate
X509v3 Subject Alternative Name:
DNS:foreman.macrocom.de, DNS:puppet, DNS:puppet.macrocom.de
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage: critical
TLS Web Server Authentication, TLS Web Client Authentication
X509v3 Basic Constraints: critical
CA:FALSE
X509v3 Subject Key Identifier:
2D:58:E3:20:AC:A4:F3:6E:F0:B8:5F:BD:CA:5A:57:06:E6:95:CA:55
X509v3 Authority Key Identifier:
keyid:1F:33:0B:B2:33:C5:5D:E4:5A:AE:A6:01:07:88:CC:80:32:41:55:4A
Signature Algorithm: sha256WithRSAEncryption
4f:cb:72:36:96:53:68:13:1c:6a:b3:bd:2c:1c:ee:1a:be:f8:
07:40:5b:d8:3b:af:8e:77:d5:bf:f8:d5:2c:9b:1c:26:69:ac:
83:54:2d:89:3c:9f:7d:cd:e6:4d:20:55:8b:e6:99:d8:08:14:
2d:7f:37:75:5f:2f:71:94:94:ae:44:f5:75:f0:18:51:7c:ec:
68:24:5b:26:76:f8:be:8d:2c:bf:e6:58:c7:db:e0:93:f0:75:
af:ee:17:0a:77:8e:73:d7:58:08:b9:72:4b:7e:4c:a2:f2:3f:
a1:3e:0a:0e:2a:8a:40:7b:25:7c:89:06:04:d9:95:32:50:c5:
f1:bf:ba:81:1a:91:e7:46:f3:af:bc:db:dd:8d:17:b0:da:cc:
dd:9d:2d:79:cc:8e:d2:97:6f:7c:f6:39:0a:ff:39:eb:db:45:
60:8f:b0:bd:19:9a:5b:2e:4f:cf:6b:de:26:6e:1b:a4:42:c0:
de:92:b3:6a:3c:69:38:13:9e:44:0d:b1:60:49:0d:7d:b9:11:
b1:5f:e3:1d:48:66:cb:2f:84:e5:af:b0:cb:f8:e9:f1:ad:c1:
4c:c9:f8:55:8b:6b:c0:83:d9:aa:51:a5:9a:64:08:80:bb:3c:
45:7c:7c:dc:a0:d5:93:a3:47:94:54:43:36:85:fd:70:25:07:
a9:ef:bd:1f:11:e0:24:3a:5e:8c:e6:94:c3:6b:a8:b4:34:80:
19:c0:94:82:1d:00:b8:f3:48:b2:2c:d8:c8:06:7f:19:1c:10:
de:99:8a:67:2c:d9:5f:4c:74:bb:6a:84:ce:00:b2:a2:ec:cd:
c2:58:8e:3f:07:bf:01:5d:55:04:55:28:ea:dd:5d:be:af:59:
31:e7:b1:d9:25:1a:72:3f:ae:33:a0:e9:5e:a4:85:36:32:f7:
2d:bb:9c:af:da:b9:05:e4:38:6d:58:4b:d7:a1:ff:dc:7a:b9:
ad:7a:36:62:4e:76:c2:a5:0e:02:af:7a:a3:c1:71:06:c8:3c:
37:49:ae:e6:4e:18:3e:8b:9c:72:a7:c2:3b:b2:d3:c1:3c:07:
55:7a:b4:49:e3:df:4a:a8:89:5c:9e:3b:9d:f5:fe:64:ab:94:
27:96:9d:80:40:37:8e:ab:c3:52:5c:37:36:bb:72:07:88:ea:
2e:18:ea:f6:21:ae:a9:53:24:37:3e:58:26:8f:01:d9:47:f5:
14:89:87:7e:fe:4f:83:c6:28:44:22:f5:38:7c:09:30:d1:82:
92:fb:ed:c2:7b:9d:60:49:db:de:ca:5e:94:f7:df:4c:55:9d:
69:8b:db:a7:e1:6f:66:b0:11:f7:ac:fe:b7:2c:50:f0:a8:0e:
d4:c6:4b:a2:b1:52:70:eb
2019-05-23T14:08:16 [I] WEBrick::HTTPServer#start: pid=22965 port=8443
2019-05-23T14:08:16 [I] Smart proxy has launched on 1 socket(s), waiting for requests
2019-05-23T14:08:16 [I] Finished puppet class cache initialization
W, [2019-05-23T14:11:41.521572 #22965] WARN -- : Could not open DB for dynflow at '', will keep data in memory. Restart will drop all dynflow data.
2019-05-23T14:11:41 [I] [2019-05-23 14:11:41.599 #22965] 1 -- dynflow: Execution plan cleaner removing 0 execution plans.
2019-05-23T14:11:41 e01f9f8a [I] [2019-05-23 14:11:41.607 #22965] 1 -- Started GET /pubkey
2019-05-23T14:11:41 e01f9f8a [I] [2019-05-23 14:11:41.612 #22965] 1 -- Finished GET /pubkey with 200 (4.48 ms)
2019-05-23T14:11:53 cb91d56d [I] [2019-05-23 14:11:53.959 #22965] 1 -- Started GET /pubkey
2019-05-23T14:11:53 cb91d56d [I] [2019-05-23 14:11:53.959 #22965] 1 -- Finished GET /pubkey with 200 (0.22 ms)
2019-05-23T14:12:56 1ac0a6e4 [I] [2019-05-23 14:12:56.947 #22965] 1 -- Started GET /version
2019-05-23T14:12:56 1ac0a6e4 [I] [2019-05-23 14:12:56.947 #22965] 1 -- Finished GET /version with 200 (0.26 ms)
2019-05-23T14:18:44 4cac21e6 [I] [2019-05-23 14:18:44.545 #22965] 1 -- Started GET /tasks/count state=running
2019-05-23T14:18:44 4cac21e6 [I] [2019-05-23 14:18:44.546 #22965] 1 -- Finished GET /tasks/count with 200 (1.26 ms)
2019-05-23T14:18:44 b11d866d [I] [2019-05-23 14:18:44.873 #22965] 1 -- Started POST /tasks/
2019-05-23T14:18:44 b11d866d [I] [2019-05-23 14:18:44.890 #22965] 1 -- Finished POST /tasks/ with 200 (16.97 ms)
2019-05-23T14:19:10 121fc771 [I] [2019-05-23 14:19:10.363 #22965] 1 -- Started GET /tasks/count state=running
2019-05-23T14:19:10 121fc771 [I] [2019-05-23 14:19:10.366 #22965] 1 -- Finished GET /tasks/count with 200 (3.23 ms)
2019-05-23T14:19:10 3f1fb973 [I] [2019-05-23 14:19:10.574 #22965] 1 -- Started POST /tasks/
2019-05-23T14:19:10 3f1fb973 [I] [2019-05-23 14:19:10.591 #22965] 1 -- Finished POST /tasks/ with 200 (16.99 ms)
2019-05-23T14:28:43 c7ecf49c [I] [2019-05-23 14:28:43.956 #22965] 1 -- Started POST /tasks/status
2019-05-23T14:28:43 c7ecf49c [I] [2019-05-23 14:28:43.968 #22965] 1 -- Finished POST /tasks/status with 200 (11.33 ms)
2019-05-23T14:29:10 8dcec1e4 [I] [2019-05-23 14:29:10.287 #22965] 1 -- Started POST /tasks/status
2019-05-23T14:29:10 8dcec1e4 [I] [2019-05-23 14:29:10.290 #22965] 1 -- Finished POST /tasks/status with 200 (3.25 ms)
Production Logs:
2019-05-23T14:35:13 [I|app|fd354] Current user set to admin (admin)
2019-05-23T14:35:13 [I|app|fd354] Completed 200 OK in 23ms (Views: 0.5ms | ActiveRecord: 3.0ms)
2019-05-23T14:35:14 [I|app|] Started GET "/job_invocations/3?hosts_needs_refresh=&host_ids_needing_status_update%5B%5D=1&_=1558622069083" for 10.0.0.1 at 2019-05-23 14:35:14 +0000
2019-05-23T14:35:14 [I|app|4ce87] Processing by JobInvocationsController#show as JS
2019-05-23T14:35:14 [I|app|4ce87] Parameters: {"hosts_needs_refresh"=>"", "host_ids_needing_status_update"=>["1"], "_"=>"1558622069083", "id"=>"3"}