Foreman 2.4 Plugins RPM Test pipeline failed:
https://ci.theforeman.org/job/foreman-plugins-2.4-rpm-test-pipeline/1/
foreman-pipeline-plugins-2.4-centos7-install (passed)
foreman-pipeline-plugins-2.4-centos8-install (passed)
foreman-pipeline-plugins-2.4-centos7-upgrade (failed)
foreman-pipeline-plugins-2.4-centos8-upgrade (failed)
rex and ansible execution failed with
04:31:08 e[0;32m not ok 1 run 'uptime' via Remote Executione[0m
04:31:08 e[0;32m # (in test file fb-test-foreman-rex.bats, line 7)e[0m
04:31:08 e[0;32m # `hammer job-invocation create --job-template 'Run Command - SSH Default' --inputs 'command=uptime' --search-query "name = $HOSTNAME"' failed with status 70e[0m
04:31:08 e[0;32m # Task fb935b16-af31-429c-a576-3ae64b42de9e running: 0.0/1, 0%, elapsed: 00:00:00e[0m
04:31:08 e[0;32m # Task fb935b16-af31-429c-a576-3ae64b42de9e running: 0.0/1, 0%, 0.0/s, elapsed: 00:00:02e[0m
04:31:08 e[0;32m # Task fb935b16-af31-429c-a576-3ae64b42de9e running: 0.0/1, 0%, 0.0/s, elapsed: 00:00:04e[0m
04:31:08 e[0;32m # Task fb935b16-af31-429c-a576-3ae64b42de9e running: 0.0/1, 0%, 0.0/s, elapsed: 00:00:06e[0m
04:31:08 e[0;32m # Task fb935b16-af31-429c-a576-3ae64b42de9e running: 0.0/1, 0%, 0.0/s, elapsed: 00:00:09e[0m
04:31:08 e[0;32m # Task fb935b16-af31-429c-a576-3ae64b42de9e warning: 1.0/1, 100%, 0.1/s, elapsed: 00:00:11e[0m
04:31:08 e[0;32m # Task fb935b16-af31-429c-a576-3ae64b42de9e warning: 1.0/1, 100%, 0.1/s, elapsed: 00:00:11e[0m
04:31:08 e[0;32m # Error: A sub task failede[0m
04:31:08 e[0;32m # Job invocation 1 createde[0m
04:31:08 e[0;32m # 1 task(s), 0 success, 1 faile[0m
04:31:08 e[0;32m 1..1e[0m
04:31:08 e[0;32m not ok 1 run 'uptime' via Ansiblee[0m
04:31:08 e[0;32m # (in test file fb-test-foreman-ansible.bats, line 7)e[0m
04:31:08 e[0;32m # `hammer job-invocation create --job-template 'Run Command - Ansible Default' --inputs 'command=uptime' --search-query "name = $HOSTNAME"' failed with status 70e[0m
04:31:08 e[0;32m # Task c2c1fe92-9740-49e2-a4dd-9ee6379807c3 running: 0.0/1, 0%, elapsed: 00:00:00e[0m
04:31:08 e[0;32m # Task c2c1fe92-9740-49e2-a4dd-9ee6379807c3 running: 0.0/1, 0%, 0.0/s, elapsed: 00:00:02e[0m
04:31:08 e[0;32m # Task c2c1fe92-9740-49e2-a4dd-9ee6379807c3 running: 0.0/1, 0%, 0.0/s, elapsed: 00:00:04e[0m
04:31:08 e[0;32m # Task c2c1fe92-9740-49e2-a4dd-9ee6379807c3 running: 0.0/1, 0%, 0.0/s, elapsed: 00:00:06e[0m
04:31:08 e[0;32m # Task c2c1fe92-9740-49e2-a4dd-9ee6379807c3 running: 0.0/1, 0%, 0.0/s, elapsed: 00:00:09e[0m
04:31:08 e[0;32m # Task c2c1fe92-9740-49e2-a4dd-9ee6379807c3 warning: 1.0/1, 100%, 0.1/s, elapsed: 00:00:11e[0m
04:31:08 e[0;32m # Task c2c1fe92-9740-49e2-a4dd-9ee6379807c3 warning: 1.0/1, 100%, 0.1/s, elapsed: 00:00:11e[0m
04:31:08 e[0;32m # Error: A sub task failede[0m
04:31:08 e[0;32m # Job invocation 2 createde[0m
04:31:08 e[0;32m # 1 task(s), 0 success, 1 faile[0m
Looking at the sosreport, it seems dynflow is unhappy after the upgrade.
production.log:
2021-03-12T03:30:45 [E|bac|] The only applicable proxy pipe-up-plugins-2-4-centos7.n49.example.com is down (RuntimeError)
proxy.log:
2021-03-12 03:30:45 - Errno::ECONNREFUSED - Failed to open TCP connection to pipe-up-plugins-2-4-centos7.n49.example.com:8008 (Connection refused - connect(2) for "pipe-up-plugins-2-4-centos7.n49.example.com" port 8008):
restarting smart_proxy_dynflow_core before running the tests seems to fix it, but why?
coming closer…
[root@pipe-up-plugins-2-4-centos7 ~]# netstat -tulpen |grep 8008
tcp 0 0 127.0.0.1:8008 0.0.0.0:* LISTEN 994 92221 23154/ruby
[root@pipe-up-plugins-2-4-centos7 ~]# systemctl restart smart_proxy_dynflow_core
[root@pipe-up-plugins-2-4-centos7 ~]# netstat -tulpen |grep 8008
tcp 0 0 0.0.0.0:8008 0.0.0.0:* LISTEN 994 116848 29191/ruby
tcp6 0 0 :::8008 :::* LISTEN 994 116849 29191/ruby
Seems before the restart, it was listening on localhost only (and HTTP), and after, it’s 0.0.0.0 and HTTPS (with cert auth).
Looking at the log, we see Puppet re-configuring dynflow_core to listen on * and use HTTPS:
2021-03-12 11:31:32 [DEBUG ] [configure] /Stage[main]/Foreman_proxy::Plugin::Dynflow/File[/etc/smart_proxy_dynflow_core/settings.yml]: Starting to evaluate the resource (1398 of 1492)
2021-03-12 11:31:32 [DEBUG ] [configure] Executing: 'diff -u /etc/smart_proxy_dynflow_core/settings.yml /tmp/puppet-file20210312-23756-5dx57y'
2021-03-12 11:31:32 [INFO ] [configure] /Stage[main]/Foreman_proxy::Plugin::Dynflow/File[/etc/smart_proxy_dynflow_core/settings.yml]/content:
2021-03-12 11:31:32 [INFO ] [configure] --- /etc/smart_proxy_dynflow_core/settings.yml 2020-11-20 18:09:55.000000000 +0000
2021-03-12 11:31:32 [INFO ] [configure] +++ /tmp/puppet-file20210312-23756-5dx57y 2021-03-12 11:31:32.344315562 +0000
2021-03-12 11:31:32 [INFO ] [configure] -:listen: 127.0.0.1
2021-03-12 11:31:32 [INFO ] [configure] +:listen: "*"
2021-03-12 11:31:32 [INFO ] [configure] -# :use_https: false
2021-03-12 11:31:32 [INFO ] [configure] +:use_https: true
2021-03-12 11:31:32 [INFO ] [configure] /Stage[main]/Foreman_proxy::Plugin::Dynflow/File[/etc/smart_proxy_dynflow_core/settings.yml]/content: content changed '{md5}8870be0336a0bc33d7f796b9b14c98b3' to '{md5}fe34a96945914909a80ff493f730f8d5'
2021-03-12 11:31:32 [DEBUG ] [configure] /Stage[main]/Foreman_proxy::Plugin::Dynflow/File[/etc/smart_proxy_dynflow_core/settings.yml]: The container Class[Foreman_proxy::Plugin::Dynflow] will propagate my refresh event
2021-03-12 11:31:32 [DEBUG ] [configure] /Stage[main]/Foreman_proxy::Plugin::Dynflow/File[/etc/smart_proxy_dynflow_core/settings.yml]: Evaluated in 0.02 seconds
Which, when you read the puppet code should trigger a restart of the service:
And in the log there is also no mention of a notify relationship between the file and the service:
2021-03-12 11:25:55 [DEBUG ] [configure] Adding relationship from Systemd::Service_limits[smart_proxy_dynflow_core.service] to Service[smart_proxy_dynflow_core] with 'notify'
2021-03-12 11:25:55 [DEBUG ] [configure] Adding relationship from Foreman_proxy::Settings_file[ansible] to Service[smart_proxy_dynflow_core] with 'notify'
2021-03-12 11:25:55 [DEBUG ] [configure] Adding relationship from Foreman_proxy::Settings_file[remote_execution_ssh] to Service[smart_proxy_dynflow_core] with 'notify'
2021-03-12 11:25:55 [DEBUG ] [configure] /Service[smart_proxy_dynflow_core]: Provider systemd does not support features manages_logon_credentials; not managing attribute logonpassword
2021-03-12 11:25:55 [DEBUG ] [configure] /Service[smart_proxy_dynflow_core]: Provider systemd does not support features configurable_timeout; not managing attribute timeout
2021-03-12 11:25:56 [DEBUG ] [configure] /Stage[main]/Foreman_proxy::Plugin::Dynflow/Systemd::Service_limits[smart_proxy_dynflow_core.service]/notify: notify to Service[smart_proxy_dynflow_core]
2021-03-12 11:25:56 [DEBUG ] [configure] /Stage[main]/Foreman_proxy::Plugin::Ansible/Foreman_proxy::Plugin::Module[ansible]/Foreman_proxy::Module[ansible]/Foreman_proxy::Settings_file[ansible]/notify: notify to Service[smart_proxy_dynflow_core]
2021-03-12 11:25:56 [DEBUG ] [configure] /Stage[main]/Foreman_proxy::Plugin::Remote_execution::Ssh/Foreman_proxy::Plugin::Module[remote_execution_ssh]/Foreman_proxy::Module[remote_execution_ssh]/Foreman_proxy::Settings_file[remote_execution_ssh]/notify: notify to Service[smart_proxy_dynflow_core]
2021-03-12 11:31:37 [DEBUG ] [configure] /Service[smart_proxy_dynflow_core]: Starting to evaluate the resource (1480 of 1492)
2021-03-12 11:31:37 [DEBUG ] [configure] /Service[smart_proxy_dynflow_core]: Evaluated in 0.03 seconds
And this should fix it:
This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.