The job results is failed although remote Job was successfully executed

Support
1

Problem:
as captioned, after I submitted a remote job(just simply run command job: “/usr/bin/uptime; /usr/bin/hostname;”) to host. I saw the expected output from the Job > Host page as attachment “output.png”, but at the job page it still showed pending for 10-15 mins and then become faild as attachment “failed.png”. Firewall from both side were disabled.

Expected outcome:
The command is executed successfully (done) and the job return success status on foreman gui (not yet done).

Foreman and Proxy versions:
Foreman and Proxy plugin versions:
Foreman Version 1.19.1
Bastion Version 6.1.11
Foreman-tasks 0.13.4
Foreman_docker 4.1.0
Foreman_remote_execution 1.5.6
Katello 3.8.1
Master/Proxy OS version: rhel 7.6
Client Host OS version: rhel 7.6

Other relevant data:
[e.g. logs from Foreman and/or the Proxy, modified templates, commands issued, etc]
(for logs, surround with three back-ticks to get proper formatting, e.g.)

As attachment
![failed|690x197](upload://LKzSBFGlSJsxmfmbvwb67wpKDP.png) ![output|690x197](upload://5NfOBOB9XskOazMUaJjJD1YJs71.png) 
Log file:
https://drive.google.com/open?id=1wAuL1qNZ9vj4SnlwTxOXXkbdch0Hk3FO

Thanks all orz!

2

output
output1280×367 41.8 KB

failed
failed1280×366 43 KB

3

Hi,
check out this thread Plugin Job: All Jobs i run failed

4

Thanks. I just retry again and find below error in /var/log/foreman-proxy/smart_proxy_dynflow_core.log

[2018-11-29 17:32:11.102 #7232] ERROR – action: Failed to open TCP connection to localhost:3000 (Connection refused - connect(2) for “localhost” port 3000) (Errno::ECONNREFUSED)

[2018-11-29 17:32:11.102 #7232] DEBUG – dynflow: Step 22ec8d81-a1d9-4813-88d9-1b2ff074910c: 5 running >> error in phase Run SmartProxyDynflowCore::Callback::Action
[2018-11-29 17:32:11.104 #7232] DEBUG – dynflow: ExecutionPlan 22ec8d81-a1d9-4813-88d9-1b2ff074910c running >> paused

So, I checked /etc/foreman-proxy/settings.d/dynflow.yml and /etc/smart_proxy_dynflow_core/settings.yml. Is it correct?

/etc/smart_proxy_dynflow_core/settings.yml:
:foreman_url: ‘http://localhost:3000’
:console_auth: false
:listen: 127.0.0.1
:port: 8008

/etc/foreman-proxy/settings.d/dynflow.yml:
:core_url: ‘http://127.0.0.1:8008’

Truncated /var/log/foreman-proxy/smart_proxy_dynflow_core.log:
https://drive.google.com/open?id=14JncfQWhlJNPXV3Pd29Gn2Ob3kcIXvzs

5

Actually, I think it is strange that the :foreman_url parameter of /etc/smart_proxy_dynflow_core/settings.yml is ‘http://localhost:3000’, so I just change it according to GUI > Administer > Settings > General > Foreman URL (‘https://hk3cvdv00483.oocl.com’) and give it a try but no luck.

New log:
https://drive.google.com/open?id=1TERY3MXtUBxHycr8j9x_IFknlnK6rpel

6

When a job is completed the smart proxy dynflow core calls back to foreman to let it know that it finished. The foreman_url is used to determine where that callback should be sent.

localhost:3000 really isn’t the right url for production in most cases. By changing it to the fqdn you made it actually reach foreman, but apparently foreman rejected the incoming request (and it then failed with 403: Forbidden). The best course of action now would be to look through foreman’s production.log for any mentions of 403 or Forbidden. When you find it it should give you a hint why it was rejected and how to fix that.

7

Really great clue for me to solve the issue. Thanks. Find that it returned the status to ipv6 address, so I was going to disable it.
Disabling ipv6:
https://access.redhat.com/solutions/8709
After ipv6 is disabled and restarting the service, my foreman-proxy failed to start and here is solution:
https://bugzilla.redhat.com/show_bug.cgi?id=1643183

/var/log/foreman/production.log:

2018-12-03T09:14:28 [I|app|] Started POST “/foreman_tasks/api/tasks/callback” for ::1 at 2018-12-03 09:14:28 +0800

…skipping…

2018-12-03T09:14:28 [I|app|a146a] Completed 403 Forbidden in 16ms (Views: 5.5ms | ActiveRecord: 2.1ms)

Really appreciate and thanks for your help :slight_smile: