proxy.log (44.9 KB)
Attaching the whole file of proxy.log
If you need any other logs do please let me know
This is the relevant bit
2022-08-03T09:03:43 [W] Error details for Couldn't enable 'ssh': <RuntimeError>: Ssh public key file /var/lib/foreman-proxy/ssh/id_rsa.pub doesn't exist
/opt/theforeman/tfm/root/usr/share/gems/gems/smart_proxy_remote_execution_ssh-0.5.2/lib/smart_proxy_remote_execution_ssh.rb:20:in `validate!'
/opt/theforeman/tfm/root/usr/share/gems/gems/smart_proxy_remote_execution_ssh-0.5.2/lib/smart_proxy_remote_execution_ssh/plugin.rb:36:in `block in <class:Plugin>'
/usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:228:in `class_eval'
/usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:228:in `configure_plugin'
/usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:109:in `block in configure'
/usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:109:in `each'
/usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:109:in `configure'
/usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:187:in `each'
/usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:187:in `initialize_plugins'
/usr/share/foreman-proxy/lib/launcher.rb:173:in `launch'
/usr/share/foreman-proxy/bin/smart-proxy:6:in `<main>'
this is the root cause, or at least for now
After fixing the above issue now I am getting the below issue
#######################################
Host,stdout,stderr,debug,Result,Finished
xdevsl.admiral.es,Exit status: EXCEPTION,"","Error initializing command: RuntimeError - Unable to create directory on remote system /var/tmp/foreman-ssh-cmd-ae3e2cb0-8936-44b7-99ca-fa31e5a07b6c: exit code: 255
Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
",error,2022-8-3 7:40:15 +0000
####################################################################
Hey Guys,
One more thing I have an old foreman setup in which this remote execution is working fine and the new one is configures exactly same as the old one. I am attaching the old foreman proxy logs as well.
old_foreman_proxy.log (13.4 KB)
There was a thread opened yesterday by you where the main topic was the exact error you’re hitting now. Maybe you didn’t really resolve that?
On a side note, claiming that two foreman instances are configured exactly the same way is a bold claim, even though it might appear so on the surface.
That’s exactly the error message from your thread Remote Execution Fails
So this should continue there and you should really fix your SSH keys…
It got resolved with changing the keys. What I really understood from yesterday till now that on new foreman setup dynflow is not working but on old its running.
############## New Foreman Output ###############################
[root@xproorchestrator05 dynflow]# ps -ef |grep -i dynflow
root 35263 33918 0 10:15 pts/0 00:00:00 grep --color=auto -i dynflow
[root@xproorchestrator05 dynflow]# systemctl status dynflowd.service
Unit dynflowd.service could not be found.
[root@xproorchestrator05 dynflow]#
#########################################################
############ Old Foreman Output #########################
[root@xproorchestrator03:/etc/smart_proxy_dynflow_core#] ps -ef |grep -i dynflow
foreman+ 2824 1 0 Jul13 ? 00:10:02 ruby /usr/bin/smart_proxy_dynflow_core -d -p /var/run/foreman-proxy/smart_proxy_dynflow_core.pid
foreman 2905 1 0 Jul13 ? 00:00:22 dynflow_executor_monitor
root 22770 20631 0 10:15 pts/2 00:00:00 grep --color=auto -i dynflow
foreman 61083 1 0 Jul27 ? 00:57:35 dynflow_executor
[root@xproorchestrator03:/etc/smart_proxy_dynflow_core#] systemctl status dynflowd.service
● dynflowd.service - Foreman jobs daemon
Loaded: loaded (/usr/lib/systemd/system/dynflowd.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2022-07-13 10:44:35 CEST; 2 weeks 6 days ago
Docs: https://theforeman.org
Process: 1721 ExecStart=/usr/sbin/dynflowd start (code=exited, status=0/SUCCESS)
Tasks: 19
CGroup: /system.slice/dynflowd.service
├─ 2905 dynflow_executor_monitor
└─61083 dynflow_executor
Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
#################################################################################
So if you could help me in this how I could really start dynflow on the new setup
A lot has changed between foreman 1.17 and foreman 3.1.
Simply run
# foreman-maintain health check
and it tells you if everything needed is running.
Do not attempt to activate or start deprecated services only because they were running on foreman 1.17…
Looks Like Everything is all good on new foreman
############################################################
[root@xproorchestrator05 smart_proxy_dynflow]# foreman-maintain health check
Running ForemanMaintain::Scenario::FilteredScenario
Check number of fact names in database: [OK]
Check whether all services are running: [OK]
Check whether all services are running using the ping call: [OK]
Check for paused tasks: [OK]
Check to verify no empty CA cert requests exist: [OK]
[root@xproorchestrator05 smart_proxy_dynflow]#
#####################################################################################
I also found out that dynflow core rubygem is of foreman 2 version is that ok?
[root@xproorchestrator05 smart_proxy_dynflow]# rpm -qa|grep -i dynflow
tfm-rubygem-dynflow-1.6.3-1.fm3_1.el7.noarch
tfm-rubygem-smart_proxy_dynflow-0.6.2-1.fm3_1.el7.noarch
tfm-rubygem-smart_proxy_dynflow_core-0.4.1-1.fm2_6.el7.noarch
foreman-dynflow-sidekiq-3.1.2-2.el7.noarch
dynflow-utils-1.6.3-1.el7.x86_64
[root@xproorchestrator05 smart_proxy_dynflow]#
Well, it depends. You wrote in the other thread that this is a new server. Now it looks like you have upgraded. If you have upgraded from 1.17 to 3.1.2 you do understand that you have to go through all intermediate versions as well? If you install 3.1.2 on a new server it won’t install that rpm.
I have installed it on new server. i never did upgrade. As you could see in my outputs I have 2 server running.
Well, then it’s probably still in use, but never been repackaged specifically for 3.1. It’s going to be removed with 3.2 anyway.
Ok now back to my problem of remote execution
How could we fix it ?
Please help me guys I am working on it from last 2 weeks
But I havent come to any conclusion.
What problem? You wrote you have resolved the issue, the one here as well as the one in the other thread.
I resolved the key issue but still remote execution is failing. I have pasted the logs above.
Sorry, but I still don’t understand.
The 404 error from this thread was due to
Ssh public key file /var/lib/foreman-proxy/ssh/id_rsa.pub doesn't exist
which you have fixed:
At the same time you have posted this error:
which is the exact same error as in the other thread.
So same error, same solution unless you provide other information showing it’s really a different problem. But again, that belongs into the other thread because that is your thread about that issue with that exact error message…
Yeah I solved the one ```
Ssh public key file /var/lib/foreman-proxy/ssh/id_rsa.pub doesn’t exist
But again after that I got another error which I mentioned in the below thread
https://community.theforeman.org/t/initialization-error-restclient-notfound-404-not-found/29731/8?u=shakunj
So whatever I do my remote execution is not working
So please help me in solving the remote execution problem.
So if you guys need any other logs just let me knowYes. And as have been pointed multiple times, that is the exact same error as in your thread Remote Execution Fails
This is the error from post 8 above:
and this is the error from Remote Execution Fails:
As you can see it’s the same error message, just without the debug log… So it’s the error from the other thread. And it doesn’t belong into this thread.