But you never wrote whether it uses the JumpHost or not. That part is in the verbose output.
This, again, is highly ambiguous and unclear. It’s really difficult to know if you really used the command I have posted before or not. Things are so much easier if you just post the details including the command. That’s precise and clear. More often then less it happens that people write they did something but then they didn’t or with variation and it’s impossible to know.
Again: where exactly is what key for what purpose? I can assume you mean the public key is in /root/.ssh/authorized_keys of those “admin and end boxes”. But again, you don’t clear write it.
Again: unclear. Does it mean when using the CLI command to log into the end boxes it does use the proxy host or not?
The issue in place is about using a proxy host and using the necessary configuration on the foreman server to do so. Thus, the relevant logs should cover that part at least…
I don’t understand what your problem is: you do a remote login into a server. For a remote login you need a username which is what you pass with -l or before the @ to the ssh client. For all that, it’s absolutely irrelevant what user you may be on your local system. I don’t see your problem. You can log into as root or any other if you wish. You can use any method you want to elevate to root if you didn’t log in as root to begin with. So basically you can run jobs as any user you want on the remote system. What else do you need? Why does it seem to matter whether the process/service initiating this on the local foreman server is running as root, foreman-proxy or anything else for that matter?
You can configure the defaults in the settings. You can change any of those for each job you submit. What else do you need? I think in most cases people use it to run something as root on the remote systems and if you can do that you can do anything you want. The configuration only gives you options on how to achieve that exactly…
And again: without not even knowing the foreman/katello version you are using and all that other information usually asked about when starting a thread it’s really futile. At some point in the past they have switched from using some netssh module to directly using ssh via shell. I think the module was very limited in what options it would actually use from the ssh config. So, you know, without knowing all the facts and details it’s really just guesswork. You may be just using an old foreman version and it doesn’t support the proxyhost at all.
O.K. But what is happening when you are using foreman-proxy, i.e.:
# sudo -u foreman-proxy ssh -i /var/lib/foreman-proxy/ssh/id_rsa_foreman_proxy root@test.example.com date
You don’t write it. I can only guess what you mean.
Well, look at the command I have been posting and just posted again and how often I wrote where foreman expects the key to be. It’s not /var/lib/foreman-proxy/ssh/sv_id_rsa. It is passed on the command line which takes precedence.
The foreman proxy currently runs ssh with command line option -o ProxyCommand=none which again takes precedence… If it’s a current foreman version you are using…
Which of course, kills the ProxyJump as well because ProxyJump is competing with ProxyCommand… I am wondering whether there is already an issue/bug report for that…
OK, when i say adding keys to servers, then yes I mean to the authorized_keys file for the user I am connecting in as. I didn’ think I’d need to explain that part, given this is quite a technical thread
My problem is very straightforward - I want Foreman to use the proxy host we have before reaching all of our end clients. It doesn’t really matter why, but just in case it helps - we have hardered all servers to only allow our admin jump hosts SSH access in. I would therefore like to connect to all of these servers via the allowed jump box and not directly.
yes, I mean when connecting to the end client with the jump proxy config in place, it proxies successfully, as expected, but only via CLI, not when running a Foreman remote job.
i did and it does, yes. All of my tests via CLI from foreman to end client with proxy config in place, connect to that proxy first.
both using foreman-proxy and/or root keys
debug1: /etc/ssh/ssh_config line 58: Applying options for *
debug1: Executing proxy command: exec ssh proxyadmin1 -W endclient:22
debug1: Offering RSA public key: /var/lib/foreman-proxy/ssh/id_rsa_foreman_proxy
debug1: Server accepts key: pkalg rsa-sha2-512 blen 279
debug1: Authentication succeeded (publickey).
Authenticated to endclient (via proxy).
So, in a nut shell, my initial ‘problem’ is how can i request Foreman uses the admin proxy host for all SSH remote jobs. Either by reading a config file or by other means
Well, unless you use an older but not too old version of foreman, it’s not possible at the moment which I would consider a bug but I am not sure if someone posted one already. See the other thread…
Presuming this works, that’s all I needed to know. kind of funny that we had lots of back and forths about users/keys when I just needed to remove the proxy command from Foremans config.
I don’t use proxies for ssh thus most of the time I have the original issue with the ProxyCommand in mind which I have had a year ago and which was the reason for that command line option being added there. I never think about it killing ProxyJump as well.
I am wondering what the effect of the /bin/false default shell for foreman-proxy is exactly. I thought that was why the ProxyCommand failed to begin with…
Spent some time grepping and I’ve only found some ruby SSH options in the following path but does not contain the all important ProxyCommand variable. I’m running version 3.2.1 (soon to be upgrade to latest though)
Well, I have asked multiple times for the exact ssh config and lacking that information gave you information on how it generally works. If you had posted the config file and the foreman version in the beginning, it would have gone must quicker…
The code is in the smart proxy package for remote execution rubygem-smart_proxy_remote_execution_ssh. Since the switch to ssh it should be in the file multiplexed_ssh_connection.rb, i.e. you can search
will list all files in the package if it does not contain that file and indicate the path.
Of course, the next update of rubygem-smart_proxy_remote_execution_ssh will require the same change and you may want to clean out the remnant file and directories after that as the update won’t remove the modified file with each new gem version.
hmm, not sure about that. I did explain it was a basic SSH config with a ProxyCommand to our admin host. Again, I should have just read the first reply with that link to see that Foreman has a proxy variable in the ruby code.
Yup one of the first searches I ran was for ’ multiplexed_ssh_connection.rb’ but couldnt find it. Anyway, I’ve checked and also no joy from the commands you suggest as the package is not there - likely renamed in a newer release?
package rubygem-smart_proxy_remote_execution_ssh is not installed
the only smart proxy remote execution location I have is as stated:
/opt/theforeman/tfm/root/usr/share/gems/gems/smart_proxy_remote_execution_ssh-0.5.3/ - but nothing in there for ProxyCommand variable
to list all packages containing “remote” and it should list the smart proxy package for remote execution.
Then you can use rpm -ql on the package name (without the version if you want) to list the files included.
Either way, I find it somewhat peculiar that you don’t have rubygem-smart_proxy_remote_execution_ssh installed. It’s in the foreman plugin repository for 3.2 with that exact name so I don’t know why it would be installed if you have it.
I have pasted my searches in the other thread which you are part of
Seems I have the correct path for all ssh_options variable but does not include the proxy
Well, yes, another reason why you should really give the information which is asked when you start a thread: you are still running on a el7 os. Information like that is really helpful if we know it right from the start. Then from the very beginning we would all know that you are running 3.2.1 on el7 and instead of guessing what you might use and looking at code of recent versions, we would know what you have and what it looks like.
And that’s then the reason why you cannot find the ProxyCommand command line option in the code: it hasn’t been added, yet, at that time.
It was added with
You could have save a lot of time here…
So you have a ProxyCommand configuration in ~foreman-proxy/.ssh/config and it’s actually running but not using the ProxyCommand although you think it should do so? Odd. I thought it that case it shouldn’t work at all because the ProxyCommand would run with default shell which should be /bin/false which would indicate failure.
What is the default shell for foreman-proxy on your foreman server?
I would suggest to set :ssh_log_level in /etc/foreman-proxy/settings.d/remote_execution_ssh.yml to verbose (or even debug) and run again and check the foreman proxy log. It contains the output from the ssh client just like with -v and it should tell what files and configuration it uses…
Understood, I’ll accept that I should have provided more info on the OS and version first. Apologies for that
No, the config was copied to /var/lib/foreman-proxy/ssh which symlinks to /usr/share… it was copied there because you said this is where the ssh home path is by default for keys etc
There is no user home dir setup for foreman-proxy on the OS. I would need to create one and the .ssh subfolder. Should I?
urgh ignore me, i miss-read
yes, the config was also added to /usr/share/foreman-proxy which is the home dir and the task immediately failed, prob because as you say, there’s no shell for it?
Well, I don’t know if it’s different with 3.2.1 or on el7, but on my 3.5 with el8 /usr/share/foreman-proxy/.ssh is a symlink to /var/lib/foreman-proxy/ssh.
I am not sure, if you can use ProxyJump instead. I guess, internally ssh will just rewrite that into another proxy command and it won’t work either. But I am not sure. You could try.
And the patching gets worse after you upgrade to a later version which has the ProxyCommand override on the command line.
Bottom line: it all comes down to what they do with the issue I have opened. Currently, it’s not supported and only possible with some patching… Let’s hope someone comes up with a solution.
So, technically the quick band-aid at this point would be to change the default
Ok, I did try proxyjump but same result. will try changing the foreman-proxy user to /bin/bash and re-try. If this works, I will try and remember to revert it back if when we run foreman-installer
no joy after changing foreman-proxy from /bin/false to /bin/bash
Will give up on this proxy solution until we have upgraded
again thanks for your help on here. i will up the logging to verbose for curiousity but may have to give up on the idea and allow Foremans source IP in to our servers for SSH
That’s odd. What’s the exact error in this case? You should also set :ssh_log_level to verbose and check the ssh verbose output to see what is going wrong.
