Remote execution Authentication failure

Support
1

Problem:

Remote execution via SSH authentication fails when connecting to remote machine.

Solution was to re-enable SHA1 in the client crypto policy using:

update-crypto-policies --set DEFAULT:SHA1; /sbin/init 6

Expected outcome:

Remote execution works as expected.

Foreman and Proxy versions:

Foreman: 2.5.1
Foreman-proxy: 2.5.1

Foreman and Proxy plugin versions:

foreman-tasks 4.1.2
foreman_bootdisk 17.1.0
foreman_discovery 17.0.1
foreman_expire_hosts 7.0.4
foreman_hooks 0.3.17
foreman_openscap 4.3.2
foreman_remote_execution 4.5.0
foreman_setup 7.0.0
foreman_statistics 1.1.1
foreman_templates 9.1.0

Distribution and version:

Foreman: RHEL 8.4
Client: RHEL 9.1

Other relevant data:

Related messages noted in /var/log/secure:

userauth_pubkey: key type ssh-rsa not in PubkeyAcceptedAlgorithms [preauth]

Assume SSH library is using older SHA1 by default. Request a method to be published (eg. in FAQ) detailing enabling more recent algorithms and test procedure.

2

But technically this has nothing to do with foreman. That’s a general ssh issue. Any ssh connection with a ssh-rsa key would fail. Your users will notice this, too, if they they don’t have newer ec keys.

And how to solve this issue is a matter of your security policies and whether you want to keep using older keys or whether you want people to switch to newer, more secure keys. So it needs careful consideration.

So your “solution” may be acceptable for your environment but in it’s brevity it’s ill-advised to simply follow it without fully understanding the consequences.

The better solution would be to use this opportunity to retire old rsa keys and use new keys. That wouldn’t require to alter security policies of el9 and it solves the connection issue in a secure way.

3

So… the objection to publishing an FAQ entry or similar is…?

4

Some additional information:

This works fine:

[root@foreman]# sudo -u foreman-proxy ssh -vvv -i /var/lib/foreman-proxy/ssh/id_rsa_foreman_proxy USERNAME@IP_ADDRESS

However attempting to connect via foreman doesn’t work, I assume because the algorithms supported within the foreman remote execution module require SHA1 or are configured to expect SHA1. This appears, therefore, to be a foreman issue, not an underlying SSH issue.

Request a method to be published (eg. in FAQ) detailing enabling support for more recent algorithms within foreman remote execution and test procedure.

5

O.K. Looking deeper into it, it’s not an issue with ssh-rsa but really only with the sha1 signature. So the problem you have it why or foreman server sends sha1 signatures and not sha256. My foreman 3.4.1 server on alma8 uses sha256 signatures just fine with ssh-rsa.

Weakening security on all your el9 clients is no solution. It’s a temporary quick workaround at best. But from experience those workarounds tend to stick around because noone remembers to undo that change at some point. It’s just like those “solutions” suggesting to disable selinux…

So the real problem is that you are running an outdated foreman server with an old version of remote execution which IIRC uses an old netssh module which seems only to support sha1 for signatures. At some point, they switched to using the standard ssh client. That’s why you can use ssh but not remote execution.

So the solution is to upgrade your foreman server to a current version. That will fix your problem without touching any client.

If you cannot upgrade your server then your suggestion is a temporary workaround until you can upgrade. It’s not a general solution. And you shouldn’t really never forget to undo the change on all your el9 clients once you have upgraded.

There is no point putting this in a FAQ of foreman because it’s not an issue with any currently supported version. That issue is your outdated server…

6

I don’t have this problem on my foreman 3.5.1 instance, either. I do​, however, have customers with older versions of foreman installed and are unwilling to spend $$ right now for the time required to perform upgrades on their foreman deployments.

We live in the Real World, not an Ideal World. How often are users expected to upgrade their foreman instances in the Real World, in any case?

So - if there are folks out in foreman_userland™ who deploy RHEL9 boxes into their environment and hit the same issue…? Its not worth informing them? Not worth saving them a half day of head-scratching?

I reenabled SHA1 to identify the source of the problem, not to become the ‘fix’, for the reasons you identified. That still, however, leaves customers with older foreman 2.x.x deployments that can’t work with the new clients that they have​ paid to be upgraded.

7

But as you can see, you’ll run into problems if you run outdated, unsupported software…

If you are running outdated, unsupported software? Do you really expect to support all old versions for all kind of issues which may arise if people running outdated software doesn’t work anymore with the latest operating systems? When foreman 2.5.1 was released el9 hasn’t been released, yet.

Yes. However, your “solution” doesn’t really fix the issue in place. And as workaround it does far more than expected. For instance, it allows sha1 certificates to be used. sha1 certificates have been deprecated for years. It allows for SHA1 to be used in ciphers. What you do is severely crippling current security practices. It is simply a bad idea to enable the SHA1 submodule unless it’s absolutely necessary in your environment.

As I wrote before: it’s similar to those people who disable selinux because their server cannot write a specific file. The best workaround would be to label the file or directory correctly and it works. The solution would be to comply with the files/directory structure to comply with selinux,

Thus, to accept ssh-rsa for ssh connections from your foreman server, it’s no solution but only a bad workaround to enable SHA1 for all crypto functions. The workaround would be to allow ssh-rsa in the sshd server on your el9 clients, if possible even in a Match block to limit this workaround exclusively for the ssh connections from your foreman server/proxy. That would allow your foreman server to use remote execution without jeopardizing the security of the whole client. Of course, you should make sure that this modification is remembered when it’s not necessary anymore. Because, otherwise changes like these tend to stick around forever…

But people who have this issue may see this thread, see your “solution” and apply it without thinking it through. That’s why it’s kind of dangerous to suggest far reaching solutions like this without a serious disclaimer. It has much bigger consequences as allow those ssh remote execution connections.

8

Wow. I think you missed the part where I said: “I used this I reenabled SHA1 to identify the source of the problem, not to become the ‘fix’, for the reasons you identified.”

As a diagnostic tool. To identify the source of the problem.

I think leave it there.

Regards,

Ross

9

Yes. Later in the thread. But who reads through the whole thread if the solution is right there in the beginning and in large letters:

As I have tried to explain with my selinux analogy: you’ll end up with people picking it up and simply following it, because it “works”.

And it doesn’t explain why you didn’t simply enable ssh-rsa in the sshd_config, because I would consider that a “diagnostic tool”.

10

But who reads through the whole thread…?

Illiterate people without the time and care to actually read stuff.

And it doesn’t explain why you didn’t simply enable ssh-rsa in the sshd_config, because I would consider that a “diagnostic tool”.

Because that approach doesn’t work. I tend not to post things that don’t work.

This is probably because netssh doesn’t observe sshd_config settings?

As I have tried to explain with my selinux analogy: you’ll end up with people picking it up and simply following it, because it “works”.

Well - I did post this warning: “NOTE: re-enabling SHA1 may violate your security policy.” in my post to this topic: Remote action failed - #12 by gonzi.

I think you may have missed that, too, since it was at the end of the post and you didn’t read that far?

11

Let’s take a little detour into the past.

When remote execution was first introduced to Foreman in 1.9, it used net/ssh, a pure-ruby reimplementation of the ssh protocol, under the hood to connect to the remote hosts. At that time, we were quite satisfied with how it worked and with the level of control it gave us. However as time went by, new standards were established and older ones deprecated, net/ssh itself was having a bit of a hard time keeping up and to make things worse, we found ourselves locked to an old version of net/ssh for quite a long time due to other dependencies. If you look at Red Hat bugzilla, you’ll find around 8 bugzillas describing this very issue in different words, most of them have their counterpart in Foreman’s issue tracker as well.

Later it became clear that net/ssh wasn’t cutting it for us anymore, so we switched it out for using the ssh binary which is provided in openssh-clients package on EL* based distributions. This switch went live with Foreman 3.1.

Yes, that is sadly the truth.

Sadly, there is no way around this. There is no knob that can be turned, no switch to be flicked, the only way out is to upgrade as we have reached the limits of what the software can do. I admit it is unfortunate, but here we are.

New version of Foreman comes every 3 months give or take, with two latest releases being supported. There are other products built on Foreman, where the life cycle is longer.

To be fair, Foreman 2.5 was released in June 2021, which was some time ago. I know some customers cannot move that fast, but not everything is as future-proof as it should be. It really is up to the users to decide, but there is only so much of what can be supported.

Usually I’d end my comment right here, where most of the purely technical stuff was said, however in this case I’ll make an exception to touch on a few points.

This whole thread got a little bit heated and as it went, we strayed away from the problem we originally wanted to solve. The key thing is, that all parties come from a different background, but are motivated by good intentions. So please keep the focus on the outcome. As with everything, nothing is perfect and there are always tradeoffs that need to be made and it is up to each of us to decide where we draw the line.

On one hand, I must agree with rosco. The assumption that things just keep working is a sensible one, especially when we don’t really state anywhere that they don’t.

On the other side, I must also agree with gvde that the wording of the original post is a bit unfortunate. The bold letters and everything with a warning being posted in another thread really makes it easy for tunnel vision to kick in and have folks just copy-paste it everywhere. As much as it makes me feel ashamed to admit it, I’ve been there and I did that. When you spend a day or two banging your head against the wall, you don’t care about the small print. Whatever works. What makes it slightly better is that there is no post in this thread that is actually marked as a solution.

2 Likes
12

Hi Adam,

Please advise whether you would like me to formulate a FAQ entry for this issue.

WRT the use of SHA1 / SHA256, each user would need to perform a risk assessment and mitigate as appropriate.

WRT customers, I made them aware of this topic. Two have responded thus far:

  • One has examined the thread and decided to assess OpenStack as a (partial) replacement for Foreman. I won’t repeat here his comments regarding his perception of the ‘support’ he believes is available via the Community. It was not complementary.
  • One has identified Foreman for upgrade in Q2 2023.

One other customer running an even older Foreman version is yet to respond.

All of these 3 customers were inherited as part of an aquisition ~2022-09, so I only have the customers’ information to go on WRT the previous SP’s advice and the customers’ resulting decisions. Upgrading foreman was not placed high on their agendas.

Under these circumstances, there isn’t much point in a ‘heated’ discussion. IMHO.

An unexpected ‘feature’ of the forum is that using ‘#’ at the beginning of a line results in bold text. I commonly use ‘#’ at the beginning of a my own notes to indicate a command that should be run as root or equivalent.

I also formulate posts outside a forum and then paste them in and submit. Lines don’t appear bold prior to pasting, and I sometimes “fire and forget” once I’m happy with wording

In a subsequent post, I used: “root@[hostname] #” to indicate a command that could / should be run as root, but was unable to return to the original post to modify its appearance. It seems gvde fixated on this for reasons unknown.

13

Overall, foreman does its job well - saves a huge amount of time overall, however this ‘gotcha’ was unusual in that a newer OS wouldn’t work alongside an otherwise fully working Foreman. Somewhat unexpected, but I’d suspect this isn’t the first time a user has noted this, nor will it be that last, hence my initial request for a FAQ to make others’ lives easier.

14

Insulting people isn’t helping. In particular insulting people who simply stating facts. The command is prominently posted on your initial post, in large letters, called “solution” with all additional context information point exclusively to ssh and sha1, thus not even indicating any further implications that may have. You don’t explain why simply enabling “ssh-rsa” isn’t enough, which would - given the information in your initial post - be the solution to the error.

That is very unfortunate and risky, in particular because command has a lot of effects which people may never be aware of until it’s too late. I tried to explain this with my selinux analogy because there I have seen how people picked up a quick solution presented in this style, never thought about the full consequences and had servers running with selinux disabled only because some service couldn’t write a file into a location where it usually should write to. “Solution” applied - “problem” solved - forgotten.

So don’t shoot the messenger. Picking on people or insulting them won’t do you any good.

15

Not sure where I ‘insulted’ anyone?

Being ‘fixated’ on anything is generally bad. Other than my offering to formulate an FAQ entry, I think this conversation has run its course.

16

Illiteracy is a real handicap of real people. I don’t have anything else to add to that.

This is a community forum for open source software. So basically, this topic is in the end that FAQ you want. If you want to write a comprehensive answer on this topic, then you are free to do so. As this is only a problem of older, unsupported foreman versions there is only limited audience here who require this information let alone write a comprehensive mitigation for this problem.

As I have pointed out, if it comes down to changing the crypto-policies more detailed instructions would be good, as I think there are a lot of people out there who really didn’t know about crypto-policies in EL8/9, yet.

And changes should be minimal. Looking into the SHA1 pmod:

# cat /usr/share/crypto-policies/policies/modules/SHA1.pmod 
# This subpolicy adds SHA1 hash and signature support

hash = SHA1+

sign = ECDSA-SHA1+ RSA-PSS-SHA1+ RSA-SHA1+

sha1_in_certs = 1

It seems to add a lot more then necessary to allow ssh-rsa with SHA1.