Encryption decryption oauth_consumer_key

It looks like sometimes decrypting of oauth key doesn’t work. I’m using Foreman 1.23
In foreman server logs I see entries like that:
2019-12-13T14:43:26 [W|app|55cf6b64] oauth_consumer_key should be 'encrypted-ZmhWSzVjaHVWTXp2MWFyd05oUXRCUGN1NUxaVENkaEpGak1jbkdBWEQzTXhEY3NhbGhjbHNWRzFTakdFdVlPbFc5M2JCY2hVc3JkRWw4bGUybWo2cWc9PS0tbDA5djFFaXhqVEN1UmNVOENIL1NGdz09--f72682c13e351858f6de9b4bd5ef44f6bafd3534' but was '7dUQG3uvW3ypmsfryKJWGyNrTbZjhMAA' 2019-12-13T14:43:26 [W|app|55cf6b64] SSO failed

It seems like issues with encrypting / decrypting. It causes that foreman proxy can’t be registered with foreman server.
Usually, it works well for a few minutes.

This sounds like a bug indeed, could you report it on projects.theforeman.org please? Does it only fail sometimes or all the time? for all hosts or just one? was it working properly in 1.22? I don’t think we’ve touched this area of the code in a while, so any further information is appreciated.

Probably this issue is much more complex. I’ve installed foreman with puppet.
I have three instances of foreman behind the load balancer. They share a session with one memcached. When I leave just one instance of foreman there is no issue with encrypting / decrypting key. I have this issue when I start more than one foreman instances.

It fails sometimes. Not all the time. It looks like is ok for 5 minutes and another 5 is not ok. I can’t check if it was good with 1.22 because I started from 1.23.

When you deploy multiple application servers, they need to share the same secret key. That’s because /etc/foreman/encryption_key.rb is used to encrypt secrets (like the oauth credentials). AFAIK we have no way to manage /etc/foreman/encryption_key.rb so you’ll need to take care of that yourself.

We have been seeing this issue too since we started using 2 Foreman instances.
We also have 2 puppetservers that register as proxies.
We also use puppet to set this up and there is only 1 secret key defined in Hiera for all (both) nodes.

Sure, we have everywhere the same secret. This is managed by puppet. Unfortunately it doesn’t work even with the same key on every server.

we have a foreman installation where we lost the :oauth_* config from the settings.yml, also we always get the following error when running foreman-rake db:migrate:

root@foreman:/root # foreman-rake db:migrate
At least one field decryption failed, check ENCRYPTION_KEY

does anyone have an idea how to fix this?

solved by re-entering all passwords