Smart proxy upgrade receives 401 http code from Foreman

Support
,
1

Problem:
We have upgraded foreman and foreman proxy from 2.3.5 to 2.4.1 using our puppet agents and theforeman puppet modules, the foreman upgrade went successful without issues, the db migration and seed went well as well, however after upgrading the foreman proxy, the puppet agent is now throwing the following error:

Error: /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[smartproxy1.domain.tld]: Could not evaluate: Error making GET request to Foreman at https://foreman.domain.tld/api/v2/smart_proxies: Response: 401 Unauthorized: Often this is caused by invalid Oauth credentials

I have checked the oauth key and secret which we are passing them through hiera to the foreman server and foreman proxies as follow:

foreman_proxy.yaml:

profiles::foreman::smart_proxy::parameters:
  version: '2.4.1'
  trusted_hosts:
    - 'smartproxy1.domain.tld'
    - 'smartproxy2.domain.tld'
    - 'smartproxy3.domain.tld'
    - 'smartproxy4.domain.tld'
  foreman_base_url: 'https://foreman.domain.tld'
  oauth_consumer_key: aaabbbccc
  oauth_consumer_secret: xxxyyyzzz
  puppetca_provider: 'puppetca_hostname_whitelisting'
  registration: false

The puppet class profiles::foreman::smart_proxy passes the parameters as follow:

class{ '::foreman_proxy' :
    * => $parameters,
  }

foreman.yaml:

profiles::foreman::parameters:
  initial_admin_username: 'admin'
  initial_admin_password: redacted
  initial_admin_first_name: redacted
  initial_admin_last_name: redacted
  initial_admin_email: redacted
  db_manage: false
  email_delivery_method: 'smtp'
  email_smtp_address: redacted
  email_smtp_port: 25
  email_smtp_domain: redacted
  foreman_url: 'https://foreman.domain.tld'
  unattended: true
  unattended_url: 'http://foreman.domain.tld'
  serveraliases:
    - 'foreman.domain.tld'
  ssl: false
  version: '2.4.1'
  plugin_version: 'latest'
  db_host: redacted
  db_port: 5432
  db_database : 'foreman'
  db_username: redacted
  db_password: redacted
  db_sslmode: 'UNSET'
  db_manage_rake: false
  server_port: 21002
  client_ssl_ca: '/etc/puppetlabs/puppet/ssl/certs/ca.pem'
  client_ssl_cert: '/etc/puppetlabs/puppet/ssl/certs/foreman_proxy_client.pem'
  client_ssl_key: '/etc/puppetlabs/puppet/ssl/private_keys/foreman_proxy_client.pem'
  ipa_manage_sssd: false
  websockets_ssl_key: '/etc/puppetlabs/puppet/ssl/private_keys/foreman_websockets.pem'
  websockets_ssl_cert: '/etc/puppetlabs/puppet/ssl/certs/foreman_websockets.pem'
  logging_level: 'info'
  logging_type: 'file'
  logging_layout: 'json'
  jobs_manage_service: false
  oauth_consumer_key: aaabbbccc
  oauth_consumer_secret: xxxyyyzzz

The puppet class profiles::foreman is loading the parameters as follow:

class { '::foreman' :
    * => $parameters,
  }

I have set the logging to Debug and this is what I can see in the foreman production.log file:

{"logger":"app","timestamp":"2023-06-06T10:56:50.201081-05:00","level":"INFO","message":"Started GET \"/api/v2/smart_proxies?search=name%3D%22smartproxy1.domain.tld%22\" for 192.168.0.10 at 2023-06-06 10:56:50 -0500","mdc":{"remote_ip":"192.168.0.10","request":"1cd7d91e-637b-4d94-b21f-6e5703175835"},"ndc":[]}
{"logger":"app","timestamp":"2023-06-06T10:56:50.204476-05:00","level":"INFO","message":"Processing by Api::V2::SmartProxiesController#index as JSON","mdc":{"remote_ip":"192.168.0.10","request":"1cd7d91e-637b-4d94-b21f-6e5703175835","session":"1cd7d91e-637b-4d94-b21f-6e5703175835"},"ndc":[]}
{"logger":"app","timestamp":"2023-06-06T10:56:50.204707-05:00","level":"INFO","message":"  Parameters: {\"search\"=\u003e\"name=\\\"smartproxy1.domain.tld\\\"\", \"apiv\"=\u003e\"v2\", \"smart_proxy\"=\u003e{}}","mdc":{"remote_ip":"192.168.0.10","request":"1cd7d91e-637b-4d94-b21f-6e5703175835","session":"1cd7d91e-637b-4d94-b21f-6e5703175835"},"ndc":[]}
{"logger":"app","timestamp":"2023-06-06T10:56:50.205169-05:00","level":"DEBUG","message":"Cache read: settings/failed_login_attempts_limit","mdc":{"remote_ip":"192.168.0.10","request":"1cd7d91e-637b-4d94-b21f-6e5703175835","session":"1cd7d91e-637b-4d94-b21f-6e5703175835"},"ndc":[]}
{"logger":"app","timestamp":"2023-06-06T10:56:50.246558-05:00","level":"DEBUG","message":"Cache fetch_hit: settings/failed_login_attempts_limit","mdc":{"remote_ip":"192.168.0.10","request":"1cd7d91e-637b-4d94-b21f-6e5703175835","session":"1cd7d91e-637b-4d94-b21f-6e5703175835"},"ndc":[]}
{"logger":"app","timestamp":"2023-06-06T10:56:50.246824-05:00","level":"DEBUG","message":"Cache read: failed_login_192.168.0.10","mdc":{"remote_ip":"192.168.0.10","request":"1cd7d91e-637b-4d94-b21f-6e5703175835","session":"1cd7d91e-637b-4d94-b21f-6e5703175835"},"ndc":[]}
{"logger":"app","timestamp":"2023-06-06T10:56:50.247207-05:00","level":"DEBUG","message":"Cache fetch_hit: failed_login_192.168.0.10","mdc":{"remote_ip":"192.168.0.10","request":"1cd7d91e-637b-4d94-b21f-6e5703175835","session":"1cd7d91e-637b-4d94-b21f-6e5703175835"},"ndc":[]}
{"logger":"app","timestamp":"2023-06-06T10:56:50.247422-05:00","level":"DEBUG","message":"Cache read: settings/authorize_login_delegation","mdc":{"remote_ip":"192.168.0.10","request":"1cd7d91e-637b-4d94-b21f-6e5703175835","session":"1cd7d91e-637b-4d94-b21f-6e5703175835"},"ndc":[]}
{"logger":"app","timestamp":"2023-06-06T10:56:50.247725-05:00","level":"DEBUG","message":"Cache fetch_hit: settings/authorize_login_delegation","mdc":{"remote_ip":"192.168.0.10","request":"1cd7d91e-637b-4d94-b21f-6e5703175835","session":"1cd7d91e-637b-4d94-b21f-6e5703175835"},"ndc":[]}
{"logger":"app","timestamp":"2023-06-06T10:56:50.247986-05:00","level":"DEBUG","message":"Cache read: settings/oauth_active","mdc":{"remote_ip":"192.168.0.10","request":"1cd7d91e-637b-4d94-b21f-6e5703175835","session":"1cd7d91e-637b-4d94-b21f-6e5703175835"},"ndc":[]}
{"logger":"app","timestamp":"2023-06-06T10:56:50.248272-05:00","level":"DEBUG","message":"Cache generate: settings/oauth_active","mdc":{"remote_ip":"192.168.0.10","request":"1cd7d91e-637b-4d94-b21f-6e5703175835","session":"1cd7d91e-637b-4d94-b21f-6e5703175835"},"ndc":[]}
{"logger":"app","timestamp":"2023-06-06T10:56:50.250723-05:00","level":"DEBUG","message":"Cache write: settings/oauth_active","mdc":{"remote_ip":"192.168.0.10","request":"1cd7d91e-637b-4d94-b21f-6e5703175835","session":"1cd7d91e-637b-4d94-b21f-6e5703175835"},"ndc":[]}
{"logger":"app","timestamp":"2023-06-06T10:56:50.251255-05:00","level":"DEBUG","message":"Cache read: settings/oauth_consumer_key","mdc":{"remote_ip":"192.168.0.10","request":"1cd7d91e-637b-4d94-b21f-6e5703175835","session":"1cd7d91e-637b-4d94-b21f-6e5703175835"},"ndc":[]}
{"logger":"app","timestamp":"2023-06-06T10:56:50.292769-05:00","level":"DEBUG","message":"Cache fetch_hit: settings/oauth_consumer_key","mdc":{"remote_ip":"192.168.0.10","request":"1cd7d91e-637b-4d94-b21f-6e5703175835","session":"1cd7d91e-637b-4d94-b21f-6e5703175835"},"ndc":[]}
{"logger":"app","timestamp":"2023-06-06T10:56:50.293025-05:00","level":"DEBUG","message":"Cache read: settings/oauth_consumer_secret","mdc":{"remote_ip":"192.168.0.10","request":"1cd7d91e-637b-4d94-b21f-6e5703175835","session":"1cd7d91e-637b-4d94-b21f-6e5703175835"},"ndc":[]}
{"logger":"app","timestamp":"2023-06-06T10:56:50.293390-05:00","level":"DEBUG","message":"Cache fetch_hit: settings/oauth_consumer_secret","mdc":{"remote_ip":"192.168.0.10","request":"1cd7d91e-637b-4d94-b21f-6e5703175835","session":"1cd7d91e-637b-4d94-b21f-6e5703175835"},"ndc":[]}
{"logger":"app","timestamp":"2023-06-06T10:56:50.294295-05:00","level":"WARN","message":"OAuth signature verification failed.","mdc":{"remote_ip":"192.168.0.10","request":"1cd7d91e-637b-4d94-b21f-6e5703175835","session":"1cd7d91e-637b-4d94-b21f-6e5703175835"},"ndc":[]}
{"logger":"app","timestamp":"2023-06-06T10:56:50.294512-05:00","level":"WARN","message":"SSO failed","mdc":{"remote_ip":"192.168.0.10","request":"1cd7d91e-637b-4d94-b21f-6e5703175835","session":"1cd7d91e-637b-4d94-b21f-6e5703175835"},"ndc":[]}
{"logger":"app","timestamp":"2023-06-06T10:56:50.294655-05:00","level":"DEBUG","message":"Cache read: failed_login_192.168.0.10","mdc":{"remote_ip":"192.168.0.10","request":"1cd7d91e-637b-4d94-b21f-6e5703175835","session":"1cd7d91e-637b-4d94-b21f-6e5703175835"},"ndc":[]}
{"logger":"app","timestamp":"2023-06-06T10:56:50.294851-05:00","level":"DEBUG","message":"Cache fetch_hit: failed_login_192.168.0.10","mdc":{"remote_ip":"192.168.0.10","request":"1cd7d91e-637b-4d94-b21f-6e5703175835","session":"1cd7d91e-637b-4d94-b21f-6e5703175835"},"ndc":[]}
{"logger":"app","timestamp":"2023-06-06T10:56:50.295098-05:00","level":"DEBUG","message":"Cache write: failed_login_192.168.0.10 ({:expires_in=\u003e5 minutes})","mdc":{"remote_ip":"192.168.0.10","request":"1cd7d91e-637b-4d94-b21f-6e5703175835","session":"1cd7d91e-637b-4d94-b21f-6e5703175835"},"ndc":[]}
{"logger":"app","timestamp":"2023-06-06T10:56:50.297129-05:00","level":"INFO","message":"  Rendering api/v2/errors/unauthorized.json.rabl within api/v2/layouts/error_layout","mdc":{"remote_ip":"192.168.0.10","request":"1cd7d91e-637b-4d94-b21f-6e5703175835","session":"1cd7d91e-637b-4d94-b21f-6e5703175835"},"ndc":[]}
{"logger":"app","timestamp":"2023-06-06T10:56:50.298224-05:00","level":"INFO","message":"  Rendered api/v2/errors/unauthorized.json.rabl within api/v2/layouts/error_layout (Duration: 0.9ms | Allocations: 339)","mdc":{"remote_ip":"192.168.0.10","request":"1cd7d91e-637b-4d94-b21f-6e5703175835","session":"1cd7d91e-637b-4d94-b21f-6e5703175835"},"ndc":[]}
{"logger":"app","timestamp":"2023-06-06T10:56:50.298886-05:00","level":"INFO","message":"Filter chain halted as :authorize rendered or redirected","mdc":{"remote_ip":"192.168.0.10","request":"1cd7d91e-637b-4d94-b21f-6e5703175835","session":"1cd7d91e-637b-4d94-b21f-6e5703175835"},"ndc":[]}
{"logger":"app","timestamp":"2023-06-06T10:56:50.299145-05:00","level":"INFO","message":"Completed 401 Unauthorized in 94ms (Views: 3.3ms | ActiveRecord: 0.9ms | Allocations: 7632)","mdc":{"remote_ip":"192.168.0.10","request":"1cd7d91e-637b-4d94-b21f-6e5703175835","session":"1cd7d91e-637b-4d94-b21f-6e5703175835"},"ndc":[]}

I have also enabled httpd forensics module to validate if the right oauth keys are being sent in the headers which they look to be the same:

+117875:647f5742:8|GET /api/v2/**smart_proxies**?search=name%253D%2522smartproxy1.domain.tld%2522 HTTP/1.1|Accept:application/json|Content-Type:application/json|Foreman_user:admin|Accept-Encoding:gzip;q=1.0,deflate;q=0.6,identity;q=0.3|User-Agent:OAuth gem v0.5.1|Authorization:OAuth oauth_consumer_key="aaabbbccc", oauth_nonce="redacted_value", oauth_signature="redacted_value", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1686067010", oauth_version="1.0"|Connection:close|Host:foreman.domain.tld|X-Forwarded-For:192.168.0.10|X-Forwarded-Proto:https|X-TLS-Name:TLS13-CHACHA20-POLY1305-SHA256|X-TLS-Version:TLSv1.3|X-TLS-Bits:256

I am not sure why the Foreman_smartproxy function fails to make the API call to foreman if the oauth keys are the same in both setups (foreman and foreman_proxy)
Expected outcome:

Foreman and Proxy versions:
2.4.1
Foreman and Proxy plugin versions:

Distribution and version:
CentOS 7
Other relevant data:
The puppet modules version are as follow:
mod theforeman/foreman_proxy => 17.1.1
mod theforeman/foreman => 16.1.0

2

Any help to understand what the problem is and be able to solve it is greatly appreciated, I would just also like a good place to start to get more info about why it is not able to authenticate when the oauth values seem to be the same.
Thank you,

3

I have upgraded to Foreman and Foreman proxy 3.0.2 and this issue still persist. What could be the reason where Foreman is not validating the Oauth signature when it is setup to use the oauth key and secret?
I have also tried changing those values to another random alphanumeric 32 char string with the same error message.

4

Does anyone know if there is any param or setting that needs to be changed in the newest foreman versions? Something like allowing auto-registration from Smart proxy?
That would probably explain why puppet is not able to do a get to Foreman while using the same oauth secret and key for the signatures.

I went through the manifests here puppet-foreman/manifests at 18.2.0 · theforeman/puppet-foreman · GitHub but cant find anything that sheds some light as to why this would fail.

This just started failing when I went from foreman 2.3 to 2.4, previous versions was properly working.

I am now sitting on Foreman 3.0.2