We upgraded our Redhat IDM from RHEL 8.5 to RHEL 8.6, since that we have those errors and can’t create new vm’s via foreman
Problem:
2022-07-07T16:34:35 7f97fc86 [I] Started POST /realm/EXAMPLE.COM/
2022-07-07T16:34:35 7f97fc86 [D] require_ssl_client_verification: skipping, non-HTTPS request
2022-07-07T16:34:35 7f97fc86 [D] freeipa: realm EXAMPLE.COM
2022-07-07T16:34:35 7f97fc86 [D] freeipa: uri is https://ipa-ihs-prod-r83.lx.example.comt/ipa/xml
2022-07-07T16:34:35 7f97fc86 [D] Making IPA call: ["host_show", ["edgar-alwine.lx.example.com"]]
2022-07-07T16:34:35 7f97fc86 [D] Requesting credentials for Kerberos principal realm-proxy@EXAMPLE.COM using keytab /etc/foreman-proxy/freeipa.keytab
2022-07-07T16:34:35 7f97fc86 [D] Kerberos credential cache initialised with principal: realm-proxy@EXAMPLE.COM
2022-07-07T16:34:36 7f97fc86 [D] Making IPA call: ["host_add", ["edgar-alwine.lx.example.com"], {:random=>1, :force=>1, :setattr=>["userclass=\u00D6AMTC Lx/Ubuntu Focal/IHS-dev-focal"]}]
2022-07-07T16:34:36 7f97fc86 [E] Authorization failed.
HTTP-Error: 401 Unauthorized
2022-07-07T16:34:36 7f97fc86 [W] Error details for Authorization failed.
HTTP-Error: 401 Unauthorized: <RuntimeError>: Authorization failed.
HTTP-Error: 401 Unauthorized
/usr/lib/ruby/vendor_ruby/xmlrpc/client.rb:507:in `do_rpc'
/usr/lib/ruby/vendor_ruby/xmlrpc/client.rb:287:in `call2'
/usr/lib/ruby/vendor_ruby/xmlrpc/client.rb:268:in `call'
/usr/share/foreman-proxy/modules/realm_freeipa/provider.rb:152:in `ipa_call'
/usr/share/foreman-proxy/modules/realm_freeipa/provider.rb:110:in `do_host_create'
/usr/share/foreman-proxy/modules/realm_freeipa/provider.rb:77:in `create'
/usr/share/foreman-proxy/modules/realm/realm_api.rb:13:in `block in <class:Api>'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1636:in `call'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1636:in `block in compile!'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:987:in `block (3 levels) in route!'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1006:in `route_eval'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:987:in `block (2 levels) in route!'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1035:in `block in process_route'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1033:in `catch'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1033:in `process_route'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:985:in `block in route!'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:984:in `each'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:984:in `route!'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1098:in `block in dispatch!'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1072:in `block in invoke'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1072:in `catch'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1072:in `invoke'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1095:in `dispatch!'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:919:in `block in call!'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1072:in `block in invoke'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1072:in `catch'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1072:in `invoke'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:919:in `call!'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:908:in `call'
/usr/share/foreman-proxy/lib/proxy/log.rb:105:in `call'
/usr/share/foreman-proxy/lib/proxy/request_id_middleware.rb:11:in `call'
/usr/lib/ruby/vendor_ruby/rack/protection/xss_header.rb:18:in `call'
/usr/lib/ruby/vendor_ruby/rack/protection/path_traversal.rb:16:in `call'
/usr/lib/ruby/vendor_ruby/rack/protection/json_csrf.rb:26:in `call'
/usr/lib/ruby/vendor_ruby/rack/protection/base.rb:50:in `call'
/usr/lib/ruby/vendor_ruby/rack/protection/base.rb:50:in `call'
/usr/lib/ruby/vendor_ruby/rack/protection/frame_options.rb:31:in `call'
/usr/lib/ruby/vendor_ruby/rack/null_logger.rb:9:in `call'
/usr/lib/ruby/vendor_ruby/rack/head.rb:12:in `call'
/usr/lib/ruby/vendor_ruby/sinatra/show_exceptions.rb:22:in `call'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:194:in `call'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1951:in `call'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1503:in `block in call'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1730:in `synchronize'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1503:in `call'
/usr/lib/ruby/vendor_ruby/rack/urlmap.rb:68:in `block in call'
/usr/lib/ruby/vendor_ruby/rack/urlmap.rb:53:in `each'
/usr/lib/ruby/vendor_ruby/rack/urlmap.rb:53:in `call'
/usr/lib/ruby/vendor_ruby/rack/builder.rb:153:in `call'
/usr/lib/ruby/vendor_ruby/rack/handler/webrick.rb:86:in `service'
/usr/lib/ruby/2.7.0/webrick/httpserver.rb:140:in `service'
/usr/lib/ruby/2.7.0/webrick/httpserver.rb:96:in `run'
/usr/lib/ruby/2.7.0/webrick/server.rb:307:in `block in start_thread'
/usr/lib/ruby/vendor_ruby/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2022-07-07T16:34:36 7f97fc86 [W] Authorization failed.
HTTP-Error: 401 Unauthorized: <RuntimeError>: Authorization failed.
HTTP-Error: 401 Unauthorized
/usr/lib/ruby/vendor_ruby/xmlrpc/client.rb:507:in `do_rpc'
/usr/lib/ruby/vendor_ruby/xmlrpc/client.rb:287:in `call2'
/usr/lib/ruby/vendor_ruby/xmlrpc/client.rb:268:in `call'
/usr/share/foreman-proxy/modules/realm_freeipa/provider.rb:152:in `ipa_call'
/usr/share/foreman-proxy/modules/realm_freeipa/provider.rb:110:in `do_host_create'
/usr/share/foreman-proxy/modules/realm_freeipa/provider.rb:77:in `create'
/usr/share/foreman-proxy/modules/realm/realm_api.rb:13:in `block in <class:Api>'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1636:in `call'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1636:in `block in compile!'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:987:in `block (3 levels) in route!'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1006:in `route_eval'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:987:in `block (2 levels) in route!'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1035:in `block in process_route'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1033:in `catch'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1033:in `process_route'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:985:in `block in route!'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:984:in `each'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:984:in `route!'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1098:in `block in dispatch!'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1072:in `block in invoke'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1072:in `catch'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1072:in `invoke'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1095:in `dispatch!'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:919:in `block in call!'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1072:in `block in invoke'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1072:in `catch'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1072:in `invoke'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:919:in `call!'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:908:in `call'
/usr/share/foreman-proxy/lib/proxy/log.rb:105:in `call'
/usr/share/foreman-proxy/lib/proxy/request_id_middleware.rb:11:in `call'
/usr/lib/ruby/vendor_ruby/rack/protection/xss_header.rb:18:in `call'
/usr/lib/ruby/vendor_ruby/rack/protection/path_traversal.rb:16:in `call'
/usr/lib/ruby/vendor_ruby/rack/protection/json_csrf.rb:26:in `call'
/usr/lib/ruby/vendor_ruby/rack/protection/base.rb:50:in `call'
/usr/lib/ruby/vendor_ruby/rack/protection/base.rb:50:in `call'
/usr/lib/ruby/vendor_ruby/rack/protection/frame_options.rb:31:in `call'
/usr/lib/ruby/vendor_ruby/rack/null_logger.rb:9:in `call'
/usr/lib/ruby/vendor_ruby/rack/head.rb:12:in `call'
/usr/lib/ruby/vendor_ruby/sinatra/show_exceptions.rb:22:in `call'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:194:in `call'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1951:in `call'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1503:in `block in call'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1730:in `synchronize'
/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1503:in `call'
/usr/lib/ruby/vendor_ruby/rack/urlmap.rb:68:in `block in call'
/usr/lib/ruby/vendor_ruby/rack/urlmap.rb:53:in `each'
/usr/lib/ruby/vendor_ruby/rack/urlmap.rb:53:in `call'
/usr/lib/ruby/vendor_ruby/rack/builder.rb:153:in `call'
/usr/lib/ruby/vendor_ruby/rack/handler/webrick.rb:86:in `service'
/usr/lib/ruby/2.7.0/webrick/httpserver.rb:140:in `service'
/usr/lib/ruby/2.7.0/webrick/httpserver.rb:96:in `run'
/usr/lib/ruby/2.7.0/webrick/server.rb:307:in `block in start_thread'
/usr/lib/ruby/vendor_ruby/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2022-07-07T16:34:36 7f97fc86 [I] Finished POST /realm/EXAMPLE.COM/ with 400 (187.5 ms)
Logs on Ipa side:
==> /var/log/httpd/access_log <==
10.203.0.22 - - [07/Jul/2022:14:48:07 +0000] "GET /ipa/session/cookie HTTP/1.1" 301 264 "-" "python-requests/2.20.0"
10.203.0.22 - realm-proxy@EXAMPLE.COM [07/Jul/2022:14:48:07 +0000] "GET /ipa/session/cookie HTTP/1.1" 200 -
==> /var/log/httpd/ssl_request_log <==
[07/Jul/2022:14:48:07 +0000] 10.203.0.22 TLSv1.3 TLS_AES_256_GCM_SHA384 "GET /ipa/session/cookie HTTP/1.1" -
==> /var/log/httpd/access_log <==
10.20.30.22 - realm-proxy@EXAMPLE.COM [07/Jul/2022:14:48:07 +0000] "POST /ipa/session/login_kerberos HTTP/1.1" 200 20
==> /var/log/httpd/ssl_request_log <==
[07/Jul/2022:14:48:07 +0000] 10.20.30.22 TLSv1.3 TLS_AES_256_GCM_SHA384 "POST /ipa/session/login_kerberos HTTP/1.1" 20
==> /var/log/httpd/error_log <==
[Thu Jul 07 14:48:07.165270 2022] [wsgi:error] [pid 63187:tid 140392687339264] [remote 10.20.30.22:45150] ipa: INFO: [xmlserver_session] realm-proxy@EXAMPLE.COM: host_show('edgar-alwine.lx.example.com', version='2.51'): NotFound
==> /var/log/httpd/access_log <==
10.20.30.22 - realm-proxy@EXAMPLE.COM[07/Jul/2022:14:48:07 +0000] "POST /ipa/session/xml HTTP/1.1" 200 209
==> /var/log/httpd/ssl_request_log <==
[07/Jul/2022:14:48:07 +0000] 10.20.30.22 TLSv1.3 TLS_AES_256_GCM_SHA384 "POST /ipa/session/xml HTTP/1.1" 209
==> /var/log/httpd/access_log <==
10.20.30.22 - - [07/Jul/2022:14:48:07 +0000] "POST /ipa/session/xml HTTP/1.1" 401 3210
==> /var/log/httpd/ssl_request_log <==
[07/Jul/2022:14:48:07 +0000] 10.20.30.22 TLSv1.3 TLS_AES_256_GCM_SHA384 "POST /ipa/session/xml HTTP/1.1" 3210
Config:
cat /etc/foreman-proxy/settings.d/realm.yml
---
:enabled: http
:use_provider: realm_freeipa
# Unparsed options, please review
:use_provider: realm_freeipa
cat /etc/foreman-proxy/settings.d/realm_freeipa.yml
---
# Authentication for Kerberos-based Realms
:keytab_path: /etc/foreman-proxy/freeipa.keytab
:principal: realm-proxy@EXAMPLE.COM
:ipa_config: /etc/ipa/default.conf
# Remove from DNS when deleting the FreeIPA entry
:remove_dns: true
# verify IPA API HTTPS server certificate
:verify_ca: true
what we already tried (a lot of times)
foreman-prepare-realm admin realm-proxy
[...]
Keytab successfully retrieved and stored in: freeipa.keytab
Realm Proxy User: realm-proxy
Realm Proxy Keytab: /root/freeipa.keytab
chown foreman-proxy /etc/foreman-proxy/freeipa.keytab
chmod 600 /etc/foreman-proxy/freeipa.keytab
ipa-getkeytab -s ipa-ihs-prod-r83.lx.example.com -p realm-proxy@EXAMPLE.COM -k /tmp/freeipa.keytab
Failed to load translations
Keytab successfully retrieved and stored in: /tmp/freeipa.keytab
chown foreman-proxy /etc/foreman-proxy/freeipa.keytab
chmod 600 /etc/foreman-proxy/freeipa.keytab
create a new user with foreman-prepare-realm and use it with his keytab
Foreman and Proxy versions:
Foreman: 3.3.0
Ipa Client: 4.8.6
Ipa Server: 4.9.8
Distribution and version:
Ubuntu 20.04