Remote execution failing with 403 Forbidden

Support
,
1

Problem:
Remote execution failing with 403 Forbidden.

Rails logs-

2020-06-16T19:43:21 [I|app|060d826b] Processing by ForemanTasks::Api::TasksController#callback as /
2020-06-16T19:43:21 [I|app|060d826b] Parameters: {“callback”=>{“task_id”=>“a693e81e-d8db-4938-806f-50b4daae5414”, “step_id”=>3}, “data”=>{“result”=>[{“output_type”=>“stdout”, “output”=>“Wed Jun 17 01:13:19 IST 2020\r\n”, “timestamp”=>1592336601.2842703}], “runner_id”=>“7e99580e-6bd9-486e-bf25-ae2d56b5deb4”, “exit_status”=>0}, “task”=>{}}
2020-06-16T19:43:21 [W|app|060d826b] No SSL cert with CN supplied - request from 192.168.121.236
2020-06-16T19:43:21 [I|app|060d826b] Rendering api/v2/errors/access_denied.json.rabl within api/v2/layouts/error_layout
2020-06-16T19:43:21 [I|app|060d826b] Rendered api/v2/errors/access_denied.json.rabl within api/v2/layouts/error_layout (Duration: 1.7ms | Allocations: 323)
2020-06-16T19:43:21 [I|app|060d826b] Filter chain halted as #Proc:0x00007fe269326f10@/home/vagrant/foreman/app/controllers/concerns/foreman/controller/smart_proxy_auth.rb:14 rendered or redirected
2020-06-16T19:43:21 [I|app|060d826b] Completed 403 Forbidden in 28ms (Views: 8.1ms | ActiveRecord: 5.5ms | Allocations: 3533)

It was found that smart_proxy_dynflow_core is sending the proper request to the foreman. I have edited file /opt/theforeman/tfm/root/usr/share/gems/gems/smart_proxy_dynflow_core-0.2.5/lib/smart_proxy_dynflow_core/callback.rb and inserted few Log statements on line 61.

Run SmartProxyDynflowCore::Callback::Action==================================SSL option - {:verify_ssl=>1, :ssl_client_key=>#OpenSSL::PKey::RSA:0x00007fa300061670, :ssl_client_cert=>#<OpenSSL::X509::Certificate: subject=#<OpenSSL::X509::Name CN=centos7-katello-devel.localhost.example.com,OU=FOREMAN_PROXY,O=FOREMAN,ST=North Carolina,C=US>, issuer=#<OpenSSL::X509::Name CN=centos7-katello-devel.localhost.example.com,OU=SomeOrgUnit,O=Katello,L=Raleigh,ST=North Carolina,C=US>, serial=#OpenSSL::BN:0x00007fa300061418, not_before=2020-05-31 10:07:01 UTC, not_after=2038-01-18 10:07:02 UTC>, :ssl_ca_file=>"/etc/foreman-proxy/foreman_ssl_ca.pem"}Foreman URL - https://centos7-katello-devel.localhost.example.com
403 Forbidden (RestClient::Forbidden)
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/abstract_response.rb:223:in exception_with_response' /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/abstract_response.rb:103:in return!’

I have also added “centos7-katello-devel.localhost.example.com” in Settings > Authentication > Trusted hosts.

foreman is failing in auth_smart_proxy - https://github.com/theforeman/foreman/blob/bcee8f4f624650d8ee2c89feef8acb42af51696f/app/controllers/concerns/foreman/controller/smart_proxy_auth.rb#L79

After further debugging, it was found that the issue is with the way foreman/rails is parsing the request.

auth_smart_proxy looks from request.env[“SSL_CLIENT_CERT”] but the request contains “HTTP_SSL_CLIENT_CERT”.
Same is the case which ssl_client_dn_env foreman looks for request.env[“SSL_CLIENT_S_DN”] but the request contains “HTTP_SSL_CLIENT_S_DN”.

Why is “HTTP_” appended? What is responsible for this? Is this due to any change in HTTP library or any gem?

2

Hi, nice work tracking it down so far.
All the SSL_* headers should be set by httpd when it proxies the request to the application server, not sure where the HTTP_ prefix comes from. What version is this? I haven’t seen this before

3

@aruzicka Thanks for pointing that httpd is responsible for setting those headers. I will continue debugging in that direction.

Foreman version - 2.2.0-develop I might have pull the develop branch a week ago or so.
foreman_remote_execution - latest master branch.
foreman-proxy - 2.2.0-0.2.develop.20200514124310gite138e23.el7.noarch
smart_proxy_dynflow_core - tfm-rubygem-smart_proxy_dynflow_core0.2.5-1.fm2_1.el7.noarch

I have seen the below-related issue. But didn’t found anything that will help.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1596636
[2] No SSL cert with CN supplied - request from xxxxxxx

4

In my dev environment below RequestHeader were set in /etc/httpd/conf.d/05-foreman.conf and /etc/httpd/conf.d/05-foreman-ssl.conf.

In /etc/httpd/conf.d/05-foreman.conf -

##Request header rules
##as per http://httpd.apache.org/docs/2.2/mod/mod_headers.html#requestheader
RequestHeader set X_FORWARDED_PROTO “https”
RequestHeader set SSL_CLIENT_S_DN “%{SSL_CLIENT_S_DN}s”
RequestHeader set SSL_CLIENT_CERT “%{SSL_CLIENT_CERT}s”
RequestHeader set SSL_CLIENT_VERIFY “%{SSL_CLIENT_VERIFY}s”

In /etc/httpd/conf.d/05-foreman-ssl.conf -

##Request header rules
##as per http://httpd.apache.org/docs/2.2/mod/mod_headers.html#requestheader
RequestHeader set X_FORWARDED_PROTO “http”
RequestHeader set SSL_CLIENT_S_DN “”
RequestHeader set SSL_CLIENT_CERT “”
RequestHeader set SSL_CLIENT_VERIFY “”

After removing this RequestHeader remote execution was successful. I also didn’t find these RequestHeader in the downstream satellite.
If RequestHeader are purposely added then we might need to change the default value to HTTP_* in settings

5

Looks like the katello-devel scenario isn’t setting things up exactly right