Openscap and smart-proxy?

Support
1

Problem:

Openscap module for smart-proxy is not uploading ARF reports to foreman. It looks like the client is uploading to the proxy successfully, but there is a failure in smart-proxy-arf-json. The extended error information back to the client says the certificate verify failed. There’s a stack from proxy.log below. Nothing interesting in the main foreman log.

I have a server cert issued by mycorp.net both for the foreman web UI and to get login information from LDAP. That part works fine.

I left the self-signed puppet cert in place for the proxy. The comments in the proxy settings file say that proxy clients have to use the same CA.

Both CA’s are in the openssl trust store. Trust list shows them as anchor/authority and not blacklisted.

I can see the puppet CA certs through the foreman UI.

Expected outcome:

ARF report uploaded to foreman and visible in the web UI

Foreman and Proxy versions:

Foreman and proxy 1.16

Foreman and Proxy plugin versions:

rubygem-smart_proxy_openscap 0.6.8-1

Other relevant data:
From /var/log/foreman-proxy/proxy.log

D, [2018-07-06T09:36:17.505275 ] DEBUG -- : accept: 10.164.87.48:45330
D, [2018-07-06T09:36:17.506350 ] DEBUG -- : Rack::Handler::WEBrick is invoked.
D, [2018-07-06T09:36:17.522843 ] DEBUG -- : Executing: smart-proxy-arf-json /var/tmp/aa199cf4-9187-44f9-918c-497f27d2e822-testlnv1413.corp.intuit.net-1-1530894977-20180706-1282-1aaxs0g /var/tmp/aa199cf4-9187-44f9-918c-497f27d2e822-testlnv1413.corp.intuit.net-1-1530894977-json-20180706-1282-ote7tc
D, [2018-07-06T09:36:19.127348 ] DEBUG -- : /usr/share/ruby/net/http.rb:921:in `connect'
  /usr/share/ruby/net/http.rb:921:in `block in connect'
  /usr/share/ruby/timeout.rb:52:in `timeout'
  /usr/share/ruby/net/http.rb:921:in `connect'
  /usr/share/ruby/net/http.rb:862:in `do_start'
  /usr/share/ruby/net/http.rb:851:in `start'
  /usr/share/ruby/net/http.rb:1373:in `request'
  /usr/share/gems/gems/smart_proxy_openscap-0.6.8/lib/smart_proxy_openscap/foreman_forwarder.rb:35:in `send_request'
  /usr/share/gems/gems/smart_proxy_openscap-0.6.8/lib/smart_proxy_openscap/foreman_forwarder.rb:9:in `post_arf_report'
  /usr/share/gems/gems/smart_proxy_openscap-0.6.8/lib/smart_proxy_openscap/openscap_api.rb:39:in `block in <class:Api>'
  /usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:1611:in `call'
  /usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:1611:in `block in compile!'
  /usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:975:in `[]'
  /usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:975:in `block (3 levels) in route!'
  /usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:994:in `route_eval'
  /usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:975:in `block (2 levels) in route!'
  /usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:1015:in `block in process_route'
  /usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:1013:in `catch'
  /usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:1013:in `process_route'
  /usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:973:in `block in route!'
  /usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:972:in `each'
  /usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:972:in `route!'
  /usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:1085:in `block in dispatch!'
  /usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:1067:in `block in invoke'
  /usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:1067:in `catch'
  /usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:1067:in `invoke'
  /usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:1082:in `dispatch!'
  /usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:907:in `block in call!'
  /usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:1067:in `block in invoke'
  /usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:1067:in `catch'
  /usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:1067:in `invoke'
 /usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:907:in `call!'
  /usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:895:in `call'
  /usr/share/gems/gems/rack-1.6.4/lib/rack/commonlogger.rb:33:in `call'
  /usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:219:in `call'
  /usr/share/foreman-proxy/lib/proxy/log.rb:109:in `call'
  /usr/share/foreman-proxy/lib/proxy/request_id_middleware.rb:9:in `call'
  /usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/xss_header.rb:18:in `call'
  /usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/path_traversal.rb:16:in `call'
  /usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/json_csrf.rb:18:in `call'
  /usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
  /usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
  /usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/frame_options.rb:31:in `call'
  /usr/share/gems/gems/rack-1.6.4/lib/rack/nulllogger.rb:9:in `call'
  /usr/share/gems/gems/rack-1.6.4/lib/rack/head.rb:13:in `call'
  /usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/show_exceptions.rb:25:in `call'
  /usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:182:in `call'
  /usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:2013:in `call'
  /usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:1487:in `block in call'
  /usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:1787:in `synchronize'
  /usr/share/gems/gems/sinatra-1.4.8/lib/sinatra/base.rb:1487:in `call'
  /usr/share/gems/gems/rack-1.6.4/lib/rack/urlmap.rb:66:in `block in call'
 /usr/share/gems/gems/rack-1.6.4/lib/rack/urlmap.rb:50:in `each'
  /usr/share/gems/gems/rack-1.6.4/lib/rack/urlmap.rb:50:in `call'
  /usr/share/gems/gems/rack-1.6.4/lib/rack/builder.rb:153:in `call'
  /usr/share/gems/gems/rack-1.6.4/lib/rack/handler/webrick.rb:88:in `service'
  /usr/share/ruby/webrick/httpserver.rb:138:in `service'
  /usr/share/ruby/webrick/httpserver.rb:94:in `run'
  /usr/share/ruby/webrick/server.rb:295:in `block in start_thread'
D, [2018-07-06T09:36:19.179894 ] DEBUG -- : close: 10.164.87.48:45330
2

Hi,
if it really is a cert issue, I would expect unauthorized requests in the foreman log. Do you receive facts? If so, then problem is elsewhere as the endpoints for facts and report upload both use the same mechanism to authenticate requests from proxy.

The error about verifying certificate, is it displayed when you run foreman_scap_client $policy_id? Could you show the whole output when you run foreman_scap_client manually?

What scap content and profile do you use to generate report? Is it one of those created by a rake task or do you have a custom one?

3

Okay - so I wound up putting some debug code in the openscap proxy and also running wireshark. I also turned off SSLVerifyClient to none (temporarily). It turned out that the proxy was rejecting the server cert as coming from an unknown CA. Once I fixed that the 500 unknown CA error went away and the proxy_client can upload to the proxy just fine.

Now, however (with the original openscap_proxy), I get:

D, [2018-07-19T13:28:52.872770 ] DEBUG – : Executing: smart-proxy-arf-json /var/tmp/6cf89cdb-dd20-4d58-91a0-659a157ac573-testlnv1412.mycorp.net-1-1532032132-20180719-3785-26sni /var/tmp/6cf89cdb-dd20-4d58-91a0-659a157ac573-testlnv1412.mycorp.net-1-1532032132-json-20180719-3785-1hio9p5
D, [2018-07-19T13:28:54.841522 ] DEBUG – : {
“error”: {“message”:“Access denied”,“details”:“Missing one of the required permissions: create_arf_reports”}
}
E, [2018-07-19T13:28:54.841864 ] ERROR – : Failed to upload to Foreman, saving in spool. Failed with: 403 “Forbidden”
D, [2018-07-19T13:28:54.845832 ] DEBUG – : File /var/spool/foreman-proxy/openscap/arf/testlnv1412.mycorp.net/1/1532032132/d86b5bda7b5e4c6e1757545b5a323b93eed18de01531bbe75725e2874c0b37e2 stored in reports dir.
I, [2018-07-19T13:28:54.846365 ] INFO – : 10.164.87.46 - - [19/Jul/2018:13:28:54 -0700] “POST /compliance/arf/1 HTTP/1.1” 200 - 1.9748

I’m now trying to figure out how to assign the create_arf_reports role to the proxy. Any hints?

4

I think Foreman fails to authorize your proxy using the cert, then it falls back to user access permissions. Because no current user is set for smart proxy requests, it fails with the missing permissions.

You can try switching either ‘Require SSL for smart proxies’ or ‘Restrict registered smart proxies’ setting to ‘false’ in the authentication tab to confirm.

The relevant method tries to get the distinguished name and client cert from request env and verify. The environment variable names are configurable, relevant settings are:

  • SSL client cert env
  • SSL client DN env
  • SSL client verify env

You should find something in the Foreman server logs that would point you to what is going on as the method logs any problem it encounters.

5

Okay I got it working.

I dug through the code and the logs and yeah SSL_CLIENT_S_DN can’t get populated unless SSLVerifyClient is on. If that’s not populated, like you said, Foreman doesn’t know what rights the incoming request has so it just fails.

I turned SSLVerifyClient back on but it kept failing even though I had the right CA file. I turned up mod_ssl logging and it turns out Apache was mad because it couldn’t do the CRL check for mycorp.net and just failing the authorization. It couldn’t do the CRL check because our servers have to use proxies to reach the internet, and I didn’t have proxy configured (we’re discouraged from using it unless absolutely necessary).

So I set SSLCARevocationCheck to none, bounced httpd, and all of a sudden it works like a champ.

Thanks for your help!

1 Like