Cannot login to foreman 1.16.0/Katello 3.5

I successfully upgraded Katello from 3.4 to 3.5 and was able to login. Now 2 weeks latter I am trying to login to the foreman gui and cannot. I get the splash page. But after I put in my login and hit enter it just spins forever.

I restarted and rebooted the server to clear everything up. No good. In looking around I found this error in the tomcat localhost.log file:

Mar 20, 2018 10:00:00 AM org.apache.catalina.core.StandardContext listenerStop
SEVERE: Exception sending context destroyed event to listener instance of class org.candlepin.guice.CandlepinContextListener
java.lang.NullPointerException

Not sure what that means. I am seeing this a lot in the catalina.log file:

WARNING: Method [public void org.candlepin.model.EntitlementCertificateCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@55b455c0]. This could indicate a bug. The method may be intercepted twice, or may not be intercepted at all.

In the httpd directory I am seeing this in the foreman-ssl_error_ssl.log file:

[Tue Mar 20 10:06:53.070472 2018] [ssl:warn] [pid 4305] [client 10.25.4.41:60226] AH02227: Failed to set r->user to ‘SSL_CLIENT_S_DN_CN’

Anyone know whats going on here???

As I said after I upgraded I had no issues logging in to the GUI and no one has done anything to the system except add new content hosts.

Louis

if you open your browser console, do you see any error?

I had totally forgotten about the console so I pulled up Chrome and tried it. Nothing showed up in the console.

on a hunch (big guess) I tried some foreman clean up scripts:

foreman-rake foreman_tasks:cleanup TASK_SEARCH=‘label=“Actions::Katello::Repository::Sync”’ STATES="paused"
foreman-rake katello:delete_orphaned_content --trace
foreman-rake foreman_tasks:cleanup
foreman-rake katello:reimport --trace

Once they ran I was able to login… Very odd.