Katello 3.14.0-4/Foreman 1.24.0-15: new install problem

On a completely clean install (CentOS7, fully patched,) the pulp-admin backend fails:

$ pulp-admin status
±---------------------------------------------------------------------+
Status of the server
±---------------------------------------------------------------------+

There was an internal server error while trying to access the Pulp application.
One possible cause is that the database needs to be migrated to the latest
version. If this is the case, run pulp-manage-db and restart the services. More
information may be found in Apache’s log.

What I should see:
$ pulp-admin status
±---------------------------------------------------------------------+
Status of the server
±---------------------------------------------------------------------+

Api Version: 2
Database Connection:
Connected: True
Known Workers:
_id: scheduler@ahdcos019.ent.lolcentral.com
_ns: workers
Last Heartbeat: 2020-01-09T16:51:51Z
_id: reserved_resource_worker-2@ahdcos019.ent.lolcentral.com
_ns: workers
Last Heartbeat: 2020-01-09T16:51:52Z
_id: reserved_resource_worker-0@ahdcos019.ent.lolcentral.com
_ns: workers
Last Heartbeat: 2020-01-09T16:51:52Z
_id: reserved_resource_worker-1@ahdcos019.ent.lolcentral.com
_ns: workers
Last Heartbeat: 2020-01-09T16:51:52Z
_id: reserved_resource_worker-3@ahdcos019.ent.lolcentral.com
_ns: workers
Last Heartbeat: 2020-01-09T16:51:52Z
_id: resource_manager@ahdcos019.ent.lolcentral.com
_ns: workers
Last Heartbeat: 2020-01-09T16:51:52Z
Messaging Connection:
Connected: True
Versions:
Platform Version: 2.21

Katello 3.14.0-4
Foreman 1.24.0-15
CentOS 7, fully patched

I’ve been struggling with this for a couple of days, after I decided to completely burn down and rebuild my provisioning server, with a clean install of Katello. It finally dawned on me to dig down into the underlying services as I’d been having significant issues with pulp failures. I finally decided on a long shot to run ‘foreman-installer --scenario katello --upgrade’ which appears to have solved the issue.

Has anyone else experienced this? Or can reproduce it? I would not be surprised in the least if it was something I did.

"More information may be found in Apache’s log."

Was there anything in there? Anything in foreman/production.log?

I think the real nut of the problem is what ‘pulp-admin status’ said: that the Pulp database schema version wasn’t right. Simply running the katello upgrade from foreman-installer seems to have fixed it.

I have a parallel version of this at home, I’ll nuke it from orbit in the same way, and see if I can reproduce it.

Finally managed to nuke my trial system from orbit and start from scratch. Either something in the online repositories has changed, or (more likely,) there was something left dangling in my office PoC.

Sorry for the delayed response, good to know things are working now, let us know if you have any more trouble!