Latest develop branch is broken on production

I just opened Bug #23842: unable to run rake tasks in production - Foreman, this disallow running any rake task in production installs (as it missed the test dependencies).

I am not so much worried about this particular line (someone will send a pr i guess) but very worried how this did not trigger any alters what so ever, i find it hard to believe we have no single test that does rake db:migrate in production?

What would you suggest we change, so such trivial, yet so destructive failures don’t repeat themselves?

Our pipeline did run rake db:migrate and rake db:seed because the installer always does that. I’m wondering why that worked because we do rely on a working database in the tests we run.

A possible explanation is that the actual offending code hadn’t made it to RPM nightlies because test_develop was red. We build nightly RPMs from the latest green build. Without more info in the bug report (like what command was run, which software versions were present, what platform) it’s really hard to say something useful about this. I have created a PR to fix the current issue:

https://github.com/theforeman/foreman/pull/5661

The RPM build has been failing due to this which is why it’s not seen in production nightly:

http://koji.katello.org/kojifiles/work/tasks/2922/102922/build.log