Random test failures - fact values API controller

Hey,

we are suffering from some random errors on Jenkins, this is blocking core from merging PRs:

SeedsTest.test_0006_doesn't add a template back that was renamed
SeedsTest.test_0005_doesn't add a template back that was deleted

These I was able hopefully fix:

https://github.com/theforeman/foreman/pull/6058

Api::V2::FactValuesControllerTest.test_0007_should search facts by organiztion
Api::V2::FactValuesControllerTest.test_0006_should search facts by location

I spent some time trying to figure these out, but I am giving up. This is lot of scoped_search code. I think @tbrisker tried to fix these, it is probably related:

https://github.com/theforeman/foreman/pull/6013

Ideas?