I did some investigation of this issue. The TLDR of the story is: this is not a problem with Rabl, in fact, this particular API does not use Rabl rendering at all.
The Rabl gem (and Foreman) hook into the rendering process by supplying a template handler to ActionView. The template handling engine looks through available handlers performing a lookup based on the name of the handler and the available file extensions. Since there is not a create.json.rabl present, the rendering engine skips the Rabl handler all together. For example, when hitting api/v2/discovery_rules/393 in the logs you can see the rendering path:
2020-09-30T19:01:34 [I|app|1f6417f1] Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_discovery-16.2.0/app/views/api/v2/discovery_rules/show.json.rabl
OK, so what is happening here?
When a POST is made to api/v2/discovery_rules in order to create a discovery rule, the rendering engine finds no corresponding template handler (i.e. Rabl template) to process and falls back to standard Rails processing methods. That is, the object is converted to_json and then sent back in the response.
Why do we get two different response formats then?
The fallback in Foreman’s API layer is to follow the API V2 standard of not including the root object type in the response by disabling this configuration. This disable is done per response, not application wide.
Conclusion
The move to Puma in a multi-thread deployment exposed a place in the Foreman code (that is over 6 years old) where per response we are attempting to manipulate a class level attribute which is not thread safe. You can see from this output where the processing starts to get out of order setting and re-setting this:
Sep 30 18:48:42 centos7-foreman-nightly.war.example.com foreman[1524]: DISABLE JSON ROOT
Sep 30 18:48:42 centos7-foreman-nightly.war.example.com foreman[1524]: PROCESS RESPONSE
Sep 30 18:48:42 centos7-foreman-nightly.war.example.com foreman[1524]: RESPONSE: 223177500
Sep 30 18:48:42 centos7-foreman-nightly.war.example.com foreman[1524]: RE-ENABLE JSON ROOT
Sep 30 18:48:43 centos7-foreman-nightly.war.example.com foreman[1524]: DISABLE JSON ROOT
Sep 30 18:48:43 centos7-foreman-nightly.war.example.com foreman[1524]: DISABLE JSON ROOT
Sep 30 18:48:43 centos7-foreman-nightly.war.example.com foreman[1524]: PROCESS RESPONSE
Sep 30 18:48:43 centos7-foreman-nightly.war.example.com foreman[1524]: RESPONSE: 292601022
Sep 30 18:48:43 centos7-foreman-nightly.war.example.com foreman[1524]: RE-ENABLE JSON ROOT
Sep 30 18:48:43 centos7-foreman-nightly.war.example.com foreman[1524]: PROCESS RESPONSE
Sep 30 18:48:43 centos7-foreman-nightly.war.example.com foreman[1524]: RESPONSE: 184514629
Sep 30 18:48:43 centos7-foreman-nightly.war.example.com foreman[1524]: RE-ENABLE JSON ROOT
Sep 30 18:48:43 centos7-foreman-nightly.war.example.com foreman[1524]: DISABLE JSON ROOT
Sep 30 18:48:43 centos7-foreman-nightly.war.example.com foreman[1524]: PROCESS RESPONSE
Sep 30 18:48:43 centos7-foreman-nightly.war.example.com foreman[1524]: RESPONSE: 1101731735
Sep 30 18:48:43 centos7-foreman-nightly.war.example.com foreman[1524]: RE-ENABLE JSON ROOT
Sep 30 18:48:43 centos7-foreman-nightly.war.example.com foreman[1524]: DISABLE JSON ROOT
Sep 30 18:48:43 centos7-foreman-nightly.war.example.com foreman[1524]: PROCESS RESPONSE
Sep 30 18:48:43 centos7-foreman-nightly.war.example.com foreman[1524]: RESPONSE: 2178710316
Sep 30 18:48:43 centos7-foreman-nightly.war.example.com foreman[1524]: RE-ENABLE JSON ROOT
Sep 30 18:48:43 centos7-foreman-nightly.war.example.com foreman[1524]: DISABLE JSON ROOT
I have proposed the following fixes: