Generation of cached API doc and dynamic content types

Background

In Foreman and every plugin with an API uses apipie-rails. This describes the API (locations, methods, parameters). It can also generate human readable documentation that’s served on /apidoc and what we also provide on apidocs.theforeman.org.

To do this, Foreman generates apidocs and ships them in its package. The same is done for every plugin.

Then after installation the indexes are regenerated (using the apipie:cache:index rake task). This is because there’s an index page with all endpoints that can only be assembled once you know which plugins are present. There is also a JSON file which has the complete description, which again can only be generated with all plugins present.

This is also the reason why it’s generally recommended to visit /apidoc on your own instance rather then our apidocs: they document your installed plugins.

The problem

Katello behaves differently depending on which content types are present. This means some parts are only visible if Pulp is installed with Debian or OSTree support. For the UI this works well, because that’s generated dynamically anyway.

The problem appears in the API documentation. During package building it’s not known yet which content types are available. That means it’s incomplete.

An example is always clearer:

This means RepositoryTypeManager.removable_content_types(false).map(&:label) is executed during packaging and only the types available during packaging are visible by default. There may be more examples.

In https://github.com/theforeman/puppet-foreman_proxy_content/pull/393 we at least set up some events that when a content type is added the index is refreshed, but realistically this isn’t going to cover everything.

Solutions

Generate with all options by default in Katello

It is possible to make Katello generate all options. This could be achieved by pretending all types are present during packaging. This would generate complete API documentation, but it may be too broad.

This could be seen as a workaround.

Rebuild API documentation on the target system

It is possible to run the apipie:cache (which is what packaging does). This is slow: on my machine with Katello it took about 50 seconds per language plus some other common overhead. It would also need to be done after every Foreman/plugin package installation which makes it very expensive.

Not a solution I’d like to see due to the cost for end user systems

Disable apipie cache

Essentially this is like a caching problem. apipie-rails can generate the apidocs on the fly. This is what happens in development. It’s technically a bit slower, but given how little the apidocs are used it’s probably an acceptable overhead.

Perhaps the JSON view (as used by Hammer, FAM and other apipie clients) could be cached by Foreman using its regular caching system (file based by default, possibly memcache or Redis).

This would also have faster package builds and smaller packages (thus faster installation) as an advantage.

Rewrite API docs to use the JSON document as a source

We already generate a JSON document that describes the whole API. Perhaps a Javascript based UI could be built instead of using ERB templates. That would remove the need to ship so many files. It places the cost of rendering on the user’s browser.

This sounds like a costly rewrite that would take time.

Conclusion

I’m leaning to investigate option 3 (disabling the cache) but perhaps I’m missing another solution. I’ll readily admit I’m not too familiar with it.

I tested this on my production system by disabling the cache, and could not tell the difference in speed of page rendering the API docs from cached to not cached. My vote would be to drop caching all together.

Not only would dropping caching solve the inherent problem but it would, as noted, speed up builds by a significant amount. In some cases, cache generation can take ~10-15 minutes of build time. Further this would also reduce complexity, bring parity to environments and generally make the operations we do a lot (build and install) faster for developers and users.

My test process:

vim config/initializers/apipie.rb 

Then update line 42:

  config.use_cache = false #Rails.env.production? || File.directory?(config.cache_dir)

And for good measure delete cache on disk and restart services:

rm -rf /usr/share/foreman/public/apipie-cache
systemctl restart foreman

This made me remember Bug #33956: serve assets directly via Apache, not via Puma/Rails - Installer - Foreman. We’re already serving the request via Puma now rather than directly via Apache (which would be faster). Perhaps that’s why you hardly notice a difference.

Currently there is code in Foreman that tries to detect that the cache is stale and tells users to regenerate it. This should be removed if we decide to not cache anymore.