Proposal: Move apidocs from git to infra web

Hey,

we currently generate and carry Foreman apipie docs in our theforeman.org github repo. These are autogenerated HTML pages and there is no reason to have them in the git repo - once these are generated we never change them. The api directory is currently sized 672 MB, we recently dropped some old versions:

https://theforeman.org/api/

There is some value in archiving those API documents however, people use really old Foreman versions and it might be handy to have them online. Also with the foreman new generation documentation which is hosted on a brand new site docs.theforeman.org we need to move apipie docs as well.

So here is my proposal: let’s move all current apipie documents from our main site to one of infra subdomains or we can create a dedicated domain too. Then we will update links and create redirect document or httpd rules so old links are kept working. Finally we set up some process how to upload new documentation with coordination from the infra team.

We would start linking apipie docs from the new documentation as well. BIg advantage is that we can afford not to delete old versions, there should be plenty of space on our infra, we can even hardlink same files which many of them are not changing for years (via fsdupe utility or similar tools).

Opinions? @ehelms @ekohl @evgeni @jjeffers @mcorr

3 Likes

Hmmm I am trying to find Katello apidocs and Google got me here:

https://theforeman.org/plugins/katello/3.5/api/apidoc/v2/products.html

Anyway, the proposal would also include Katello. Eithert as a separate directory on the new site or maybe we can find a way to merge them together.

Katello isn’t the only plugin we have – just the only one that publishes own API docs today.
I’d love to see something like https://api.theforeman.org/<type>/<version>/ where type is foreman, katello, foreman_ansible etc, and version is well, the version.

3 Likes

A nice side effect of removing the apidocs from the main website is that generating and deploying it would be much faster.

1 Like

This is exactly what I am going after.

I volunteer to do the work, however I will need assistance from the infra team to set me up (I do have ssh access so I can copy once things are set up) and also I need some info on how to create the docs. I mean, I know there is the rake task, shall I use production installation, dev installation? What is the process? Who actually create those? Maybe we can automate this and add more plugins.

Heretical question: Do we need the apidoc or would it be better to tell users to query there own installation for the apidoc? This would always give them the matching version including all plugins.

1 Like

Not heretical, but it is nice to be able to look things up without a server for discussions, investigations or doing development work on something like our Ansible modules.

:+1: for api.theforeman.org or apidoc.theforeman.org as @evgeni described. I do wonder if we should consider storing it in git. I have used it in the past to check if examples rendered ok. However, it can be split from our main repo and just be plain text files without Jekyll.

I think I’ve used it in the past to link it to people. Their own installation is always better but if you’re writing a script or helping a user then it can be easy to have older versions easily available.

I like to link to the docs in PRs or Discussions, which is rather hard to local installs :wink:

docs generation is essentially rake plugin:apipie:cache[$PLUGIN] right? I think we can trivially build a Job that given a plugin name, plugin version (and foreman version, for compatibility) can execute that and store the result in a git repo that will be used to deploy the api-site.

then plugin authors just need to be able to trigger said job once they release a new version.

2 Likes

If you also run the tests, it can record examples from that. Today many of our examples are bad and it may be better to use a different method to build examples, but that’s an implementation detail.

Yes, this sounds good. I’d say that ideally the job would create a PR that’s still manually merged, but generation should be automated.

1 Like

Given the size of the diffs, I don’t think they are reviewable.

That really depends. I’ve certainly done it on theforeman.org in the past. Usually reviewing a small number of files is sufficient to check if the whole generation went well. A common thing was that on a rerun it would remove all examples if someone forgot to record tests. On updates in a stable branch, a diff should also be small and if it’s big, that’s a signal on its own that something went wrong.

That all sounds nice, I would love to start moving the current content into the new subdomain and if anyone can pick it up from there and improve the actual build process that would be fantastic.

I guess we need a puppet codebase change to create a new vhost and DNS change. I can attempt to do the former if you guys guide me. I am trying to find “docs.theforeman.org” domain in our puppet codebase but I cannot find it. I do see “downloads.theforeman.org” there, weird. Am I in a wrong place?

Do I simply create a copy of that pp file and include it somewhere? I am really bad in Puppet so I will be shooting in the dark here. :slight_smile:

My suggestion would be to create a new git repo (I suggest apidoc). The use the Github Pages feature to publish it. If you create a file .nojekyll in the root, no Jekyll processing will happen. Then just submit PRs to it. We can then set up the right subdomain just like we did with foreman-documentation.

1 Like

docs.tfm is hosted on GH pages, not on our infra, so that’s why it doesn’t have any Puppet.

I like Ewouds idea.

1 Like

Yeah that is probably the easiest thing to do. @tbrisker can I ask for creation of another repo? :slight_smile: foreman-apidoc perhaps to keep it short

Created:

I added the website and documentation teams with write access and made you the admin of the repo. Enjoy!

In the project, should we use gh-pages as the default (and only) branch? This is the convention Github uses for their Github Pages and also what we use in GitHub - theforeman/theforeman.org: The new and improved Foreman website.

1 Like

Sure, what was the other option then?