Currently we can only run hammer tests against the API version specified in the test_helper.rb. This points to an apipie cache of the current release, but other past releases still reside in the hammer repository. It would be great to be able to test a hammer version against multiple versions of the API to determine if a specific hammer release works with certain Foreman releases.
Note: This also applies to other hammer plugins, such as hammer-cli-katello.
Slightly off topic, or maybe a new but related RFC: Once we are able to test against multiple major versions of Foreman/Katello, hammer should know which versions of the API it supports, and if users try to use a mismatched version of hammer/Foreman, hammer would give a message such as “API version mismatch: This version of hammer only works with Foreman 1.16 and 1.17 and Katello 3.5 and 3.6”.
I have helped a handful of confused users debug their hammer issues only to find that they were using the most recently released hammer with Foreman 1.14 or Katello 3.2. This change would prevent such situations from happening in the future, and it would help us determine if a version of hammer can be used with older versions of Foreman.
Do you mean that we don’t do this in our current CI job but all the code is there and suggest to modify the CI job? I’d consider it a bug in our current CI if we don’t do this
I’d say that technically, all the code is not there. The CI just runs the tests without knowing anything about API version. I’d like to keep it that way and have the test_helper run the tests multiple times for each API version.
I like the idea!
Pretty sure offtopic: wouldn’t it be possible to create the hammer cli automatically on-the-fly based on the API which is available on the foreman host?
But if we define a matrix and run the tests with TEST_API_VERSION=1.14 =1.15 =1.16 then we’d get the correct result? If hardcoding it for now works, then I’d be happy to do that.
Are you thinking about how in some cases if you use swagger, for example, a binding or CLI can be auto-generated from it? I am a bit curious what the potential GraphQL API will do for CLI design.
I think, when we look at CI, having the ability to break things down by API version can be useful. If you were to think of being able to look at a CI job and easily see the results for each API version (e.g. http://ci.theforeman.org/blue/organizations/jenkins/katello-pr-test/detail/katello-pr-test/570/pipeline) and which one failed. For local development, having a single rake task that knows how to re-run the tasks could ease developer workflows. Something to think about, both ease of use and conveying information easily.
I like the idea of running tests against multiple versions of API.
I was thinking about checking the API versions and printing the warnings per command. Because the breakage is usually local and we know that certain commands won’t work, but the rest could be ok. It will also enable devs to add such version checks already during development of new features.
FYI we’re already generating good portion of the CLI from our apipie documentation. There’s a lot of places where we need to intervene manually and tune things so I’d stay back from having it fully auto-generated.
GraphQL shouldn’t have any dramatic impact on the architecture of the CLI that we currently have. There are some refactorings that would make it easier to migrate, but technically, it should be possible already today.
