Indeed we only test upgrades one version at a time. Upgrading more than one might work, but there might be issues as well. As a side note, if foreman is running in a VM, I would recommend taking a snapshot before starting the upgrade and then another snapshot for each version so in case one upgrade goes wrong you don’t need to start all over from 1.24.
