Upgrade foreman from 1.10 to 1.20

Support
1

Hello team,
We are on a verge of upgrading our foreman setup from 1.10.3 to 1.20 and would like to keep the database on a different setup.
Didn’t find anything specific with Postgres version to be chosen as a dependency on any of the docs, so is it fine if we go ahead with Postgres version 11 with foreman 1.20.
Also, has anyone done a full upgrade from 1.10 to 1.20?

Our current setup

  1. foreman server - 1.10.3 version - runs on ubuntu14.04

Our plans for an upgrade.

  1. Install foreman server 1.20 on a Ubuntu 18.04 machine.
  2. Do a pgdump of the current setup (1.10) and restore the same on (1.20)

Any suggestion is really appreciated.

2

I did a upgrade from 1.3 to 1.15 in one big chunk and it was an absolute nightmare of trying to migrate data with my own scripts.

When upgrading foreman multiple versions, you really need to do all the intermediate upgrades individually, as you will need to run all the database migrations, which depend on the code in the specific version of foreman they were released with (you can’t just run them all at latest version)

3

With the support for bionic from 1.19 we are looking to install 1.20 on a new bionic machine.
Restore the dump from 1.10 to 1.20 which is something I’m a bit confused.
How to approach DB migrate for each version is what I’m really need to understand.

4

Upgrades across multiple versions are not tested and there may be dragons involved.
The recommended way of doing such upgrade is going one version at a time, each time running the database migrations and seeds, and following the release notes and upgrade warnings to.

Considering the number of versions involved, you may need to also have some steps where you upgrade your underlying OS or some other dependencies to allow proceeding with the upgrade to the next version, and there may also be unexpected results due to the age of the OS versions - if some updates came out at the OS level that broke a release after it has been out of support, you might have to figure out some workarounds.

Restoring a DB dumb from 1.10 on a 1.20 machine will likely not work at all as the database schema has changed greatly in the ~3 years between those releases. Your other option is standing up a new Foreman 1.20 and migrating the hosts gradually from the old foreman to the new one.

1 Like
5

On the other hand, we still have all the database migrations. Doing a fresh 1.20 install, stop services, drop database, import database, migrate and seed might work. No guarantees but for such a big jump it might be much quicker. I’d try it in a lab and see if you see any dragons.

6

Thanks for the info guys let me test it out.

7

I have setup foreman 1.20.2 on my ubuntu18 (bionic) machine.
I did a restore of db from foreman 1.10 (ubuntu14) to the foreman 1.20.2 machine.
While doing db:migrate things broke.

8

So basically this is not a upgrade but just install 1.20 on a new operating system (ubuntu1804) and try to restore the dump from the old 1.10 version of foreman running on ubuntu1404.

9

Looks like the 10 versions as a time is too big of a step. At some point we became much stricter about database models. I’d recommend upgrading to the latest version that was supported with 14.04 and retrying from that, perhaps to the oldest supported version on 18.04.

10

@ekohl thanks for the update.
I have noted the following details:

1.12 - this is when ubuntu16 got support
1.15 - will be the last major release for Ubuntu/trusty (ubuntu 14.04)
1.19 - support started for ubuntu bionic(18.04)

Looks like I have to update my ubuntu14 machine till 1.12 and then shift to ubuntu 16 and so on.

11

You could still try to upgrade to 1.15 on 14.04 and then jump to 1.19. As long as the DB migrations pass I’d trust it, at least if you verify it in a lab setup.

12

I was able to install foreman 1.20 on my ubuntu18 machine.
Restore the dump from 1.10 to 1.20 and by nullifying the description field in operatingsystems table the db:migrate went fine.
Thanks for the help guys, really appreciated.

I’m stuck with installing the module through puppet though.
This is what my Puppetfile looks like:

mod 'theforeman/puppet-foreman',
  git: 'http://github.com/theforeman/puppet-foreman',
  ref: '9.2-stable'

When I run librarian-puppet install, I get the following error:

[Librarian] Resolving theforeman-puppet-foreman (>= 0) <http://github.com/theforeman/puppet-foreman#9.2-stable>
[Librarian]   Checking manifests
[Librarian]     --- No output
[Librarian]     --- No output
[Librarian]     --- No output
[Librarian]     --- No output
[Librarian]     --> origin
[Librarian]     -->   origin/1.4-stable
[Librarian]     -->   origin/2.1-stable
[Librarian]     -->   origin/2.2-stable
[Librarian]     -->   origin/2.3-stable
[Librarian]     -->   origin/3.0-stable
[Librarian]     -->   origin/5.0-stable
[Librarian]     -->   origin/5.2-stable
[Librarian]     -->   origin/6.0-stable
[Librarian]     -->   origin/7.2-stable
[Librarian]     -->   origin/8.1-stable
[Librarian]     -->   origin/9.2-stable
[Librarian]     -->   origin/HEAD -> origin/master
[Librarian]     -->   origin/master
[Librarian]     --> 258afab8caa44a63ae4c40b4c45984c5725b6862
[Librarian]     --> 258afab8caa44a63ae4c40b4c45984c5725b6862
[Librarian]     Checking theforeman-puppet-foreman/9.2.1 <http://github.com/theforeman/puppet-foreman#9.2-stable>
[Librarian]       Resolved theforeman-puppet-foreman (>= 0) <http://github.com/theforeman/puppet-foreman#9.2-stable> at theforeman-puppet-foreman/9.2.1 <http://github.com/theforeman/puppet-foreman#9.2-stable>
[Librarian]   Resolved theforeman-puppet-foreman (>= 0) <http://github.com/theforeman/puppet-foreman#9.2-stable>
[Librarian] Conflict between puppetlabs-apache (>= 1.2.0, < 2.0.0) <https://forgeapi.puppetlabs.com> and puppetlabs-apache (>= 2.0.0, < 4.0.0) <https://forgeapi.puppetlabs.com>
Could not resolve the dependencies.
13

I was able to fix the apache dependency but not the concat one.

[Librarian] Resolving puppetlabs-concat (= 5.3.0) <https://forgeapi.puppetlabs.com>
[Librarian]   Checking manifests
[Librarian]   Module puppetlabs-concat found versions: 5.3.0, 5.2.0, 5.1.0, 5.0.0, 4.2.1, 4.2.0, 4.1.1, 4.1.0, 4.0.1, 4.0.0, 3.0.0, 2.2.1, 2.2.0, 2.1.0, 1.2.5, 1.2.4, 1.2.3, 1.2.2, 1.2.1, 1.2.0, 1.1.2, 1.1.1, 1.1.0, 1.1.0-rc1, 1.0.4, 1.0.3, 1.0.2, 1.0.1, 1.0.0, 1.0.0-rc1
[Librarian]     Checking puppetlabs-concat/5.3.0 <https://forgeapi.puppetlabs.com>
[Librarian]       Conflict between puppetlabs-concat/5.3.0 <https://forgeapi.puppetlabs.com> and puppetlabs-concat (>= 1.1.1, < 2.0.0) <https://forgeapi.puppetlabs.com>
[Librarian]       Backtracking from puppetlabs-concat/5.3.0 <https://forgeapi.puppetlabs.com>
[Librarian]     Checking puppetlabs-concat/5.2.0 <https://forgeapi.puppetlabs.com>
[Librarian]       Conflict between puppetlabs-concat/5.2.0 <https://forgeapi.puppetlabs.com> and puppetlabs-concat (= 5.3.0) <https://forgeapi.puppetlabs.com>
[Librarian]       Backtracking from puppetlabs-concat/5.2.0 <https://forgeapi.puppetlabs.com>
[Librarian]     Checking puppetlabs-concat/5.1.0 <https://forgeapi.puppetlabs.com>
[Librarian]       Conflict between puppetlabs-concat/5.1.0 <https://forgeapi.puppetlabs.com> and puppetlabs-concat (= 5.3.0) <https://forgeapi.puppetlabs.com>
[Librarian]       Backtracking from puppetlabs-concat/5.1.0 <https://forgeapi.puppetlabs.com>
[Librarian]     Checking puppetlabs-concat/5.0.0 <https://forgeapi.puppetlabs.com>
[Librarian]       Conflict between puppetlabs-concat/5.0.0 <https://forgeapi.puppetlabs.com> and puppetlabs-concat (= 5.3.0) <https://forgeapi.puppetlabs.com>
[Librarian]       Backtracking from puppetlabs-concat/5.0.0 <https://forgeapi.puppetlabs.com>
[Librarian]     Checking puppetlabs-concat/4.2.1 <https://forgeapi.puppetlabs.com>
[Librarian]       Conflict between puppetlabs-concat/4.2.1 <https://forgeapi.puppetlabs.com> and puppetlabs-concat (= 5.3.0) <https://forgeapi.puppetlabs.com>
[Librarian]       Backtracking from puppetlabs-concat/4.2.1 <https://forgeapi.puppetlabs.com>
[Librarian]     Checking puppetlabs-concat/4.2.0 <https://forgeapi.puppetlabs.com>
[Librarian]       Conflict between puppetlabs-concat/4.2.0 <https://forgeapi.puppetlabs.com> and puppetlabs-concat (= 5.3.0) <https://forgeapi.puppetlabs.com>
[Librarian]       Backtracking from puppetlabs-concat/4.2.0 <https://forgeapi.puppetlabs.com>
[Librarian]     Checking puppetlabs-concat/4.1.1 <https://forgeapi.puppetlabs.com>
[Librarian]       Conflict between puppetlabs-concat/4.1.1 <https://forgeapi.puppetlabs.com> and puppetlabs-concat (= 5.3.0) <https://forgeapi.puppetlabs.com>
[Librarian]       Backtracking from puppetlabs-concat/4.1.1 <https://forgeapi.puppetlabs.com>
[Librarian]     Checking puppetlabs-concat/4.1.0 <https://forgeapi.puppetlabs.com>
[Librarian]       Conflict between puppetlabs-concat/4.1.0 <https://forgeapi.puppetlabs.com> and puppetlabs-concat (= 5.3.0) <https://forgeapi.puppetlabs.com>
[Librarian]       Backtracking from puppetlabs-concat/4.1.0 <https://forgeapi.puppetlabs.com>
[Librarian]     Checking puppetlabs-concat/4.0.1 <https://forgeapi.puppetlabs.com>
[Librarian]       Conflict between puppetlabs-concat/4.0.1 <https://forgeapi.puppetlabs.com> and puppetlabs-concat (= 5.3.0) <https://forgeapi.puppetlabs.com>
[Librarian]       Backtracking from puppetlabs-concat/4.0.1 <https://forgeapi.puppetlabs.com>
[Librarian]     Checking puppetlabs-concat/4.0.0 <https://forgeapi.puppetlabs.com>
[Librarian]       Conflict between puppetlabs-concat/4.0.0 <https://forgeapi.puppetlabs.com> and puppetlabs-concat (= 5.3.0) <https://forgeapi.puppetlabs.com>
[Librarian]       Backtracking from puppetlabs-concat/4.0.0 <https://forgeapi.puppetlabs.com>
[Librarian]     Checking puppetlabs-concat/3.0.0 <https://forgeapi.puppetlabs.com>
[Librarian]       Conflict between puppetlabs-concat/3.0.0 <https://forgeapi.puppetlabs.com> and puppetlabs-concat (= 5.3.0) <https://forgeapi.puppetlabs.com>
[Librarian]       Backtracking from puppetlabs-concat/3.0.0 <https://forgeapi.puppetlabs.com>
[Librarian]     Checking puppetlabs-concat/2.2.1 <https://forgeapi.puppetlabs.com>
[Librarian]       Conflict between puppetlabs-concat/2.2.1 <https://forgeapi.puppetlabs.com> and puppetlabs-concat (= 5.3.0) <https://forgeapi.puppetlabs.com>
[Librarian]       Backtracking from puppetlabs-concat/2.2.1 <https://forgeapi.puppetlabs.com>
[Librarian]     Checking puppetlabs-concat/2.2.0 <https://forgeapi.puppetlabs.com>
[Librarian]       Conflict between puppetlabs-concat/2.2.0 <https://forgeapi.puppetlabs.com> and puppetlabs-concat (= 5.3.0) <https://forgeapi.puppetlabs.com>
[Librarian]       Backtracking from puppetlabs-concat/2.2.0 <https://forgeapi.puppetlabs.com>
[Librarian]     Checking puppetlabs-concat/2.1.0 <https://forgeapi.puppetlabs.com>
[Librarian]       Conflict between puppetlabs-concat/2.1.0 <https://forgeapi.puppetlabs.com> and puppetlabs-concat (= 5.3.0) <https://forgeapi.puppetlabs.com>
[Librarian]       Backtracking from puppetlabs-concat/2.1.0 <https://forgeapi.puppetlabs.com>
[Librarian]     Checking puppetlabs-concat/1.2.5 <https://forgeapi.puppetlabs.com>
[Librarian]       Conflict between puppetlabs-concat/1.2.5 <https://forgeapi.puppetlabs.com> and puppetlabs-concat (= 5.3.0) <https://forgeapi.puppetlabs.com>
[Librarian]       Backtracking from puppetlabs-concat/1.2.5 <https://forgeapi.puppetlabs.com>
[Librarian]     Checking puppetlabs-concat/1.2.4 <https://forgeapi.puppetlabs.com>
[Librarian]       Conflict between puppetlabs-concat/1.2.4 <https://forgeapi.puppetlabs.com> and puppetlabs-concat (= 5.3.0) <https://forgeapi.puppetlabs.com>
[Librarian]       Backtracking from puppetlabs-concat/1.2.4 <https://forgeapi.puppetlabs.com>
[Librarian]     Checking puppetlabs-concat/1.2.3 <https://forgeapi.puppetlabs.com>
[Librarian]       Conflict between puppetlabs-concat/1.2.3 <https://forgeapi.puppetlabs.com> and puppetlabs-concat (= 5.3.0) <https://forgeapi.puppetlabs.com>
[Librarian]       Backtracking from puppetlabs-concat/1.2.3 <https://forgeapi.puppetlabs.com>
[Librarian]     Checking puppetlabs-concat/1.2.2 <https://forgeapi.puppetlabs.com>
[Librarian]       Conflict between puppetlabs-concat/1.2.2 <https://forgeapi.puppetlabs.com> and puppetlabs-concat (= 5.3.0) <https://forgeapi.puppetlabs.com>
[Librarian]       Backtracking from puppetlabs-concat/1.2.2 <https://forgeapi.puppetlabs.com>
[Librarian]     Checking puppetlabs-concat/1.2.1 <https://forgeapi.puppetlabs.com>
[Librarian]       Conflict between puppetlabs-concat/1.2.1 <https://forgeapi.puppetlabs.com> and puppetlabs-concat (= 5.3.0) <https://forgeapi.puppetlabs.com>
[Librarian]       Backtracking from puppetlabs-concat/1.2.1 <https://forgeapi.puppetlabs.com>
[Librarian]     Checking puppetlabs-concat/1.2.0 <https://forgeapi.puppetlabs.com>
[Librarian]       Conflict between puppetlabs-concat/1.2.0 <https://forgeapi.puppetlabs.com> and puppetlabs-concat (= 5.3.0) <https://forgeapi.puppetlabs.com>
[Librarian]       Backtracking from puppetlabs-concat/1.2.0 <https://forgeapi.puppetlabs.com>
[Librarian]     Checking puppetlabs-concat/1.1.2 <https://forgeapi.puppetlabs.com>
[Librarian]       Conflict between puppetlabs-concat/1.1.2 <https://forgeapi.puppetlabs.com> and puppetlabs-concat (= 5.3.0) <https://forgeapi.puppetlabs.com>
[Librarian]       Backtracking from puppetlabs-concat/1.1.2 <https://forgeapi.puppetlabs.com>
[Librarian]     Checking puppetlabs-concat/1.1.1 <https://forgeapi.puppetlabs.com>
[Librarian]       Conflict between puppetlabs-concat/1.1.1 <https://forgeapi.puppetlabs.com> and puppetlabs-concat (= 5.3.0) <https://forgeapi.puppetlabs.com>
[Librarian]       Backtracking from puppetlabs-concat/1.1.1 <https://forgeapi.puppetlabs.com>
[Librarian]     Checking puppetlabs-concat/1.1.0 <https://forgeapi.puppetlabs.com>
[Librarian]       Conflict between puppetlabs-concat/1.1.0 <https://forgeapi.puppetlabs.com> and puppetlabs-concat (= 5.3.0) <https://forgeapi.puppetlabs.com>
[Librarian]       Backtracking from puppetlabs-concat/1.1.0 <https://forgeapi.puppetlabs.com>
[Librarian]     Checking puppetlabs-concat/1.1.0-rc1 <https://forgeapi.puppetlabs.com>
[Librarian]       Conflict between puppetlabs-concat/1.1.0-rc1 <https://forgeapi.puppetlabs.com> and puppetlabs-concat (= 5.3.0) <https://forgeapi.puppetlabs.com>
[Librarian]       Backtracking from puppetlabs-concat/1.1.0-rc1 <https://forgeapi.puppetlabs.com>
[Librarian]     Checking puppetlabs-concat/1.0.4 <https://forgeapi.puppetlabs.com>
[Librarian]       Conflict between puppetlabs-concat/1.0.4 <https://forgeapi.puppetlabs.com> and puppetlabs-concat (= 5.3.0) <https://forgeapi.puppetlabs.com>
[Librarian]       Backtracking from puppetlabs-concat/1.0.4 <https://forgeapi.puppetlabs.com>
[Librarian]     Checking puppetlabs-concat/1.0.3 <https://forgeapi.puppetlabs.com>
[Librarian]       Conflict between puppetlabs-concat/1.0.3 <https://forgeapi.puppetlabs.com> and puppetlabs-concat (= 5.3.0) <https://forgeapi.puppetlabs.com>
[Librarian]       Backtracking from puppetlabs-concat/1.0.3 <https://forgeapi.puppetlabs.com>
[Librarian]     Checking puppetlabs-concat/1.0.2 <https://forgeapi.puppetlabs.com>
[Librarian]       Conflict between puppetlabs-concat/1.0.2 <https://forgeapi.puppetlabs.com> and puppetlabs-concat (= 5.3.0) <https://forgeapi.puppetlabs.com>
[Librarian]       Backtracking from puppetlabs-concat/1.0.2 <https://forgeapi.puppetlabs.com>
[Librarian]     Checking puppetlabs-concat/1.0.1 <https://forgeapi.puppetlabs.com>
[Librarian]       Conflict between puppetlabs-concat/1.0.1 <https://forgeapi.puppetlabs.com> and puppetlabs-concat (= 5.3.0) <https://forgeapi.puppetlabs.com>
[Librarian]       Backtracking from puppetlabs-concat/1.0.1 <https://forgeapi.puppetlabs.com>
[Librarian]     Checking puppetlabs-concat/1.0.0 <https://forgeapi.puppetlabs.com>
[Librarian]       Conflict between puppetlabs-concat/1.0.0 <https://forgeapi.puppetlabs.com> and puppetlabs-concat (= 5.3.0) <https://forgeapi.puppetlabs.com>
[Librarian]       Backtracking from puppetlabs-concat/1.0.0 <https://forgeapi.puppetlabs.com>
[Librarian]     Checking puppetlabs-concat/1.0.0-rc1 <https://forgeapi.puppetlabs.com>
[Librarian]       Conflict between puppetlabs-concat/1.0.0-rc1 <https://forgeapi.puppetlabs.com> and puppetlabs-concat (= 5.3.0) <https://forgeapi.puppetlabs.com>
[Librarian]       Backtracking from puppetlabs-concat/1.0.0-rc1 <https://forgeapi.puppetlabs.com>
[Librarian]   Failed to resolve puppetlabs-concat (= 5.3.0) <https://forgeapi.puppetlabs.com>
Could not resolve the dependencies.
/Users/abalakrishnan/.rvm/gems/ruby-2.1.9/gems/librarianp-0.6.4/lib/librarian/action/persist_resolution_mixin.rb:11:in `persist_resolution'
/Users/abalakrishnan/.rvm/gems/ruby-2.1.9/gems/librarianp-0.6.4/lib/l
14

librarian-puppet output is always hard to follow so I generally try to use the latest versions. Why are you pinning to 9.20-stable? With 1.20 you should be able to the latest version.

15

I updated the code to latest but the error still stays.

[Librarian] Resolved puppetlabs-apache (= 2.0.0) https://forgeapi.puppetlabs.com
[Librarian] Conflict between puppetlabs-concat (= 1.2.5) https://forgeapi.puppetlabs.com and puppetlabs-concat (>= 2.2.1, < 5.0.0) https://forgeapi.puppetlabs.com
Could not resolve the dependencies.

16

Something is pulling in puppetlabs-concat 1.x and I wonder what it is. It would be useful to see your entire Puppetfile.

17

this is what it looks like

forge 'https://forgeapi.puppetlabs.com'

#
# 3rd-Party modules from the Forge
#
mod 'golja/gnupg', '1.2.1'
mod 'thias/vsftpd', '0.2.1'
mod 'spiette/selinux', '0.5.4'
mod 'saz/sudo', '3.0.9'
mod 'saz/rsyslog', '5.0.0'
mod 'puppetlabs/java', '1.6.0'
mod 'puppetlabs/mysql', '3.0.0'
mod 'puppetlabs/apt', '2.2.2'
mod 'jonnyx/msuac', '0.0.3'
mod 'jonnyx/iesec', '0.0.1'
mod 'puppetlabs/stdlib', '4.25.0'
mod 'puppetlabs/ntp', '3.3.0'
mod 'puppetlabs/apache', '2.0.0'
mod 'puppetlabs/concat', '1.2.5'
mod 'puppetlabs/registry', '1.0.3'
mod 'puppetlabs/powershell', '1.0.4'
18

You explicitly request the old version of concat, but we’re no longer compatible with that.

19

Makes sense, my team is looking into it.
The upgrade of only the foreman server has been done now with the dump/restore from 1.10to 1.20 going well.

While trying to create a new host I see the following :-

019-03-18T10:49:20 [I|app|24d5d] Adding Compute instance for jenkins-test.com
2019-03-18T10:49:24 [I|app|] Started GET "/notification_recipients" for 10.0.15.138 at 2019-03-18 10:49:24 +0000
2019-03-18T10:49:24 [I|app|e2b1a] Processing by NotificationRecipientsController#index as JSON
2019-03-18T10:49:24 [I|app|e2b1a] Current user set to ashwin (admin)
2019-03-18T10:49:24 [I|app|e2b1a] Completed 200 OK in 121ms (Views: 0.4ms | ActiveRecord: 27.0ms)
2019-03-18T10:49:42 [W|app|24d5d] Failed to create a compute QA_cluster (VMware) instance jenkins-test.com: undefined method `resourcePool' for nil:NilClass
 |
 | NoMethodError: undefined method `resourcePool' for nil:NilClass
 | /usr/share/foreman/vendor/ruby/2.5.0/gems/fog-vsphere-2.5.0/lib/fog/vsphere/requests/compute/get_resource_pool.rb:16:in `get_raw_resource_pool'
 | /usr/share/foreman/vendor/ruby/2.5.0/gems/fog-vsphere-2.5.0/lib/fog/vsphere/requests/compute/vm_clone.rb:120:in `vm_clone'
 | /usr/share/foreman/app/models/compute_resources/foreman/model/vmware.rb:525:in `clone_vm'
 | /usr/share/foreman/app/models/compute_resources/foreman/model/vmware.rb:446:in `create_vm'
 | /usr/share/foreman/app/models/concerns/orchestration/compute.rb:86:in `setCompute'
 | /usr/share/foreman/app/models/concerns/orchestration.rb:222:in `execute'
 | /usr/share/foreman/app/models/concerns/orchestration.rb:145:in `block in process'
 | /usr/share/foreman/app/models/concerns/orchestration.rb:137:in `each'
 | /usr/share/foreman/app/models/concerns/orchestration.rb:137:in `process'
 | /usr/share/foreman/app/models/concerns/orchestration.rb:44:in `around_save_orchestration'
 | /usr/share/foreman/vendor/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
 | /usr/share/foreman/vendor/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:136:in `run_callbacks'
 | /usr/share/foreman/vendor/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:816:in `_run_save_callbacks'
 | /usr/share/foreman/vendor/ruby/2.5.0/gems/activerecord-5.2.1/lib/active_record/callbacks.rb:342:in `create_or_update'
 | /usr/share/foreman/vendor/ruby/2.5.0/gems/activerecord-5.2.1/lib/active_record/persistence.rb:275:in `save'
 | /usr/share/foreman/vendor/ruby/2.5.0/gems/activerecord-5.2.1/lib/active_record/validations.rb:46:in `save'
 | /usr/share/foreman/vendor/ruby/2.5.0/gems/activerecord-5.2.1/lib/active_record/transactions.rb:310:in `block (2 levels) in save'
 | /usr/share/foreman/vendor/ruby/2.5.0/gems/activerecord-5.2.1/lib/active_record/transactions.rb:387:in `block in with_transaction_returning_status'
 | /usr/share/foreman/vendor/ruby/2.5.0/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/database_statements.rb:254:in `block in transaction'
 | /usr/share/foreman/vendor/ruby/2.5.0/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/transaction.rb:239:in `block in within_new_transaction'
 | /usr/lib/ruby/2.5.0/monitor.rb:226:in `mon_synchronize'
20

Looks like the cluster name which used to work as a Datacenter in foreman (1.10) API has changed to actual cluster name in foreman (1.20).
Changing that helped to move forward.