Upgrade Guidance

Support
1

Problem:
Looking for some general guidance on best practices or “gotchas” while I am planning a foreman upgrade. For instance:

  • Do I have to upgrade to point releases until I get to the current stable release?
  • Any caches to clear?
  • Do my capsules need to match versions? Do I need to upgrade them in parallel?
  • Backup strategies incase a rollback is necessary
  • ???

Expected outcome:
Zero downtime :smile:

Foreman and Proxy versions:
katello-3.4.5-1
foreman-1.15.6
Proxies running 1.15.6 and 1.15.1

2

Hi,

we are currently in the process of upgrading from 1.15/3.4 ourselfs :smiley:
For general gotchas:
In our dev environment, upgrading to 1.20/3.9 went quite flawlessly except for some issues regarding certain plugins. We ran into some bugs with 1.20.1, but if you upgrade to a newer version, those should be fixed already :wink:
In general, plugins will be where you will most likely run into gotchas. During our upgrade, we encountered dragons with:

  • scc_manager
  • a weird interaction between snapshot_manager and bootdisk

We did update in the following way:

  • Update Foreman and proxy servers to the latest bugfix release
  • Update Foreman and proxy servers, one release at a time (so, 1.16, 1.17, 1.18, etc). For each release we did Foreman first, then all the proxys in whatever order; rinse and repeat for the next release.
  • Check whether we broke something :laughing:
  • Done

I think from some version on it is supported to upgrade from there straight to latest, but better be safe than sorry :wink:

For a backup/rollback strategy, we did an offline clone of all the Foreman and proxy VMs. If you are on hardware, that’s a totally different beast of course. You can do a data backup with katello-backup, but you will have to reinstall the whole stack with 1.15 before being able to restore that.

Zero downtime… Yeah, would have liked that, too :laughing:

If you have more questions, go on :wink:

Regards

1 Like
3

Oh, one more gotcha you might run into:
We encountered problems with the old Foreman version when updating OS and EPEL packages prior to the Foreman update. Especially, there were some incompatibilities between pulp and the EPEL django package. Maybe you already hit that on a regular OS update. That issue should be resolved in more recent pulp version, but you will want to look out for that, there will be dragons. I thought there was a forum toppic about that from back when it happened, but I could not find it on the spot.

Also, I found out that skipping updates is indeed not supported and you should upgrade one Release at a time.

4

From our experience, step-by-step upgrades are the way to go. Some versions got major changes especially in database structure and we got huge problems with them. From 1.17. to 1.19. for example.

Some issues occurred because of our growing environment, some actually were bugs. Generally it should a safe way to backup the machine, export database and import with migrations.

5

Recommendations for upgrading this combination?
foreman 1.20.1
with features: Templates, Pulp, TFTP, Puppet, Puppet CA, Logs, Dynflow, Discovery, Openscap, and SSH
katello 3.10.0
puppet 5.5.10

Is there an easy way to determine the latest versions available, and the appropriate compatibility matrix?

Then, what is the correct sequence of events for actually applying the upgrades?
Backing up first seems like a good idea, until you actually try it.

I would rather not have to do a fresh install with all new versions from scratch, and reconfigure everything…

6

Thanks for feedback everyone. We are dealing with a physical box so vm snapshots aren’t really in the realm of possibility :frowning:

We plan on trimming down old content views before hand in a hope that it creates less “stuff” to mind during the backup process.

Part of our plan between point releases is to validate some functionality of all services, ports, and what not using inspec tests. Perhaps when it is all said and done, we can share our tests with the community if there is interest?

Looking forward to hearing more about these dragons before we make the dive :smiley:

7

Perhaps when it is all said and done, we can share our tests with the community if there is interest?

Sharing with the community is what community means IMHO :wink:

1 Like