Hey @jamboNum5!
I’ve recently gave a talk about this at FOSDEM: your management layer should be cattle too. The recording isn’t available yet (should be in the next days/weeks), but there are slides and a demo repository linked from the talk page and I can give a brief TL;DR version here:
- use the same stack as your prod – if you’re running Foreman on CentOS7 in a VM in prod, try to do the same in your lab/test/dev.
- keep your resource definitions (hostgroups, variables, etc) in a git-manageable file that you can re-play in every environment
- preferably with Ansible using our collection as @quba42 pointed out
- we also building an operations collection right now, that is supposed to help with install/upgrade steps
Following that will allow you to have very similar environments, that hopefully behave the same. Obviously this is nothing you can do in a day, especially if you already have a prod setup that has had seen some changes over a longer period of time.
As for the actual upgrade testing: I’d verify that all features work using the standalone test environment, and then try an upgrade on a cloned prod VM, to see if the database can be properly migrated – as it will contain more data (reports, clients) than your test environment. If both are good, upgrade prod.
There are also older talks of mine linked on the Foreman Media page – some are in German, sorry.