Need help with upgrading Katello/Foreman to newer versions

Need help with upgrading Katello/Capsule servers

Running into a lot of issues lately due to being on old versions. Still new to katello/foreman.

When I upgrade to newer version, do i expect to upgrade both foreman/proxy versions as well as plugins?

We are running our katello/capsule servers on Centos 7.

Appreciate all the help.

Here are the versions:

  • candlepin-0.9.54.6-1.el7.noarch
  • candlepin-selinux-0.9.54.6-1.el7.noarch
  • foreman-1.13.4-1.el7.noarch
  • foreman-compute-1.13.4-1.el7.noarch
  • foreman-debug-1.13.4-1.el7.noarch
  • foreman-installer-1.13.4-1.el7.noarch
  • foreman-installer-katello-3.2.3-1.el7.noarch
  • foreman-postgresql-1.13.4-1.el7.noarch
  • foreman-proxy-1.13.4-1.el7.noarch
  • foreman-release-1.13.4-1.el7.noarch
  • foreman-release-scl-3-1.el7.noarch
  • foreman-selinux-1.13.4-1.el7.noarch
  • foreman-vmware-1.13.4-1.el7.noarch
  • katello-3.2.2-1.el7.noarch
  • katello-certs-tools-2.4.0-1.el7.noarch
  • katello-client-bootstrap-1.1.0-1.el7.noarch
  • katello-common-3.2.2-1.el7.noarch
  • katello-debug-3.2.2-1.el7.noarch
  • katello-default-ca-1.0-1.noarch
  • katello-installer-base-3.2.3-1.el7.noarch
  • katello-repos-3.2.0-4.el7.noarch
  • katello-selinux-3.0.1-1.el7.noarch
  • katello-server-ca-1.0-1.noarch
  • katello-service-3.2.2-1.el7.noarch
  • pulp-admin-client-2.9.3-1.el7.noarch
  • pulp-client-1.0-1.noarch
  • pulp-docker-plugins-2.0.4-1.el7.noarch
  • pulp-katello-1.0.2-1.el7.noarch
  • pulp-puppet-plugins-2.9.3-1.el7.noarch
  • pulp-puppet-tools-2.9.3-1.el7.noarch
  • pulp-rpm-plugins-2.9.3-1.el7.noarch
  • pulp-selinux-2.9.3-1.el7.noarch
  • pulp-server-2.9.3-1.el7.noarch
  • python-gofer-qpid-2.7.6-1.el7.noarch
  • python-isodate-0.5.0-4.pulp.el7.noarch
  • python-kombu-3.0.33-6.pulp.el7.noarch
  • python-pulp-bindings-2.9.3-1.el7.noarch
  • python-pulp-client-lib-2.9.3-1.el7.noarch
  • python-pulp-common-2.9.3-1.el7.noarch
  • python-pulp-docker-common-2.0.4-1.el7.noarch
  • python-pulp-oid_validation-2.9.3-1.el7.noarch
  • python-pulp-puppet-common-2.9.3-1.el7.noarch
  • python-pulp-repoauth-2.9.3-1.el7.noarch
  • python-pulp-rpm-common-2.9.3-1.el7.noarch
  • python-pulp-streamer-2.9.3-1.el7.noarch
  • python-qpid-1.35.0-3.el7.noarch
  • python-qpid-proton-0.17.0-1.el7.x86_64
  • python-qpid-qmf-1.35.0-3.el7.x86_64
  • python-urllib3-1.10.2-2.katello.el7.noarch
  • qpid-cpp-client-1.35.0-3.el7.x86_64
  • qpid-cpp-client-devel-1.35.0-3.el7.x86_64
  • qpid-cpp-server-1.35.0-3.el7.x86_64
  • qpid-cpp-server-linearstore-1.35.0-3.el7.x86_64
  • qpid-dispatch-router-0.7.0-1.el7.x86_64
  • qpid-proton-c-0.17.0-1.el7.x86_64
  • qpid-qmf-1.35.0-3.el7.x86_64
  • qpid-tools-1.35.0-3.el7.noarch
  • rubygem-smart_proxy_pulp-1.3.0-1.el7.noarch
  • sl1mmgplsat0001-foreman-client-1.0-1.noarch
  • sl1mmgplsat0001-foreman-proxy-1.0-1.noarch
  • sl1mmgplsat0001-foreman-proxy-client-1.0-1.noarch
1 Like

Hi and welcome to the forum :slight_smile:

The update workflow is usually always the same for each version, and quite simple. You do always update the Foreman server with all plugins first, then update all your smart-proxies servers (if you have additional ones besides the Foreman server). You have to update one version at a time, though, so updating from your current version to a recent one will take some time.

The complete workflow for upgrades usually looks like this. I will assume you have a setup with additional dedicated smart-proxy servers. If not, just skip those parts. In general, you can just follow Katelloā€™s upgrade instructions for each version, but here is the general workflow:

  1. Make a backup
  2. Start out with reading the release notes for all versions of both Foreman and Katello you are going to pass throughout your upgrades. Especially deprecation notices and upgrade warnings are important, but it is usually worth the time to at least zoom through everything.
  3. Run ā€œyum update -yā€ on all of your Foreman and smart-proxy systems. Reboot if there was a kernel update.
  4. On your foreman servers, install the release packages for the next versions of Foreman/Katello.
  5. Run ā€œyum clean all && yum update -yā€ on the foreman server.
  6. Run " foreman-installer --scenario katello --upgrade" on the foreman server. If your system has had manual changes to foreman related configs, this will override those. If you are unsure, I would recommend running " foreman-installer --scenario katello --upgrade -v --noop" first. This will print out the logs of the upgrade procedure and might point you to config changes that could break your setup. If you spot any changes that will affect your environment, you should pass the desired config parameters to foreman-installer as well on the actual upgrade. That way they will be stored for the future.
  7. Install the matching release packages for Foreman/Katello repos on any dedicated smart-proxy servers and ā€œyum clean all && yum update -yā€ on them.
  8. Regenerate the smart-proxy certificates on your Foreman server and copy the new tarball over (needs to be done per dedicated smart-proxy system).
  9. Run " foreman-installer --scenario foreman-proxy-content --upgrade --foreman-proxy-content-certs-tar ~/myproxy.example.com-certs.tar --certs-update-all --certs-regenerate --certs-deploy" (replace filename with ones that match your environment) on your smart-proxy servers. Like with the Foreman server itself, this will upgrade any manual config changes that might have been done and you might want to run the installer with an additional ā€œ-v --noopā€ first.
  10. Repeat 3.-9. until you are on the desired version.
  11. Update the foreman-client repos for your managed servers to the release you are on now and update the client tools on each system.

I would recommend checking the update documentation for each release, though, since there might be additional steps needed on some occasions.
Here are the one for the first version you will have to update to:
Foreman/Katello Update
Smart-Proxy Update

Feel free to ask any additional questions you might have on this :slight_smile:

Regards

4 Likes

would not running yum update -y will bring me to the latest version instead of going one version at a time in your step 3?

I am at the version 1.13 at this moment. Do i need to go to 1.14 first?

On this documentation from foreman for 1.14 version , it only explains about installer not upgrade process.

https://theforeman.org/manuals/1.14/quickstart_guide.html

at step 8 you mentioned about renegerating smart proxy certs, can you explain more on how to do on Centos 7?

Appreciate your help and respond on this.

upgrades are only supported one version at a time. Please follow the upgrade instructions for each version you are updating to, starting with the links provided above by @areyus. Be sure to go over the upgrade warnings and release notes for both foreman and katello in the manuals for every version you are upgrading prior to the upgrade, there have been a lot of changes over the past several years since 1.13/3.2 were released.
If you are running on a VM, I recommend taking a snapshot after each successful version upgrade so you donā€™t have to restart if one step fails.

The alternative option, which may be easier depending on your setup, is to install a fresh new Katello 3.15 server and manually migrate your configuration and all hosts over to it. if you have a small-ish estate, it might be easier than doing 12 version upgrades one after the other.

1 Like

As an addition to what @tbrisker already explained, if you are not able to setup a new instance or willing to invest the extra effort:

No, this will not bring you directly to latest. This will just bring your OS to the latest point and upgrade your Foreman/Katello Stack to the latest bugfix Release (X.YZ.N). Each Foreman and Katello minor version (X.YZ) have their own repository. Those are switched as part of step 4.

You should use the Katello upgrade guide, since you have a Katello stack. That upgrade documentation also handles Foreman upgrades in sync with the Katello version. I linked the documentation for your first upgrade (to 1.14/3.3) in my earlier post. Do not use the ā€œplainā€ Foreman documentation for upgrading a Katello stack, you will break your setup and get yourself in a quite nasty position.

Regards

thank you for your responseā€¦i should really consider setting up fresh Katello server on Centos 8 maybe.
need to do some research on how to migrate configuration and hosts to it.

got it sir, thank you for detailed response!

When I try to run foreman-proxy-certs-generate on katello server to generate certs I am getting this error. Any thoughts?

/usr/share/gems/gems/kafo-0.9.8/lib/kafo/param_builder.rb:83:in const_defined?': wrong constant name Stdlib::absolutepath (NameError) from /usr/share/gems/gems/kafo-0.9.8/lib/kafo/param_builder.rb:83:in get_typeā€™
from /usr/share/gems/gems/kafo-0.9.8/lib/kafo/param_builder.rb:55:in build' from /usr/share/gems/gems/kafo-0.9.8/lib/kafo/param_builder.rb:30:in block in build_paramsā€™
from /usr/share/gems/gems/kafo-0.9.8/lib/kafo/param_builder.rb:29:in map' from /usr/share/gems/gems/kafo-0.9.8/lib/kafo/param_builder.rb:29:in build_paramsā€™
from /usr/share/gems/gems/kafo-0.9.8/lib/kafo/puppet_module.rb:79:in parse' from /usr/share/gems/gems/kafo-0.9.8/lib/kafo/configuration.rb:89:in block in modulesā€™
from /usr/share/gems/gems/kafo-0.9.8/lib/kafo/configuration.rb:89:in map' from /usr/share/gems/gems/kafo-0.9.8/lib/kafo/configuration.rb:89:in modulesā€™
from /usr/share/gems/gems/kafo-0.9.8/lib/kafo/configuration.rb:189:in params' from /usr/share/gems/gems/kafo-0.9.8/lib/kafo/configuration.rb:199:in preset_defaults_from_puppetā€™
from /usr/share/gems/gems/kafo-0.9.8/lib/kafo/kafo_configure.rb:270:in set_parameters' from /usr/share/gems/gems/kafo-0.9.8/lib/kafo/kafo_configure.rb:99:in initializeā€™
from /usr/share/gems/gems/clamp-1.0.0/lib/clamp/command.rb:133:in new' from /usr/share/gems/gems/clamp-1.0.0/lib/clamp/command.rb:133:in runā€™
from /usr/share/gems/gems/kafo-0.9.8/lib/kafo/kafo_configure.rb:154:in run' from /sbin/foreman-proxy-certs-generate:79:in ā€™

I had to install foreman-installer-katello package to be able to run foreman-proxy-certs-generate script.

I havenā€™t seen that. Can you share the output of yum list installed foreman-installer\*'? I suspect there may be a mismatch in versions.

2 Likes

yup that was it. Once I have updated foreman-installer package to foreman-installer-1.14.3-1.el7.noarch from 1.13ā€¦it worked.

Appreciate the help! Thank you so much!

1 Like

When I try to update packages on the server

I keep getting these errors? can you see if I am missing certain repos?

Error: Package: pulp-server-2.9.3-1.el7.noarch (@katello-pulp)
Requires: python-celery < 3.2.0
Removing: python-celery-3.1.11-1.el7.noarch (@katello-pulp)
python-celery = 3.1.11-1.el7
Obsoleted By: python2-celery-4.2.1-3.el7.noarch (epel)
python-celery = 4.2.1-3.el7
You could try using --skip-broken to work around the problem
** Found 4 pre-existing rpmdb problem(s), ā€˜yum checkā€™ output follows:

@katello after upgrading to Katello 3.3/Foreman 1.14 I am seeing this error

pulp: gofer.messaging.adapter.connect:ERROR: connect: qpid+ssl://hostname:5671, failed: [Errno 111] Connection refused

Hi,

I assume you did a ā€œyum update -yā€ against all repositories? Could you provide your /etc/qpid-dispatch/qdrouterd.conf? You can mask servernames and the like I you want to.
There has been a major version upgrade to qpid from qpid that requires a new config file syntax in the past, but old foreman-installer commands are not able to produce that config version. This caused a lot of trouble back in the day, though I am currently unable to find any related thread.
If this is the problem, I can provide you with a working config template that should fix the issue going forward until you reach a release that can handle the new qdroter config.