Problem:
I have encountered a strange problem trying to upgrade my Foreman server itself from 3.6 to 3.7, in that I’m experiencing GPG Key Failure errors on Packages from the 3.7 Plugin repo, despite having updated the key that is in use:
Package rubygem-et-orbi-1.2.7-1.el8.noarch.rpm is not signed
Package rubygem-fugit-1.8.1-1.el8.noarch.rpm is not signed
Package rubygem-raabro-1.4.0-1.el8.noarch.rpm is not signed
Package rubygem-foreman_openscap-7.0.0-1.fm3_7.el8.noarch.rpm is not signed
Package rubygem-foreman_remote_execution-10.1.1-1.fm3_7.el8.noarch.rpm is not signed
Package rubygem-foreman-tasks-8.1.4-1.fm3_7.el8.noarch.rpm is not signed
Package rubygem-hammer_cli_foreman_remote_execution-0.2.3-1.fm3_7.el8.noarch.rpm is not signed
Package rubygem-hammer_cli_foreman_tasks-0.0.19-1.fm3_7.el8.noarch.rpm is not signed
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.
Error: GPG check FAILED
First, a little background. I have a possibly slightly unusual setup in that I have 2 Foreman servers, Prod and PreProd, and they receive updates from each other. Prroducts and Content Views on both are synced at the same time monthly, I update PreProd from the repos on Prod, perform some tests, and then update Prod using the same content/package versions as Prod (for my use case I want to keep package versions consistent through my estate and it is less overhead that deploying an additional proxy purely for the purposes of updating the Foreman box itself). It works fine though.
Foreman 3.6 has recently reached EOL, so a week or two back I incremented the Foreman and Katello repos to 3.7/4.9, generated a fresh Sync of the Foreman repos and a fresh Content View. I then ran the upgrade as per the process in the Foreman documentation. During my initial update attempt I encountered a GPG Key error and discovered that the Foreman 3.7 repos use a different GPG Key, so I updated my Foreman GPG Key (used by all my Foreman repos) in my Credentials, continued the process and all went well.
Post upgrade I ran fresh set of tests (Product Sync, Content View generation, updated a “PreProd” test box, Promoted the Content View and upgraded a “Prod” test box. All fine.
As a final step, within 2 hours of starting, I synced the Products on the PreProd box, took a fresh Content View, and updated the Foreman GPG Key so everything was ready to start on my Foreman Prod box. It’s possible a minor package version slipped in but likely not given the time of day and I don’t think it’s relevant to the issue at hand anyway.
Today I began the upgrade process on my Prod box, after running “dnf update” I approved the new GPG key, and immediately got the above error. All my Foreman Repos use the same GPG key, but it only seems to be the packages from the 3.7 Plugins repo that are failing.
I’ve tried most of the following, multiple times:
- Confirming no config issues with the Product repos
- Checking/reading the GPG key in the Credentials on the Foreman box
- I’ve confirmed using the direct URL in a browser that it matches the key currently on the Foreman Repos
- Unregistering and reregistering the client box
- Generating a completely fresh Product Sync and Content View
- I ran a ‘dnf clean all’ before every update attempt
I’ve checked forums and I can’t see anybody else reporting similar problems, and I think it’s highly unlikely if there was an integrity problem repo end that it wouldn’t have been flagged in this timeframe.
The only way I can get around it and allow the upgrade to succeed is by disabling the GPG check on the Plugins repo, which obviously isn’t great from a security perspective, and I’m concerned that this will become a long running problem/persist into future major upgrades.
I can’t see what’s different either in terms of config, package versions or my process between the to upgrades.
Can anybody suggest what I can try next to try and resolve this error?
Expected outcome:
Successful upgrade of Foreman/Katello from 3.6/4.8 > Foreman/Katello 3.7/4.9.
Foreman and Proxy versions:
Current version: Foreman- 3.6.2-1.el8
Target version: Foreman 3.7.1-1.el8
Distribution and version:
Alma 8
