Upgrade to Katello 4.17 - katello:correct_repositories fails

Support
1

Problem:
I have just upgraded our katello server from 4.15 → 4.16 → 4.17

The foreman-installer step produces this error:

2025-06-11 10:28:49 [DEBUG ] [root] Upgrade Step 1/2: katello:correct_repositories. This may take a long while.
2025-06-11 10:28:49 [DEBUG ] [root] Processing Repository 1/962: BaseOS x86_64 os (1)
2025-06-11 10:28:49 [DEBUG ] [root] Failed upgrade task: katello:correct_repositories, see logs for more information.

There is no other information in the log about the error. The system seems to be running fine but i’m concerned about future effects of this error and potential future upgrade problems

Expected outcome:

foreman-installer completes without errors

Foreman and Proxy versions:

Foreman and Proxy plugin versions:
Foreman: 3.15.0
Katello: 4.17.0

Distribution and version:

Other relevant data:
Running foreman-rake katello:correct_repositories manually gives this error:

All operations will be skipped.  Re-run with COMMIT=true to perform corrections.
Processing Repository 1/962: BaseOS x86_64 os (1)
rake aborted!
NoMethodError: undefined method `code' for #<ArgumentError: package_signing_fingerprint cannot be nil>
/usr/share/gems/gems/katello-4.17.0/lib/katello/tasks/repository.rake:101:in `rescue in repo_exists?'
/usr/share/gems/gems/katello-4.17.0/lib/katello/tasks/repository.rake:91:in `repo_exists?'
/usr/share/gems/gems/katello-4.17.0/lib/katello/tasks/repository.rake:41:in `block (3 levels) in <top (required)>'
/usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/relation/batches.rb:71:in `each'
/usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/relation/batches.rb:71:in `block in find_each'
/usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/relation/batches.rb:138:in `block in find_in_batches'
/usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/relation/batches.rb:245:in `block in in_batches'
/usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/relation/batches.rb:229:in `loop'
/usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/relation/batches.rb:229:in `in_batches'
/usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/relation/batches.rb:137:in `find_in_batches'
/usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/relation/batches.rb:70:in `find_each'
/usr/share/gems/gems/katello-4.17.0/lib/katello/tasks/repository.rake:39:in `with_index'
/usr/share/gems/gems/katello-4.17.0/lib/katello/tasks/repository.rake:39:in `block (2 levels) in <top (required)>'
/usr/share/gems/gems/rake-13.0.3/exe/rake:27:in `<top (required)>'

Caused by:
ArgumentError: package_signing_fingerprint cannot be nil
/usr/share/gems/gems/pulp_rpm_client-3.29.2/lib/pulp_rpm_client/models/rpm_rpm_repository_response.rb:393:in `package_signing_fingerprint='
/usr/share/gems/gems/pulp_rpm_client-3.29.2/lib/pulp_rpm_client/models/rpm_rpm_repository_response.rb:276:in `initialize'
/usr/share/gems/gems/pulp_rpm_client-3.29.2/lib/pulp_rpm_client/models/rpm_rpm_repository_response.rb:509:in `new'
/usr/share/gems/gems/pulp_rpm_client-3.29.2/lib/pulp_rpm_client/models/rpm_rpm_repository_response.rb:509:in `build_from_hash'
/usr/share/gems/gems/pulp_rpm_client-3.29.2/lib/pulp_rpm_client/api_client.rb:328:in `convert_to_type'
/usr/share/gems/gems/pulp_rpm_client-3.29.2/lib/pulp_rpm_client/api_client.rb:288:in `deserialize'
/usr/share/gems/gems/pulp_rpm_client-3.29.2/lib/pulp_rpm_client/api_client.rb:86:in `call_api'
/usr/share/gems/gems/pulp_rpm_client-3.29.2/lib/pulp_rpm_client/api/repositories_rpm_api.rb:726:in `read_with_http_info'
/usr/share/gems/gems/pulp_rpm_client-3.29.2/lib/pulp_rpm_client/api/repositories_rpm_api.rb:672:in `read'
/usr/share/gems/gems/katello-4.17.0/lib/katello/tasks/repository.rake:95:in `repo_exists?'
/usr/share/gems/gems/katello-4.17.0/lib/katello/tasks/repository.rake:41:in `block (3 levels) in <top (required)>'
/usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/relation/batches.rb:71:in `each'
/usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/relation/batches.rb:71:in `block in find_each'
/usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/relation/batches.rb:138:in `block in find_in_batches'
/usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/relation/batches.rb:245:in `block in in_batches'
/usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/relation/batches.rb:229:in `loop'
/usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/relation/batches.rb:229:in `in_batches'
/usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/relation/batches.rb:137:in `find_in_batches'
/usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/relation/batches.rb:70:in `find_each'
/usr/share/gems/gems/katello-4.17.0/lib/katello/tasks/repository.rake:39:in `with_index'
/usr/share/gems/gems/katello-4.17.0/lib/katello/tasks/repository.rake:39:in `block (2 levels) in <top (required)>'
/usr/share/gems/gems/rake-13.0.3/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:correct_repositories
2

I do have the same issue after upgrade to 4.17.
Also I get this when publishing a content view: “package_signing_fingerprint cannot be nil”
and the publish task is State “paused” with a result of “error”

3

also affected by the same issue

4

We’ll be asking the Pulp team about this issue today.

5

Were services stopped before the upgrade? I wonder if the old Pulp is still running, which could be the case if services weren’t restarted.

For someone who is still hitting this issue, can you please show the output of:

sudo pulp status
6

Some users have seen issues with the katello 4.17 upgrade issue reported here: Latest Katello 4.17.0 release packages

Worthwhile to check the upgraded katello version as well.

7

I have followed the instructions from here, so I suppose the services were stopped
https://docs.theforeman.org/3.15/Upgrading_Project/index-katello.html

I have installed the RC1 packages first, then ‘downgraded’ to the release version

8

redid the upgrade now and ensured the services are stopped.

{
  "versions": [
    {
      "component": "core",
      "version": "3.73.9",
      "package": "pulpcore",
      "module": "pulpcore.app",
      "domain_compatible": true
    },
    {
      "component": "certguard",
      "version": "3.73.9",
      "package": "pulpcore",
      "module": "pulp_certguard.app",
      "domain_compatible": true
    },
    {
      "component": "file",
      "version": "3.73.9",
      "package": "pulpcore",
      "module": "pulp_file.app",
      "domain_compatible": true
    },
    {
      "component": "ansible",
      "version": "0.24.6",
      "package": "pulp-ansible",
      "module": "pulp_ansible.app",
      "domain_compatible": false
    },
    {
      "component": "container",
      "version": "2.24.2",
      "package": "pulp-container",
      "module": "pulp_container.app",
      "domain_compatible": true
    },
    {
      "component": "deb",
      "version": "3.5.2",
      "package": "pulp_deb",
      "module": "pulp_deb.app",
      "domain_compatible": false
    },
    {
      "component": "python",
      "version": "3.13.5",
      "package": "pulp-python",
      "module": "pulp_python.app",
      "domain_compatible": true
    },
    {
      "component": "rpm",
      "version": "3.29.2",
      "package": "pulp-rpm",
      "module": "pulp_rpm.app",
      "domain_compatible": true
    }
  ],
  "online_workers": [
    {
      "pulp_href": "/pulp/api/v3/workers/0197632d-95d3-7867-b22b-febd9126cdf6/",
      "prn": "prn:core.worker:0197632d-95d3-7867-b22b-febd9126cdf6",
      "pulp_created": "2025-06-12T08:07:01.332512Z",
      "pulp_last_updated": "2025-06-12T08:07:01.332532Z",
      "name": "7551@fqdn",
      "last_heartbeat": "2025-06-12T08:07:41.407786Z",
      "versions": {
        "deb": "3.5.2",
        "rpm": "3.29.2",
        "core": "3.73.9",
        "file": "3.73.9",
        "python": "3.13.5",
        "ansible": "0.24.6",
        "certguard": "3.73.9",
        "container": "2.24.2"
      },
      "current_task": null
    },
    {
      "pulp_href": "/pulp/api/v3/workers/0197632d-97c2-7859-9fb0-e24615099ddf/",
      "prn": "prn:core.worker:0197632d-97c2-7859-9fb0-e24615099ddf",
      "pulp_created": "2025-06-12T08:07:01.835882Z",
      "pulp_last_updated": "2025-06-12T08:07:01.835903Z",
      "name": "7579@fqdn",
      "last_heartbeat": "2025-06-12T08:07:41.903037Z",
      "versions": {
        "deb": "3.5.2",
        "rpm": "3.29.2",
        "core": "3.73.9",
        "file": "3.73.9",
        "python": "3.13.5",
        "ansible": "0.24.6",
        "certguard": "3.73.9",
        "container": "2.24.2"
      },
      "current_task": null
    },
    {
      "pulp_href": "/pulp/api/v3/workers/0197632d-994f-7a6a-a89d-01805eff208b/",
      "prn": "prn:core.worker:0197632d-994f-7a6a-a89d-01805eff208b",
      "pulp_created": "2025-06-12T08:07:02.224428Z",
      "pulp_last_updated": "2025-06-12T08:07:02.224447Z",
      "name": "7607@fqdn",
      "last_heartbeat": "2025-06-12T08:07:42.291967Z",
      "versions": {
        "deb": "3.5.2",
        "rpm": "3.29.2",
        "core": "3.73.9",
        "file": "3.73.9",
        "python": "3.13.5",
        "ansible": "0.24.6",
        "certguard": "3.73.9",
        "container": "2.24.2"
      },
      "current_task": null
    },
    {
      "pulp_href": "/pulp/api/v3/workers/0197632d-9aa2-72d3-8064-8ea7b213d7dd/",
      "prn": "prn:core.worker:0197632d-9aa2-72d3-8064-8ea7b213d7dd",
      "pulp_created": "2025-06-12T08:07:02.563301Z",
      "pulp_last_updated": "2025-06-12T08:07:02.563319Z",
      "name": "7636@fqdn",
      "last_heartbeat": "2025-06-12T08:07:42.640924Z",
      "versions": {
        "deb": "3.5.2",
        "rpm": "3.29.2",
        "core": "3.73.9",
        "file": "3.73.9",
        "python": "3.13.5",
        "ansible": "0.24.6",
        "certguard": "3.73.9",
        "container": "2.24.2"
      },
      "current_task": null
    }
  ],
  "online_api_apps": [
    {
      "name": "7724@fqdn",
      "last_heartbeat": "2025-06-12T08:07:46.811530Z",
      "versions": {
        "deb": "3.5.2",
        "rpm": "3.29.2",
        "core": "3.73.9",
        "file": "3.73.9",
        "python": "3.13.5",
        "ansible": "0.24.6",
        "certguard": "3.73.9",
        "container": "2.24.2"
      }
    },
    {
      "name": "7730@fqdn",
      "last_heartbeat": "2025-06-12T08:07:46.807515Z",
      "versions": {
        "deb": "3.5.2",
        "rpm": "3.29.2",
        "core": "3.73.9",
        "file": "3.73.9",
        "python": "3.13.5",
        "ansible": "0.24.6",
        "certguard": "3.73.9",
        "container": "2.24.2"
      }
    },
    {
      "name": "7754@fqdn",
      "last_heartbeat": "2025-06-12T08:07:46.811531Z",
      "versions": {
        "deb": "3.5.2",
        "rpm": "3.29.2",
        "core": "3.73.9",
        "file": "3.73.9",
        "python": "3.13.5",
        "ansible": "0.24.6",
        "certguard": "3.73.9",
        "container": "2.24.2"
      }
    },
    {
      "name": "7756@fqdn",
      "last_heartbeat": "2025-06-12T08:07:46.815835Z",
      "versions": {
        "deb": "3.5.2",
        "rpm": "3.29.2",
        "core": "3.73.9",
        "file": "3.73.9",
        "python": "3.13.5",
        "ansible": "0.24.6",
        "certguard": "3.73.9",
        "container": "2.24.2"
      }
    },
    {
      "name": "7757@fqdn",
      "last_heartbeat": "2025-06-12T08:07:46.811693Z",
      "versions": {
        "deb": "3.5.2",
        "rpm": "3.29.2",
        "core": "3.73.9",
        "file": "3.73.9",
        "python": "3.13.5",
        "ansible": "0.24.6",
        "certguard": "3.73.9",
        "container": "2.24.2"
      }
    }
  ],
  "online_content_apps": [
    {
      "name": "7794@fqdn",
      "last_heartbeat": "2025-06-12T08:07:41.673498Z",
      "versions": {
        "deb": "3.5.2",
        "rpm": "3.29.2",
        "core": "3.73.9",
        "file": "3.73.9",
        "python": "3.13.5",
        "ansible": "0.24.6",
        "certguard": "3.73.9",
        "container": "2.24.2"
      }
    },
    {
      "name": "7839@fqdn",
      "last_heartbeat": "2025-06-12T08:07:41.968520Z",
      "versions": {
        "deb": "3.5.2",
        "rpm": "3.29.2",
        "core": "3.73.9",
        "file": "3.73.9",
        "python": "3.13.5",
        "ansible": "0.24.6",
        "certguard": "3.73.9",
        "container": "2.24.2"
      }
    },
    {
      "name": "7796@fqdn",
      "last_heartbeat": "2025-06-12T08:07:41.683751Z",
      "versions": {
        "deb": "3.5.2",
        "rpm": "3.29.2",
        "core": "3.73.9",
        "file": "3.73.9",
        "python": "3.13.5",
        "ansible": "0.24.6",
        "certguard": "3.73.9",
        "container": "2.24.2"
      }
    },
    {
      "name": "7843@fqdn",
      "last_heartbeat": "2025-06-12T08:07:42.026678Z",
      "versions": {
        "deb": "3.5.2",
        "rpm": "3.29.2",
        "core": "3.73.9",
        "file": "3.73.9",
        "python": "3.13.5",
        "ansible": "0.24.6",
        "certguard": "3.73.9",
        "container": "2.24.2"
      }
    },
    {
      "name": "7805@fqdn",
      "last_heartbeat": "2025-06-12T08:07:41.771015Z",
      "versions": {
        "deb": "3.5.2",
        "rpm": "3.29.2",
        "core": "3.73.9",
        "file": "3.73.9",
        "python": "3.13.5",
        "ansible": "0.24.6",
        "certguard": "3.73.9",
        "container": "2.24.2"
      }
    },
    {
      "name": "7846@fqdn",
      "last_heartbeat": "2025-06-12T08:07:42.094558Z",
      "versions": {
        "deb": "3.5.2",
        "rpm": "3.29.2",
        "core": "3.73.9",
        "file": "3.73.9",
        "python": "3.13.5",
        "ansible": "0.24.6",
        "certguard": "3.73.9",
        "container": "2.24.2"
      }
    },
    {
      "name": "7828@fqdn",
      "last_heartbeat": "2025-06-12T08:07:41.832503Z",
      "versions": {
        "deb": "3.5.2",
        "rpm": "3.29.2",
        "core": "3.73.9",
        "file": "3.73.9",
        "python": "3.13.5",
        "ansible": "0.24.6",
        "certguard": "3.73.9",
        "container": "2.24.2"
      }
    },
    {
      "name": "7830@fqdn",
      "last_heartbeat": "2025-06-12T08:07:41.831356Z",
      "versions": {
        "deb": "3.5.2",
        "rpm": "3.29.2",
        "core": "3.73.9",
        "file": "3.73.9",
        "python": "3.13.5",
        "ansible": "0.24.6",
        "certguard": "3.73.9",
        "container": "2.24.2"
      }
    },
    {
      "name": "7835@fqdn",
      "last_heartbeat": "2025-06-12T08:07:41.919724Z",
      "versions": {
        "deb": "3.5.2",
        "rpm": "3.29.2",
        "core": "3.73.9",
        "file": "3.73.9",
        "python": "3.13.5",
        "ansible": "0.24.6",
        "certguard": "3.73.9",
        "container": "2.24.2"
      }
    }
  ],
  "database_connection": {
    "connected": true
  },
  "redis_connection": {
    "connected": true
  },
  "storage": {
    "total": 146680619008,
    "used": 128382910464,
    "free": 18297708544
  },
  "content_settings": {
    "content_origin": "https://fqdn",
    "content_path_prefix": "/pulp/content/"
  },
  "domain_enabled": false
}

installed pulp/katello packages are

katello-certs-tools-2.10.0-1.el9.noarch
katello-client-bootstrap-1.7.9-2.el9.noarch
katello-selinux-5.2.0-1.el9.noarch
katello-repos-4.17.0-1.el9.noarch
katello-common-4.17.0-1.el9.noarch
rubygem-katello-4.17.0-1.el9.noarch
foreman-installer-katello-3.15.0-1.el9.noarch
katello-4.17.0-1.el9.noarch

rubygem-smart_proxy_pulp-3.4.0-1.fm3_13.el9.noarch
rubygem-pulpcore_client-3.73.9-1.el9.noarch
rubygem-pulp_rpm_client-3.29.2-1.el9.noarch
rubygem-pulp_python_client-3.13.5-1.el9.noarch
rubygem-pulp_ostree_client-2.4.8-1.el9.noarch
rubygem-pulp_file_client-3.73.9-1.el9.noarch
rubygem-pulp_deb_client-3.5.2-1.el9.noarch
rubygem-pulp_container_client-2.24.2-2.el9.noarch
rubygem-pulp_certguard_client-3.73.9-1.el9.noarch
rubygem-pulp_ansible_client-0.24.6-1.el9.noarch
python3.12-pulp-glue-0.32.3-1.el9.noarch
python3.12-pulpcore-3.73.9-1.el9.noarch
python3.12-pulp-ansible-0.24.6-1.el9.noarch
python3.12-pulp-container-2.24.2-2.el9.noarch
python3.12-pulp-deb-3.5.2-1.el9.noarch
python3.12-pulp-python-3.13.5-1.el9.noarch
python3.12-pulp-rpm-3.29.2-1.el9.noarch
pulpcore-selinux-2.0.1-2.el9.x86_64
python3.12-pulp-cli-0.32.3-1.el9.noarch
pulpcore-obsolete-packages-1.3.0-1.el9.noarch
9

You forgot to mention if you got the same error again or the upgrade succeeded this time :wink:

10

true, error is still there of course :slight_smile:
Not sure if just a coincidence but I also upgraded from 4.15 (first time directly to 4.17…) and second time with making the proper in between step to 4.16

11

same issue here, i checked if pulp is stopped and it is.

12

i did 4.16 > 4.17 upgrade, it makes no difference.

13

Regarding my previously mentioned error, “package_signing_fingerprint cannot be nil”.
I copied a content view to a new name and published a new version.
Now it went ok. :slightly_smiling_face:

14

Experiencing the issue as well. Upgrade from 4.16->4.17 with services stopped beforehand.

Note that, unless I am mistaken, an observed side effect of continuing to use the upgraded version in this broken state results in NO packages in the repos configured under Katello/Pulp being signed. I am sure there are other issues as well, but it’s probably advised to rollback.

15

Since I upgraded my machine from 4.15 → 4.16 → 4.17 and did not reproduce this issue, I’m wondering if it’s a data issue with the signing fingerprint since it was introduced in Pulp RPM 3.27 Support for signing RPM Packages on upload (previously: Native obs-signd support for signing PULP hosted content) · Issue #2986 · pulp/pulp_rpm · GitHub. The API bindings are what are erroring out, so it could be that an update to the bindings just now makes the issue noticeable.

Anyway, it’s pretty clear this is a bug somewhere between Pulp RPM’s Ruby bindings and Pulp RPM itself, so I’ll file an issue.

4 Likes
16

hi the same problem

foreman-rake katello:correct_repositories
Rubocop not loaded.
All operations will be skipped.  Re-run with COMMIT=true to perform corrections.
Processing Repository 1/251: Docker_stable (38)
rake aborted!
NoMethodError: undefined method `code' for #<ArgumentError: package_signing_fingerprint cannot be nil>
/usr/share/gems/gems/katello-4.17.0/lib/katello/tasks/repository.rake:101:in `rescue in repo_exists?'
/usr/share/gems/gems/katello-4.17.0/lib/katello/tasks/repository.rake:91:in `repo_exists?'
/usr/share/gems/gems/katello-4.17.0/lib/katello/tasks/repository.rake:41:in `block (3 levels) in <top (required)>'
/usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/relation/batches.rb:71:in `each'
/usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/relation/batches.rb:71:in `block in find_each'
/usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/relation/batches.rb:138:in `block in find_in_batches'
/usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/relation/batches.rb:245:in `block in in_batches'
/usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/relation/batches.rb:229:in `loop'
/usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/relation/batches.rb:229:in `in_batches'
/usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/relation/batches.rb:137:in `find_in_batches'
/usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/relation/batches.rb:70:in `find_each'
/usr/share/gems/gems/katello-4.17.0/lib/katello/tasks/repository.rake:39:in `with_index'
/usr/share/gems/gems/katello-4.17.0/lib/katello/tasks/repository.rake:39:in `block (2 levels) in <top (required)>'
/usr/share/gems/gems/rake-13.0.3/exe/rake:27:in `<top (required)>'

Caused by:
ArgumentError: package_signing_fingerprint cannot be nil
/usr/share/gems/gems/pulp_rpm_client-3.29.2/lib/pulp_rpm_client/models/rpm_rpm_repository_response.rb:393:in `package_signing_fingerprint='
/usr/share/gems/gems/pulp_rpm_client-3.29.2/lib/pulp_rpm_client/models/rpm_rpm_repository_response.rb:276:in `initialize'
/usr/share/gems/gems/pulp_rpm_client-3.29.2/lib/pulp_rpm_client/models/rpm_rpm_repository_response.rb:509:in `new'
/usr/share/gems/gems/pulp_rpm_client-3.29.2/lib/pulp_rpm_client/models/rpm_rpm_repository_response.rb:509:in `build_from_hash'
/usr/share/gems/gems/pulp_rpm_client-3.29.2/lib/pulp_rpm_client/api_client.rb:328:in `convert_to_type'
/usr/share/gems/gems/pulp_rpm_client-3.29.2/lib/pulp_rpm_client/api_client.rb:288:in `deserialize'
/usr/share/gems/gems/pulp_rpm_client-3.29.2/lib/pulp_rpm_client/api_client.rb:86:in `call_api'
/usr/share/gems/gems/pulp_rpm_client-3.29.2/lib/pulp_rpm_client/api/repositories_rpm_api.rb:726:in `read_with_http_info'
/usr/share/gems/gems/pulp_rpm_client-3.29.2/lib/pulp_rpm_client/api/repositories_rpm_api.rb:672:in `read'
/usr/share/gems/gems/katello-4.17.0/lib/katello/tasks/repository.rake:95:in `repo_exists?'
/usr/share/gems/gems/katello-4.17.0/lib/katello/tasks/repository.rake:41:in `block (3 levels) in <top (required)>'
/usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/relation/batches.rb:71:in `each'
/usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/relation/batches.rb:71:in `block in find_each'
/usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/relation/batches.rb:138:in `block in find_in_batches'
/usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/relation/batches.rb:245:in `block in in_batches'
/usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/relation/batches.rb:229:in `loop'
/usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/relation/batches.rb:229:in `in_batches'
/usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/relation/batches.rb:137:in `find_in_batches'
/usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/relation/batches.rb:70:in `find_each'
/usr/share/gems/gems/katello-4.17.0/lib/katello/tasks/repository.rake:39:in `with_index'
/usr/share/gems/gems/katello-4.17.0/lib/katello/tasks/repository.rake:39:in `block (2 levels) in <top (required)>'
/usr/share/gems/gems/rake-13.0.3/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:correct_repositories
(See full trace by running task with --trace)


foreman-rake upgrade:run
Rubocop not loaded.
=============================================
Upgrade Step 1/2: katello:correct_repositories. This may take a long while.
Processing Repository 1/251: Docker_stable (38)
Failed upgrade task: katello:correct_repositories, see logs for more information.
=============================================
Upgrade Step 2/2: katello:clean_backend_objects. This may take a long while.
0 orphaned consumer id(s) found in candlepin.
Candlepin orphaned consumers: []

katello version: katello-4.17.0-1.el9.noarch

17

Hello,

Same problem after upgrade to katello 4.17:

foreman-rake katello:correct_repositories
All operations will be skipped.  Re-run with COMMIT=true to perform corrections.
Processing Repository 1/313: rockylinux8-appstream-x86_64 (77)
rake aborted!
NoMethodError: undefined method `code' for #<ArgumentError: package_signing_fingerprint cannot be nil>
/usr/share/gems/gems/katello-4.17.0/lib/katello/tasks/repository.rake:101:in `rescue in repo_exists?'
/usr/share/gems/gems/katello-4.17.0/lib/katello/tasks/repository.rake:91:in `repo_exists?'
/usr/share/gems/gems/katello-4.17.0/lib/katello/tasks/repository.rake:41:in `block (3 levels) in <top (required)>'
/usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/relation/batches.rb:71:in `each'
/usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/relation/batches.rb:71:in `block in find_each'
/usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/relation/batches.rb:138:in `block in find_in_batches'
/usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/relation/batches.rb:245:in `block in in_batches'
/usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/relation/batches.rb:229:in `loop'
/usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/relation/batches.rb:229:in `in_batches'
/usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/relation/batches.rb:137:in `find_in_batches'
/usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/relation/batches.rb:70:in `find_each'
/usr/share/gems/gems/katello-4.17.0/lib/katello/tasks/repository.rake:39:in `with_index'
/usr/share/gems/gems/katello-4.17.0/lib/katello/tasks/repository.rake:39:in `block (2 levels) in <top (required)>'
/usr/share/gems/gems/rake-13.0.3/exe/rake:27:in `<top (required)>'

Caused by:
ArgumentError: package_signing_fingerprint cannot be nil
/usr/share/gems/gems/pulp_rpm_client-3.29.2/lib/pulp_rpm_client/models/rpm_rpm_repository_response.rb:393:in `package_signing_fingerprint='
/usr/share/gems/gems/pulp_rpm_client-3.29.2/lib/pulp_rpm_client/models/rpm_rpm_repository_response.rb:276:in `initialize'
/usr/share/gems/gems/pulp_rpm_client-3.29.2/lib/pulp_rpm_client/models/rpm_rpm_repository_response.rb:509:in `new'
/usr/share/gems/gems/pulp_rpm_client-3.29.2/lib/pulp_rpm_client/models/rpm_rpm_repository_response.rb:509:in `build_from_hash'
/usr/share/gems/gems/pulp_rpm_client-3.29.2/lib/pulp_rpm_client/api_client.rb:328:in `convert_to_type'
/usr/share/gems/gems/pulp_rpm_client-3.29.2/lib/pulp_rpm_client/api_client.rb:288:in `deserialize'
/usr/share/gems/gems/pulp_rpm_client-3.29.2/lib/pulp_rpm_client/api_client.rb:86:in `call_api'
/usr/share/gems/gems/pulp_rpm_client-3.29.2/lib/pulp_rpm_client/api/repositories_rpm_api.rb:726:in `read_with_http_info'
/usr/share/gems/gems/pulp_rpm_client-3.29.2/lib/pulp_rpm_client/api/repositories_rpm_api.rb:672:in `read'
/usr/share/gems/gems/katello-4.17.0/lib/katello/tasks/repository.rake:95:in `repo_exists?'
/usr/share/gems/gems/katello-4.17.0/lib/katello/tasks/repository.rake:41:in `block (3 levels) in <top (required)>'
/usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/relation/batches.rb:71:in `each'
/usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/relation/batches.rb:71:in `block in find_each'
/usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/relation/batches.rb:138:in `block in find_in_batches'
/usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/relation/batches.rb:245:in `block in in_batches'
/usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/relation/batches.rb:229:in `loop'
/usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/relation/batches.rb:229:in `in_batches'
/usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/relation/batches.rb:137:in `find_in_batches'
/usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/relation/batches.rb:70:in `find_each'
/usr/share/gems/gems/katello-4.17.0/lib/katello/tasks/repository.rake:39:in `with_index'
/usr/share/gems/gems/katello-4.17.0/lib/katello/tasks/repository.rake:39:in `block (2 levels) in <top (required)>'
/usr/share/gems/gems/rake-13.0.3/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:correct_repositories
(See full trace by running task with --trace)
18

There is a pulp-rpm PR out with the fix: Ensure Repository.package_signing_fingerprint doesnt return None by pedro-psb · Pull Request #3996 · pulp/pulp_rpm · GitHub

The bindings will need to be upgraded in foreman-packaging for users to take advantage.

With how the PR is written, it looks like the Pulpcore DB could potentially be modified manually to have the Repository package_signing_fingerprint field become "" (empty-string) as a hack workaround. I haven’t tested this, so if you try it, please take a backup first.

1 Like
19

Finally also made it back to the upgrade, and yeah same issue,
great that there is already a fix on the way!

20

Applied the fix manually and can confirm it mitigates the migration failure :+1:

3 Likes