Katello Upgrade failed with pulp migration issue

Problem:
I trying to upgrade Foreman 3.5.3 with Katello 4.7.4 to Foreman 3.6.1 with Katello 4.8.1 but foreman-installer failed when pulp migration happens.

I used the upgrade guide from:
https://theforeman.org/manuals/3.6/index.html#3.6Upgradeto3.6
https://docs.theforeman.org/3.6/Upgrading_and_Updating/index-katello.html

Expected outcome:
Running foreman-installer without errors.

Foreman and Proxy versions:

Installed Packages

  • ansible-collection-theforeman-foreman-3.10.0-1.el8.noarch
  • candlepin-4.2.13-1.el8.noarch
  • candlepin-selinux-4.2.13-1.el8.noarch
  • -foreman-client-1.0-1.noarch
  • -foreman-proxy-1.0-1.noarch
  • -foreman-proxy-client-1.0-1.noarch
  • foreman-3.6.1-1.el8.noarch
  • foreman-cli-3.6.1-1.el8.noarch
  • foreman-debug-3.6.1-1.el8.noarch
  • foreman-dynflow-sidekiq-3.6.1-1.el8.noarch
  • foreman-ec2-3.6.1-1.el8.noarch
  • foreman-installer-3.6.1-1.el8.noarch
  • foreman-installer-katello-3.6.1-1.el8.noarch
  • foreman-ovirt-3.6.1-1.el8.noarch
  • foreman-postgresql-3.6.1-1.el8.noarch
  • foreman-proxy-3.6.1-1.el8.noarch
  • foreman-release-3.6.1-1.el8.noarch
  • foreman-selinux-3.6.1-1.el8.noarch
  • foreman-service-3.6.1-1.el8.noarch
  • foreman-vmware-3.6.1-1.el8.noarch
  • katello-4.8.1-1.el8.noarch
  • katello-certs-tools-2.9.0-1.el8.noarch
  • katello-client-bootstrap-1.7.9-1.el8.noarch
  • katello-common-4.8.1-1.el8.noarch
  • katello-debug-4.8.1-1.el8.noarch
  • katello-default-ca-1.0-1.noarch
  • katello-repos-4.8.1-1.el8.noarch
  • katello-selinux-4.0.2-3.el8.noarch
  • katello-server-ca-1.0-1.noarch
  • pulp-client-1.0-1.noarch
  • pulpcore-selinux-1.3.2-1.el8.x86_64
  • python39-pulp-ansible-0.16.0-1.el8.noarch
  • python39-pulp-certguard-1.5.6-1.el8.noarch
  • python39-pulp-cli-0.14.0-4.el8.noarch
  • python39-pulp-container-2.14.5-1.el8.noarch
  • python39-pulp-deb-2.20.2-1.el8.noarch
  • python39-pulp-file-1.12.0-1.el8.noarch
  • python39-pulp-python-3.8.0-1.el8.noarch
  • python39-pulp-rpm-3.19.2-1.el8.noarch
  • python39-pulpcore-3.22.2-4.el8.noarch
  • qpid-proton-c-0.37.0-1.el8.x86_64
  • rubygem-foreman-tasks-7.2.1-2.fm3_6.el8.noarch
  • rubygem-foreman-tasks-core-0.3.6-1.fm2_5.el8.noarch
  • rubygem-foreman_ansible-11.2.0-1.fm3_6.el8.noarch
  • rubygem-foreman_ansible_core-4.1.2-1.fm2_5.el8.noarch
  • rubygem-foreman_azure_rm-2.2.9-1.fm3_6.el8.noarch
  • rubygem-foreman_bootdisk-21.0.4-1.fm3_6.el8.noarch
  • rubygem-foreman_fog_proxmox-0.14.0-4.fm3_5.el8.noarch
  • rubygem-foreman_maintain-1.3.0-1.el8.noarch
  • rubygem-foreman_openscap-6.0.0-1.fm3_6.el8.noarch
  • rubygem-foreman_puppet-5.1.2-1.fm3_6.el8.noarch
  • rubygem-foreman_remote_execution-9.1.0-1.fm3_6.el8.noarch
  • rubygem-foreman_remote_execution-cockpit-9.1.0-1.fm3_6.el8.noarch
  • rubygem-foreman_remote_execution_core-1.4.8-1.el8.noarch
  • rubygem-foreman_statistics-2.0.1-4.fm3_5.el8.noarch
  • rubygem-foreman_templates-9.3.0-2.fm3_5.el8.noarch
  • rubygem-foreman_virt_who_configure-0.5.13-1.fm3_6.el8.noarch
  • rubygem-foreman_webhooks-3.0.5-1.fm3_5.el8.noarch
  • rubygem-hammer_cli-3.6.0-1.el8.noarch
  • rubygem-hammer_cli_foreman-3.6.0-1.el8.noarch
  • rubygem-hammer_cli_foreman_puppet-0.0.6-1.fm3_4.el8.noarch
  • rubygem-hammer_cli_foreman_remote_execution-0.2.2-1.fm3_0.el8.noarch
  • rubygem-hammer_cli_foreman_tasks-0.0.18-1.fm3_5.el8.noarch
  • rubygem-hammer_cli_katello-1.8.1-1.el8.noarch
  • rubygem-katello-4.8.1-1.el8.noarch
  • rubygem-pulp_ansible_client-0.16.0-1.el8.noarch
  • rubygem-pulp_certguard_client-1.5.7-1.el8.noarch
  • rubygem-pulp_container_client-2.14.3-1.el8.noarch
  • rubygem-pulp_deb_client-2.20.2-1.el8.noarch
  • rubygem-pulp_file_client-1.12.0-1.el8.noarch
  • rubygem-pulp_ostree_client-2.0.0-0.1.a1.el8.noarch
  • rubygem-pulp_python_client-3.8.0-1.el8.noarch
  • rubygem-pulp_rpm_client-3.19.0-1.el8.noarch
  • rubygem-pulpcore_client-3.22.2-1.el8.noarch
  • rubygem-qpid_proton-0.37.0-1.el8.x86_64
  • rubygem-smart_proxy_pulp-3.2.0-3.fm3_3.el8.noarch

Distribution and version:
AlmaLinux 8.8

Other relevant data:

I checked katello.log and found issues with pulp upgrade, i ran pulp upgrade again to get the log for this report.

[root@fm /]# sudo -u pulp PULP_SETTINGS=‘/etc/pulp/settings.py’ pulpcore-manager migrate
Operations to perform:
Apply all migrations: ansible, auth, certguard, container, contenttypes, core, deb, file, python, rpm, sessions
Running migrations:
Applying rpm.0048_artifacts_dependencies_fix…Traceback (most recent call last):
File “/bin/pulpcore-manager”, line 33, in
sys.exit(load_entry_point(‘pulpcore==3.22.2’, ‘console_scripts’, ‘pulpcore-manager’)())
File “/usr/lib/python3.9/site-packages/pulpcore/app/manage.py”, line 11, in manage
execute_from_command_line(sys.argv)
File “/usr/lib/python3.9/site-packages/django/core/management/init.py”, line 419, in execute_from_command_line
utility.execute()
File “/usr/lib/python3.9/site-packages/django/core/management/init.py”, line 413, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File “/usr/lib/python3.9/site-packages/django/core/management/base.py”, line 354, in run_from_argv
self.execute(*args, **cmd_options)
File “/usr/lib/python3.9/site-packages/django/core/management/base.py”, line 398, in execute
output = self.handle(*args, **options)
File “/usr/lib/python3.9/site-packages/django/core/management/base.py”, line 89, in wrapped
res = handle_func(*args, **kwargs)
File “/usr/lib/python3.9/site-packages/django/core/management/commands/migrate.py”, line 244, in handle
post_migrate_state = executor.migrate(
File “/usr/lib/python3.9/site-packages/django/db/migrations/executor.py”, line 117, in migrate
state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
File “/usr/lib/python3.9/site-packages/django/db/migrations/executor.py”, line 147, in _migrate_all_forwards
state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
File “/usr/lib/python3.9/site-packages/django/db/migrations/executor.py”, line 227, in apply_migration
state = migration.apply(state, schema_editor)
File “/usr/lib/python3.9/site-packages/django/db/migrations/migration.py”, line 126, in apply
operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
File “/usr/lib/python3.9/site-packages/django/db/migrations/operations/special.py”, line 190, in database_forwards
self.code(from_state.apps, schema_editor)
File “/usr/lib/python3.9/site-packages/pulp_rpm/app/migrations/0048_artifacts_dependencies_fix.py”, line 31, in fixup_modulemd_artifacts_dependencies
mmd.artifacts = modulemd[“data”].get(“artifacts”, {}).get(“rpms”, )
TypeError: ‘NoneType’ object is not subscriptable

Anyone can help me solve this issue?

@sjansen I know it’s not easy to understand what is the root cause whether pulp or katello, but in case you see in the traceback pulp code, most likely it’s a pulp issue. In that case, please rather open an issue directly in Pulp tracker Issues · pulp/pulp_rpm · GitHub or start a pulp discourse thread Support - Pulp Community
FYI there is already the migration issue filed Upgrade fails on rpm.0048_artifacts_dependencies_fix migration with TypeError: 'NoneType' object is not subscriptable · Issue #3177 · pulp/pulp_rpm · GitHub

1 Like

@sjansen Have you at any point used import / export functionality, or deleted any contents of /var/lib/pulp/artifacts/?

Hello @dralley

do you mean the import function to import rpm packages into rpm repos? yes i used it a few times when smaller uptream repos had temporary issues and i had to update and important package. I never touched any pulp folders. Sometimes i use the reclaim space function in the smart proxies overview to get rid of old files after an point release upgrade of the OS (all repos run in on demand download).

How many repos do you have synced? If it’s just a few can you list them?

There are a few repos on my installation :wink:

-----|---------------------------|-----------------------|--------------|---------------------------------------------------------------------------------

ID NAME PRODUCT CONTENT TYPE URL
187 al8_appstream al8 yum Index of /almalinux/8/AppStream/x86_64/os/
186 al8_baseos al8 yum Index of /almalinux/8/BaseOS/x86_64/os/
188 al8_extras al8 yum Index of /almalinux/8/extras/x86_64/os/
493 al8_foreman_client al8 yum Index of /client/3.5/el8/x86_64
190 al8_highavailability al8 yum Index of /almalinux/8/HighAvailability/x86_64/os/
1651 al8_plus al8 yum Index of /almalinux/8/plus/x86_64/os/
189 al8_powertools al8 yum Index of /almalinux/8/PowerTools/x86_64/os/
1650 al8_resilientstorage al8 yum Index of /almalinux/8/ResilientStorage/x86_64/os/
6086 al8_sap al8 yum Index of /almalinux/8/SAP/x86_64/os/
6087 al8_saphana al8 yum Index of /almalinux/8/SAPHANA/x86_64/os/
3671 al9_appstream al9 yum Index of /almalinux/9/AppStream/x86_64/os/
3670 al9_baseos al9 yum Index of /almalinux/9/BaseOS/x86_64/os/
3689 al9_crb al9 yum Index of /almalinux/9/CRB/x86_64/os/
3672 al9_extras al9 yum Index of /almalinux/9/extras/x86_64/os/
4363 al9_foreman_client al9 yum Index of /client/3.5/el9/x86_64
3673 al9_highavailability al9 yum Index of /almalinux/9/HighAvailability/x86_64/os/
3678 al9_kickstart al9 yum Index of /almalinux/9/BaseOS/x86_64/kickstart/
3675 al9_plus al9 yum Index of /almalinux/9/plus/x86_64/os/
3677 al9_resilientstorage al9 yum Index of /almalinux/9/ResilientStorage/x86_64/os/
3687 al9_rt al9 yum Index of /almalinux/9/RT/x86_64/os/
3691 al9_sap al9 yum Index of /almalinux/9/SAP/x86_64/os/
3690 al9_saphana al9 yum Index of /almalinux/9/SAPHANA/x86_64/os/
3538 el8_caddy Caddy yum Index of /results/@caddy/caddy/epel-8-x86_64/
3505 el8_crowdsec CrowdSec yum https://packagecloud.io/crowdsec/crowdsec/el/8/x86_64/
9 el8_docker Docker yum Index of linux/centos/8/x86_64/stable/
3572 el8_envoy Envoy yum https://rpm.dl.getenvoy.io/public/rpm/el/8/x86_64/
393 el8_epel EPEL yum Index of /pub/epel/8/Everything/x86_64
16 el8_harbottle harbottle yum Index of /results/harbottle/main/epel-8-x86_64/
1021 el8_hashicorp_stable HashiCorp yum https://rpm.releases.hashicorp.com/RHEL/8/x86_64/stable
14 el8_influxdata influxdata yum InfluxData - Package Repository
17 el8_msprod Microsoft Prod yum Index of /rhel/8/prod/
18 el8_mssql_2019 Microsoft SQL Server yum Index of /rhel/8/mssql-server-2019/
19 el8_netdata netdata yum https://packagecloud.io/netdata/netdata/el/8/x86_64/
394 el8_ocsng OCS Inventory NG yum Index of /enterprise/8/x86_64
4984 el8_tailscale Tailscale yum https://pkgs.tailscale.com/stable/rhel/8/x86_64
395 el8_trivy Aqua Security - Trivy yum https://aquasecurity.github.io/trivy-repo/rpm/releases/8/x86_64/
22 el8_zabbix Zabbix yum Zabbix Official Repository
23 el8_zerotier Zerotier yum Index of /redhat/el/8/
3686 el9_caddy Caddy yum Index of /results/@caddy/caddy/epel-9-x86_64/
3685 el9_crowdsec CrowdSec yum https://packagecloud.io/crowdsec/crowdsec/el/9/x86_64/
3684 el9_docker Docker yum Index of linux/centos/9/x86_64/stable/
3683 el9_envoy Envoy yum https://rpm.dl.getenvoy.io/public/rpm/el/9/x86_64/
3682 el9_epel EPEL yum Index of /pub/epel/9/Everything/x86_64
3681 el9_hashicorp_stable HashiCorp yum https://rpm.releases.hashicorp.com/RHEL/9/x86_64/stable
8464 el9_headscale Headscale yum https://download.copr.fedorainfracloud.org/results/jonathanspw/headscale/rhel
3680 el9_influxdata influxdata yum InfluxData - Package Repository
3679 el9_msprod Microsoft Prod yum Index of /rhel/9/prod/
4508 el9_netdata netdata yum https://packagecloud.io/netdata/netdata/el/9/x86_64/
3688 el9_nfv al9 yum Index of /almalinux/9/NFV/x86_64/os/
7965 el9_ocsng OCS Inventory NG yum Index of /enterprise/9/x86_64
4985 el9_tailscale Tailscale yum https://pkgs.tailscale.com/stable/rhel/9/x86_64
7296 el9_trivy Aqua Security - Trivy yum https://aquasecurity.github.io/trivy-repo/rpm/releases/9/x86_64/
4510 el9_zabbix Zabbix yum https://repo.zabbix.com/zabbix/5.0/rhel/9/x86_64/
4509 el9_zerotier Zerotier yum https://download.zerotier.com/redhat/el/9/
12 Grafana Grafana yum https://rpm.grafana.com/
13 lsc_poroject LSC yum https://lsc-project.org/rpm/noarch/
5172 OpenSearch-2.x OpenSearch yum https://artifacts.opensearch.org/releases/bundle/opensearch/2.x/yum
5173 OpenSearch-Dashboards-2.x OpenSearch yum https://artifacts.opensearch.org/releases/bundle/opensearch-dashboards/2.x/yum
----- --------------------------- ----------------------- -------------- ---------------------------------------------------------------------------------

Another upgrade try with newer pulp packages was successful. issue is solved for me.