Deb package sync seems to be broken since Katello 4.8

Support
1

Problem:

Since the upgrade to 4.8 or one of the minor package updates afterwards I’m getting the following error for all deb repo syncs:

column deb_packageindex.release_id does not exist
LINE 1: ...f_interest", "deb_packageindex"."content_ptr_id", "deb_packa...

Expected outcome:

Deb syncs work normally.

Foreman and Proxy versions:

Foreman 3.6.1
Katello 4.8.0

Foreman and Proxy plugin versions:

Tasks 7.2.1
Ansible 11.2.0
Bootdisk 21.0.4
Puppet 5.1.2
Remote-Execution 9.1.0
Snapshot-Management 2.0.3
Statistics 2.0.1
Templates 9.3.0

Distribution and version:

Rocky Linux 8.7 (this system didn’t get any 8.8 updates so far)

Other relevant data:

And here is the full error from the production log:

2023-05-20T17:10:45 [E|bac|082a985a] column deb_packageindex.release_id does not exist
 082a985a | LINE 1: ...f_interest", "deb_packageindex"."content_ptr_id", "deb_packa...
 082a985a |                                                              ^
 082a985a |  (Katello::Errors::Pulp3Error)
 082a985a | /usr/share/gems/gems/katello-4.8.0/app/lib/actions/pulp3/abstract_async_task.rb:108:in `block in check_for_errors'
 082a985a | /usr/share/gems/gems/katello-4.8.0/app/lib/actions/pulp3/abstract_async_task.rb:106:in `each'
 082a985a | /usr/share/gems/gems/katello-4.8.0/app/lib/actions/pulp3/abstract_async_task.rb:106:in `check_for_errors'
 082a985a | /usr/share/gems/gems/katello-4.8.0/app/lib/actions/pulp3/abstract_async_task.rb:160:in `poll_external_task'
 082a985a | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action/polling.rb:100:in `poll_external_task_with_rescue'
 082a985a | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action/polling.rb:22:in `run'
 082a985a | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action/cancellable.rb:14:in `run'
 082a985a | /usr/share/gems/gems/katello-4.8.0/app/lib/actions/pulp3/abstract_async_task.rb:10:in `run'
 082a985a | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:582:in `block (3 levels) in execute_run'
 082a985a | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 082a985a | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
 082a985a | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:32:in `run'
 082a985a | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
 082a985a | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 082a985a | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
 082a985a | /usr/share/gems/gems/katello-4.8.0/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
 082a985a | /usr/share/gems/gems/katello-4.8.0/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
 082a985a | /usr/share/gems/gems/katello-4.8.0/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
 082a985a | /usr/share/gems/gems/katello-4.8.0/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
 082a985a | /usr/share/gems/gems/katello-4.8.0/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
 082a985a | /usr/share/gems/gems/katello-4.8.0/app/lib/actions/middleware/remote_action.rb:16:in `run'
 082a985a | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
 082a985a | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 082a985a | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
 082a985a | /usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
 082a985a | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/execution_wrapper.rb:91:in `wrap'
 082a985a | /usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
 082a985a | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
 082a985a | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 082a985a | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
 082a985a | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
 082a985a | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action/progress.rb:17:in `run'
 082a985a | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
 082a985a | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 082a985a | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
 082a985a | /usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/load_setting_values.rb:20:in `run'
 082a985a | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
 082a985a | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 082a985a | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
 082a985a | /usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
 082a985a | /usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_request_id.rb:52:in `restore_current_request_id'
 082a985a | /usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
 082a985a | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
 082a985a | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 082a985a | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
 082a985a | /usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
 082a985a | /usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
 082a985a | /usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
 082a985a | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
 082a985a | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 082a985a | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
 082a985a | /usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
 082a985a | /usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
 082a985a | /usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
 082a985a | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
 082a985a | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 082a985a | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
 082a985a | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:32:in `run'
 082a985a | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
 082a985a | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 082a985a | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
 082a985a | /usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
 082a985a | /usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_user.rb:54:in `restore_curent_user'
 082a985a | /usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
 082a985a | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
 082a985a | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/world.rb:31:in `execute'
 082a985a | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:581:in `block (2 levels) in execute_run'
 082a985a | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:580:in `catch'
 082a985a | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:580:in `block in execute_run'
 082a985a | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:483:in `block in with_error_handling'
 082a985a | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:483:in `catch'
 082a985a | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:483:in `with_error_handling'
 082a985a | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:575:in `execute_run'
 082a985a | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:296:in `execute'
 082a985a | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
 082a985a | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
 082a985a | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
 082a985a | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
 082a985a | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
 082a985a | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/director.rb:94:in `execute'
 082a985a | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
 082a985a | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors.rb:18:in `run_user_code'
 082a985a | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
 082a985a | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
 082a985a | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
 082a985a | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
 082a985a | [ sidekiq ]
 082a985a | [ concurrent-ruby ]

If I missed any needed details I can of course hand them in later :slight_smile:

Cheers, Lukas

2

This most likely means that Pulp DB migrations where not properly applied. Normally because foreman-installer did not run successfully after upgrade.

You can check if there are unapplied pulp_deb migrations in particular by running:

PULP_SETTINGS=/etc/pulp/settings.py pulpcore-manager showmigrations deb

If there are you, can fix this by running:

PULP_SETTINGS=/etc/pulp/settings.py pulpcore-manager migrate deb

Assuming the above did not produce any errors, you should also re-run foreman-installer. If you are missing pulp_deb migrations, you are likely missing other steps that should have been performed by foreman-installer. Make sure you chase up any errors reported by foreman-installer and re-run it until it finishes completely.

If none of this solves your issue, please report back, I have one other theory but it seems far less likely than unapplied migrations.

3

Thanks for taking the time!

Hm this was actually my first guess, so I immediately ran foreman-installer after I noticed it.
But that didn’t really help.

And it looks like all migrations have been completed as well:

# PULP_SETTINGS=/etc/pulp/settings.py pulpcore-manager showmigrations deb
deb
 [X] 0001_initial
 [X] 0002_auto_20190905_1000
 [X] 0003_rename_package_field
 [X] 0004_auto_20191014_1722
 [X] 0005_rename_release_file
 [X] 0006_debrepository
 [X] 0007_create_metadata_models
 [X] 0008_debremote_gpgkey
 [X] 0009_apt_release_signing_service
 [X] 0010_debpublication_signing_service
 [X] 0011_rename_models_to_apt
 [X] 0012_auto_20200803_1337
 [X] 0013_aptremote_ignore_missing_package_indices
 [X] 0014_swap_distribution_model
 [X] 0015_add_custom_fields_to_packages
 [X] 0016_switch_to_newer_jsonfield_model
 [X] 0017_allow_longer_string_lists
 [X] 0018_textfield_conversion
 [X] 0019_immutable_metadata_constraints
 [X] 0020_remove_fk_relations_in_packageindex_and_installerfileindex
# PULP_SETTINGS=/etc/pulp/settings.py pulpcore-manager migrate deb
Operations to perform:
  Apply all migrations: deb
Running migrations:
  No migrations to apply.

Another thing I noticed, that might help, the Remove orphans task stopped working again and shows the same error, which might give a bit of a smaller timewindow!
The last one that finished was on the 7th of May, after that on the 12th I did an dnf update (following an foreman-installer) and the next one on the 14th failed, so it must be one of the packages in this update! (good to have the transactions):

rubygem-foreman_ansible     11.1.2-1          -> 11.2.0-1
python39-pulp-container     2.14.3-2          -> 2.13.5-1
python39-pulp-deb           2.20.1-1          -> 2.20.2-1
puppet7-release             7.0.0-11          -> 7.0.0-13
java-11-openjdk             1:11.0.18.0.10-2  -> 1:11.0.19.0.7-1
java-11-openjdk-headless    1:11.0.18.0.10-2  -> 1:11.0.19.0.7-1
java-1.8.0-openjdk          1:1.8.0.362.b09-2 -> 1:1.8.0.372.b07-1
java-1.8.0-openjdk-devel    1:1.8.0.362.b09-2 -> 1:1.8.0.372.b07-1
java-1.8.0-openjdk-headless 1:1.8.0.362.b09-2 -> 1:1.8.0.372.b07-1

I hope this is exactly what you already had in mind!

4

So it looks like you have python39-pulp-deb version 2.20.2-1, which is the version with the fix for the orphans cleanup, and it looks like the migration (number 0020) from that version is applied.

My one other idea what might be causing this is a client bindings incompatibility. Katello 4.8 currently has access to rubygem-pulp_deb_client-2.20.1-1.el8.noarch.rpm, the rubygem-pulp_deb_client-2.20.2-1.el8.noarch.rpm client is in the pipeline for Katello 4.8, but has not been added to the repo yet. However, you can get it from the nightly Katello repo. So try downloading the following onto your Katello host:

http://yum.theforeman.org/katello/nightly/katello/el8/x86_64/rubygem-pulp_deb_client-2.20.2-1.el8.noarch.rpm

Then just install it with dnf installl ./rubygem-pulp_deb_client-2.20.2-1.el8.noarch.rpm, then restart everything with foreman-maintain service restart and see if that fixes your issues. If that solves the problem, I will need to ping the Katello team to put a rush on adding the newer client version.

1 Like
5

Note: The newer client should become available to the Katello 4.8 repository with Katello 4.8.1 release process

1 Like
6

Okay thank you that did the trick!

# dnf update https://yum.theforeman.org/katello/nightly/katello/el8/x86_64/rubygem-pulp_deb_client-2.20.2-1.el8.noarch.rpm

To make sure really nothing is missing I did the full install:

# foreman-maintain service stop
# foreman-installer

Had a look at the log for some more time while running syncs and orphane removals, and yep that’s it that fixed it completely.