Docker container sync "NoMethodError: undefined method `repository_href' for nil:NilClass"

Problem:
When upgrading from Foreman 1.24/Katello 3.14 to Forman 2.0/Katello 3.15 and Foreman 2.1/Katello 3.16RC5, existing defined container products/repos fail to sync with the error “NoMethodError: undefined method `repository_href’ for nil:NilClass”. Problem manifested first with Katello upgrade instructions for 3.15, which had stuck tasks after “foreman-maintain advanced procedure run content-prepare” because it claimed that file content and docker content had already been migrated. Upgraded to Katello 3.16RC5, attempted the same content prepare/ content migrate steps, which also resulted in stuck tasks that stated the same thing. At least I could force cancel the tasks. Tried the sync operations with https://mcr.microsoft.com/, https://registry-1.docker.io/, and a private GitLab Container registry, all resulting in the same failure.

Deleting and recreating the Repo with the exact same information works properly. Looks like a migration issue from pulp2 to pulp3, as newly defined repos under pulp3 sync fine.

Expected outcome:
Sync behaviour should work the same as under Katello 3.14

Distribution and version:
CentOS 7.8.2003

Other relevant data:

Installed Packages

ansiblerole-foreman_scap_client-0.0.4-1.el7.noarch
candlepin-3.1.16-1.el7.noarch
candlepin-selinux-3.1.16-1.el7.noarch
foreman-2.1.1-1.el7.noarch
foreman-cli-2.1.1-1.el7.noarch
foreman-client-release-1.24.3-1.el7.noarch
foreman-debug-2.1.1-1.el7.noarch
foreman-dynflow-sidekiq-2.1.1-1.el7.noarch
foreman-installer-2.1.1-2.el7.noarch
foreman-installer-katello-2.1.1-2.el7.noarch
foreman-libvirt-2.1.1-1.el7.noarch
foreman-postgresql-2.1.1-1.el7.noarch
foreman-proxy-2.1.1-1.el7.noarch
foreman-release-2.1.1-1.el7.noarch
foreman-release-scl-7-3.el7.noarch
foreman-selinux-2.1.1-1.el7.noarch
foreman-service-2.1.1-1.el7.noarch
katello-3.16.0-0.4.rc5.1.el7.noarch
katello-certs-tools-2.7.0-1.el7.noarch
katello-client-bootstrap-1.7.5-1.el7.noarch
katello-common-3.16.0-0.4.rc5.1.el7.noarch
katello-debug-3.16.0-0.4.rc5.1.el7.noarch
katello-default-ca-1.0-1.noarch
katello-host-tools-3.5.1-2.el7.noarch
katello-host-tools-fact-plugin-3.5.1-2.el7.noarch
katello-repos-3.16.0-0.4.rc5.1.el7.noarch
katello-selinux-3.4.0-1.el7.noarch
katello-server-ca-1.0-1.noarch
katello-service-3.14.1-1.el7.noarch
pulp-client-1.0-1.noarch
pulp-deb-plugins-1.10.1-1.el7.noarch
pulp-docker-plugins-3.2.6-1.el7.noarch
pulp-katello-1.0.3-1.el7.noarch
pulp-puppet-plugins-2.21.3-1.el7.noarch
pulp-puppet-tools-2.21.3-1.el7.noarch
pulp-rpm-plugins-2.21.3-1.el7.noarch
pulp-selinux-2.21.3-1.el7.noarch
pulp-server-2.21.3-1.el7.noarch
pulpcore-selinux-1.0.0-2.el7.x86_64
python-gofer-qpid-2.12.5-3.el7.noarch
python-pulp-common-2.21.3-1.el7.noarch
python-pulp-deb-common-1.10.1-1.el7.noarch
python-pulp-docker-common-3.2.6-1.el7.noarch
python-pulp-manifest-2.21.3-1.el7.noarch
python-pulp-oid_validation-2.21.3-1.el7.noarch
python-pulp-puppet-common-2.21.3-1.el7.noarch
python-pulp-repoauth-2.21.3-1.el7.noarch
python-pulp-rpm-common-2.21.3-1.el7.noarch
python-pulp-streamer-2.21.3-1.el7.noarch
python2-qpid-1.37.0-4.el7.noarch
python2-qpid-proton-0.31.0-2.el7.x86_64
python2-qpid-qmf-1.39.0-1.el7.x86_64
python3-pulp-2to3-migration-0.2.0-0.1.b3.el7.noarch
python3-pulp-certguard-0.1.0-0.2.rc5.el7.noarch
python3-pulp-container-1.4.1-1.el7.noarch
python3-pulp-file-1.0.1-1.el7.noarch
python3-pulp-rpm-3.4.1-2.el7.noarch
python3-pulpcore-3.4.1-1.el7.noarch
qpid-cpp-client-1.39.0-1.el7.x86_64
qpid-cpp-client-devel-1.39.0-1.el7.x86_64
qpid-cpp-server-1.39.0-1.el7.x86_64
qpid-cpp-server-linearstore-1.39.0-1.el7.x86_64
qpid-dispatch-router-1.12.0-1.el7.x86_64
qpid-proton-c-0.31.0-2.el7.x86_64
qpid-qmf-1.39.0-1.el7.x86_64
qpid-tools-1.39.0-1.el7.noarch
rubygem-foreman_maintain-0.6.9-1.el7.noarch
tfm-rubygem-actioncable-6.0.3.1-1.el7.noarch
tfm-rubygem-actionmailbox-6.0.3.1-1.el7.noarch
tfm-rubygem-actionmailer-6.0.3.1-1.el7.noarch
tfm-rubygem-actionpack-6.0.3.1-1.el7.noarch
tfm-rubygem-actiontext-6.0.3.1-1.el7.noarch
tfm-rubygem-actionview-6.0.3.1-1.el7.noarch
tfm-rubygem-activejob-6.0.3.1-1.el7.noarch
tfm-rubygem-activemodel-6.0.3.1-1.el7.noarch
tfm-rubygem-activerecord-6.0.3.1-1.el7.noarch
tfm-rubygem-activerecord-import-1.0.0-2.el7.noarch
tfm-rubygem-activerecord-session_store-1.1.1-4.el7.noarch
tfm-rubygem-activestorage-6.0.3.1-1.el7.noarch
tfm-rubygem-activesupport-6.0.3.1-1.el7.noarch
tfm-rubygem-addressable-2.6.0-2.el7.noarch
tfm-rubygem-algebrick-0.7.3-7.el7.noarch
tfm-rubygem-amazing_print-1.1.0-1.el7.noarch
tfm-rubygem-ancestry-3.0.7-1.el7.noarch
tfm-rubygem-anemone-0.7.2-17.el7.noarch
tfm-rubygem-angular-rails-templates-1.1.0-1.el7.noarch
tfm-rubygem-ansi-1.5.0-2.el7.noarch
tfm-rubygem-apipie-bindings-0.3.0-2.el7.noarch
tfm-rubygem-apipie-dsl-2.2.2-2.el7.noarch
tfm-rubygem-apipie-params-0.0.5-4.el7.noarch
tfm-rubygem-apipie-rails-0.5.17-3.el7.noarch
tfm-rubygem-arel-9.0.0-4.el7.noarch
tfm-rubygem-audited-4.9.0-3.el7.noarch
tfm-rubygem-awesome_print-1.8.0-5.el7.noarch
tfm-rubygem-bcrypt-3.1.12-3.el7.x86_64
tfm-rubygem-builder-3.2.4-1.el7.noarch
tfm-rubygem-bundler_ext-0.4.1-5.el7.noarch
tfm-rubygem-clamp-1.1.2-6.el7.noarch
tfm-rubygem-coffee-rails-5.0.0-1.el7.noarch
tfm-rubygem-coffee-script-2.4.1-4.el7.noarch
tfm-rubygem-coffee-script-source-1.12.2-4.el7.noarch
tfm-rubygem-concurrent-ruby-1.1.6-2.el7.noarch
tfm-rubygem-concurrent-ruby-edge-0.6.0-2.fm2_1.el7.noarch
tfm-rubygem-connection_pool-2.2.2-2.el7.noarch
tfm-rubygem-crass-1.0.6-1.el7.noarch
tfm-rubygem-css_parser-1.4.7-4.el7.noarch
tfm-rubygem-daemons-1.2.3-6.el7.noarch
tfm-rubygem-deacon-1.0.0-4.el7.noarch
tfm-rubygem-deep_cloneable-3.0.0-3.el7.noarch
tfm-rubygem-deface-1.5.3-2.el7.noarch
tfm-rubygem-domain_name-0.5.20160310-4.el7.noarch
tfm-rubygem-dynflow-1.4.6-1.fm2_1.el7.noarch
tfm-rubygem-erubi-1.9.0-1.el7.noarch
tfm-rubygem-ethon-0.12.0-1.el7.noarch
tfm-rubygem-excon-0.58.0-5.el7.noarch
tfm-rubygem-execjs-2.7.0-4.el7.noarch
tfm-rubygem-facter-2.4.0-7.el7.x86_64
tfm-rubygem-faraday-0.15.4-2.el7.noarch
tfm-rubygem-fast_gettext-1.4.1-4.el7.noarch
tfm-rubygem-ffi-1.12.2-1.el7.x86_64
tfm-rubygem-fog-core-2.1.0-3.el7.noarch
tfm-rubygem-fog-json-1.2.0-3.el7.noarch
tfm-rubygem-fog-libvirt-0.7.0-2.el7.noarch
tfm-rubygem-fog-proxmox-0.12.0-1.el7.noarch
tfm-rubygem-fog-xml-0.1.2-8.el7.noarch
tfm-rubygem-foreman-tasks-2.0.2-1.fm2_1.el7.noarch
tfm-rubygem-foreman-tasks-core-0.3.4-1.fm2_1.el7.noarch
tfm-rubygem-foreman_ansible-5.1.1-1.fm2_1.el7.noarch
tfm-rubygem-foreman_ansible_core-3.0.3-1.fm2_1.el7.noarch
tfm-rubygem-foreman_dhcp_browser-0.0.8-4.fm2_1.el7.noarch
tfm-rubygem-foreman_fog_proxmox-0.11.1-1.fm2_0.el7.noarch
tfm-rubygem-foreman_openscap-4.0.1-1.fm2_1.el7.noarch
tfm-rubygem-foreman_remote_execution-3.3.5-1.fm2_1.el7.noarch
tfm-rubygem-foreman_remote_execution_core-1.3.0-1.el7.noarch
tfm-rubygem-formatador-0.2.1-12.el7.noarch
tfm-rubygem-friendly_id-5.3.0-1.el7.noarch
tfm-rubygem-fx-0.5.0-1.el7.noarch
tfm-rubygem-get_process_mem-0.2.1-4.el7.noarch
tfm-rubygem-gettext_i18n_rails-1.8.0-2.el7.noarch
tfm-rubygem-gitlab-sidekiq-fetcher-0.5.2-2.el7.noarch
tfm-rubygem-globalid-0.4.2-1.el7.noarch
tfm-rubygem-graphql-1.8.14-2.el7.noarch
tfm-rubygem-graphql-batch-0.3.10-2.el7.noarch
tfm-rubygem-gssapi-1.2.0-7.el7.noarch
tfm-rubygem-hammer_cli-2.1.1-1.el7.noarch
tfm-rubygem-hammer_cli_csv-2.3.1-2.el7.noarch
tfm-rubygem-hammer_cli_foreman-2.1.2-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_bootdisk-0.3.0-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_docker-0.0.5-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_openscap-0.1.11-1.fm2_1.el7.noarch
tfm-rubygem-hammer_cli_foreman_tasks-0.0.14-1.fm2_1.el7.noarch
tfm-rubygem-hammer_cli_katello-0.22.2-1.el7.noarch
tfm-rubygem-hashie-3.6.0-2.el7.noarch
tfm-rubygem-highline-1.7.8-5.el7.noarch
tfm-rubygem-http-cookie-1.0.2-4.el7.noarch
tfm-rubygem-i18n-1.8.2-1.el7.noarch
tfm-rubygem-ipaddress-0.8.0-12.el7.noarch
tfm-rubygem-jwt-2.2.1-2.el7.noarch
tfm-rubygem-kafo-4.1.0-3.el7.noarch
tfm-rubygem-kafo_parsers-1.1.0-3.el7.noarch
tfm-rubygem-kafo_wizards-0.0.1-4.el7.noarch
tfm-rubygem-katello-3.16.0-0.16.rc5.1.el7.noarch
tfm-rubygem-ldap_fluff-0.4.7-5.el7.noarch
tfm-rubygem-little-plugger-1.1.4-2.el7.noarch
tfm-rubygem-locale-2.0.9-14.el7.noarch
tfm-rubygem-logging-2.2.2-5.el7.noarch
tfm-rubygem-loofah-2.4.0-1.el7.noarch
tfm-rubygem-mail-2.7.1-1.el7.noarch
tfm-rubygem-marcel-0.3.3-1.el7.noarch
tfm-rubygem-method_source-0.9.2-1.el7.noarch
tfm-rubygem-mime-types-3.2.2-4.el7.noarch
tfm-rubygem-mime-types-data-3.2018.0812-4.el7.noarch
tfm-rubygem-mimemagic-0.3.5-1.el7.noarch
tfm-rubygem-mini_mime-1.0.2-1.el7.noarch
tfm-rubygem-mini_portile2-2.4.0-1.el7.noarch
tfm-rubygem-multi_json-1.14.1-1.el7.noarch
tfm-rubygem-multipart-post-2.0.0-2.el7.noarch
tfm-rubygem-mustermann-1.0.2-4.el7.noarch
tfm-rubygem-net-ldap-0.16.1-2.el7.noarch
tfm-rubygem-net-ping-2.0.1-4.el7.noarch
tfm-rubygem-net-scp-1.2.1-4.el7.noarch
tfm-rubygem-net-ssh-4.2.0-2.el7.noarch
tfm-rubygem-netrc-0.11.0-5.el7.noarch
tfm-rubygem-nio4r-2.5.2-1.el7.x86_64
tfm-rubygem-nokogiri-1.10.9-1.el7.x86_64
tfm-rubygem-oauth-0.5.4-4.el7.noarch
tfm-rubygem-openscap-0.4.9-3.el7.noarch
tfm-rubygem-parse-cron-0.1.4-4.fm2_1.el7.noarch
tfm-rubygem-passenger-4.0.18-10.13.el7.x86_64
tfm-rubygem-passenger-native-4.0.18-10.13.el7.x86_64
tfm-rubygem-passenger-native-libs-4.0.18-10.13.el7.x86_64
tfm-rubygem-pg-1.1.4-3.el7.x86_64
tfm-rubygem-polyglot-0.3.5-2.el7.noarch
tfm-rubygem-powerbar-2.0.1-2.el7.noarch
tfm-rubygem-promise.rb-0.7.4-2.el7.noarch
tfm-rubygem-public_suffix-3.0.3-2.el7.noarch
tfm-rubygem-pulp_2to3_migration_client-0.2.0-0.1.b2.el7.noarch
tfm-rubygem-pulp_ansible_client-0.2.0b13.dev01588546902-1.el7.noarch
tfm-rubygem-pulp_certguard_client-0.1.0rc5-1.el7.noarch
tfm-rubygem-pulp_container_client-1.4.1-1.el7.noarch
tfm-rubygem-pulp_docker_client-4.0.0b6.dev01562331743-1.el7.noarch
tfm-rubygem-pulp_file_client-1.0.1-1.el7.noarch
tfm-rubygem-pulp_rpm_client-3.4.2-1.el7.noarch
tfm-rubygem-pulpcore_client-3.4.1-1.el7.noarch
tfm-rubygem-puma-4.3.3-4.el7.x86_64
tfm-rubygem-qpid_messaging-1.38.0-1.el7.x86_64
tfm-rubygem-rabl-0.14.3-1.el7.noarch
tfm-rubygem-rack-2.2.2-1.el7.noarch
tfm-rubygem-rack-cors-1.0.2-2.el7.noarch
tfm-rubygem-rack-jsonp-1.3.1-9.el7.noarch
tfm-rubygem-rack-protection-2.0.3-4.el7.noarch
tfm-rubygem-rack-test-1.1.0-4.el7.noarch
tfm-rubygem-rails-6.0.3.1-1.el7.noarch
tfm-rubygem-rails-dom-testing-2.0.3-6.el7.noarch
tfm-rubygem-rails-html-sanitizer-1.3.0-1.el7.noarch
tfm-rubygem-rails-i18n-6.0.0-2.el7.noarch
tfm-rubygem-railties-6.0.3.1-1.el7.noarch
tfm-rubygem-rainbow-2.2.1-3.el7.noarch
tfm-rubygem-rake-compiler-1.0.7-3.el7.noarch
tfm-rubygem-rb-inotify-0.9.7-5.el7.noarch
tfm-rubygem-record_tag_helper-1.0.1-3.el7.noarch
tfm-rubygem-redis-4.1.2-2.el7.noarch
tfm-rubygem-responders-3.0.0-3.el7.noarch
tfm-rubygem-rest-client-2.0.2-3.el7.noarch
tfm-rubygem-rkerberos-0.1.5-18.el7.x86_64
tfm-rubygem-roadie-3.4.0-3.el7.noarch
tfm-rubygem-roadie-rails-2.1.1-2.el7.noarch
tfm-rubygem-robotex-1.0.0-21.el7.noarch
tfm-rubygem-rprogram-0.3.2-2.el7.noarch
tfm-rubygem-rsec-0.4.3-4.el7.noarch
tfm-rubygem-ruby-libvirt-0.7.1-1.el7.x86_64
tfm-rubygem-ruby-nmap-0.9.3-3.el7.noarch
tfm-rubygem-ruby2ruby-2.4.2-3.el7.noarch
tfm-rubygem-ruby_parser-3.10.1-3.el7.noarch
tfm-rubygem-rubyipmi-0.10.0-6.el7.noarch
tfm-rubygem-runcible-2.13.1-1.el7.noarch
tfm-rubygem-safemode-1.3.5-3.el7.noarch
tfm-rubygem-scoped_search-4.1.8-1.el7.noarch
tfm-rubygem-secure_headers-6.3.0-2.el7.noarch
tfm-rubygem-sequel-5.7.1-3.el7.noarch
tfm-rubygem-sexp_processor-4.10.0-6.el7.noarch
tfm-rubygem-sidekiq-5.2.7-3.el7.noarch
tfm-rubygem-sinatra-2.0.3-4.el7.noarch
tfm-rubygem-smart_proxy_ansible-3.0.1-6.fm2_1.el7.noarch
tfm-rubygem-smart_proxy_dynflow-0.2.4-6.fm2_1.el7.noarch
tfm-rubygem-smart_proxy_dynflow_core-0.2.5-1.fm2_1.el7.noarch
tfm-rubygem-smart_proxy_openscap-0.7.3-1.fm2_1.el7.noarch
tfm-rubygem-smart_proxy_pulp-2.1.0-3.fm2_1.el7.noarch
tfm-rubygem-smart_proxy_remote_execution_ssh-0.3.0-4.fm2_1.el7.noarch
tfm-rubygem-sprockets-3.7.2-6.el7.noarch
tfm-rubygem-sprockets-rails-3.2.1-6.el7.noarch
tfm-rubygem-sqlite3-1.3.13-6.el7.x86_64
tfm-rubygem-sshkey-1.9.0-4.el7.noarch
tfm-rubygem-statsd-instrument-2.1.4-3.el7.noarch
tfm-rubygem-stomp-1.4.9-1.el7.noarch
tfm-rubygem-thor-1.0.1-2.el7.noarch
tfm-rubygem-thread_safe-0.3.6-5.el7.noarch
tfm-rubygem-tilt-2.0.8-4.el7.noarch
tfm-rubygem-typhoeus-1.3.1-1.el7.noarch
tfm-rubygem-tzinfo-1.2.6-1.el7.noarch
tfm-rubygem-unf-0.1.3-8.el7.noarch
tfm-rubygem-unf_ext-0.0.7.2-3.el7.x86_64
tfm-rubygem-unicode-0.4.4.4-3.el7.x86_64
tfm-rubygem-unicode-display_width-1.0.5-4.el7.noarch
tfm-rubygem-validates_lengths_from_database-0.5.0-7.el7.noarch
tfm-rubygem-webpack-rails-0.9.8-5.el7.noarch
tfm-rubygem-websocket-driver-0.7.1-1.el7.x86_64
tfm-rubygem-websocket-extensions-0.1.4-1.el7.noarch
tfm-rubygem-will_paginate-3.1.7-3.el7.noarch
tfm-rubygem-x-editable-rails-1.5.5-5.el7.noarch
tfm-rubygem-xmlrpc-0.3.0-2.el7.noarch
tfm-rubygem-zeitwerk-2.2.2-1.el7.noarch
tfm-rubygem-zest-0.0.4-1.el7.noarch
tfm-runtime-6.1-1.el7.x86_64

Hi @Mkbrown,

Can you tell me what you see on your main smart proxy under Services -> Pulpcore? (https://<server_name>/smart_proxies/<your_pulp_master>#services). I’m looking for your content types being under Pulpcore. I’m thinking Yum will say it’s overridden to Pulp 2 but File and Docker will not.

Also, did you try the switchover? Or just the prepare procedure?

What was supposed to happen back in Katello 3.15 was that your Pulp 3 would be installed, but the configuration would be set to use Pulp 2 for Docker and File (Yum would be Pulp 2 as well since the Pulp 3 RPM plugin wasn’t installed). I expect there was a bug at some point (perhaps with the installer) and your Pulp 3 was fully switched over before you could do your migration. The migration will not run if it sees that File and Docker are set to use Pulp 3 instead of Pulp 2. If you’re curious, the code to check is here.

I suspect your repositories aren’t syncing because it’s trying to interact with Pulp 2 repos using Pulp 3. If I’m right about Pulp 3 already being in use (let’s confirm this first before trying the steps below), there are a couple ways we could try to fix this:

  1. You can delete all of your old Pulp 2 File and Docker content and re-create it. This is the simplest route from a technical perspective, but if you have lots of content I can see it being a pain.
  2. Or, we can try the migration again. If you’ve already started using Pulp 3, however, we will need to delete all of your newly created Pulp 3 repos (and content views) since the migration isn’t meant to be run after using Pulp 3. We’d then reset the Pulp 3 database with the following:
sudo foreman-maintain service stop --exclude=postgresql
sudo runuser - postgres -c 'dropdb pulpcore'
sudo runuser - postgres -c 'createdb pulpcore'
cd /usr/lib/python3.6/site-packages/pulpcore
sudo -u pulp PULP_SETTINGS='/etc/pulp/settings.py' DJANGO_SETTINGS_MODULE='pulpcore.app.settings' python3-django-admin migrate --no-input
sudo -u pulp PULP_SETTINGS='/etc/pulp/settings.py' DJANGO_SETTINGS_MODULE='pulpcore.app.settings' python3-django-admin reset-admin-password --password password
sudo foreman-maintain service start
  1. (continued from above) After the reset, we would need to switch you back to Pulp 2. To do so, run the following foreman-installer commands:
sudo foreman-installer --katello-use-pulp-2-for-file=true --katello-use-pulp-2-for-docker=true

Then you can try the content-prepare and content-switchover.

One thing to keep in mind: While on Pulp 2, you can keep doing the content prepare procedure to bring content over to Pulp 3. Once you do the switchover, however, you will be on Pulp 3 from then on for Docker and File. The Yum migration to Pulp 3 is still being worked on.

I tried both prepare and switchover. Both stated already migrated,and both got stuck.

Here’s the container statuses. I recreated the Containers product repos, but not the GitLab Custom containers.

I can do either approach (I figured I’d have to recreate them, and this is more of a dev system than production)

Here’s the error and backtrace from a failed sync job
Exception:

NoMethodError: undefined method `repository_href’ for nil:NilClass

Backtrace:

/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.rc5.1/app/services/katello/pulp3/repository.rb:201:in sync' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.rc5.1/app/lib/actions/pulp3/repository/sync.rb:13:in invoke_external_task’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/action/polling.rb:84:in initiate_external_action' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/action/polling.rb:19:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/action/cancellable.rb:14:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.rc5.1/app/lib/actions/pulp3/abstract_async_task.rb:45:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/action.rb:563:in block (3 levels) in execute_run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:32:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.rc5.1/app/lib/actions/middleware/remote_action.rb:16:in block in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.rc5.1/app/lib/actions/middleware/remote_action.rb:40:in block in as_remote_user' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.rc5.1/app/models/katello/concerns/user_extensions.rb:21:in cp_config’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.rc5.1/app/lib/actions/middleware/remote_action.rb:27:in as_cp_user' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.rc5.1/app/lib/actions/middleware/remote_action.rb:39:in as_remote_user’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.rc5.1/app/lib/actions/middleware/remote_action.rb:16:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/rails_executor_wrap.rb:14:in block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/execution_wrapper.rb:88:in wrap’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/rails_executor_wrap.rb:13:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/action/progress.rb:31:in with_progress_calculation' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/action/progress.rb:17:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in block in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_request_id.rb:49:in restore_current_request_id' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in block in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_timezone.rb:44:in restore_curent_timezone' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_user.rb:15:in block in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_user.rb:44:in restore_curent_user' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_user.rb:15:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in block in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in restore_current_taxonomies' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:32:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/world.rb:31:in execute’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/action.rb:562:in block (2 levels) in execute_run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/action.rb:561:in catch’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/action.rb:561:in block in execute_run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/action.rb:475:in block in with_error_handling’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/action.rb:475:in catch' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/action.rb:475:in with_error_handling’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/action.rb:556:in execute_run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/action.rb:285:in execute’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in block (2 levels) in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/execution_plan/steps/abstract.rb:167:in with_meta_calculation’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in block in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in open_action’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/director.rb:68:in execute’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in block (2 levels) in perform' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/executors.rb:18:in run_user_code’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in block in perform' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in with_telemetry’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in perform' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/executors/sidekiq/serialization.rb:27:in perform’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in execute_job' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in block (2 levels) in process’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in block in invoke' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in invoke’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in block in process' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in block (6 levels) in dispatch’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in local' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in block (5 levels) in dispatch’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq.rb:37:in block in <module:Sidekiq>' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in block (4 levels) in dispatch’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in stats' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in block (3 levels) in dispatch’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in block (2 levels) in dispatch’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in global' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in block in dispatch’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in with_context' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in with_job_hash_context’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in process’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in process_one' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in watchdog' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in block in safe_thread’
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context’

Ok, so I see that Docker and File are using Pulp 3 (Pulpcore) right now, which is just as I expected. I can’t say for sure what went wrong with the upgrade to get your server in the situation, I’ll have to try an upgrade myself to see if it’s still a problem.

In your situation, I think the safest route will be to just delete the old Pulp 2 repositories and recreate them for Pulp 3.

Let us know for future upgrades if you do have more troubles with the Pulp 2 -> Pulp 3 content migrations. There will be one more that you’ll have to do for Yum before Katello 4.0 comes along. By then, we’ll have changed the code to detect which content types are already migrated so you shouldn’t see the same error.

I am having this problem when upgrading from katello-3.15.3 to katello-3.16.0

I get this error when trying to migrate the content:

[root@katello-testing ~]# foreman-maintain advanced procedure run content-switchover
Running ForemanMaintain::Scenario
================================================================================
Switch support for certain content from Pulp 2 to Pulp 3: 
Performing final content migration before switching content
Content Migration completed successfully
Performing a check to verify everything that is needed has been migrated
                                                                      [FAIL]
Failed executing foreman-rake katello:pulp3_post_migration_check, exit status 1:
 rake aborted!
The Dynflow world was not initialized yet. If your plugin uses it, make sure to call Rails.application.dynflow.require! in some initializer
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/rails.rb:75:in `world'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/lib/foreman_tasks.rb:23:in `trigger'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/lib/foreman_tasks.rb:29:in `block in trigger_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/lib/foreman_tasks.rb:49:in `block in rails_safe_trigger_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/lib/foreman_tasks.rb:48:in `rails_safe_trigger_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/lib/foreman_tasks.rb:27:in `trigger_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/lib/foreman_tasks.rb:58:in `sync_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0/lib/katello/tasks/pulp3_post_migration_check.rake:10:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_post_migration_check
(See full trace by running task with --trace)
--------------------------------------------------------------------------------
Scenario [ForemanMaintain::Scenario] failed.

The following steps ended up in failing state:

  [content-switchover]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="content-switchover"

If I try to remove the repositories to recreate them I get an error Repository guacamole_dockerhub cannot be deleted since it has already been included in a published Content View.

These are my proxy settings:

Is there any workaround? or any way I can force to remove the repository without deleting the content views?

I tried to rollback my devel instance to katello-3.15.3 and executed foreman-maintain advanced procedure run content-prepare and I am getting error Pulp 3 migration cannot run. Types file, docker have already been migrated.

[root@katello-testing ~]# foreman-maintain advanced procedure run content-prepare
Running ForemanMaintain::Scenario
================================================================================
Prepare content for Pulp 3:                                           [FAIL]
Failed executing foreman-rake katello:pulp3_migration, exit status 1:
 rake aborted!
ForemanTasks::TaskError: Task 3120e425-e8aa-45d7-a694-97caa1a42059: Katello::Errors::Pulp3MigrationError: Pulp 3 migration cannot run. Types file, docker have already been migrated.
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.3.1/lib/katello/tasks/pulp3_migration.rake:15:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_migration
(See full trace by running task with --trace)
--------------------------------------------------------------------------------
Scenario [ForemanMaintain::Scenario] failed.

The following steps ended up in failing state:

  [content-prepare]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="content-prepare"

Hi @Pablo_Escobar

Have you previously run foreman-maintain advanced procedure run content-switchover and had it succeed? Perhaps when you started using Katello 3.15? The content-switchover task is a one-time deal for the content types it switches from Pulp 2 to Pulp 3. Your Docker and File content types are already using Pulp 3, which is why the migration is failing.

Some more info: At the moment, only Docker and File are being migrated to Pulp 3. Once the switchover is run successfully, all content for Docker and File will be permanently on Pulp 3. In a later release, you will do these steps again for Yum content.

If you definitely haven’t run the switchover, we can try some manual steps to fix your environment. Most of what I said above to @Mkbrown would apply to you too.

We had the same issue, confined to Docker repositories. I can confirm that deleting the repository and recreating it resolved the problem.

Hi,

Sorry for the late reply but I have been offline for 2 weeks. I will be back to this post in coming days when I manage to clean my email backlog.

@poulterer I also tried to remove the repositories but those are already in a CV so I cannot remove them without removing the CV. I have no problem about deleting/adding the repos. Is there any way to force it?

Yes. Remove them from the CV, publish a new version and purge the old version. Then, remove the repository.

1 Like

I could delete the docker repos and add them again in my prod environment but in testing environment I keep getting this error when I try to delete the docker repo:

Repository 199 cannot be deleted since it has already been included in a published Content View.

But I only have one CV and when I go to the CV >> container images >> repositories I can only read There are currently no Docker Repositories associated with this Content View, you can add some by clicking on the "Add" tab above.

any suggestion about how I can force the removal of the docker repository to recreate it again?

It must be in an old version. I cannot look at this directly because we do not use Content Views for Docker repositories. We pull directly from the Foreman URL for a given repository.

2 Likes

Thanks for the pointer. The CV had more than 100 versions and one of the old ones had the containers included (I guess one of my colleagues did a test). After removing the old CV version I could remove the docker repos, upgrade katello and recreate the repos. Problem solved by now. Both my prod and testing environment are running latest katello 3.16.0. thanks

Glad it worked. You can easily purge all old versions by issuing:
hammer content-view purge --async --count 0 --name "<conrent view name>"