Pulp 2 to Pulp3 upgrade fails

Support
, ,
1

Problem:

Running Switch support for certain content from Pulp 2 to Pulp 3

Switch support for certain content from Pulp 2 to Pulp 3:
Performing final content migration before switching content [FAIL]
Failed executing foreman-rake katello:pulp3_migration, exit status 1:
Rubocop not loaded.
Migration failed, You will want to investigate: https://foreman.test.com/foreman_tasks/tasks/d3d2c34b-bb7b-41bc-871a-88dd20d5392b
rake aborted!
ForemanTasks::TaskError: Task d3d2c34b-bb7b-41bc-871a-88dd20d5392b: ActiveRecord::StatementInvalid: PG::UndefinedColumn: ERROR: column katello_debs.migrated_pulp3_href does not exist
LINE 1: SELECT COUNT(*) FROM “katello_debs” WHERE “katello_debs”."mi…
^
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/lib/katello/tasks/pulp3_migration.rake:31:in block (2 levels) in <top (required)>' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in <top (required)>’
Tasks: TOP => katello:pulp3_migration
(See full trace by running task with --trace)
Starting task.
2021-02-01 14:21:26 +0100: Content migration starting.

Scenario [Switch support for certain content from Pulp 2 to Pulp 3] failed.

The following steps ended up in failing state:

[content-switchover]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist=“content-switchover”

Expected outcome:
Pulp2 should upgrade to Pulp 3.

Foreman and Proxy versions:
Installed Packages
ansible-collection-theforeman-foreman-1.4.0-1.el8.noarch
ansiblerole-foreman_scap_client-0.1.0-1.el7.noarch
candlepin-3.1.22-1.el7.noarch
candlepin-selinux-3.1.22-1.el7.noarch
foreman-2.3.2-1.el7.noarch
foreman-bootloaders-redhat-202005201200-1.el7.noarch
foreman-bootloaders-redhat-tftpboot-202005201200-1.el7.noarch
foreman-cli-2.3.2-1.el7.noarch
foreman-console-2.3.2-1.el7.noarch
foreman-debug-2.3.2-1.el7.noarch
foreman-discovery-image-service-1.0.0-4.el7.x86_64
foreman-dynflow-sidekiq-2.3.2-1.el7.noarch
foreman-ec2-2.3.2-1.el7.noarch
foreman-installer-2.3.2-1.el7.noarch
foreman-installer-katello-2.3.2-1.el7.noarch
foreman-libvirt-2.3.2-1.el7.noarch
foreman-postgresql-2.3.2-1.el7.noarch
foreman-proxy-2.3.2-1.el7.noarch
foreman-release-2.3.2-1.el7.noarch
foreman-selinux-2.3.2-1.el7.noarch
foreman-service-2.3.2-1.el7.noarch
foreman.test.com-apache-1.0-1.noarch
foreman.test.com-foreman-client-1.0-1.noarch
foreman.test.com-foreman-proxy-1.0-1.noarch
foreman.test.com-foreman-proxy-client-1.0-1.noarch
foreman.test.com-puppet-client-1.0-1.noarch
foreman.test.com-qpid-broker-1.0-1.noarch
foreman.test.com-qpid-client-cert-1.0-1.noarch
foreman.test.com-qpid-router-client-1.0-1.noarch
foreman.test.com-qpid-router-server-1.0-1.noarch
katello-3.18.1-1.el7.noarch
katello-certs-tools-2.7.3-1.el7.noarch
katello-client-bootstrap-1.7.5-1.el7.noarch
katello-common-3.18.1-1.el7.noarch
katello-debug-3.18.1-1.el7.noarch
katello-default-ca-1.0-1.noarch
katello-repos-3.18.1-1.el7.noarch
katello-selinux-3.5.0-1.el7.noarch
katello-server-ca-1.0-1.noarch
pulp-admin-client-2.21.5-1.el7.noarch
pulp-client-1.0-1.noarch
pulp-consumer-client-2.21.5-1.el7.noarch
pulp-deb-plugins-1.10.2-1.el7.noarch
pulp-docker-plugins-3.2.8-1.el7.noarch
pulp-katello-1.0.3-1.el7.noarch
pulp-puppet-plugins-2.21.5-1.el7.noarch
pulp-puppet-tools-2.21.5-1.el7.noarch
pulp-rpm-admin-extensions-2.21.5-1.el7.noarch
pulp-rpm-consumer-extensions-2.21.5-1.el7.noarch
pulp-rpm-handlers-2.21.5-1.el7.noarch
pulp-rpm-plugins-2.21.5-1.el7.noarch
pulp-rpm-yumplugins-2.21.5-1.el7.noarch
pulp-selinux-2.21.5-1.el7.noarch
pulp-server-2.21.5-1.el7.noarch
pulpcore-selinux-1.2.3-2.el7.x86_64
puppet-foreman_scap_client-0.4.0-1.el7.noarch
python-gofer-qpid-2.12.5-3.el7.noarch
python-pulp-agent-lib-2.21.5-1.el7.noarch
python-pulp-bindings-2.21.5-1.el7.noarch
python-pulp-client-lib-2.21.5-1.el7.noarch
python-pulp-common-2.21.5-1.el7.noarch
python-pulp-deb-common-1.10.2-1.el7.noarch
python-pulp-docker-common-3.2.8-1.el7.noarch
python-pulp-manifest-2.21.5-1.el7.noarch
python-pulp-oid_validation-2.21.5-1.el7.noarch
python-pulp-puppet-common-2.21.5-1.el7.noarch
python-pulp-repoauth-2.21.5-1.el7.noarch
python-pulp-rpm-common-2.21.5-1.el7.noarch
python-pulp-streamer-2.21.5-1.el7.noarch
python2-qpid-1.37.0-5.el7.noarch
python2-qpid-proton-0.33.0-1.el7.x86_64
python2-qpid-qmf-1.39.0-1.el7.x86_64
python3-pulp-2to3-migration-0.6.0-2.el7.noarch
python3-pulp-certguard-1.0.3-1.el7.noarch
python3-pulp-container-2.1.0-1.el7.noarch
python3-pulp-deb-2.7.0-1.el7.noarch
python3-pulp-file-1.3.0-1.el7.noarch
python3-pulp-rpm-3.7.0-1.el7.noarch
python3-pulpcore-3.7.3-1.el7.noarch
qpid-cpp-client-1.39.0-1.el7.x86_64
qpid-cpp-client-devel-1.39.0-1.el7.x86_64
qpid-cpp-server-1.39.0-1.el7.x86_64
qpid-cpp-server-linearstore-1.39.0-1.el7.x86_64
qpid-dispatch-router-1.14.0-1.el7.x86_64
qpid-proton-c-0.33.0-1.el7.x86_64
qpid-qmf-1.39.0-1.el7.x86_64
qpid-tools-1.39.0-1.el7.noarch
rubygem-foreman_maintain-0.7.1-1.el7.noarch
rubygem-foreman_scap_client-0.4.0-1.el7.noarch
tfm-rubygem-actioncable-6.0.3.4-1.el7.noarch
tfm-rubygem-actionmailbox-6.0.3.4-1.el7.noarch
tfm-rubygem-actionmailer-6.0.3.4-1.el7.noarch
tfm-rubygem-actionpack-6.0.3.4-1.el7.noarch
tfm-rubygem-actiontext-6.0.3.4-1.el7.noarch
tfm-rubygem-actionview-6.0.3.4-1.el7.noarch
tfm-rubygem-activejob-6.0.3.4-1.el7.noarch
tfm-rubygem-activemodel-6.0.3.4-1.el7.noarch
tfm-rubygem-activerecord-6.0.3.4-1.el7.noarch
tfm-rubygem-activerecord-import-1.0.0-2.el7.noarch
tfm-rubygem-activerecord-session_store-1.1.1-4.el7.noarch
tfm-rubygem-activestorage-6.0.3.4-1.el7.noarch
tfm-rubygem-activesupport-6.0.3.4-1.el7.noarch
tfm-rubygem-addressable-2.6.0-2.el7.noarch
tfm-rubygem-algebrick-0.7.3-7.el7.noarch
tfm-rubygem-amazing_print-1.1.0-1.el7.noarch
tfm-rubygem-ancestry-3.0.7-1.el7.noarch
tfm-rubygem-anemone-0.7.2-17.el7.noarch
tfm-rubygem-angular-rails-templates-1.1.0-1.el7.noarch
tfm-rubygem-ansi-1.5.0-2.el7.noarch
tfm-rubygem-apipie-bindings-0.4.0-1.el7.noarch
tfm-rubygem-apipie-dsl-2.3.0-1.el7.noarch
tfm-rubygem-apipie-params-0.0.5-4.el7.noarch
tfm-rubygem-apipie-rails-0.5.17-3.el7.noarch
tfm-rubygem-audited-4.9.0-3.el7.noarch
tfm-rubygem-azure_mgmt_compute-0.18.7-1.el7.noarch
tfm-rubygem-azure_mgmt_network-0.19.0-1.el7.noarch
tfm-rubygem-azure_mgmt_resources-0.17.6-1.el7.noarch
tfm-rubygem-azure_mgmt_storage-0.17.10-1.el7.noarch
tfm-rubygem-azure_mgmt_subscriptions-0.18.2-1.el7.noarch
tfm-rubygem-bcrypt-3.1.12-3.el7.x86_64
tfm-rubygem-builder-3.2.4-1.el7.noarch
tfm-rubygem-bundler_ext-0.4.1-5.el7.noarch
tfm-rubygem-clamp-1.1.2-6.el7.noarch
tfm-rubygem-concurrent-ruby-1.1.6-2.el7.noarch
tfm-rubygem-concurrent-ruby-edge-0.6.0-2.fm2_1.el7.noarch
tfm-rubygem-connection_pool-2.2.2-2.el7.noarch
tfm-rubygem-crass-1.0.6-1.el7.noarch
tfm-rubygem-css_parser-1.4.7-4.el7.noarch
tfm-rubygem-daemons-1.2.3-6.el7.noarch
tfm-rubygem-deacon-1.0.0-4.el7.noarch
tfm-rubygem-deep_cloneable-3.0.0-3.el7.noarch
tfm-rubygem-deface-1.5.3-2.el7.noarch
tfm-rubygem-domain_name-0.5.20160310-4.el7.noarch
tfm-rubygem-dynflow-1.4.7-1.fm2_3.el7.noarch
tfm-rubygem-erubi-1.9.0-1.el7.noarch
tfm-rubygem-excon-0.76.0-1.el7.noarch
tfm-rubygem-facter-2.4.0-7.el7.x86_64
tfm-rubygem-faraday-0.17.3-1.el7.noarch
tfm-rubygem-faraday-cookie_jar-0.0.6-1.el7.noarch
tfm-rubygem-fast_gettext-1.4.1-4.el7.noarch
tfm-rubygem-ffi-1.12.2-1.el7.x86_64
tfm-rubygem-fog-aws-3.6.5-1.el7.noarch
tfm-rubygem-fog-core-2.1.0-3.el7.noarch
tfm-rubygem-fog-json-1.2.0-3.el7.noarch
tfm-rubygem-fog-libvirt-0.7.0-2.el7.noarch
tfm-rubygem-fog-xml-0.1.2-8.el7.noarch
tfm-rubygem-foreman-tasks-3.0.3-1.fm2_3.el7.noarch
tfm-rubygem-foreman-tasks-core-0.3.4-1.fm2_1.el7.noarch
tfm-rubygem-foreman_ansible-6.1.1-1.fm2_3.el7.noarch
tfm-rubygem-foreman_ansible_core-4.0.0-1.fm2_3.el7.noarch
tfm-rubygem-foreman_azure_rm-2.1.2-1.fm2_2.el7.noarch
tfm-rubygem-foreman_bootdisk-17.0.2-2.fm2_2.el7.noarch
tfm-rubygem-foreman_dhcp_browser-0.0.8-4.fm2_1.el7.noarch
tfm-rubygem-foreman_discovery-16.3.4-1.fm2_3.el7.noarch
tfm-rubygem-foreman_openscap-4.1.2-1.fm2_3.el7.noarch
tfm-rubygem-foreman_remote_execution-4.2.2-1.fm2_3.el7.noarch
tfm-rubygem-foreman_remote_execution-cockpit-4.2.2-1.fm2_3.el7.noarch
tfm-rubygem-foreman_remote_execution_core-1.4.0-1.el7.noarch
tfm-rubygem-foreman_snapshot_management-1.7.1-1.fm2_1.el7.noarch
tfm-rubygem-formatador-0.2.1-12.el7.noarch
tfm-rubygem-friendly_id-5.3.0-1.el7.noarch
tfm-rubygem-fx-0.5.0-1.el7.noarch
tfm-rubygem-get_process_mem-0.2.1-4.el7.noarch
tfm-rubygem-gettext-3.1.4-10.el7.noarch
tfm-rubygem-gettext_i18n_rails-1.8.0-2.el7.noarch
tfm-rubygem-gitlab-sidekiq-fetcher-0.6.0-1.el7.noarch
tfm-rubygem-globalid-0.4.2-1.el7.noarch
tfm-rubygem-graphql-1.8.14-2.el7.noarch
tfm-rubygem-graphql-batch-0.3.10-2.el7.noarch
tfm-rubygem-gssapi-1.2.0-7.el7.noarch
tfm-rubygem-hammer_cli-2.3.0-1.el7.noarch
tfm-rubygem-hammer_cli_foreman-2.3.0-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_ansible-0.3.2-1.fm2_1.el7.noarch
tfm-rubygem-hammer_cli_foreman_azure_rm-0.2.0-1.fm2_2.el7.noarch
tfm-rubygem-hammer_cli_foreman_bootdisk-0.3.0-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_docker-0.0.7-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_openscap-0.1.12-1.fm2_3.el7.noarch
tfm-rubygem-hammer_cli_foreman_remote_execution-0.2.1-1.fm2_3.el7.noarch
tfm-rubygem-hammer_cli_foreman_tasks-0.0.15-1.fm2_2.el7.noarch
tfm-rubygem-hammer_cli_katello-0.24.2-1.el7.noarch
tfm-rubygem-hashie-3.6.0-2.el7.noarch
tfm-rubygem-highline-1.7.8-5.el7.noarch
tfm-rubygem-http-cookie-1.0.2-4.el7.noarch
tfm-rubygem-i18n-1.8.2-1.el7.noarch
tfm-rubygem-ipaddress-0.8.0-12.el7.noarch
tfm-rubygem-jwt-2.2.1-2.el7.noarch
tfm-rubygem-kafo-6.1.2-1.el7.noarch
tfm-rubygem-kafo_parsers-1.1.0-3.el7.noarch
tfm-rubygem-kafo_wizards-0.0.1-4.el7.noarch
tfm-rubygem-katello-3.18.1-1.el7.noarch
tfm-rubygem-ldap_fluff-0.4.7-5.el7.noarch
tfm-rubygem-little-plugger-1.1.4-2.el7.noarch
tfm-rubygem-locale-2.0.9-14.el7.noarch
tfm-rubygem-logging-2.3.0-1.el7.noarch
tfm-rubygem-loofah-2.4.0-1.el7.noarch
tfm-rubygem-mail-2.7.1-1.el7.noarch
tfm-rubygem-marcel-0.3.3-1.el7.noarch
tfm-rubygem-method_source-0.9.2-2.el7.noarch
tfm-rubygem-mime-types-3.2.2-4.el7.noarch
tfm-rubygem-mime-types-data-3.2018.0812-4.el7.noarch
tfm-rubygem-mimemagic-0.3.5-1.el7.noarch
tfm-rubygem-mini_mime-1.0.2-1.el7.noarch
tfm-rubygem-mini_portile2-2.4.0-1.el7.noarch
tfm-rubygem-ms_rest-0.7.4-2.el7.noarch
tfm-rubygem-ms_rest_azure-0.11.1-2.el7.noarch
tfm-rubygem-multi_json-1.14.1-2.el7.noarch
tfm-rubygem-multipart-post-2.0.0-2.el7.noarch
tfm-rubygem-mustermann-1.0.2-4.el7.noarch
tfm-rubygem-net-ldap-0.16.1-2.el7.noarch
tfm-rubygem-net-ping-2.0.1-4.el7.noarch
tfm-rubygem-net-scp-1.2.1-4.el7.noarch
tfm-rubygem-net-ssh-4.2.0-2.el7.noarch
tfm-rubygem-netrc-0.11.0-5.el7.noarch
tfm-rubygem-nio4r-2.5.4-1.el7.x86_64
tfm-rubygem-nokogiri-1.10.9-1.el7.x86_64
tfm-rubygem-oauth-0.5.4-4.el7.noarch
tfm-rubygem-openscap-0.4.9-4.el7.noarch
tfm-rubygem-paint-0.8.7-9.el7.noarch
tfm-rubygem-parse-cron-0.1.4-4.fm2_1.el7.noarch
tfm-rubygem-pg-1.1.4-3.el7.x86_64
tfm-rubygem-polyglot-0.3.5-2.el7.noarch
tfm-rubygem-powerbar-2.0.1-2.el7.noarch
tfm-rubygem-promise.rb-0.7.4-2.el7.noarch
tfm-rubygem-public_suffix-3.0.3-2.el7.noarch
tfm-rubygem-pulp_2to3_migration_client-0.5.0-1.el7.noarch
tfm-rubygem-pulp_ansible_client-0.4.2-1.el7.noarch
tfm-rubygem-pulp_certguard_client-1.0.3-1.el7.noarch
tfm-rubygem-pulp_container_client-2.1.0-1.el7.noarch
tfm-rubygem-pulp_deb_client-2.7.0-1.el7.noarch
tfm-rubygem-pulp_file_client-1.3.0-1.el7.noarch
tfm-rubygem-pulp_rpm_client-3.7.0-1.el7.noarch
tfm-rubygem-pulpcore_client-3.7.1-1.el7.noarch
tfm-rubygem-puma-4.3.6-1.el7.x86_64
tfm-rubygem-puma-plugin-systemd-0.1.5-1.el7.noarch
tfm-rubygem-rabl-0.14.3-1.el7.noarch
tfm-rubygem-rack-2.2.3-1.el7.noarch
tfm-rubygem-rack-cors-1.0.2-2.el7.noarch
tfm-rubygem-rack-jsonp-1.3.1-9.el7.noarch
tfm-rubygem-rack-protection-2.0.3-4.el7.noarch
tfm-rubygem-rack-test-1.1.0-4.el7.noarch
tfm-rubygem-rails-6.0.3.4-1.el7.noarch
tfm-rubygem-rails-dom-testing-2.0.3-6.el7.noarch
tfm-rubygem-rails-html-sanitizer-1.3.0-1.el7.noarch
tfm-rubygem-rails-i18n-6.0.0-2.el7.noarch
tfm-rubygem-railties-6.0.3.4-1.el7.noarch
tfm-rubygem-rainbow-2.2.1-3.el7.noarch
tfm-rubygem-rb-inotify-0.9.7-5.el7.noarch
tfm-rubygem-record_tag_helper-1.0.1-3.el7.noarch
tfm-rubygem-redfish_client-0.5.2-1.el7.noarch
tfm-rubygem-redis-4.1.2-2.el7.noarch
tfm-rubygem-responders-3.0.0-3.el7.noarch
tfm-rubygem-rest-client-2.0.2-3.el7.noarch
tfm-rubygem-rkerberos-0.1.5-19.el7.x86_64
tfm-rubygem-roadie-3.4.0-3.el7.noarch
tfm-rubygem-roadie-rails-2.1.1-2.el7.noarch
tfm-rubygem-robotex-1.0.0-21.el7.noarch
tfm-rubygem-rsec-0.4.3-4.el7.noarch
tfm-rubygem-ruby-libvirt-0.7.1-1.el7.x86_64
tfm-rubygem-ruby2ruby-2.4.2-3.el7.noarch
tfm-rubygem-ruby_parser-3.10.1-3.el7.noarch
tfm-rubygem-rubyipmi-0.10.0-6.el7.noarch
tfm-rubygem-runcible-2.13.1-1.el7.noarch
tfm-rubygem-safemode-1.3.6-1.el7.noarch
tfm-rubygem-scoped_search-4.1.9-1.el7.noarch
tfm-rubygem-sd_notify-0.1.0-1.el7.noarch
tfm-rubygem-secure_headers-6.3.0-2.el7.noarch
tfm-rubygem-sequel-5.7.1-3.el7.noarch
tfm-rubygem-server_sent_events-0.1.2-1.el7.noarch
tfm-rubygem-sexp_processor-4.10.0-6.el7.noarch
tfm-rubygem-sidekiq-5.2.7-3.el7.noarch
tfm-rubygem-sinatra-2.0.3-4.el7.noarch
tfm-rubygem-smart_proxy_ansible-3.0.1-6.fm2_2.el7.noarch
tfm-rubygem-smart_proxy_discovery-1.0.5-6.fm2_2.el7.noarch
tfm-rubygem-smart_proxy_discovery_image-1.3.2-1.fm2_3.el7.noarch
tfm-rubygem-smart_proxy_dynflow-0.3.0-2.fm2_3.el7.noarch
tfm-rubygem-smart_proxy_dynflow_core-0.3.2-1.fm2_3.el7.noarch
tfm-rubygem-smart_proxy_openscap-0.7.4-1.fm2_2.el7.noarch
tfm-rubygem-smart_proxy_pulp-2.1.0-3.fm2_2.el7.noarch
tfm-rubygem-smart_proxy_remote_execution_ssh-0.3.1-1.fm2_3.el7.noarch
tfm-rubygem-sprockets-4.0.2-1.el7.noarch
tfm-rubygem-sprockets-rails-3.2.1-6.el7.noarch
tfm-rubygem-sqlite3-1.3.13-6.el7.x86_64
tfm-rubygem-sshkey-1.9.0-4.el7.noarch
tfm-rubygem-statsd-instrument-2.1.4-3.el7.noarch
tfm-rubygem-stomp-1.4.9-1.el7.noarch
tfm-rubygem-text-1.3.0-7.el7.noarch
tfm-rubygem-thor-1.0.1-2.el7.noarch
tfm-rubygem-thread_safe-0.3.6-5.el7.noarch
tfm-rubygem-tilt-2.0.8-4.el7.noarch
tfm-rubygem-timeliness-0.3.10-1.el7.noarch
tfm-rubygem-tzinfo-1.2.6-1.el7.noarch
tfm-rubygem-unf-0.1.3-8.el7.noarch
tfm-rubygem-unf_ext-0.0.7.2-3.el7.x86_64
tfm-rubygem-unicode-0.4.4.4-3.el7.x86_64
tfm-rubygem-unicode-display_width-1.0.5-4.el7.noarch
tfm-rubygem-validates_lengths_from_database-0.5.0-7.el7.noarch
tfm-rubygem-webpack-rails-0.9.8-5.el7.noarch
tfm-rubygem-websocket-driver-0.7.1-1.el7.x86_64
tfm-rubygem-websocket-extensions-0.1.5-1.el7.noarch
tfm-rubygem-will_paginate-3.1.7-3.el7.noarch
tfm-rubygem-wirb-1.0.3-6.el7.noarch
tfm-rubygem-xmlrpc-0.3.0-2.el7.noarch
tfm-rubygem-zeitwerk-2.2.2-1.el7.noarch
tfm-runtime-6.1-4.el7.x86_64

Foreman and Proxy plugin versions:

Distribution and version:
CentOS Linux release 7.9.2009 (Core)

Other relevant data:

3

I have two test systems and have just completely (re)built one using Katello 3.18 and Foreman 2.3 and then tried the Pulp migration.
Unfortunately it made no difference so the problem is definitely within the Pulp 2 to Pulp 3 migration.

4

I run into the same problem during upgrade. Does somebody know more?

5

Same for me:

blank installed Katello 3.17 -> update to 3.18 error during pulp migration with missing table

Anyone else?

6

Hey!

We support yum,files,container content so far for pulp3 and debian migration support should be in soon. You’ll not be able to do a content switchover with debian content till the support is added.

1 Like
7

yes, I know. But I had no Debian content in Foreman, just 4 Centos7-Repos. And I’m really sure about that.

@klaus-peter and @techietubby: did you have debian content inside when you got this error?

Maybe something in the script is misinterpreting things as debian… just an idea

8

Same for me I only have Centos-7, Centos-8, and some Docker test repos. I want to go to Pulp-3 as I am told I can then pull my Docker images from Satellite.

9

You can make sure you have no debian content with
Katello::Deb.count
and
Katello::Repository.where(:root => Katello::RootRepository.where(:content_type => "deb")).count
in the foreman console.

If both of these are 0 then you can safely disable deb content in katello before migration by running foreman-installer --katello-enable-deb=false
Upon migration re-run the installer with --katello-enable-deb=true to enable deb on pulp3.

10

I have filed an issue around this: Bug #31875: Disable/ Work around deb migration before upgrading if there's no debian content - Katello - Foreman here. Thanks for raising this.

11

I tried to disable deb content as described with:

foreman-installer --katello-enable-deb=false

So, the error message is different now:

RuntimeError: Cannot find repository type deb, is it enabled

3 Likes
12

I tried the above and the migration was successful

foreman-installer --katello-enable-deb=false
foreman-rake katello:pulp3_migration
foreman-installer --katello-enable-deb=true
1 Like
13

Some extra info. I have another test machine where I had been experimenting with Debian. I had to remove the product and then clean-up my database using:

foreman-installer --katello-enable-deb=false
foreman-rake foreman_tasks:cleanup
foreman-rake foreman_tasks:cleanup:config
foreman-rake katello:pulp3_migration
foreman-installer --katello-enable-deb=true
14

Now I need to be able to do a docker pull on my containers.

15

Have same issue.

16

What is strange is that I can list the docker repos:

# docker search $(hostname)/fedora
NAME                                                                                            DESCRIPTION   STARS     OFFICIAL   AUTOMATED
test-centos_8_content-2_0-fedora_docker_containers-fedora_docker_container_repo                         0
test-development-centos_8_content-fedora_docker_containers-fedora_docker_container_repo                 0
test-fedora_docker_containers-fedora_docker_container_repo                                              0
test-library-centos_8_content-fedora_docker_containers-fedora_docker_container_repo                     0

Pulp3 now also shows the repos:

# pulp container repository list
[
  {
    "pulp_href": "/pulp/api/v3/repositories/container/container/9de7f6e5-f32b-4c26-a080-366e0bd3fa8f/",
    "pulp_created": "2021-01-22T08:12:30.889021Z",
    "versions_href": "/pulp/api/v3/repositories/container/container/9de7f6e5-f32b-4c26-a080-366e0bd3fa8f/versions/",
    "latest_version_href": "/pulp/api/v3/repositories/container/container/9de7f6e5-f32b-4c26-a080-366e0bd3fa8f/versions/0/",
    "name": "Fedora_Docker_container_repo-472668",
    "description": null,
    "remote": null
  },
..

I have tried all the ways I can think of to pull the container but none of the examples in the Foreman docs work.

17

Exactly the same behavior here (Oracle Linux 7.9, no deb content)

18

Hey @techietubby , can you share the command(s) you’ve tried, their output, and any errors you’re seeing show up in journalctl?

19

Hi Jonathon,

Yes of course, gladly! :slight_smile:

I migrated Pulp as follows:

# foreman-installer --katello-enable-deb=false
# foreman-rake foreman_tasks:cleanup
# foreman-rake foreman_tasks:cleanup:config foreman-rake
# foreman-rake katello:pulp3_migration
# foreman-installer --katello-enable-deb=true

I did a restorecon to ensure all SELinux labels were up date and the ran:

# docker pull foreman.test.com/test-centos_8_content-2_0-fedora_docker_containers-fedora_docker_container_repo

This gives:

# journalctl -l --since=09:26 --no-pager
-- Logs begin at Mon 2021-02-01 13:52:47 CET, end at Thu 2021-02-18 09:26:15 CET. --
Feb 18 09:26:15 foreman.test.com dockerd[8018]: time="2021-02-18T09:26:15.935487079+01:00" level=info msg="Attempting next endpoint for pull after error: Head https://foreman.test.com/v2/test-centos_8_content-2_0-fedora_docker_containers-fedora_docker_container_repo/manifests/latest: received unexpected HTTP status: 500 Internal Server Error"
Feb 18 09:26:15 foreman.test.com dockerd[8018]: time="2021-02-18T09:26:15.940765618+01:00" level=error msg="Handler for POST /v1.41/images/create returned error: Head https://foreman.test.com/v2/test-centos_8_content-2_0-fedora_docker_containers-fedora_docker_container_repo/manifests/latest: received unexpected HTTP status: 500 Internal Server Error"

If I add fedora to the tag the results are:

# docker pull foreman.test.com/test-centos_8_content-2_0-fedora_docker_containers-fedora_docker_container_repo/fedora
Using default tag: latest
Error response from daemon: unknown: test-centos_8_content-2_0-fedora_docker_containers-fedora_docker_container_repo/fedora was not found!

The journalctl gives:

Feb 18 09:27:37 foreman.test.com pulp[21363]: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred)
Feb 18 09:27:37 foreman.test.com pulp[21404]: celery.worker.strategy:INFO: Received task: pulp.server.controllers.repository.queue_download_deferred[2b716a9f-5926-4261-861b-30a0b8fe9e34]
Feb 18 09:27:37 foreman.test.com pulp[21407]: celery.worker.strategy:INFO: Received task: pulp.server.controllers.repository.download_deferred[aba775b8-f924-4ddc-a8b1-5a7c27c617d6]
Feb 18 09:27:37 foreman.test.com pulp[21472]: celery.app.trace:INFO: [2b716a9f] Task pulp.server.controllers.repository.queue_download_deferred[2b716a9f-5926-4261-861b-30a0b8fe9e34] succeeded in 0.00635926402174s: None
Feb 18 09:27:38 foreman.test.com pulp[21473]: celery.app.trace:INFO: [aba775b8] Task pulp.server.controllers.repository.download_deferred[aba775b8-f924-4ddc-a8b1-5a7c27c617d6] succeeded in 1.01021712599s: None
Feb 18 09:30:01 foreman.test.com systemd[1]: Created slice User Slice of foreman-proxy.
Feb 18 09:30:01 foreman.test.com systemd[1]: Started Session 2149 of user foreman-proxy.
Feb 18 09:30:01 foreman.test.com systemd[1]: Created slice User Slice of foreman.
Feb 18 09:30:01 foreman.test.com systemd[1]: Started Session 2148 of user foreman.
Feb 18 09:30:01 foreman.test.com CROND[8794]: (foreman-proxy) CMD (smart-proxy-openscap-send >> /var/log/foreman-proxy/cron.log)
Feb 18 09:30:01 foreman.test.com CROND[8795]: (foreman) CMD (   /usr/sbin/foreman-rake ldap:refresh_usergroups >>/var/log/foreman/cron.log 2>&1)
Feb 18 09:30:02 foreman.test.com systemd[1]: Removed slice User Slice of foreman-proxy.
Feb 18 09:30:20 foreman.test.com dockerd[8018]: time="2021-02-18T09:30:20.819236660+01:00" level=error msg="Not continuing with pull after error: unknown: test-centos_8_content-2_0-fedora_docker_containers-fedora_docker_container_repo/fedora was not found!"
Feb 18 09:30:20 foreman.test.com dockerd[8018]: time="2021-02-18T09:30:20.823651356+01:00" level=error msg="Handler for POST /v1.41/images/create returned error: unknown: test-centos_8_content-2_0-fedora_docker_containers-fedora_docker_container_repo/fedora was not found!"
Feb 18 09:30:24 foreman.test.com systemd[1]: Removed slice User Slice of foreman.
Feb 18 09:30:39 foreman.test.com pulpcore-worker-4[15884]: pulp: rq.worker:INFO: Cleaning registries for queue: 15884@foreman.test.com
Feb 18 09:30:39 foreman.test.com pulpcore-worker-2[15883]: pulp: rq.worker:INFO: Cleaning registries for queue: 15883@foreman.test.com
Feb 18 09:30:41 foreman.test.com pulpcore-worker-1[15882]: pulp: rq.worker:INFO: Cleaning registries for queue: 15882@foreman.test.com
Feb 18 09:30:42 foreman.test.com pulpcore-worker-3[15881]: pulp: rq.worker:INFO: Cleaning registries for queue: 15881@foreman.test.com

Do you need more examples?

Regards,
Andrew

20

Looks like there should be an ISE traceback somewhere. Can try tailing /var/log/foreman/production.log while you docker pull?

21

I get:

2021-02-18T18:31:58 [I|app|0141592d] Filter chain halted as :authorize_repository_read rendered or redirected
2021-02-18T18:31:58 [I|app|0141592d] Completed 404 Not Found in 20ms (Views: 0.2ms | ActiveRecord: 7.3ms | Allocations: 6946)
2021-02-18T18:31:58 [I|app|6b706c73] Started GET "/v2/token?account=admin&scope=repository%3Atest-centos_8_content-2_0-fedora_docker_containers-fedora_docker_container_repo%2Ffedora%3Apull&service=foreman.test.com" for 127.0.0.1 at 2021-02-18 18:31:58 +0100
2021-02-18T18:31:58 [I|app|6b706c73] Processing by Katello::Api::Registry::RegistryProxiesController#token as HTML
2021-02-18T18:31:58 [I|app|6b706c73]   Parameters: {"account"=>"admin", "scope"=>"repository:test-centos_8_content-2_0-fedora_docker_containers-fedora_docker_container_repo/fedora:pull", "service"=>"foreman.test.com"}
2021-02-18T18:31:58 [I|kat|6b706c73] Authorized user admin(Admin User)
2021-02-18T18:31:58 [I|aud|6b706c73] PersonalAccessToken (1) update event on expires_at 2021-02-18 17:37:58 UTC, 2021-02-18 17:37:58 UTC
2021-02-18T18:31:58 [I|app|6b706c73] Completed 200 OK in 118ms (Views: 0.3ms | ActiveRecord: 19.9ms | Allocations: 18621)
2021-02-18T18:31:58 [I|app|3c0ac17a] Started GET "/v2/test-centos_8_content-2_0-fedora_docker_containers-fedora_docker_container_repo/fedora/manifests/latest" for 127.0.0.1 at 2021-02-18 18:31:58 +0100
2021-02-18T18:31:58 [I|app|3c0ac17a] Processing by Katello::Api::Registry::RegistryProxiesController#pull_manifest as JSON
2021-02-18T18:31:58 [I|app|3c0ac17a]   Parameters: {"repository"=>"test-centos_8_content-2_0-fedora_docker_containers-fedora_docker_container_repo/fedora", "tag"=>"latest"}
2021-02-18T18:31:58 [I|app|3c0ac17a] Filter chain halted as :authorize_repository_read rendered or redirected
2021-02-18T18:31:58 [I|app|3c0ac17a] Completed 404 Not Found in 24ms (Views: 0.3ms | ActiveRecord: 9.5ms | Allocations: 7070)