[Content-Migration] katello:correct_repositories and correct_puppet_environments fails blocking migration

Mkbrown · April 1, 2021, 2:59pm

Problem:
foreman-installer

Executing: foreman-rake upgrade:run
=============================================
Upgrade Step 1/3: katello:correct_repositories. This may take a long while.
Processing Repository 1/367: Puppet EL 7 - x86_64 (26)
Processing Repository 2/367: Puppet EL 8 - x86_64 (27)
Processing Repository 3/367: Puppet Debian (39)
Processing Repository 4/367: Puppet Ubuntu (40)
Processing Repository 5/367: ISOs (44)
Failed upgrade task: katello:correct_repositories, see logs for more information.
=============================================
Upgrade Step 2/3: katello:correct_puppet_environments. This may take a long while.
Processing Puppet Environment 1/39: 1-Docker_EL_7-v1_0-puppet-483eff48-f056-442f-afb3-71443bfb35cc (15)
Failed upgrade task: katello:correct_puppet_environments, see logs for more information.
=============================================
Upgrade Step 3/3: katello:clean_backend_objects. This may take a long while.
0 orphaned consumer id(s) found in candlepin.
Candlepin orphaned consumers: []
0 orphaned consumer id(s) found in pulp.
Pulp orphaned consumers: []
  Success!

Expected outcome:
Successful migration to pulp 3 for the remaining content types (rpm and deb). Note, I previously had issues with File and Docker content as well, and ended up deleting those repos and recreating them under Pulp 3. I’d rather not have to nuke the content views and repos to do the same again this time…
Foreman and Proxy versions:
Foreman 2.3.3 and Katello 3.18.2
Foreman and Proxy plugin versions:

Installed Packages

ansible-collection-theforeman-foreman-2.0.1-1.el7.noarch
ansiblerole-foreman_scap_client-0.1.0-1.el7.noarch
candlepin-3.1.22-1.el7.noarch
candlepin-selinux-3.1.22-1.el7.noarch
foreman-2.3.3-1.el7.noarch
foreman-cli-2.3.3-1.el7.noarch
foreman-client-release-2.3.3-1.el7.noarch
foreman-debug-2.3.3-1.el7.noarch
foreman-dynflow-sidekiq-2.3.3-1.el7.noarch
foreman-installer-2.3.3-1.el7.noarch
foreman-installer-katello-2.3.3-1.el7.noarch
foreman-libvirt-2.3.3-1.el7.noarch
foreman-postgresql-2.3.3-1.el7.noarch
foreman-proxy-2.3.3-1.el7.noarch
foreman-release-2.3.3-1.el7.noarch
foreman-release-scl-7-3.el7.noarch
foreman-selinux-2.3.3-1.el7.noarch
foreman-service-2.3.3-1.el7.noarch
foreman-telemetry-2.3.3-1.el7.noarch
foreman.solutionsathand.ca-apache-1.0-4.noarch
foreman.solutionsathand.ca-foreman-client-1.0-1.noarch
foreman.solutionsathand.ca-foreman-proxy-1.0-4.noarch
foreman.solutionsathand.ca-foreman-proxy-client-1.0-1.noarch
foreman.solutionsathand.ca-puppet-client-1.0-1.noarch
foreman.solutionsathand.ca-qpid-broker-1.0-2.noarch
foreman.solutionsathand.ca-qpid-client-cert-1.0-1.noarch
foreman.solutionsathand.ca-qpid-router-client-1.0-1.noarch
foreman.solutionsathand.ca-qpid-router-server-1.0-1.noarch
foreman.solutionsathand.ca-tomcat-1.0-1.noarch
katello-3.18.2-1.el7.noarch
katello-ca-consumer-foreman.solutionsathand.ca-1.0-1.noarch
katello-certs-tools-2.7.3-1.el7.noarch
katello-client-bootstrap-1.7.5-1.el7.noarch
katello-common-3.18.2-1.el7.noarch
katello-debug-3.18.2-1.el7.noarch
katello-default-ca-1.0-1.noarch
katello-host-tools-3.5.4-1.el7.noarch
katello-host-tools-fact-plugin-3.5.4-1.el7.noarch
katello-repos-3.18.2-1.el7.noarch
katello-selinux-3.5.0-1.el7.noarch
katello-server-ca-1.0-4.noarch
katello-service-3.14.1-1.el7.noarch
pulp-client-1.0-1.noarch
pulp-deb-plugins-1.10.2-1.el7.noarch
pulp-docker-plugins-3.2.9-1.el7.noarch
pulp-katello-1.0.3-1.el7.noarch
pulp-puppet-plugins-2.21.5-1.el7.noarch
pulp-puppet-tools-2.21.5-1.el7.noarch
pulp-rpm-plugins-2.21.5-1.el7.noarch
pulp-selinux-2.21.5-1.el7.noarch
pulp-server-2.21.5-1.el7.noarch
pulpcore-selinux-1.2.3-2.el7.x86_64
python-gofer-qpid-2.12.5-3.el7.noarch
python-pulp-common-2.21.5-1.el7.noarch
python-pulp-deb-common-1.10.2-1.el7.noarch
python-pulp-docker-common-3.2.9-1.el7.noarch
python-pulp-manifest-2.21.5-1.el7.noarch
python-pulp-oid_validation-2.21.5-1.el7.noarch
python-pulp-puppet-common-2.21.5-1.el7.noarch
python-pulp-repoauth-2.21.5-1.el7.noarch
python-pulp-rpm-common-2.21.5-1.el7.noarch
python-pulp-streamer-2.21.5-1.el7.noarch
python2-qpid-1.37.0-5.el7.noarch
python2-qpid-proton-0.33.0-1.el7.x86_64
python2-qpid-qmf-1.39.0-1.el7.x86_64
python3-pulp-2to3-migration-0.10.0-1.el7.noarch
python3-pulp-certguard-1.0.3-1.el7.noarch
python3-pulp-container-2.1.0-1.el7.noarch
python3-pulp-deb-2.7.0-1.el7.noarch
python3-pulp-file-1.3.0-1.el7.noarch
python3-pulp-rpm-3.9.0-2.el7.noarch
python3-pulpcore-3.7.3-2.el7.noarch
qpid-cpp-client-1.39.0-1.el7.x86_64
qpid-cpp-client-devel-1.39.0-1.el7.x86_64
qpid-cpp-server-1.39.0-1.el7.x86_64
qpid-cpp-server-linearstore-1.39.0-1.el7.x86_64
qpid-dispatch-router-1.14.0-1.el7.x86_64
qpid-proton-c-0.33.0-1.el7.x86_64
qpid-qmf-1.39.0-1.el7.x86_64
qpid-tools-1.39.0-1.el7.noarch
rubygem-foreman_maintain-0.7.5-1.el7.noarch
rubygem-foreman_scap_client-0.4.7-1.el7.noarch
tfm-rubygem-actioncable-6.0.3.4-1.el7.noarch
tfm-rubygem-actionmailbox-6.0.3.4-1.el7.noarch
tfm-rubygem-actionmailer-6.0.3.4-1.el7.noarch
tfm-rubygem-actionpack-6.0.3.4-1.el7.noarch
tfm-rubygem-actiontext-6.0.3.4-1.el7.noarch
tfm-rubygem-actionview-6.0.3.4-1.el7.noarch
tfm-rubygem-activejob-6.0.3.4-1.el7.noarch
tfm-rubygem-activemodel-6.0.3.4-1.el7.noarch
tfm-rubygem-activerecord-6.0.3.4-1.el7.noarch
tfm-rubygem-activerecord-import-1.0.0-2.el7.noarch
tfm-rubygem-activerecord-session_store-1.1.1-4.el7.noarch
tfm-rubygem-activestorage-6.0.3.4-1.el7.noarch
tfm-rubygem-activesupport-6.0.3.4-1.el7.noarch
tfm-rubygem-addressable-2.6.0-2.el7.noarch
tfm-rubygem-algebrick-0.7.3-7.el7.noarch
tfm-rubygem-amazing_print-1.1.0-1.el7.noarch
tfm-rubygem-ancestry-3.0.7-1.el7.noarch
tfm-rubygem-anemone-0.7.2-17.el7.noarch
tfm-rubygem-angular-rails-templates-1.1.0-1.el7.noarch
tfm-rubygem-ansi-1.5.0-2.el7.noarch
tfm-rubygem-apipie-bindings-0.4.0-1.el7.noarch
tfm-rubygem-apipie-dsl-2.3.0-1.el7.noarch
tfm-rubygem-apipie-params-0.0.5-4.el7.noarch
tfm-rubygem-apipie-rails-0.5.17-3.el7.noarch
tfm-rubygem-arel-9.0.0-4.el7.noarch
tfm-rubygem-audited-4.9.0-3.el7.noarch
tfm-rubygem-awesome_print-1.8.0-5.el7.noarch
tfm-rubygem-bcrypt-3.1.12-3.el7.x86_64
tfm-rubygem-builder-3.2.4-1.el7.noarch
tfm-rubygem-bundler_ext-0.4.1-5.el7.noarch
tfm-rubygem-chunky_png-1.3.10-2.el7.noarch
tfm-rubygem-clamp-1.1.2-6.el7.noarch
tfm-rubygem-coffee-rails-5.0.0-1.el7.noarch
tfm-rubygem-coffee-script-2.4.1-4.el7.noarch
tfm-rubygem-coffee-script-source-1.12.2-4.el7.noarch
tfm-rubygem-concurrent-ruby-1.1.6-2.el7.noarch
tfm-rubygem-concurrent-ruby-edge-0.6.0-2.fm2_1.el7.noarch
tfm-rubygem-connection_pool-2.2.2-2.el7.noarch
tfm-rubygem-crass-1.0.6-1.el7.noarch
tfm-rubygem-css_parser-1.4.7-4.el7.noarch
tfm-rubygem-daemons-1.2.3-6.el7.noarch
tfm-rubygem-deacon-1.0.0-4.el7.noarch
tfm-rubygem-deep_cloneable-3.0.0-3.el7.noarch
tfm-rubygem-deface-1.5.3-2.el7.noarch
tfm-rubygem-domain_name-0.5.20160310-4.el7.noarch
tfm-rubygem-dynflow-1.4.7-1.fm2_3.el7.noarch
tfm-rubygem-erubi-1.9.0-1.el7.noarch
tfm-rubygem-ethon-0.12.0-1.el7.noarch
tfm-rubygem-excon-0.76.0-1.el7.noarch
tfm-rubygem-execjs-2.7.0-4.el7.noarch
tfm-rubygem-facter-2.4.0-7.el7.x86_64
tfm-rubygem-faraday-0.17.3-1.el7.noarch
tfm-rubygem-fast_gettext-1.4.1-4.el7.noarch
tfm-rubygem-ffi-1.12.2-1.el7.x86_64
tfm-rubygem-fog-core-2.1.0-3.el7.noarch
tfm-rubygem-fog-json-1.2.0-3.el7.noarch
tfm-rubygem-fog-libvirt-0.7.0-2.el7.noarch
tfm-rubygem-fog-proxmox-0.13.0-1.el7.noarch
tfm-rubygem-fog-xml-0.1.2-8.el7.noarch
tfm-rubygem-foreman-tasks-3.0.5-1.fm2_3.el7.noarch
tfm-rubygem-foreman-tasks-core-0.3.4-1.fm2_1.el7.noarch
tfm-rubygem-foreman_ansible-6.1.1-1.fm2_3.el7.noarch
tfm-rubygem-foreman_ansible_core-4.0.0-1.fm2_3.el7.noarch
tfm-rubygem-foreman_datacenter-2.3.0-1.fm2_3.el7.noarch
tfm-rubygem-foreman_dhcp_browser-0.0.8-4.fm2_1.el7.noarch
tfm-rubygem-foreman_discovery-16.3.4-1.fm2_3.el7.noarch
tfm-rubygem-foreman_fog_proxmox-0.12.4-1.fm2_3.el7.noarch
tfm-rubygem-foreman_openscap-4.1.3-1.fm2_3.el7.noarch
tfm-rubygem-foreman_remote_execution-4.2.2-1.fm2_3.el7.noarch
tfm-rubygem-foreman_remote_execution_core-1.4.0-1.el7.noarch
tfm-rubygem-foreman_statistics-1.0.0-1.fm2_3.el7.noarch
tfm-rubygem-formatador-0.2.1-12.el7.noarch
tfm-rubygem-friendly_id-5.3.0-1.el7.noarch
tfm-rubygem-fx-0.5.0-1.el7.noarch
tfm-rubygem-get_process_mem-0.2.1-4.el7.noarch
tfm-rubygem-gettext_i18n_rails-1.8.0-2.el7.noarch
tfm-rubygem-gitlab-sidekiq-fetcher-0.6.0-1.el7.noarch
tfm-rubygem-globalid-0.4.2-1.el7.noarch
tfm-rubygem-graphql-1.8.14-2.el7.noarch
tfm-rubygem-graphql-batch-0.3.10-2.el7.noarch
tfm-rubygem-gssapi-1.2.0-7.el7.noarch
tfm-rubygem-hammer_cli-2.3.0-1.el7.noarch
tfm-rubygem-hammer_cli_csv-2.3.1-2.el7.noarch
tfm-rubygem-hammer_cli_foreman-2.3.1-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_ansible-0.3.2-1.fm2_1.el7.noarch
tfm-rubygem-hammer_cli_foreman_bootdisk-0.3.0-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_discovery-1.0.2-1.fm2_1.el7.noarch
tfm-rubygem-hammer_cli_foreman_docker-0.0.7-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_openscap-0.1.12-1.fm2_3.el7.noarch
tfm-rubygem-hammer_cli_foreman_remote_execution-0.2.1-1.fm2_3.el7.noarch
tfm-rubygem-hammer_cli_foreman_tasks-0.0.15-1.fm2_2.el7.noarch
tfm-rubygem-hammer_cli_foreman_templates-0.2.0-1.fm2_1.el7.noarch
tfm-rubygem-hammer_cli_katello-0.24.3-1.el7.noarch
tfm-rubygem-hashie-3.6.0-2.el7.noarch
tfm-rubygem-highline-1.7.8-5.el7.noarch
tfm-rubygem-http-cookie-1.0.2-4.el7.noarch
tfm-rubygem-i18n-1.8.2-1.el7.noarch
tfm-rubygem-ipaddress-0.8.0-12.el7.noarch
tfm-rubygem-jwt-2.2.1-2.el7.noarch
tfm-rubygem-kafo-6.1.2-1.el7.noarch
tfm-rubygem-kafo_parsers-1.1.0-3.el7.noarch
tfm-rubygem-kafo_wizards-0.0.1-4.el7.noarch
tfm-rubygem-katello-3.18.2.1-1.el7.noarch
tfm-rubygem-ldap_fluff-0.4.7-5.el7.noarch
tfm-rubygem-little-plugger-1.1.4-2.el7.noarch
tfm-rubygem-locale-2.0.9-14.el7.noarch
tfm-rubygem-logging-2.3.0-1.el7.noarch
tfm-rubygem-loofah-2.4.0-1.el7.noarch
tfm-rubygem-mail-2.7.1-1.el7.noarch
tfm-rubygem-marcel-0.3.3-1.el7.noarch
tfm-rubygem-method_source-0.9.2-2.el7.noarch
tfm-rubygem-mime-types-3.2.2-4.el7.noarch
tfm-rubygem-mime-types-data-3.2018.0812-4.el7.noarch
tfm-rubygem-mimemagic-0.3.5-1.el7.noarch
tfm-rubygem-mini_mime-1.0.2-1.el7.noarch
tfm-rubygem-mini_portile2-2.4.0-1.el7.noarch
tfm-rubygem-multi_json-1.14.1-2.el7.noarch
tfm-rubygem-multipart-post-2.0.0-2.el7.noarch
tfm-rubygem-mustermann-1.0.2-4.el7.noarch
tfm-rubygem-net-ldap-0.16.1-2.el7.noarch
tfm-rubygem-net-ping-2.0.1-4.el7.noarch
tfm-rubygem-net-scp-1.2.1-4.el7.noarch
tfm-rubygem-net-ssh-4.2.0-2.el7.noarch
tfm-rubygem-netrc-0.11.0-5.el7.noarch
tfm-rubygem-nio4r-2.5.4-1.el7.x86_64
tfm-rubygem-nokogiri-1.10.9-1.el7.x86_64
tfm-rubygem-oauth-0.5.4-4.el7.noarch
tfm-rubygem-openscap-0.4.9-4.el7.noarch
tfm-rubygem-parse-cron-0.1.4-4.fm2_1.el7.noarch
tfm-rubygem-passenger-4.0.18-10.13.el7.x86_64
tfm-rubygem-passenger-native-4.0.18-10.13.el7.x86_64
tfm-rubygem-passenger-native-libs-4.0.18-10.13.el7.x86_64
tfm-rubygem-pdf-core-0.7.0-2.el7.noarch
tfm-rubygem-pg-1.1.4-3.el7.x86_64
tfm-rubygem-polyglot-0.3.5-2.el7.noarch
tfm-rubygem-powerbar-2.0.1-2.el7.noarch
tfm-rubygem-prawn-2.2.2-2.el7.noarch
tfm-rubygem-prometheus-client-1.0.0-2.el7.noarch
tfm-rubygem-promise.rb-0.7.4-2.el7.noarch
tfm-rubygem-public_suffix-3.0.3-2.el7.noarch
tfm-rubygem-pulp_2to3_migration_client-0.7.0-1.el7.noarch
tfm-rubygem-pulp_ansible_client-0.4.2-1.el7.noarch
tfm-rubygem-pulp_certguard_client-1.0.3-1.el7.noarch
tfm-rubygem-pulp_container_client-2.1.0-1.el7.noarch
tfm-rubygem-pulp_deb_client-2.7.0-1.el7.noarch
tfm-rubygem-pulp_docker_client-4.0.0b6.dev01562331743-1.el7.noarch
tfm-rubygem-pulp_file_client-1.3.0-1.el7.noarch
tfm-rubygem-pulp_rpm_client-3.9.0-1.el7.noarch
tfm-rubygem-pulpcore_client-3.7.1-1.el7.noarch
tfm-rubygem-puma-4.3.6-1.el7.x86_64
tfm-rubygem-puma-plugin-systemd-0.1.5-1.el7.noarch
tfm-rubygem-qpid_messaging-1.38.0-1.el7.x86_64
tfm-rubygem-quantile-0.2.0-4.el7.noarch
tfm-rubygem-rabl-0.14.3-1.el7.noarch
tfm-rubygem-rack-2.2.3-1.el7.noarch
tfm-rubygem-rack-cors-1.0.2-2.el7.noarch
tfm-rubygem-rack-jsonp-1.3.1-9.el7.noarch
tfm-rubygem-rack-protection-2.0.3-4.el7.noarch
tfm-rubygem-rack-test-1.1.0-4.el7.noarch
tfm-rubygem-rails-6.0.3.4-1.el7.noarch
tfm-rubygem-rails-dom-testing-2.0.3-6.el7.noarch
tfm-rubygem-rails-html-sanitizer-1.3.0-1.el7.noarch
tfm-rubygem-rails-i18n-6.0.0-2.el7.noarch
tfm-rubygem-railties-6.0.3.4-1.el7.noarch
tfm-rubygem-rainbow-2.2.1-3.el7.noarch
tfm-rubygem-rake-compiler-1.0.7-3.el7.noarch
tfm-rubygem-rb-inotify-0.9.7-5.el7.noarch
tfm-rubygem-record_tag_helper-1.0.1-3.el7.noarch
tfm-rubygem-redfish_client-0.5.2-1.el7.noarch
tfm-rubygem-redis-4.1.2-2.el7.noarch
tfm-rubygem-responders-3.0.0-3.el7.noarch
tfm-rubygem-rest-client-2.0.2-3.el7.noarch
tfm-rubygem-rkerberos-0.1.5-19.el7.x86_64
tfm-rubygem-roadie-3.4.0-3.el7.noarch
tfm-rubygem-roadie-rails-2.1.1-2.el7.noarch
tfm-rubygem-robotex-1.0.0-21.el7.noarch
tfm-rubygem-rprogram-0.3.2-2.el7.noarch
tfm-rubygem-rqrcode-0.10.1-2.el7.noarch
tfm-rubygem-rsec-0.4.3-4.el7.noarch
tfm-rubygem-ruby-libvirt-0.7.1-1.el7.x86_64
tfm-rubygem-ruby-nmap-0.9.3-3.el7.noarch
tfm-rubygem-ruby2ruby-2.4.2-3.el7.noarch
tfm-rubygem-ruby_parser-3.10.1-3.el7.noarch
tfm-rubygem-rubyipmi-0.10.0-6.el7.noarch
tfm-rubygem-runcible-2.13.1-1.el7.noarch
tfm-rubygem-safemode-1.3.6-1.el7.noarch
tfm-rubygem-scoped_search-4.1.9-1.el7.noarch
tfm-rubygem-sd_notify-0.1.0-1.el7.noarch
tfm-rubygem-secure_headers-6.3.0-2.el7.noarch
tfm-rubygem-sequel-5.7.1-3.el7.noarch
tfm-rubygem-server_sent_events-0.1.2-1.el7.noarch
tfm-rubygem-sexp_processor-4.10.0-6.el7.noarch
tfm-rubygem-sidekiq-5.2.7-3.el7.noarch
tfm-rubygem-sinatra-2.0.3-4.el7.noarch
tfm-rubygem-smart_proxy_ansible-3.0.1-6.fm2_2.el7.noarch
tfm-rubygem-smart_proxy_discovery-1.0.5-6.fm2_2.el7.noarch
tfm-rubygem-smart_proxy_dynflow-0.3.0-2.fm2_3.el7.noarch
tfm-rubygem-smart_proxy_dynflow_core-0.3.2-1.fm2_3.el7.noarch
tfm-rubygem-smart_proxy_openscap-0.7.4-1.fm2_2.el7.noarch
tfm-rubygem-smart_proxy_pulp-2.1.0-3.fm2_2.el7.noarch
tfm-rubygem-smart_proxy_remote_execution_ssh-0.3.1-1.fm2_3.el7.noarch
tfm-rubygem-sprockets-4.0.2-1.el7.noarch
tfm-rubygem-sprockets-rails-3.2.1-6.el7.noarch
tfm-rubygem-sqlite3-1.3.13-6.el7.x86_64
tfm-rubygem-sshkey-1.9.0-4.el7.noarch
tfm-rubygem-statsd-instrument-2.1.4-3.el7.noarch
tfm-rubygem-stomp-1.4.9-1.el7.noarch
tfm-rubygem-thor-1.0.1-2.el7.noarch
tfm-rubygem-thread_safe-0.3.6-5.el7.noarch
tfm-rubygem-tilt-2.0.8-4.el7.noarch
tfm-rubygem-ttfunk-1.5.1-2.el7.noarch
tfm-rubygem-typhoeus-1.3.1-1.el7.noarch
tfm-rubygem-tzinfo-1.2.6-1.el7.noarch
tfm-rubygem-unf-0.1.3-8.el7.noarch
tfm-rubygem-unf_ext-0.0.7.2-3.el7.x86_64
tfm-rubygem-unicode-0.4.4.4-3.el7.x86_64
tfm-rubygem-unicode-display_width-1.0.5-4.el7.noarch
tfm-rubygem-validates_lengths_from_database-0.5.0-7.el7.noarch
tfm-rubygem-webpack-rails-0.9.8-5.el7.noarch
tfm-rubygem-websocket-driver-0.7.1-1.el7.x86_64
tfm-rubygem-websocket-extensions-0.1.5-1.el7.noarch
tfm-rubygem-will_paginate-3.1.7-3.el7.noarch
tfm-rubygem-xmlrpc-0.3.0-2.el7.noarch
tfm-rubygem-zeitwerk-2.2.2-1.el7.noarch
tfm-rubygem-zest-0.0.4-1.el7.noarch
tfm-runtime-6.1-4.el7.x86_64

Distribution and version:
CentOS 7.9
Other relevant data:

[root@foreman foreman-installer]# foreman-rake katello:pulp3_migration_stats --verbose
============Migration Summary================
Migrated/Total RPMs: 0/64643
Migrated/Total errata: 0/166785
Migrated/Total repositories: 15/361

Estimated migration time based on yum content: fewer than 5 minutes

foreman-rake katello:pulp3_post_migration_check
ERROR: yum repositories with ID [617,618,620,625,569,157,570,79,26,27,606,82,616,251,622,623,624,60,80,81,84,619,571,577,227,104,235,573,114,574,575,576,572,124,125,468,469,470,368,369,621,218,101,285,236,239,275,398,183,184,607,608,609,610,159,611,612,614,613,522,195,196,523,524,219,370,578,233,371,162,579,373,374,375,372,580,581,582,583,295,525,493,161,322,323,324,325,326,327,526,527,321,528,340,341,342,343,344,339,100,585,496,311,584,210,158,376,377,277,279,280,281,282,283,284,378,278,387,380,381,382,383,317,318,319,615,314,313,312,315,320,316,384,385,386,379,481,482,345,626,586,487,354,488,486,587,414,421,435,434,489,628,589,590,591,592,406,407,408,410,411,412,413,593,490,594,409,629,630,631,627,232,588,415,491,492,541,542,543,529,632,595,633,596,634,537,544,545,546,548,635,234,597,547,209,538,436,437,438,439,440,258,102,636,441,442,444,445,446,447,448,484,638,266,230,443,105,449,450,452,453,454,455,456,457,451,507,508,558,562,559,228,459,460,462,463,464,465,466,467,461,483,513,514,637,509,510,511,512,103,267,530,399,400,419,471,472,473,474,475,476,477,531,485,516,517,518,519,520,521,535,515,532,536,363,639,640,539,540,549,550,551,552,553,554,556,557,416,641,555,642,643,644,646,397,404,401,405,645,420,422,655,661,662,663,494,497,498,658,657,664,659,656,660,647,598,220,649,650,651,652,653,654,648,564,565,566,294,560,561,567,563,599,600,601,602,603,605,604,495,499,500,163,194,160] have a NULL value for version_href

foreman-rake katello:pulp3_migration_reset
enabled
Starting Content Migration Reset.
Content Migration Reset failed, You will want to investigate: https://foreman.solutionsathand.ca/foreman_tasks/tasks/c3424737-0a38-49be-8d4e-f66af97bdad2
rake aborted!
ForemanTasks::TaskError: Task c3424737-0a38-49be-8d4e-f66af97bdad2: ActiveRecord::StatementInvalid: PG::UndefinedColumn: ERROR:  column "migrated_pulp3_href" of relation "katello_debs" does not exist
LINE 1: UPDATE "katello_debs" SET "migrated_pulp3_href" = $1
                                  ^
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/lib/katello/tasks/pulp3_migration_reset.rake:18:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_migration_reset
(See full trace by running task with --trace)



[root@foreman foreman-installer]# foreman-rake katello:pulp3_migration
Starting task.
2021-04-01 10:19:08 -0400: Pre-migrating Pulp 2 PACKAGE_GROUP content (general info) 3400/3899Migration failed, You will want to investigate: https://foreman.solutionsathand.ca/foreman_tasks/tasks/8c4ab0b7-56bf-4915-9374-f3e2b1e0641a
rake aborted!
ForemanTasks::TaskError: Task 8c4ab0b7-56bf-4915-9374-f3e2b1e0641a: Katello::Errors::Pulp3Error: 'NoneType' object has no attribute 'replace'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/lib/katello/tasks/pulp3_migration.rake:33:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_migration
(See full trace by running task with --trace)
E, [2021-04-01T10:19:18.768827 #25802] ERROR -- /connector-database-core: Envelope Dynflow::Dispatcher::Envelope[request_id: d9424bb7-14c7-421d-ad7b-ebc385785017-1, sender_id: 9b36fd19-dbf8-4a6b-92ec-17b9d7a23dc4, receiver_id: d9424bb7-14c7-421d-ad7b-ebc385785017, message: Dynflow::Dispatcher::Done] received for stopped world

Now, it’s pretty much impossible for me to actually update any of the products and repos (deb or rpm), due to some issue that is affecting both the UI and the hammer cli.

Via the Web UI, Clicking on Products and then a specific product leads you to the Repositories page with a broken Breadcrumb trail, the error popups "wrong number of arguments (given 0, expected 1), and no ability to actually click any links on the page for products, Image tags, etc. as seen in the image below.

hammer behaves in a similar fasion when trying to do anything with products or content views:

[root@foreman foreman-installer]# hammer content-view show --id=6
wrong number of arguments (given 0, expected 1)

[root@foreman foreman-installer]# hammer repository synchronize --id 80
Could not synchronize the repository:
  wrong number of arguments (given 0, expected 1)

Help solving this would be greatly appreciated! Thanks!

jeremylenz · April 1, 2021, 3:25pm

Hi @Mkbrown, thanks for reporting! Looks like this is two issues, the failed migration and also errors in the web UI.

I think the failed migration is because of Bug #31875: Disable/ Work around deb migration before upgrading if there's no debian content - Katello - Foreman , which we’re working on a fix for. (There’s a workaround in the comments of that issue, but it may or may not be feasible for you)

I’d like to understand why we’re getting this Ruby argument error. Can you check log/production.log for that error (“wrong number of arguments”) and post the error traceback here?

thanks!

Mkbrown · April 1, 2021, 5:03pm

Jeremy,

The production log shows that the foreman-datacenter plugin may be at fault.

2021-04-01T09:18:47 [I|app|2d85ed51] Started GET "/katello/api/ping" for 127.0.0.1 at 2021-04-01 09:18:47 -0400
2021-04-01T09:18:47 [I|app|2d85ed51] Processing by Katello::Api::V2::KatelloPingController#index as JSON
2021-04-01T09:18:47 [I|app|2d85ed51]   Parameters: {"api_version"=>"v2"}
2021-04-01T09:18:47 [I|app|2d85ed51]   Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/views/katello/api/v2/katello_ping/show.json.rabl within katello/api/v2/layouts/resource
2021-04-01T09:18:47 [I|app|2d85ed51]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/views/katello/api/v2/katello_ping/show.json.rabl within katello/api/v2/layouts/resource (Duration: 14.9ms | Allocations: 2875)
2021-04-01T09:18:47 [E|app|2d85ed51] ActionView::Template::Error: wrong number of arguments (given 0, expected 1)
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_datacenter-2.3.0/app/helpers/foreman_datacenter/application_helper.rb:79:in `object_name'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/core_ext/object/try.rb:15:in `public_send'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/core_ext/object/try.rb:15:in `try'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/views/katello/api/v2/layouts/resource.json.erb:1:in `_a7e89c032c42a4413bfb1a062538244c'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.4/lib/action_view/base.rb:274:in `_run'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.4/lib/action_view/template.rb:185:in `block in render'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/notifications.rb:182:in `instrument'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.4/lib/action_view/template.rb:385:in `instrument_render_template'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.4/lib/action_view/template.rb:183:in `render'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/deface-1.5.3/lib/deface/action_view_extensions.rb:43:in `render'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.4/lib/action_view/renderer/template_renderer.rb:69:in `render_with_layout'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.4/lib/action_view/renderer/template_renderer.rb:56:in `render_template'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.4/lib/action_view/renderer/template_renderer.rb:13:in `render'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.4/lib/action_view/renderer/renderer.rb:61:in `render_template_to_object'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.4/lib/action_view/renderer/renderer.rb:29:in `render_to_object'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.4/lib/action_view/rendering.rb:117:in `block in _render_template'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.4/lib/action_view/base.rb:304:in `in_rendering_context'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.4/lib/action_view/rendering.rb:116:in `_render_template'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_controller/metal/streaming.rb:218:in `_render_template'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.4/lib/action_view/rendering.rb:103:in `render_to_body'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_controller/metal/rendering.rb:52:in `render_to_body'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_controller/metal/renderers.rb:142:in `render_to_body'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/abstract_controller/rendering.rb:25:in `render'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_controller/metal/rendering.rb:36:in `render'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_controller/metal/instrumentation.rb:44:in `block (2 levels) in render'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/core_ext/benchmark.rb:14:in `block in ms'
 2d85ed51 | /opt/rh/rh-ruby25/root/usr/share/ruby/benchmark.rb:308:in `realtime'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/core_ext/benchmark.rb:14:in `ms'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_controller/metal/instrumentation.rb:44:in `block in render'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_controller/metal/instrumentation.rb:84:in `cleanup_view_runtime'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/railties/controller_runtime.rb:34:in `cleanup_view_runtime'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_controller/metal/instrumentation.rb:43:in `render'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/katello/api/v2/rendering.rb:45:in `respond_with_template'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/katello/api/v2/rendering.rb:54:in `respond_with_template_resource'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/katello/api/v2/rendering.rb:6:in `respond_for_show'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/controllers/katello/api/v2/api_controller.rb:199:in `respond_for_show'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/controllers/katello/api/v2/katello_ping_controller.rb:15:in `index'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/abstract_controller/base.rb:195:in `process_action'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_controller/metal/rendering.rb:30:in `process_action'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:112:in `block in run_callbacks'
 2d85ed51 | /usr/share/foreman/app/controllers/concerns/foreman/controller/timezone.rb:10:in `set_timezone'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
 2d85ed51 | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
 2d85ed51 | /usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in `around'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in `around'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:139:in `run_callbacks'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/abstract_controller/callbacks.rb:41:in `process_action'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_controller/metal/rescue.rb:22:in `process_action'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_controller/metal/instrumentation.rb:33:in `block in process_action'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/notifications.rb:180:in `block in instrument'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/notifications.rb:180:in `instrument'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_controller/metal/params_wrapper.rb:245:in `process_action'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/railties/controller_runtime.rb:27:in `process_action'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/controllers/katello/concerns/api/api_controller.rb:40:in `process_action'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/abstract_controller/base.rb:136:in `process'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.4/lib/action_view/rendering.rb:39:in `process'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_controller/metal.rb:190:in `dispatch'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_controller/metal.rb:254:in `dispatch'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/routing/route_set.rb:50:in `dispatch'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/routing/route_set.rb:33:in `serve'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/routing/mapper.rb:18:in `block in <class:Constraints>'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/routing/mapper.rb:48:in `serve'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/journey/router.rb:49:in `block in serve'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/journey/router.rb:32:in `each'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/journey/router.rb:32:in `serve'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/routing/route_set.rb:834:in `call'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.4/lib/rails/engine.rb:527:in `call'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.4/lib/rails/railtie.rb:190:in `public_send'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.4/lib/rails/railtie.rb:190:in `method_missing'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/routing/mapper.rb:19:in `block in <class:Constraints>'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/routing/mapper.rb:48:in `serve'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/journey/router.rb:49:in `block in serve'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/journey/router.rb:32:in `each'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/journey/router.rb:32:in `serve'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/routing/route_set.rb:834:in `call'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/lib/katello/middleware/event_daemon.rb:10:in `call'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in `call'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in `call'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-dsl-2.3.0/lib/apipie_dsl/static_dispatcher.rb:67:in `call'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/static_dispatcher.rb:66:in `call'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/extractor/recorder.rb:137:in `call'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in `call'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in `call'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in `call'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in `call'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in `call'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in `call'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in `call'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in `call'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in `call'
 2d85ed51 | /usr/share/foreman/lib/foreman/middleware/telemetry.rb:10:in `call'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
 2d85ed51 | /usr/share/foreman/lib/foreman/middleware/catch_json_parse_errors.rb:9:in `call'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/prometheus-client-1.0.0/lib/prometheus/middleware/exporter.rb:32:in `call'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/tempfile_reaper.rb:15:in `call'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/etag.rb:27:in `call'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/conditional_get.rb:27:in `call'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/head.rb:12:in `call'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/http/content_security_policy.rb:18:in `call'
 2d85ed51 | /usr/share/foreman/lib/foreman/middleware/logging_context_session.rb:22:in `call'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/session/abstract/id.rb:266:in `context'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/session/abstract/id.rb:260:in `call'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/cookies.rb:648:in `call'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:101:in `run_callbacks'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/actionable_exceptions.rb:18:in `call'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/debug_exceptions.rb:32:in `call'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.4/lib/rails/rack/logger.rb:37:in `call_app'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.4/lib/rails/rack/logger.rb:28:in `call'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in `call'
 2d85ed51 | /usr/share/foreman/lib/foreman/middleware/logging_context_request.rb:11:in `call'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/request_id.rb:27:in `call'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/lib/katello/prevent_json_parsing.rb:12:in `call'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/method_override.rb:24:in `call'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/runtime.rb:22:in `call'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/executor.rb:14:in `call'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in `call'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/sendfile.rb:110:in `call'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/host_authorization.rb:76:in `call'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/secure_headers-6.3.0/lib/secure_headers/middleware.rb:11:in `call'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.4/lib/rails/engine.rb:527:in `call'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.4/lib/rails/railtie.rb:190:in `public_send'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.4/lib/rails/railtie.rb:190:in `method_missing'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:74:in `block in call'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `each'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `call'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.6/lib/puma/configuration.rb:228:in `call'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.6/lib/puma/server.rb:713:in `handle_request'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.6/lib/puma/server.rb:472:in `process_client'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.6/lib/puma/server.rb:328:in `block in run'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.6/lib/puma/thread_pool.rb:134:in `block in spawn_thread'
 2d85ed51 | /opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2021-04-01T09:18:47 [I|app|2d85ed51] Completed 500 Internal Server Error in 242ms (Views: 0.2ms | ActiveRecord: 3.2ms | Allocations: 20619)

Foreman-datacenter has been problematic since I re-installed it to try this current version. Is it being actively maintained anymore? I can try backing out that plugin, but that usually results in some other breakage as plugin removals don’t seem to clean up after themselves.

Also noticed a WARNING for the /about page.

||2021-04-01T09:28:35 [I|app|6dc122ff] Started GET "/about" for 127.0.0.1 at 2021-04-01 09:28:35 -0400|
|---|---|
||2021-04-01T09:28:35 [I|app|6dc122ff] Processing by AboutController#index as HTML|
||2021-04-01T09:28:35 [I|app|6dc122ff] ^[[1;32mDeface: [WARNING]^[[0m No :original defined for 'add_installed_packages_to_about', you should change its definition to include:|
|| 6dc122ff |  :original => '0be98d57a689d47f95b23f7fbc3ab40a52c160f7'|
||2021-04-01T09:28:35 [I|app|6dc122ff]   Rendering about/index.html.erb within layouts/application|
||2021-04-01T09:28:35 [I|app|a674da31] Started GET "/about" for 127.0.0.1 at 2021-04-01 09:28:35 -0400|
||2021-04-01T09:28:35 [I|app|a674da31] Processing by AboutController#index as HTML|
||2021-04-01T09:28:35 [I|app|a674da31]   Rendering about/index.html.erb within layouts/application|
||2021-04-01T09:28:36 [I|app|6dc122ff]   Rendered about/index.html.erb within layouts/application (Duration: 982.0ms | Allocations: 40302)|
||2021-04-01T09:28:36 [I|app|6dc122ff]   Rendered layouts/_application_content.html.erb (Duration: 4.6ms | Allocations: 7426)|
||2021-04-01T09:28:36 [I|app|6dc122ff]   Rendering layouts/base.html.erb|
||2021-04-01T09:28:36 [I|app|6dc122ff]   Rendered layouts/base.html.erb (Duration: 31.2ms | Allocations: 32644)|
||2021-04-01T09:28:36 [I|app|6dc122ff] Completed 200 OK in 1078ms (Views: 1025.2ms | ActiveRecord: 8.9ms | Allocations: 100848)|

Definitely seeing the Katello deb issue, so I’ll probably delete the repos and disable as per the workaround, migrate, and then recreate after.

2021-04-01T09:54:10 [E|bac|] PG::UndefinedColumn: ERROR:  column "migrated_pulp3_href" of relation "katello_debs" does not exist
 | LINE 1: UPDATE "katello_debs" SET "migrated_pulp3_href" = $1

On a side note, looks like foreman-maintain could use a log rotate job. Single log file, getting kinda large.

Mkbrown · April 1, 2021, 5:11pm

OK, just confirmed that backing out the foreman-datacenter plugin has restored the UI functioning, and can now edit products and the like. So, the plugin is currently breaking Foreman/Katello integration, at least in my environment.

jeremylenz · April 1, 2021, 5:20pm

Glad the UI errors are gone now, at least!

Indeed, looks like the issue has already been raised here: foreman_datacenter (1.20.3) - ActionView::Template::Error: wrong number of arguments (given 0, expected 1) · Issue #30 · theforeman/foreman_datacenter · GitHub

Mkbrown · April 1, 2021, 7:05pm

So, ran foreman-installer with the workaround to disable deb.

Executing: foreman-rake upgrade:run
=============================================
Upgrade Step 1/3: katello:correct_repositories. This may take a long while.
Processing Repository 1/283: Puppet EL 7 - x86_64 (26)
Processing Repository 2/283: Puppet EL 8 - x86_64 (27)
Processing Repository 3/283: ISOs (44)
Failed upgrade task: katello:correct_repositories, see logs for more information.
=============================================
Upgrade Step 2/3: katello:correct_puppet_environments. This may take a long while.
Processing Puppet Environment 1/25: 1-CentOS_8-v6_0-puppet-97d7f207-8133-4219-9867-e7dc274a980f (25)
Failed upgrade task: katello:correct_puppet_environments, see logs for more information.
=============================================
Upgrade Step 3/3: katello:clean_backend_objects. This may take a long while.
0 orphaned consumer id(s) found in candlepin.
Candlepin orphaned consumers: []
0 orphaned consumer id(s) found in pulp.
Pulp orphaned consumers: []

Still breaking on upgrade tasks for migration. Does foreman-rake log to production log, or somewhere else?

Mkbrown · April 1, 2021, 7:13pm

Ok, did find some stuff in the production log:

2021-04-01T15:03:12 [I|app|] Rails cache backend: File
2021-04-01T15:03:16 [W|app|] Creating scope :completer_scope. Overwriting existing method Organization.completer_scope.
2021-04-01T15:03:16 [W|app|] Scoped order is ignored, it's forced to be batch order.
2021-04-01T15:03:16 [W|app|] Creating scope :completer_scope. Overwriting existing method Location.completer_scope.
2021-04-01T15:03:27 [W|app|] Failed upgrade task: katello:correct_repositories
2021-04-01T15:03:27 [I|app|] Backtrace for 'Failed upgrade task: katello:correct_repositories' error (NoMethodError): undefined method `repository_href' for nil:NilClass
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/lib/katello/tasks/repository.rake:142:in `repo_exists?'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/lib/katello/tasks/repository.rake:62:in `block (3 levels) in <top (required)>'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:70:in `block (2 levels) in find_each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:70:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:70:in `block in find_each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:136:in `block in find_in_batches'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:238:in `block in in_batches'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:222:in `loop'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:222:in `in_batches'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:135:in `find_in_batches'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:69:in `find_each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/lib/katello/tasks/repository.rake:60:in `with_index'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/lib/katello/tasks/repository.rake:60:in `block (2 levels) in <top (required)>'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in `block in execute'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in `each'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in `execute'
 | /usr/share/foreman/lib/tasks/upgrade.rake:32:in `run_task'
 | /usr/share/foreman/lib/tasks/upgrade.rake:27:in `block (3 levels) in <top (required)>'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/delegation.rb:87:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/delegation.rb:87:in `each'
 | /usr/share/foreman/lib/tasks/upgrade.rake:19:in `each_with_index'
 | /usr/share/foreman/lib/tasks/upgrade.rake:19:in `block (2 levels) in <top (required)>'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in `block in execute'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in `each'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in `execute'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:195:in `block in invoke_with_call_chain'
 | /opt/rh/rh-ruby25/root/usr/share/ruby/monitor.rb:226:in `mon_synchronize'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:188:in `invoke_with_call_chain'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:181:in `invoke'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:160:in `invoke_task'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in `block (2 levels) in top_level'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in `each'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in `block in top_level'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:125:in `run_with_threads'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:110:in `top_level'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:83:in `block in run'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:186:in `standard_exception_handling'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:80:in `run'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
 | /opt/rh/rh-ruby25/root/usr/bin/rake:23:in `load'
 | /opt/rh/rh-ruby25/root/usr/bin/rake:23:in `<main>'

and

2021-04-01T15:03:27 [W|app|] Failed upgrade task: katello:correct_puppet_environments
2021-04-01T15:03:27 [I|app|] Backtrace for 'Failed upgrade task: katello:correct_puppet_environments' error (RuntimeError): Cannot find repository type puppet, is it enabled?
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/models/katello/concerns/smart_proxy_extensions.rb:194:in `pulp3_repository_type_support?'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/models/katello/concerns/smart_proxy_extensions.rb:165:in `pulp3_support?'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/lib/katello/tasks/repository.rake:140:in `repo_exists?'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/lib/katello/tasks/repository.rake:81:in `block (3 levels) in <top (required)>'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:70:in `block (2 levels) in find_each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:70:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:70:in `block in find_each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:136:in `block in find_in_batches'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:238:in `block in in_batches'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:222:in `loop'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:222:in `in_batches'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:135:in `find_in_batches'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:69:in `find_each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/lib/katello/tasks/repository.rake:79:in `with_index'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/lib/katello/tasks/repository.rake:79:in `block (2 levels) in <top (required)>'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in `block in execute'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in `each'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in `execute'
 | /usr/share/foreman/lib/tasks/upgrade.rake:32:in `run_task'
 | /usr/share/foreman/lib/tasks/upgrade.rake:27:in `block (3 levels) in <top (required)>'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/delegation.rb:87:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/delegation.rb:87:in `each'
 | /usr/share/foreman/lib/tasks/upgrade.rake:19:in `each_with_index'
 | /usr/share/foreman/lib/tasks/upgrade.rake:19:in `block (2 levels) in <top (required)>'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in `block in execute'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in `each'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in `execute'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:195:in `block in invoke_with_call_chain'
 | /opt/rh/rh-ruby25/root/usr/share/ruby/monitor.rb:226:in `mon_synchronize'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:188:in `invoke_with_call_chain'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:181:in `invoke'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:160:in `invoke_task'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in `block (2 levels) in top_level'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in `each'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in `block in top_level'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:125:in `run_with_threads'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:110:in `top_level'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:83:in `block in run'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:186:in `standard_exception_handling'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:80:in `run'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
 | /opt/rh/rh-ruby25/root/usr/bin/rake:23:in `load'
 | /opt/rh/rh-ruby25/root/usr/bin/rake:23:in `<main>'

Now, the second one is likely due to disabling the puppet content type, as I wasn’t using that feature and it’s deprecated anyway. When I have reenabled it, I usually get more errors. I can re-enable if you want to capture those errors…

jeremylenz · April 1, 2021, 8:36pm

This “ISOs” repo… Is it of the File content type? Did you recreate that one under Pulp3?

Mkbrown · April 1, 2021, 8:42pm

It was file type, and I can’t remember if I recreated it or not, so I just deleted it and will try again.

Any ideas for the puppet content type issues?

Mkbrown · April 1, 2021, 8:50pm

Successfully made through repositories after deleting the ISO one. Hitting the puppet issue now.

Upgrade Step 2/3: katello:correct_puppet_environments. This may take a long while.
Processing Puppet Environment 1/25: 1-CentOS_8-v6_0-puppet-97d7f207-8133-4219-9867-e7dc274a980f (25)
Failed upgrade task: katello:correct_puppet_environments, see logs for more information.

From production log:

2021-04-01T16:46:35 [I|app|] Rails cache backend: File
2021-04-01T16:46:39 [W|app|] Creating scope :completer_scope. Overwriting existing method Organization.completer_scope.
2021-04-01T16:46:39 [W|app|] Scoped order is ignored, it's forced to be batch order.
2021-04-01T16:46:39 [W|app|] Creating scope :completer_scope. Overwriting existing method Location.completer_scope.
2021-04-01T16:47:01 [W|app|] Failed upgrade task: katello:correct_puppet_environments
2021-04-01T16:47:01 [I|app|] Backtrace for 'Failed upgrade task: katello:correct_puppet_environments' error (RuntimeError): Cannot find repository type puppet, is it enabled?
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/models/katello/concerns/smart_proxy_extensions.rb:194:in `pulp3_repository_type_support?'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/models/katello/concerns/smart_proxy_extensions.rb:165:in `pulp3_support?'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/lib/katello/tasks/repository.rake:140:in `repo_exists?'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/lib/katello/tasks/repository.rake:81:in `block (3 levels) in <top (required)>'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:70:in `block (2 levels) in find_each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:70:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:70:in `block in find_each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:136:in `block in find_in_batches'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:238:in `block in in_batches'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:222:in `loop'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:222:in `in_batches'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:135:in `find_in_batches'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:69:in `find_each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/lib/katello/tasks/repository.rake:79:in `with_index'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/lib/katello/tasks/repository.rake:79:in `block (2 levels) in <top (required)>'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in `block in execute'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in `each'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in `execute'
 | /usr/share/foreman/lib/tasks/upgrade.rake:32:in `run_task'
 | /usr/share/foreman/lib/tasks/upgrade.rake:27:in `block (3 levels) in <top (required)>'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/delegation.rb:87:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/delegation.rb:87:in `each'
 | /usr/share/foreman/lib/tasks/upgrade.rake:19:in `each_with_index'
 | /usr/share/foreman/lib/tasks/upgrade.rake:19:in `block (2 levels) in <top (required)>'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in `block in execute'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in `each'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in `execute'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:195:in `block in invoke_with_call_chain'
 | /opt/rh/rh-ruby25/root/usr/share/ruby/monitor.rb:226:in `mon_synchronize'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:188:in `invoke_with_call_chain'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:181:in `invoke'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:160:in `invoke_task'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in `block (2 levels) in top_level'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in `each'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in `block in top_level'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:125:in `run_with_threads'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:110:in `top_level'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:83:in `block in run'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:186:in `standard_exception_handling'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:80:in `run'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
 | /opt/rh/rh-ruby25/root/usr/bin/rake:23:in `load'
 | /opt/rh/rh-ruby25/root/usr/bin/rake:23:in `<main>'

So, I’m going to delete that view (it’s old), and see what happens…

jeremylenz · April 1, 2021, 8:58pm

That error is expected if you have puppet disabled, and you can safely ignore it. (should be fixed in Katello 4.0).

Mkbrown · April 1, 2021, 9:36pm

It still triggers a fail of foreman-rake upgrade:run though…

I’m seriously housecleaning old content views to avoid triggering that issue, so not sure if you want to make a note that anything over two years old might trigger that issue.

Mkbrown · April 1, 2021, 10:19pm

Confirmed. If there are issues with the Puppet content type in the upgrade:run stage, then the content prepare will also fail.

2021-04-01 17:44:31 -0400: Initial Migration steps complete.Migration failed, You will want to investigate: https://foreman.solutionsathand.ca/foreman_tasks/tasks/97527bb0-89df-4682-831c-4cf26bcbde7b
rake aborted!
ForemanTasks::TaskError: Task 97527bb0-89df-4682-831c-4cf26bcbde7b: Katello::Errors::Pulp3Error: 'NoneType' object has no attribute 'migrate_to_pulp3'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/lib/katello/tasks/pulp3_migration.rake:33:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_migration
(See full trace by running task with --trace)
                                                                      [FAIL]
Failed executing foreman-rake katello:pulp3_migration, exit status 1
--------------------------------------------------------------------------------
Scenario [Prepare content for Pulp 3] failed.

The following steps ended up in failing state:

  [content-prepare]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="content-prepare"

From the logs

2021-04-01T17:44:36 [E|bac|] 'NoneType' object has no attribute 'migrate_to_pulp3' (Katello::Errors::Pulp3Error)
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/pulp3/abstract_async_task.rb:102:in `block in check_for_errors'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/pulp3/abstract_async_task.rb:100:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/pulp3/abstract_async_task.rb:100:in `check_for_errors'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/pulp3/abstract_async_task.rb:133:in `poll_external_task'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:100:in `poll_external_task_with_rescue'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:22:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/cancellable.rb:14:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/pulp3/abstract_async_task.rb:10:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:571:in `block (3 levels) in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/middleware/remote_action.rb:16:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/execution_wrapper.rb:88:in `wrap'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:17:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_request_id.rb:49:in `restore_current_request_id'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_user.rb:44:in `restore_curent_user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/world.rb:31:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:570:in `block (2 levels) in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in `catch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in `block in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `block in with_error_handling'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `catch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `with_error_handling'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:564:in `execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:285:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/director.rb:93:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors.rb:18:in `run_user_code'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
 | [ sidekiq ]
 | [ concurrent-ruby ]
2021-04-01T17:44:36 [I|bac|] Task {label: Actions::Pulp3::ContentMigration, id: 97527bb0-89df-4682-831c-4cf26bcbde7b, execution_plan_id: 581249cb-97b6-49aa-b245-8235054d2e97} state changed: stopped  result: warning
2021-04-01T17:44:36 [I|bac|] Task {label: Actions::Pulp3::ContentMigration, id: 97527bb0-89df-4682-831c-4cf26bcbde7b, execution_plan_id: 581249cb-97b6-49aa-b245-8235054d2e97} state changed: stopped  result: warning

Mkbrown · April 1, 2021, 10:45pm

OK, so final status.

Had to re-enable the katello-puppet content in the foreman-installer, along with disabling the deb content, and then the upgrade:run stage finally passed.

foreman-installer --katello-use-pulp-2-for-deb=false --katello-enable-deb=false --katello-enable-puppet=true

I had deleted all content views except the active one with the katello-puppet disabled, and it would not pass upgrade:run without failures on one (or more) of the content views. When I re-enabled katello-puppet, it passed. Might want to have that as a note for upgrades…

Hitting another issue though with migrations:

[root@foreman foreman]# foreman-maintain content prepare
Running Prepare content for Pulp 3
================================================================================
Prepare content for Pulp 3:
Starting task.
2021-04-01 18:41:16 -0400: Initial Migration steps complete.Migration failed, You will want to investigate: https://foreman.solutionsathand.ca/foreman_tasks/tasks/5b9c2542-fc4f-445f-93d7-6d8468873421
rake aborted!
ForemanTasks::TaskError: Task 5b9c2542-fc4f-445f-93d7-6d8468873421: Katello::Errors::Pulp3Error: 'NoneType' object has no attribute 'migrate_to_pulp3'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/lib/katello/tasks/pulp3_migration.rake:33:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_migration
(See full trace by running task with --trace)
                                                                      [FAIL]
Failed executing foreman-rake katello:pulp3_migration, exit status 1
--------------------------------------------------------------------------------
Scenario [Prepare content for Pulp 3] failed.

The following steps ended up in failing state:

  [content-prepare]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="content-prepare"

And from production.log

2021-04-01T18:41:22 [E|bac|] 'NoneType' object has no attribute 'migrate_to_pulp3' (Katello::Errors::Pulp3Error)
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/pulp3/abstract_async_task.rb:102:in `block in check_for_errors'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/pulp3/abstract_async_task.rb:100:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/pulp3/abstract_async_task.rb:100:in `check_for_errors'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/pulp3/abstract_async_task.rb:133:in `poll_external_task'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:100:in `poll_external_task_with_rescue'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:22:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/cancellable.rb:14:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/pulp3/abstract_async_task.rb:10:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:571:in `block (3 levels) in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/middleware/remote_action.rb:16:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/execution_wrapper.rb:88:in `wrap'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:17:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_request_id.rb:49:in `restore_current_request_id'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_user.rb:44:in `restore_curent_user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/world.rb:31:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:570:in `block (2 levels) in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in `catch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in `block in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `block in with_error_handling'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `catch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `with_error_handling'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:564:in `execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:285:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/director.rb:93:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors.rb:18:in `run_user_code'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
 | [ sidekiq ]
 | [ concurrent-ruby ]
2021-04-01T18:41:22 [I|bac|] Task {label: Actions::Pulp3::ContentMigration, id: 5b9c2542-fc4f-445f-93d7-6d8468873421, execution_plan_id: fe574ce0-58f2-46d1-9cc7-9f11234286b1} state changed: stopped  result: warning
2021-04-01T18:41:22 [I|bac|] Task {label: Actions::Pulp3::ContentMigration, id: 5b9c2542-fc4f-445f-93d7-6d8468873421, execution_plan_id: fe574ce0-58f2-46d1-9cc7-9f11234286b1} state changed: stopped  result: warning

Justin_Sherrill · April 2, 2021, 12:18am

That looks like a pulp3 error from running the migration, can you check ‘journalctl -u pulpcore-worker@*’ for a similar error (with its traceback)?

Mkbrown · April 8, 2021, 7:36pm

Sorry for the delay in getting back to you; slammed at work after the long weekend here.

So, here’s the foremain maintain content prepare:

[root@foreman foreman]# foreman-maintain content prepare
Running Prepare content for Pulp 3
================================================================================
Prepare content for Pulp 3:
Starting task.
2021-04-08 15:20:47 -0400: Initial Migration steps complete.Migration failed, You will want to investigate: https://foreman.solutionsathand.ca/foreman_tasks/tasks/e9c56ce0-05a5-4f6f-8e91-5171d66834ad
rake aborted!
ForemanTasks::TaskError: Task e9c56ce0-05a5-4f6f-8e91-5171d66834ad: Katello::Errors::Pulp3Error: 'NoneType' object has no attribute 'migrate_to_pulp3'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/lib/katello/tasks/pulp3_migration.rake:33:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_migration
(See full trace by running task with --trace)
                                                                      [FAIL]
Failed executing foreman-rake katello:pulp3_migration, exit status 1
--------------------------------------------------------------------------------
Scenario [Prepare content for Pulp 3] failed.

The following steps ended up in failing state:

  [content-prepare]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="content-prepare"

The foreman production log:

2021-04-08T15:20:47 [E|bac|] 'NoneType' object has no attribute 'migrate_to_pulp3' (Katello::Errors::Pulp3Error)
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/pulp3/abstract_async_task.rb:102:i
n `block in check_for_errors'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/pulp3/abstract_async_task.rb:100:i
n `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/pulp3/abstract_async_task.rb:100:i
n `check_for_errors'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/pulp3/abstract_async_task.rb:133:i
n `poll_external_task'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:100:in `poll_external_t
ask_with_rescue'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:22:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/cancellable.rb:14:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/pulp3/abstract_async_task.rb:10:in
 `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:571:in `block (3 levels) in exe
cute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/middleware/remote_action.rb:16:in
`block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/middleware/remote_action.rb:40:in
`block in as_remote_user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/models/katello/concerns/user_extensions.rb:21:
in `cp_config'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/middleware/remote_action.rb:27:in
`as_cp_user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/middleware/remote_action.rb:39:in
`as_remote_user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/middleware/remote_action.rb:16:in
`run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/rails_executor_wrap.
rb:14:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/execution_wrapper.rb:88:in
 `wrap'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/rails_executor_wrap.
rb:13:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:31:in `with_progress_c
alculation'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:17:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_request
_id.rb:15:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_request
_id.rb:49:in `restore_current_request_id'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_user.rb:44:in `restore_curent_user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/world.rb:31:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:570:in `block (2 levels) in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in `catch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in `block in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `block in with_error_handling'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `catch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `with_error_handling'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:564:in `execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:285:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/director.rb:93:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors.rb:18:in `run_user_code'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
 | [ sidekiq ]
 | [ concurrent-ruby ]
2021-04-08T15:20:48 [I|bac|] Task {label: Actions::Pulp3::ContentMigration, id: e9c56ce0-05a5-4f6f-8e91-5171d66834ad, execution_plan_id: 7722e757-6252-48f4-aa65-f4fd539e78b4} state changed: stopped  result: warning
2021-04-08T15:20:48 [I|bac|] Task {label: Actions::Pulp3::ContentMigration, id: e9c56ce0-05a5-4f6f-8e91-5171d66834ad, execution_plan_id: 7722e757-6252-48f4-aa65-f4fd539e78b4} state changed: stopped  result: warning
2021-04-08T15:20:57 [I|dyn|] start terminating throttle_limiter...
2021-04-08T15:20:57 [I|dyn|] start terminating client dispatcher...
2021-04-08T15:20:57 [I|dyn|] stop listening for new events...
2021-04-08T15:20:57 [I|dyn|] start terminating clock...

And the journalctl output.

Apr 08 15:16:50 foreman.solutionsathand.ca pulpcore-worker-2[10309]: pulp: rq.worker:INFO: 10309@foreman.solutionsathand.ca: 99711206-1150-49d9-bab5-7ea473
Apr 08 15:16:50 foreman.solutionsathand.ca pulpcore-worker-2[10309]: pulp: pulp_2to3_migration.pulp2.connection:INFO: Attempting to connect to localhost:27
Apr 08 15:19:44 foreman.solutionsathand.ca pulpcore-worker-1[10312]: pulp: rq.worker:INFO: Cleaning registries for queue: 10312@foreman.solutionsathand.ca
Apr 08 15:20:45 foreman.solutionsathand.ca pulpcore-worker-2[10309]: /usr/lib/python3.6/site-packages/django/db/models/fields/__init__.py:1427: RuntimeWarn
Apr 08 15:20:45 foreman.solutionsathand.ca pulpcore-worker-2[10309]: RuntimeWarning)
Apr 08 15:20:45 foreman.solutionsathand.ca pulpcore-worker-2[10309]: pulp: rq.worker:ERROR: Traceback (most recent call last):
Apr 08 15:20:45 foreman.solutionsathand.ca pulpcore-worker-2[10309]: File "/usr/lib/python3.6/site-packages/rq/worker.py", line 936, in perform_job
Apr 08 15:20:45 foreman.solutionsathand.ca pulpcore-worker-2[10309]: rv = job.perform()
Apr 08 15:20:45 foreman.solutionsathand.ca pulpcore-worker-2[10309]: File "/usr/lib/python3.6/site-packages/rq/job.py", line 684, in perform
Apr 08 15:20:45 foreman.solutionsathand.ca pulpcore-worker-2[10309]: self._result = self._execute()
Apr 08 15:20:45 foreman.solutionsathand.ca pulpcore-worker-2[10309]: File "/usr/lib/python3.6/site-packages/rq/job.py", line 690, in _execute
Apr 08 15:20:45 foreman.solutionsathand.ca pulpcore-worker-2[10309]: return self.func(*self.args, **self.kwargs)
Apr 08 15:20:45 foreman.solutionsathand.ca pulpcore-worker-2[10309]: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/tasks/migrate.py", line
Apr 08 15:20:45 foreman.solutionsathand.ca pulpcore-worker-2[10309]: migrate_importers(plan)
Apr 08 15:20:45 foreman.solutionsathand.ca pulpcore-worker-2[10309]: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py", line 134
Apr 08 15:20:45 foreman.solutionsathand.ca pulpcore-worker-2[10309]: remote, created = importer_migrator.migrate_to_pulp3(pulp2importer)
Apr 08 15:20:45 foreman.solutionsathand.ca pulpcore-worker-2[10309]: AttributeError: 'NoneType' object has no attribute 'migrate_to_pulp3'
Apr 08 15:20:45 foreman.solutionsathand.ca pulpcore-worker-2[10309]: Traceback (most recent call last):
Apr 08 15:20:45 foreman.solutionsathand.ca pulpcore-worker-2[10309]: File "/usr/lib/python3.6/site-packages/rq/worker.py", line 936, in perform_job
Apr 08 15:20:45 foreman.solutionsathand.ca pulpcore-worker-2[10309]: rv = job.perform()
Apr 08 15:20:45 foreman.solutionsathand.ca pulpcore-worker-2[10309]: File "/usr/lib/python3.6/site-packages/rq/job.py", line 684, in perform
Apr 08 15:20:45 foreman.solutionsathand.ca pulpcore-worker-2[10309]: self._result = self._execute()
Apr 08 15:20:45 foreman.solutionsathand.ca pulpcore-worker-2[10309]: File "/usr/lib/python3.6/site-packages/rq/job.py", line 690, in _execute
Apr 08 15:20:45 foreman.solutionsathand.ca pulpcore-worker-2[10309]: return self.func(*self.args, **self.kwargs)
Apr 08 15:20:45 foreman.solutionsathand.ca pulpcore-worker-2[10309]: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/tasks/migrate.py", line
Apr 08 15:20:45 foreman.solutionsathand.ca pulpcore-worker-2[10309]: migrate_importers(plan)
Apr 08 15:20:45 foreman.solutionsathand.ca pulpcore-worker-2[10309]: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py", line 134
Apr 08 15:20:45 foreman.solutionsathand.ca pulpcore-worker-2[10309]: remote, created = importer_migrator.migrate_to_pulp3(pulp2importer)
Apr 08 15:20:45 foreman.solutionsathand.ca pulpcore-worker-2[10309]: AttributeError: 'NoneType' object has no attribute 'migrate_to_pulp3'
Apr 08 15:20:45 foreman.solutionsathand.ca pulpcore-worker-2[10309]: pulp: rq.worker:INFO: 10309@foreman.solutionsathand.ca: f8301818-73dc-4eb5-94a8-f702ef
Apr 08 15:20:45 foreman.solutionsathand.ca pulpcore-worker-2[10309]: pulp: rq.worker:INFO: 10309@foreman.solutionsathand.ca: Job OK (f8301818-73dc-4eb5-94a
Apr 08 15:24:06 foreman.solutionsathand.ca pulpcore-worker-2[10309]: pulp: rq.worker:INFO: Cleaning registries for queue: 10309@foreman.solutionsathand.ca

Thanks for your assistance! /Mike

mcorr · April 13, 2021, 7:14pm

Hi all,

Just had a side conversation with @metalcated who seems to be experiencing this also.

Justin_Sherrill · April 14, 2021, 8:12pm

I asked the pulp team and they were curious what content types do you have enabled?

the output of: rpm -qa | grep pulp

should tell us that. The contents of /etc/foreman/katello.yaml should help too.

Mkbrown · April 14, 2021, 10:56pm

[root@foreman foreman]# rpm -qa | grep pulp
pulp-docker-plugins-3.2.9-1.el7.noarch
python-pulp-common-2.21.5-1.el7.noarch
pulp-rpm-plugins-2.21.5-1.el7.noarch
python3-pulp-file-1.3.0-1.el7.noarch
tfm-rubygem-pulp_ansible_client-0.4.2-1.el7.noarch
pulp-server-2.21.5-1.el7.noarch
tfm-rubygem-smart_proxy_pulp-2.1.0-3.fm2_2.el7.noarch
tfm-rubygem-pulp_certguard_client-1.0.3-1.el7.noarch
tfm-rubygem-pulp_2to3_migration_client-0.7.0-1.el7.noarch
tfm-rubygem-pulp_rpm_client-3.9.0-1.el7.noarch
python-pulp-puppet-common-2.21.5-1.el7.noarch
pulp-puppet-plugins-2.21.5-1.el7.noarch
python3-pulp-deb-2.7.0-1.el7.noarch
python3-pulp-certguard-1.0.3-1.el7.noarch
tfm-rubygem-pulp_container_client-2.1.0-1.el7.noarch
pulp-deb-plugins-1.10.2-1.el7.noarch
python3-pulpcore-3.7.4-1.el7.noarch
python-pulp-repoauth-2.21.5-1.el7.noarch
python-pulp-deb-common-1.10.2-1.el7.noarch
tfm-rubygem-pulpcore_client-3.7.1-1.el7.noarch
pulp-client-1.0-1.noarch
python-pulp-oid_validation-2.21.5-1.el7.noarch
python-pulp-streamer-2.21.5-1.el7.noarch
pulp-puppet-tools-2.21.5-1.el7.noarch
python-pulp-manifest-2.21.5-1.el7.noarch
pulpcore-selinux-1.2.3-2.el7.x86_64
python3-pulp-rpm-3.9.0-2.el7.noarch
python3-pulp-2to3-migration-0.10.0-1.el7.noarch
pulp-katello-1.0.3-1.el7.noarch
tfm-rubygem-pulp_docker_client-4.0.0b6.dev01562331743-1.el7.noarch
pulp-selinux-2.21.5-1.el7.noarch
tfm-rubygem-pulp_file_client-1.3.0-1.el7.noarch
python-pulp-docker-common-3.2.9-1.el7.noarch
python-pulp-rpm-common-2.21.5-1.el7.noarch
python3-pulp-container-2.1.0-1.el7.noarch

Thanks for the assistance!

Justin_Sherrill · April 15, 2021, 2:21pm

can you also upload the contents of /etc/foreman/katello.yaml (feel free to remove any hostnames that may be present)