Pulp3 migration failed (Foreman 2.2/Katello 3.17)

Problem: After upgrading my Foreman server from 2.0 -> 2.1 -> 2.2. I wanted to try the Pulp3 migration. It seems to fail with the following error, the output below is all terminal output from all steps in the manual:

[root@foreman ~]# chmod -R g+rwX /var/lib/pulp/content
[root@foreman ~]# find /var/lib/pulp/content -type d -perm -g-s -exec sudo chmod g+s {} \;
[root@foreman ~]# chgrp -R pulp /var/lib/pulp/content
[root@foreman ~]# foreman-maintain content prepare
Running Prepare content for Pulp 3
================================================================================
Prepare content for Pulp 3:                                           [FAIL]
Failed executing foreman-rake katello:pulp3_migration, exit status 1:
 Migration failed, You will want to investigate: https://foreman.lbhr.htm.lan/foreman_tasks/tasks/b3fcc8f8-c393-4ece-8d5e-5e9e33e929df
rake aborted!
ForemanTasks::TaskError: Task b3fcc8f8-c393-4ece-8d5e-5e9e33e929df: Katello::Errors::Pulp3Error: __init__() got an unexpected keyword argument 'box_settings'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.0/lib/katello/tasks/pulp3_migration.rake:17:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_migration
(See full trace by running task with --trace)
--------------------------------------------------------------------------------
Scenario [Prepare content for Pulp 3] failed.

The following steps ended up in failing state:

  [content-prepare]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="content-prepare"


[root@foreman ~]# 

As mentioned, this server was installed with Foreman 2.0 / Katello 3.15 and subsequently updated (back to back with reboots)

Expected outcome:
Content migration tool being happy

Foreman and Proxy versions:
Foreman 2.2
Katello 3.17

Foreman and Proxy plugin versions:


Installed Packages

    candlepin-3.1.22-1.el7.noarch
    candlepin-selinux-3.1.22-1.el7.noarch
    foreman-2.2.1-1.el7.noarch
    foreman-cli-2.2.1-1.el7.noarch
    foreman-debug-2.2.1-1.el7.noarch
    foreman-dynflow-sidekiq-2.2.1-1.el7.noarch
    foreman-installer-2.2.1-1.el7.noarch
    foreman-installer-katello-2.2.1-1.el7.noarch
    foreman-postgresql-2.2.1-1.el7.noarch
    foreman-proxy-2.2.1-1.el7.noarch
    foreman-release-2.2.1-1.el7.noarch
    foreman-selinux-2.2.1-1.el7.noarch
    foreman-service-2.2.1-1.el7.noarch
    foreman.lbhr.htm.lan-apache-1.0-1.noarch
    foreman.lbhr.htm.lan-foreman-client-1.0-1.noarch
    foreman.lbhr.htm.lan-foreman-proxy-1.0-1.noarch
    foreman.lbhr.htm.lan-foreman-proxy-client-1.0-1.noarch
    foreman.lbhr.htm.lan-puppet-client-1.0-1.noarch
    foreman.lbhr.htm.lan-qpid-broker-1.0-1.noarch
    foreman.lbhr.htm.lan-qpid-client-cert-1.0-1.noarch
    foreman.lbhr.htm.lan-qpid-router-client-1.0-1.noarch
    foreman.lbhr.htm.lan-qpid-router-server-1.0-1.noarch
    foreman.lbhr.htm.lan-tomcat-1.0-1.noarch
    katello-3.17.0-1.el7.noarch
    katello-certs-tools-2.7.1-2.el7.noarch
    katello-client-bootstrap-1.7.5-1.el7.noarch
    katello-common-3.17.0-1.el7.noarch
    katello-debug-3.17.0-1.el7.noarch
    katello-default-ca-1.0-1.noarch
    katello-repos-3.17.0-1.el7.noarch
    katello-selinux-3.4.0-1.el7.noarch
    katello-server-ca-1.0-1.noarch
    pulp-client-1.0-1.noarch
    pulp-deb-plugins-1.10.2-1.el7.noarch
    pulp-docker-plugins-3.2.8-1.el7.noarch
    pulp-katello-1.0.3-1.el7.noarch
    pulp-puppet-plugins-2.21.4-1.el7.noarch
    pulp-puppet-tools-2.21.4-1.el7.noarch
    pulp-rpm-plugins-2.21.4-2.el7.noarch
    pulp-selinux-2.21.4-2.el7.noarch
    pulp-server-2.21.4-2.el7.noarch
    pulpcore-selinux-1.0.0-2.el7.x86_64
    python-gofer-qpid-2.12.5-3.el7.noarch
    python-pulp-common-2.21.4-2.el7.noarch
    python-pulp-deb-common-1.10.2-1.el7.noarch
    python-pulp-docker-common-3.2.8-1.el7.noarch
    python-pulp-oid_validation-2.21.4-2.el7.noarch
    python-pulp-puppet-common-2.21.4-1.el7.noarch
    python-pulp-repoauth-2.21.4-2.el7.noarch
    python-pulp-rpm-common-2.21.4-2.el7.noarch
    python-pulp-streamer-2.21.4-2.el7.noarch
    python2-qpid-1.37.0-4.el7.noarch
    python2-qpid-proton-0.32.0-2.el7.x86_64
    python2-qpid-qmf-1.39.0-1.el7.x86_64
    python3-pulp-2to3-migration-0.5.0-1.el7.noarch
    python3-pulp-certguard-1.0.2-1.el7.noarch
    python3-pulp-container-2.0.1-1.el7.noarch
    python3-pulp-file-1.2.0-1.el7.noarch
    python3-pulp-rpm-3.6.2-1.el7.noarch
    python3-pulpcore-3.6.3-2.el7.noarch
    qpid-cpp-client-1.39.0-1.el7.x86_64
    qpid-cpp-client-devel-1.39.0-1.el7.x86_64
    qpid-cpp-server-1.39.0-1.el7.x86_64
    qpid-cpp-server-linearstore-1.39.0-1.el7.x86_64
    qpid-dispatch-router-1.14.0-1.el7.x86_64
    qpid-proton-c-0.32.0-2.el7.x86_64
    qpid-qmf-1.39.0-1.el7.x86_64
    qpid-tools-1.39.0-1.el7.noarch
    rubygem-foreman_maintain-0.6.13-1.el7.noarch
    tfm-rubygem-actioncable-6.0.3.1-1.el7.noarch
    tfm-rubygem-actionmailbox-6.0.3.1-1.el7.noarch
    tfm-rubygem-actionmailer-6.0.3.1-1.el7.noarch
    tfm-rubygem-actionpack-6.0.3.1-1.el7.noarch
    tfm-rubygem-actiontext-6.0.3.1-1.el7.noarch
    tfm-rubygem-actionview-6.0.3.1-1.el7.noarch
    tfm-rubygem-activejob-6.0.3.1-1.el7.noarch
    tfm-rubygem-activemodel-6.0.3.1-1.el7.noarch
    tfm-rubygem-activerecord-6.0.3.1-1.el7.noarch
    tfm-rubygem-activerecord-import-1.0.0-2.el7.noarch
    tfm-rubygem-activerecord-session_store-1.1.1-4.el7.noarch
    tfm-rubygem-activestorage-6.0.3.1-1.el7.noarch
    tfm-rubygem-activesupport-6.0.3.1-1.el7.noarch
    tfm-rubygem-addressable-2.6.0-2.el7.noarch
    tfm-rubygem-algebrick-0.7.3-7.el7.noarch
    tfm-rubygem-amazing_print-1.1.0-1.el7.noarch
    tfm-rubygem-ancestry-3.0.7-1.el7.noarch
    tfm-rubygem-anemone-0.7.2-17.el7.noarch
    tfm-rubygem-angular-rails-templates-1.1.0-1.el7.noarch
    tfm-rubygem-ansi-1.5.0-2.el7.noarch
    tfm-rubygem-apipie-bindings-0.4.0-1.el7.noarch
    tfm-rubygem-apipie-dsl-2.2.9-1.el7.noarch
    tfm-rubygem-apipie-params-0.0.5-4.el7.noarch
    tfm-rubygem-apipie-rails-0.5.17-3.el7.noarch
    tfm-rubygem-arel-9.0.0-4.el7.noarch
    tfm-rubygem-audited-4.9.0-3.el7.noarch
    tfm-rubygem-awesome_print-1.8.0-5.el7.noarch
    tfm-rubygem-bcrypt-3.1.12-3.el7.x86_64
    tfm-rubygem-builder-3.2.4-1.el7.noarch
    tfm-rubygem-bundler_ext-0.4.1-5.el7.noarch
    tfm-rubygem-clamp-1.1.2-6.el7.noarch
    tfm-rubygem-concurrent-ruby-1.1.6-2.el7.noarch
    tfm-rubygem-concurrent-ruby-edge-0.6.0-2.fm2_1.el7.noarch
    tfm-rubygem-connection_pool-2.2.2-2.el7.noarch
    tfm-rubygem-crass-1.0.6-1.el7.noarch
    tfm-rubygem-css_parser-1.4.7-4.el7.noarch
    tfm-rubygem-daemons-1.2.3-6.el7.noarch
    tfm-rubygem-deacon-1.0.0-4.el7.noarch
    tfm-rubygem-deep_cloneable-3.0.0-3.el7.noarch
    tfm-rubygem-deface-1.5.3-2.el7.noarch
    tfm-rubygem-domain_name-0.5.20160310-4.el7.noarch
    tfm-rubygem-dynflow-1.4.7-1.fm2_2.el7.noarch
    tfm-rubygem-erubi-1.9.0-1.el7.noarch
    tfm-rubygem-ethon-0.12.0-1.el7.noarch
    tfm-rubygem-excon-0.76.0-1.el7.noarch
    tfm-rubygem-facter-2.4.0-7.el7.x86_64
    tfm-rubygem-faraday-0.15.4-2.el7.noarch
    tfm-rubygem-fast_gettext-1.4.1-4.el7.noarch
    tfm-rubygem-ffi-1.12.2-1.el7.x86_64
    tfm-rubygem-fog-core-2.1.0-3.el7.noarch
    tfm-rubygem-foreman-tasks-3.0.1-1.fm2_2.el7.noarch
    tfm-rubygem-foreman-tasks-core-0.3.4-1.fm2_1.el7.noarch
    tfm-rubygem-foreman_discovery-16.2.0-1.fm2_2.el7.noarch
    tfm-rubygem-foreman_remote_execution-4.1.0-1.fm2_2.el7.noarch
    tfm-rubygem-foreman_remote_execution_core-1.3.1-1.el7.noarch
    tfm-rubygem-formatador-0.2.1-12.el7.noarch
    tfm-rubygem-friendly_id-5.3.0-1.el7.noarch
    tfm-rubygem-fx-0.5.0-1.el7.noarch
    tfm-rubygem-get_process_mem-0.2.1-4.el7.noarch
    tfm-rubygem-gettext_i18n_rails-1.8.0-2.el7.noarch
    tfm-rubygem-gitlab-sidekiq-fetcher-0.6.0-1.el7.noarch
    tfm-rubygem-globalid-0.4.2-1.el7.noarch
    tfm-rubygem-graphql-1.8.14-2.el7.noarch
    tfm-rubygem-graphql-batch-0.3.10-2.el7.noarch
    tfm-rubygem-gssapi-1.2.0-7.el7.noarch
    tfm-rubygem-hammer_cli-2.2.1-1.el7.noarch
    tfm-rubygem-hammer_cli_foreman-2.2.0-1.el7.noarch
    tfm-rubygem-hammer_cli_foreman_bootdisk-0.3.0-1.el7.noarch
    tfm-rubygem-hammer_cli_foreman_docker-0.0.7-1.el7.noarch
    tfm-rubygem-hammer_cli_foreman_tasks-0.0.15-1.fm2_2.el7.noarch
    tfm-rubygem-hammer_cli_katello-0.23.2-1.el7.noarch
    tfm-rubygem-hashie-3.6.0-2.el7.noarch
    tfm-rubygem-highline-1.7.8-5.el7.noarch
    tfm-rubygem-http-cookie-1.0.2-4.el7.noarch
    tfm-rubygem-i18n-1.8.2-1.el7.noarch
    tfm-rubygem-jwt-2.2.1-2.el7.noarch
    tfm-rubygem-kafo-5.0.1-1.el7.noarch
    tfm-rubygem-kafo_parsers-1.1.0-3.el7.noarch
    tfm-rubygem-kafo_wizards-0.0.1-4.el7.noarch
    tfm-rubygem-katello-3.17.0-1.el7.noarch
    tfm-rubygem-ldap_fluff-0.4.7-5.el7.noarch
    tfm-rubygem-little-plugger-1.1.4-2.el7.noarch
    tfm-rubygem-locale-2.0.9-14.el7.noarch
    tfm-rubygem-logging-2.2.2-5.el7.noarch
    tfm-rubygem-loofah-2.4.0-1.el7.noarch
    tfm-rubygem-mail-2.7.1-1.el7.noarch
    tfm-rubygem-marcel-0.3.3-1.el7.noarch
    tfm-rubygem-method_source-0.9.2-1.el7.noarch
    tfm-rubygem-mime-types-3.2.2-4.el7.noarch
    tfm-rubygem-mime-types-data-3.2018.0812-4.el7.noarch
    tfm-rubygem-mimemagic-0.3.5-1.el7.noarch
    tfm-rubygem-mini_mime-1.0.2-1.el7.noarch
    tfm-rubygem-mini_portile2-2.4.0-1.el7.noarch
    tfm-rubygem-multi_json-1.14.1-2.el7.noarch
    tfm-rubygem-multipart-post-2.0.0-2.el7.noarch
    tfm-rubygem-mustermann-1.0.2-4.el7.noarch
    tfm-rubygem-net-ldap-0.16.1-2.el7.noarch
    tfm-rubygem-net-ping-2.0.1-4.el7.noarch
    tfm-rubygem-net-scp-1.2.1-4.el7.noarch
    tfm-rubygem-net-ssh-4.2.0-2.el7.noarch
    tfm-rubygem-netrc-0.11.0-5.el7.noarch
    tfm-rubygem-nio4r-2.5.2-1.el7.x86_64
    tfm-rubygem-nokogiri-1.10.9-1.el7.x86_64
    tfm-rubygem-oauth-0.5.4-4.el7.noarch
    tfm-rubygem-parse-cron-0.1.4-4.fm2_1.el7.noarch
    tfm-rubygem-passenger-4.0.18-10.13.el7.x86_64
    tfm-rubygem-passenger-native-4.0.18-10.13.el7.x86_64
    tfm-rubygem-passenger-native-libs-4.0.18-10.13.el7.x86_64
    tfm-rubygem-pg-1.1.4-3.el7.x86_64
    tfm-rubygem-polyglot-0.3.5-2.el7.noarch
    tfm-rubygem-powerbar-2.0.1-2.el7.noarch
    tfm-rubygem-promise.rb-0.7.4-2.el7.noarch
    tfm-rubygem-public_suffix-3.0.3-2.el7.noarch
    tfm-rubygem-pulp_2to3_migration_client-0.5.0-1.el7.noarch
    tfm-rubygem-pulp_ansible_client-0.3.0-1.el7.noarch
    tfm-rubygem-pulp_certguard_client-1.0.2-1.el7.noarch
    tfm-rubygem-pulp_container_client-2.0.0-1.el7.noarch
    tfm-rubygem-pulp_file_client-1.2.0-1.el7.noarch
    tfm-rubygem-pulp_rpm_client-3.6.2-1.el7.noarch
    tfm-rubygem-pulpcore_client-3.6.0-1.el7.noarch
    tfm-rubygem-puma-4.3.3-4.el7.x86_64
    tfm-rubygem-qpid_messaging-1.38.0-1.el7.x86_64
    tfm-rubygem-rabl-0.14.3-1.el7.noarch
    tfm-rubygem-rack-2.2.3-1.el7.noarch
    tfm-rubygem-rack-cors-1.0.2-2.el7.noarch
    tfm-rubygem-rack-jsonp-1.3.1-9.el7.noarch
    tfm-rubygem-rack-protection-2.0.3-4.el7.noarch
    tfm-rubygem-rack-test-1.1.0-4.el7.noarch
    tfm-rubygem-rails-6.0.3.1-1.el7.noarch
    tfm-rubygem-rails-dom-testing-2.0.3-6.el7.noarch
    tfm-rubygem-rails-html-sanitizer-1.3.0-1.el7.noarch
    tfm-rubygem-rails-i18n-6.0.0-2.el7.noarch
    tfm-rubygem-railties-6.0.3.1-1.el7.noarch
    tfm-rubygem-rainbow-2.2.1-3.el7.noarch
    tfm-rubygem-rake-compiler-1.0.7-3.el7.noarch
    tfm-rubygem-rb-inotify-0.9.7-5.el7.noarch
    tfm-rubygem-record_tag_helper-1.0.1-3.el7.noarch
    tfm-rubygem-redis-4.1.2-2.el7.noarch
    tfm-rubygem-responders-3.0.0-3.el7.noarch
    tfm-rubygem-rest-client-2.0.2-3.el7.noarch
    tfm-rubygem-rkerberos-0.1.5-18.el7.x86_64
    tfm-rubygem-roadie-3.4.0-3.el7.noarch
    tfm-rubygem-roadie-rails-2.1.1-2.el7.noarch
    tfm-rubygem-robotex-1.0.0-21.el7.noarch
    tfm-rubygem-rsec-0.4.3-4.el7.noarch
    tfm-rubygem-ruby-libvirt-0.7.1-1.el7.x86_64
    tfm-rubygem-ruby2ruby-2.4.2-3.el7.noarch
    tfm-rubygem-ruby_parser-3.10.1-3.el7.noarch
    tfm-rubygem-rubyipmi-0.10.0-6.el7.noarch
    tfm-rubygem-runcible-2.13.1-1.el7.noarch
    tfm-rubygem-safemode-1.3.5-3.el7.noarch
    tfm-rubygem-scoped_search-4.1.9-1.el7.noarch
    tfm-rubygem-secure_headers-6.3.0-2.el7.noarch
    tfm-rubygem-sequel-5.7.1-3.el7.noarch
    tfm-rubygem-sexp_processor-4.10.0-6.el7.noarch
    tfm-rubygem-sidekiq-5.2.7-3.el7.noarch
    tfm-rubygem-sinatra-2.0.3-4.el7.noarch
    tfm-rubygem-smart_proxy_pulp-2.1.0-3.fm2_2.el7.noarch
    tfm-rubygem-sprockets-4.0.2-1.el7.noarch
    tfm-rubygem-sprockets-rails-3.2.1-6.el7.noarch
    tfm-rubygem-sshkey-1.9.0-4.el7.noarch
    tfm-rubygem-statsd-instrument-2.1.4-3.el7.noarch
    tfm-rubygem-stomp-1.4.9-1.el7.noarch
    tfm-rubygem-thor-1.0.1-2.el7.noarch
    tfm-rubygem-thread_safe-0.3.6-5.el7.noarch
    tfm-rubygem-tilt-2.0.8-4.el7.noarch
    tfm-rubygem-typhoeus-1.3.1-1.el7.noarch
    tfm-rubygem-tzinfo-1.2.6-1.el7.noarch
    tfm-rubygem-unf-0.1.3-8.el7.noarch
    tfm-rubygem-unf_ext-0.0.7.2-3.el7.x86_64
    tfm-rubygem-unicode-0.4.4.4-3.el7.x86_64
    tfm-rubygem-unicode-display_width-1.0.5-4.el7.noarch
    tfm-rubygem-validates_lengths_from_database-0.5.0-7.el7.noarch
    tfm-rubygem-webpack-rails-0.9.8-5.el7.noarch
    tfm-rubygem-websocket-driver-0.7.1-1.el7.x86_64
    tfm-rubygem-websocket-extensions-0.1.5-1.el7.noarch
    tfm-rubygem-will_paginate-3.1.7-3.el7.noarch
    tfm-rubygem-x-editable-rails-1.5.5-5.el7.noarch
    tfm-rubygem-xmlrpc-0.3.0-2.el7.noarch
    tfm-rubygem-zeitwerk-2.2.2-1.el7.noarch
    tfm-runtime-6.1-3.el7.x86_64

Distribution and version: CentOS 7

Other relevant data:

I’ll add my name to the list. I haven’t had any luck finding solutions to this exact problem and my search-fu didn’t yield any results either.