Pulp 2 to Pulp3 upgrade fails

Support
, ,
11

I tried to disable deb content as described with:

foreman-installer --katello-enable-deb=false

So, the error message is different now:

RuntimeError: Cannot find repository type deb, is it enabled

3 Likes
12

I tried the above and the migration was successful

foreman-installer --katello-enable-deb=false
foreman-rake katello:pulp3_migration
foreman-installer --katello-enable-deb=true
1 Like
13

Some extra info. I have another test machine where I had been experimenting with Debian. I had to remove the product and then clean-up my database using:

foreman-installer --katello-enable-deb=false
foreman-rake foreman_tasks:cleanup
foreman-rake foreman_tasks:cleanup:config
foreman-rake katello:pulp3_migration
foreman-installer --katello-enable-deb=true
14

Now I need to be able to do a docker pull on my containers.

15

Have same issue.

16

What is strange is that I can list the docker repos:

# docker search $(hostname)/fedora
NAME                                                                                            DESCRIPTION   STARS     OFFICIAL   AUTOMATED
test-centos_8_content-2_0-fedora_docker_containers-fedora_docker_container_repo                         0
test-development-centos_8_content-fedora_docker_containers-fedora_docker_container_repo                 0
test-fedora_docker_containers-fedora_docker_container_repo                                              0
test-library-centos_8_content-fedora_docker_containers-fedora_docker_container_repo                     0

Pulp3 now also shows the repos:

# pulp container repository list
[
  {
    "pulp_href": "/pulp/api/v3/repositories/container/container/9de7f6e5-f32b-4c26-a080-366e0bd3fa8f/",
    "pulp_created": "2021-01-22T08:12:30.889021Z",
    "versions_href": "/pulp/api/v3/repositories/container/container/9de7f6e5-f32b-4c26-a080-366e0bd3fa8f/versions/",
    "latest_version_href": "/pulp/api/v3/repositories/container/container/9de7f6e5-f32b-4c26-a080-366e0bd3fa8f/versions/0/",
    "name": "Fedora_Docker_container_repo-472668",
    "description": null,
    "remote": null
  },
..

I have tried all the ways I can think of to pull the container but none of the examples in the Foreman docs work.

17

Exactly the same behavior here (Oracle Linux 7.9, no deb content)

18

Hey @techietubby , can you share the command(s) you’ve tried, their output, and any errors you’re seeing show up in journalctl?

19

Hi Jonathon,

Yes of course, gladly! :slight_smile:

I migrated Pulp as follows:

# foreman-installer --katello-enable-deb=false
# foreman-rake foreman_tasks:cleanup
# foreman-rake foreman_tasks:cleanup:config foreman-rake
# foreman-rake katello:pulp3_migration
# foreman-installer --katello-enable-deb=true

I did a restorecon to ensure all SELinux labels were up date and the ran:

# docker pull foreman.test.com/test-centos_8_content-2_0-fedora_docker_containers-fedora_docker_container_repo

This gives:

# journalctl -l --since=09:26 --no-pager
-- Logs begin at Mon 2021-02-01 13:52:47 CET, end at Thu 2021-02-18 09:26:15 CET. --
Feb 18 09:26:15 foreman.test.com dockerd[8018]: time="2021-02-18T09:26:15.935487079+01:00" level=info msg="Attempting next endpoint for pull after error: Head https://foreman.test.com/v2/test-centos_8_content-2_0-fedora_docker_containers-fedora_docker_container_repo/manifests/latest: received unexpected HTTP status: 500 Internal Server Error"
Feb 18 09:26:15 foreman.test.com dockerd[8018]: time="2021-02-18T09:26:15.940765618+01:00" level=error msg="Handler for POST /v1.41/images/create returned error: Head https://foreman.test.com/v2/test-centos_8_content-2_0-fedora_docker_containers-fedora_docker_container_repo/manifests/latest: received unexpected HTTP status: 500 Internal Server Error"

If I add fedora to the tag the results are:

# docker pull foreman.test.com/test-centos_8_content-2_0-fedora_docker_containers-fedora_docker_container_repo/fedora
Using default tag: latest
Error response from daemon: unknown: test-centos_8_content-2_0-fedora_docker_containers-fedora_docker_container_repo/fedora was not found!

The journalctl gives:

Feb 18 09:27:37 foreman.test.com pulp[21363]: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred)
Feb 18 09:27:37 foreman.test.com pulp[21404]: celery.worker.strategy:INFO: Received task: pulp.server.controllers.repository.queue_download_deferred[2b716a9f-5926-4261-861b-30a0b8fe9e34]
Feb 18 09:27:37 foreman.test.com pulp[21407]: celery.worker.strategy:INFO: Received task: pulp.server.controllers.repository.download_deferred[aba775b8-f924-4ddc-a8b1-5a7c27c617d6]
Feb 18 09:27:37 foreman.test.com pulp[21472]: celery.app.trace:INFO: [2b716a9f] Task pulp.server.controllers.repository.queue_download_deferred[2b716a9f-5926-4261-861b-30a0b8fe9e34] succeeded in 0.00635926402174s: None
Feb 18 09:27:38 foreman.test.com pulp[21473]: celery.app.trace:INFO: [aba775b8] Task pulp.server.controllers.repository.download_deferred[aba775b8-f924-4ddc-a8b1-5a7c27c617d6] succeeded in 1.01021712599s: None
Feb 18 09:30:01 foreman.test.com systemd[1]: Created slice User Slice of foreman-proxy.
Feb 18 09:30:01 foreman.test.com systemd[1]: Started Session 2149 of user foreman-proxy.
Feb 18 09:30:01 foreman.test.com systemd[1]: Created slice User Slice of foreman.
Feb 18 09:30:01 foreman.test.com systemd[1]: Started Session 2148 of user foreman.
Feb 18 09:30:01 foreman.test.com CROND[8794]: (foreman-proxy) CMD (smart-proxy-openscap-send >> /var/log/foreman-proxy/cron.log)
Feb 18 09:30:01 foreman.test.com CROND[8795]: (foreman) CMD (   /usr/sbin/foreman-rake ldap:refresh_usergroups >>/var/log/foreman/cron.log 2>&1)
Feb 18 09:30:02 foreman.test.com systemd[1]: Removed slice User Slice of foreman-proxy.
Feb 18 09:30:20 foreman.test.com dockerd[8018]: time="2021-02-18T09:30:20.819236660+01:00" level=error msg="Not continuing with pull after error: unknown: test-centos_8_content-2_0-fedora_docker_containers-fedora_docker_container_repo/fedora was not found!"
Feb 18 09:30:20 foreman.test.com dockerd[8018]: time="2021-02-18T09:30:20.823651356+01:00" level=error msg="Handler for POST /v1.41/images/create returned error: unknown: test-centos_8_content-2_0-fedora_docker_containers-fedora_docker_container_repo/fedora was not found!"
Feb 18 09:30:24 foreman.test.com systemd[1]: Removed slice User Slice of foreman.
Feb 18 09:30:39 foreman.test.com pulpcore-worker-4[15884]: pulp: rq.worker:INFO: Cleaning registries for queue: 15884@foreman.test.com
Feb 18 09:30:39 foreman.test.com pulpcore-worker-2[15883]: pulp: rq.worker:INFO: Cleaning registries for queue: 15883@foreman.test.com
Feb 18 09:30:41 foreman.test.com pulpcore-worker-1[15882]: pulp: rq.worker:INFO: Cleaning registries for queue: 15882@foreman.test.com
Feb 18 09:30:42 foreman.test.com pulpcore-worker-3[15881]: pulp: rq.worker:INFO: Cleaning registries for queue: 15881@foreman.test.com

Do you need more examples?

Regards,
Andrew

20

Looks like there should be an ISE traceback somewhere. Can try tailing /var/log/foreman/production.log while you docker pull?

21

I get:

2021-02-18T18:31:58 [I|app|0141592d] Filter chain halted as :authorize_repository_read rendered or redirected
2021-02-18T18:31:58 [I|app|0141592d] Completed 404 Not Found in 20ms (Views: 0.2ms | ActiveRecord: 7.3ms | Allocations: 6946)
2021-02-18T18:31:58 [I|app|6b706c73] Started GET "/v2/token?account=admin&scope=repository%3Atest-centos_8_content-2_0-fedora_docker_containers-fedora_docker_container_repo%2Ffedora%3Apull&service=foreman.test.com" for 127.0.0.1 at 2021-02-18 18:31:58 +0100
2021-02-18T18:31:58 [I|app|6b706c73] Processing by Katello::Api::Registry::RegistryProxiesController#token as HTML
2021-02-18T18:31:58 [I|app|6b706c73]   Parameters: {"account"=>"admin", "scope"=>"repository:test-centos_8_content-2_0-fedora_docker_containers-fedora_docker_container_repo/fedora:pull", "service"=>"foreman.test.com"}
2021-02-18T18:31:58 [I|kat|6b706c73] Authorized user admin(Admin User)
2021-02-18T18:31:58 [I|aud|6b706c73] PersonalAccessToken (1) update event on expires_at 2021-02-18 17:37:58 UTC, 2021-02-18 17:37:58 UTC
2021-02-18T18:31:58 [I|app|6b706c73] Completed 200 OK in 118ms (Views: 0.3ms | ActiveRecord: 19.9ms | Allocations: 18621)
2021-02-18T18:31:58 [I|app|3c0ac17a] Started GET "/v2/test-centos_8_content-2_0-fedora_docker_containers-fedora_docker_container_repo/fedora/manifests/latest" for 127.0.0.1 at 2021-02-18 18:31:58 +0100
2021-02-18T18:31:58 [I|app|3c0ac17a] Processing by Katello::Api::Registry::RegistryProxiesController#pull_manifest as JSON
2021-02-18T18:31:58 [I|app|3c0ac17a]   Parameters: {"repository"=>"test-centos_8_content-2_0-fedora_docker_containers-fedora_docker_container_repo/fedora", "tag"=>"latest"}
2021-02-18T18:31:58 [I|app|3c0ac17a] Filter chain halted as :authorize_repository_read rendered or redirected
2021-02-18T18:31:58 [I|app|3c0ac17a] Completed 404 Not Found in 24ms (Views: 0.3ms | ActiveRecord: 9.5ms | Allocations: 7070)
22

I have now managed the Pulp migration work as expected and to pull containers from Foreman to Docker. The first thing that whilst obvious when you know is that you must always use the Foreman server’s FQDN when making queries and no the IP-Address as this won’t work e.g.

# curl https://$(hostname)/pulp/repos/ --insecure

This also applies if you enter this URL into the browser.

Install the Pulp administration utility:

# pip3 install pulp-cli[pygments]

Check the status of the Pulp service:

# pulp-admin status
# pulp container repository list

Install Docker on the Foreman server in order to test if the containers are now available:

# yum install docker -y
# systemctl enable --now docker

Add Foreman to the list of insecure registries:

# cat /etc/docker/daemon.json
{
"insecure-registries" : ["foreman.test.com","localhost:5000"]
}

Login to Docker

# docker login $(hostname) --username admin --password admin

If this is working you should be able to see your Foreman containers e.g.

# docker search $(hostname)/fedora

Add your containers to a Content-View, then ensure that you Publish and Promote it. If successful you can now select Content->Container Image Tags and view the URL by selecting a tag and then Lifecycle Environments.

Now all you have to do is to copy the string/URL for your lifecycle-env and use it in your “Docker Pull” command.

23

Same here with Foreman 2.3.3, Katello 3.18.1and pulp 2.21.5 while running ‘foreman-maintain content prepare’.

And we only have centos7 repos/packages!

Is it safe to go further with the last migration step ‘foreman-maintain content switchover’?

24

The table ‘katello_debs’ does not exist in the postgres database pulpcore!

25

I tried: Bug #28966: Add ‘foreman-maintain content switchover’ command - Foreman Maintain - Foreman (theforeman.org) and that doesn’t work either

26

I have also tried the manual foreman-rake tasks and get as far as:

# foreman-rake katello:pulp3_migration
# foreman-rake katello:pulp3_post_migration_check
# foreman-rake katello:pulp3_content_switchover

then it fails with can’t build task.

27

If you are not using the deb plugin, a workaround will be to do the following:

foreman-installer --katello-use-pulp-2-for-deb=false --katello-enable-deb=false
28

This helps one step further, thanks for it!

foreman-installer --katello-use-pulp-2-for-deb=false --katello-enable-deb=false
…
output: Success!

  • Foreman is running at …

next step:
foreman-maintain content prepare
output: 021-03-10 19:04:28 +0100: Content migration starting. Content Migration completed successfully

                                                                  ***[OK]***

next step:
foreman-maintain content switchover

output: Switching specified content over to pulp 3 [FAIL]
Failed executing foreman-rake katello:pulp3_content_switchover, exit status 1:
rake aborted!
The Dynflow world was not initialized yet. If your plugin uses it, make sure to call Rails.application.dynflow.require! in some initializer
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/rails.rb:75:in world' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/lib/foreman_tasks.rb:23:in trigger’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/lib/foreman_tasks.rb:29:in block in trigger_task' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/lib/foreman_tasks.rb:49:in block in rails_safe_trigger_task’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/dependencies/interlock.rb:48:in block in permit_concurrent_loads' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/share_lock.rb:187:in yield_shares’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/dependencies/interlock.rb:47:in permit_concurrent_loads' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/lib/foreman_tasks.rb:48:in rails_safe_trigger_task’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/lib/foreman_tasks.rb:27:in trigger_task' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/lib/foreman_tasks.rb:58:in sync_task’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/lib/katello/tasks/pulp3_content_switchover.rake:19:in block (2 levels) in <top (required)>' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in <top (required)>’
Tasks: TOP => katello:pulp3_content_switchover
(See full trace by running task with --trace)

Scenario [Switch support for certain content from Pulp 2 to Pulp 3] failed.

The following steps ended up in failing state:

[content-switchover]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist=“content-switchover”

May somebody have any ideas?

thanks

1 Like
29

Hi @tbec ,

The fix for that issue will likely be coming soon with Katello 3.18.2.

Here’s another workaround that should fix things:

  1. Open up the following file (correct the dynflow version):
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/rails/configuration.rb
  1. Find the line that looks like:
self.rake_tasks_with_executor = %w(db:migrate db:seed)
  1. Add katello:pulp3_content_switchover like so:
self.rake_tasks_with_executor = %w(db:migrate db:seed katello:pulp3_content_switchover)

Then, to be safe, restart all your services.

Let me know if you’re still having issues.

2 Likes
30

Hi @iballou,

the workaround did it for me.

  1. changed the configuration.rb
  2. foreman-maintain service restart
  3. foreman-maintain content prepare
  4. foreman-maintain content switchover
  5. foreman-maintain service restart
    And all is working well.
    Thanks a lot for your help!
3 Likes