Pulp 2 to Pulp3 upgrade fails

Exactly the same behavior here (Oracle Linux 7.9, no deb content)

Hey @techietubby , can you share the command(s) you’ve tried, their output, and any errors you’re seeing show up in journalctl?

Hi Jonathon,

Yes of course, gladly! :slight_smile:

I migrated Pulp as follows:

# foreman-installer --katello-enable-deb=false
# foreman-rake foreman_tasks:cleanup
# foreman-rake foreman_tasks:cleanup:config foreman-rake
# foreman-rake katello:pulp3_migration
# foreman-installer --katello-enable-deb=true

I did a restorecon to ensure all SELinux labels were up date and the ran:

# docker pull foreman.test.com/test-centos_8_content-2_0-fedora_docker_containers-fedora_docker_container_repo

This gives:

# journalctl -l --since=09:26 --no-pager
-- Logs begin at Mon 2021-02-01 13:52:47 CET, end at Thu 2021-02-18 09:26:15 CET. --
Feb 18 09:26:15 foreman.test.com dockerd[8018]: time="2021-02-18T09:26:15.935487079+01:00" level=info msg="Attempting next endpoint for pull after error: Head https://foreman.test.com/v2/test-centos_8_content-2_0-fedora_docker_containers-fedora_docker_container_repo/manifests/latest: received unexpected HTTP status: 500 Internal Server Error"
Feb 18 09:26:15 foreman.test.com dockerd[8018]: time="2021-02-18T09:26:15.940765618+01:00" level=error msg="Handler for POST /v1.41/images/create returned error: Head https://foreman.test.com/v2/test-centos_8_content-2_0-fedora_docker_containers-fedora_docker_container_repo/manifests/latest: received unexpected HTTP status: 500 Internal Server Error"

If I add fedora to the tag the results are:

# docker pull foreman.test.com/test-centos_8_content-2_0-fedora_docker_containers-fedora_docker_container_repo/fedora
Using default tag: latest
Error response from daemon: unknown: test-centos_8_content-2_0-fedora_docker_containers-fedora_docker_container_repo/fedora was not found!

The journalctl gives:

Feb 18 09:27:37 foreman.test.com pulp[21363]: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred)
Feb 18 09:27:37 foreman.test.com pulp[21404]: celery.worker.strategy:INFO: Received task: pulp.server.controllers.repository.queue_download_deferred[2b716a9f-5926-4261-861b-30a0b8fe9e34]
Feb 18 09:27:37 foreman.test.com pulp[21407]: celery.worker.strategy:INFO: Received task: pulp.server.controllers.repository.download_deferred[aba775b8-f924-4ddc-a8b1-5a7c27c617d6]
Feb 18 09:27:37 foreman.test.com pulp[21472]: celery.app.trace:INFO: [2b716a9f] Task pulp.server.controllers.repository.queue_download_deferred[2b716a9f-5926-4261-861b-30a0b8fe9e34] succeeded in 0.00635926402174s: None
Feb 18 09:27:38 foreman.test.com pulp[21473]: celery.app.trace:INFO: [aba775b8] Task pulp.server.controllers.repository.download_deferred[aba775b8-f924-4ddc-a8b1-5a7c27c617d6] succeeded in 1.01021712599s: None
Feb 18 09:30:01 foreman.test.com systemd[1]: Created slice User Slice of foreman-proxy.
Feb 18 09:30:01 foreman.test.com systemd[1]: Started Session 2149 of user foreman-proxy.
Feb 18 09:30:01 foreman.test.com systemd[1]: Created slice User Slice of foreman.
Feb 18 09:30:01 foreman.test.com systemd[1]: Started Session 2148 of user foreman.
Feb 18 09:30:01 foreman.test.com CROND[8794]: (foreman-proxy) CMD (smart-proxy-openscap-send >> /var/log/foreman-proxy/cron.log)
Feb 18 09:30:01 foreman.test.com CROND[8795]: (foreman) CMD (   /usr/sbin/foreman-rake ldap:refresh_usergroups >>/var/log/foreman/cron.log 2>&1)
Feb 18 09:30:02 foreman.test.com systemd[1]: Removed slice User Slice of foreman-proxy.
Feb 18 09:30:20 foreman.test.com dockerd[8018]: time="2021-02-18T09:30:20.819236660+01:00" level=error msg="Not continuing with pull after error: unknown: test-centos_8_content-2_0-fedora_docker_containers-fedora_docker_container_repo/fedora was not found!"
Feb 18 09:30:20 foreman.test.com dockerd[8018]: time="2021-02-18T09:30:20.823651356+01:00" level=error msg="Handler for POST /v1.41/images/create returned error: unknown: test-centos_8_content-2_0-fedora_docker_containers-fedora_docker_container_repo/fedora was not found!"
Feb 18 09:30:24 foreman.test.com systemd[1]: Removed slice User Slice of foreman.
Feb 18 09:30:39 foreman.test.com pulpcore-worker-4[15884]: pulp: rq.worker:INFO: Cleaning registries for queue: 15884@foreman.test.com
Feb 18 09:30:39 foreman.test.com pulpcore-worker-2[15883]: pulp: rq.worker:INFO: Cleaning registries for queue: 15883@foreman.test.com
Feb 18 09:30:41 foreman.test.com pulpcore-worker-1[15882]: pulp: rq.worker:INFO: Cleaning registries for queue: 15882@foreman.test.com
Feb 18 09:30:42 foreman.test.com pulpcore-worker-3[15881]: pulp: rq.worker:INFO: Cleaning registries for queue: 15881@foreman.test.com

Do you need more examples?

Regards,
Andrew

Looks like there should be an ISE traceback somewhere. Can try tailing /var/log/foreman/production.log while you docker pull?

I get:

2021-02-18T18:31:58 [I|app|0141592d] Filter chain halted as :authorize_repository_read rendered or redirected
2021-02-18T18:31:58 [I|app|0141592d] Completed 404 Not Found in 20ms (Views: 0.2ms | ActiveRecord: 7.3ms | Allocations: 6946)
2021-02-18T18:31:58 [I|app|6b706c73] Started GET "/v2/token?account=admin&scope=repository%3Atest-centos_8_content-2_0-fedora_docker_containers-fedora_docker_container_repo%2Ffedora%3Apull&service=foreman.test.com" for 127.0.0.1 at 2021-02-18 18:31:58 +0100
2021-02-18T18:31:58 [I|app|6b706c73] Processing by Katello::Api::Registry::RegistryProxiesController#token as HTML
2021-02-18T18:31:58 [I|app|6b706c73]   Parameters: {"account"=>"admin", "scope"=>"repository:test-centos_8_content-2_0-fedora_docker_containers-fedora_docker_container_repo/fedora:pull", "service"=>"foreman.test.com"}
2021-02-18T18:31:58 [I|kat|6b706c73] Authorized user admin(Admin User)
2021-02-18T18:31:58 [I|aud|6b706c73] PersonalAccessToken (1) update event on expires_at 2021-02-18 17:37:58 UTC, 2021-02-18 17:37:58 UTC
2021-02-18T18:31:58 [I|app|6b706c73] Completed 200 OK in 118ms (Views: 0.3ms | ActiveRecord: 19.9ms | Allocations: 18621)
2021-02-18T18:31:58 [I|app|3c0ac17a] Started GET "/v2/test-centos_8_content-2_0-fedora_docker_containers-fedora_docker_container_repo/fedora/manifests/latest" for 127.0.0.1 at 2021-02-18 18:31:58 +0100
2021-02-18T18:31:58 [I|app|3c0ac17a] Processing by Katello::Api::Registry::RegistryProxiesController#pull_manifest as JSON
2021-02-18T18:31:58 [I|app|3c0ac17a]   Parameters: {"repository"=>"test-centos_8_content-2_0-fedora_docker_containers-fedora_docker_container_repo/fedora", "tag"=>"latest"}
2021-02-18T18:31:58 [I|app|3c0ac17a] Filter chain halted as :authorize_repository_read rendered or redirected
2021-02-18T18:31:58 [I|app|3c0ac17a] Completed 404 Not Found in 24ms (Views: 0.3ms | ActiveRecord: 9.5ms | Allocations: 7070)

I have now managed the Pulp migration work as expected and to pull containers from Foreman to Docker. The first thing that whilst obvious when you know is that you must always use the Foreman server’s FQDN when making queries and no the IP-Address as this won’t work e.g.

# curl https://$(hostname)/pulp/repos/ --insecure

This also applies if you enter this URL into the browser.

Install the Pulp administration utility:

# pip3 install pulp-cli[pygments]

Check the status of the Pulp service:

# pulp-admin status
# pulp container repository list

Install Docker on the Foreman server in order to test if the containers are now available:

# yum install docker -y
# systemctl enable --now docker

Add Foreman to the list of insecure registries:

# cat /etc/docker/daemon.json
{
"insecure-registries" : ["foreman.test.com","localhost:5000"]
}

Login to Docker

# docker login $(hostname) --username admin --password admin

If this is working you should be able to see your Foreman containers e.g.

# docker search $(hostname)/fedora

Add your containers to a Content-View, then ensure that you Publish and Promote it. If successful you can now select Content->Container Image Tags and view the URL by selecting a tag and then Lifecycle Environments.

Now all you have to do is to copy the string/URL for your lifecycle-env and use it in your “Docker Pull” command.

Same here with Foreman 2.3.3, Katello 3.18.1and pulp 2.21.5 while running ‘foreman-maintain content prepare’.

And we only have centos7 repos/packages!

Is it safe to go further with the last migration step ‘foreman-maintain content switchover’?

The table ‘katello_debs’ does not exist in the postgres database pulpcore!

I tried: Bug #28966: Add ‘foreman-maintain content switchover’ command - Foreman Maintain - Foreman (theforeman.org) and that doesn’t work either

I have also tried the manual foreman-rake tasks and get as far as:

# foreman-rake katello:pulp3_migration
# foreman-rake katello:pulp3_post_migration_check
# foreman-rake katello:pulp3_content_switchover

then it fails with can’t build task.

If you are not using the deb plugin, a workaround will be to do the following:

foreman-installer --katello-use-pulp-2-for-deb=false --katello-enable-deb=false

This helps one step further, thanks for it!

foreman-installer --katello-use-pulp-2-for-deb=false --katello-enable-deb=false

output: Success!

  • Foreman is running at …

next step:
foreman-maintain content prepare
output: 021-03-10 19:04:28 +0100: Content migration starting. Content Migration completed successfully

                                                                  ***[OK]***

next step:
foreman-maintain content switchover

output: Switching specified content over to pulp 3 [FAIL]
Failed executing foreman-rake katello:pulp3_content_switchover, exit status 1:
rake aborted!
The Dynflow world was not initialized yet. If your plugin uses it, make sure to call Rails.application.dynflow.require! in some initializer
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/rails.rb:75:in world' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/lib/foreman_tasks.rb:23:in trigger’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/lib/foreman_tasks.rb:29:in block in trigger_task' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/lib/foreman_tasks.rb:49:in block in rails_safe_trigger_task’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/dependencies/interlock.rb:48:in block in permit_concurrent_loads' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/share_lock.rb:187:in yield_shares’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/dependencies/interlock.rb:47:in permit_concurrent_loads' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/lib/foreman_tasks.rb:48:in rails_safe_trigger_task’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/lib/foreman_tasks.rb:27:in trigger_task' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/lib/foreman_tasks.rb:58:in sync_task’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/lib/katello/tasks/pulp3_content_switchover.rake:19:in block (2 levels) in <top (required)>' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in <top (required)>’
Tasks: TOP => katello:pulp3_content_switchover
(See full trace by running task with --trace)

Scenario [Switch support for certain content from Pulp 2 to Pulp 3] failed.

The following steps ended up in failing state:

[content-switchover]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist=“content-switchover”

May somebody have any ideas?

thanks

1 Like

Hi @tbec ,

The fix for that issue will likely be coming soon with Katello 3.18.2.

Here’s another workaround that should fix things:

  1. Open up the following file (correct the dynflow version):
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/rails/configuration.rb
  1. Find the line that looks like:
self.rake_tasks_with_executor = %w(db:migrate db:seed)
  1. Add katello:pulp3_content_switchover like so:
self.rake_tasks_with_executor = %w(db:migrate db:seed katello:pulp3_content_switchover)

Then, to be safe, restart all your services.

Let me know if you’re still having issues.

2 Likes

Hi @iballou,

the workaround did it for me.

  1. changed the configuration.rb
  2. foreman-maintain service restart
  3. foreman-maintain content prepare
  4. foreman-maintain content switchover
  5. foreman-maintain service restart
    And all is working well.
    Thanks a lot for your help!
3 Likes

Hi @iballou ,

In my case it resulted in following error after applying the workaround:

[root@foreman-server ~]# foreman-maintain content switchover
Running Switch support for certain content from Pulp 2 to Pulp 3

Switch support for certain content from Pulp 2 to Pulp 3:
Performing final content migration before switching content
enabled
Starting task.
2021-03-17 12:40:56 +0100: Importing migrated yum repositories: 361/610Content Migration completed successfully
Performing a check to verify everything that is needed has been migrated

Switching specified content over to pulp 3 [FAIL]
Failed executing foreman-rake katello:pulp3_content_switchover, exit status 1:
ERROR: at least one katello_yum_metadata_files record has migrated_pulp3_href NULL value

Scenario [Switch support for certain content from Pulp 2 to Pulp 3] failed.

The following steps ended up in failing state:

[content-switchover]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist=“content-switchover”

From the logfile:

I, [2021-03-17 12:39:16+0100 #19752] INFO – : === Scenario ‘Switch support for certain content from Pulp 2 to Pulp 3’ started ===
I, [2021-03-17 12:39:16+0100 #19752] INFO – : — Execution step ‘Switch support for certain content from Pulp 2 to Pulp 3’ [content-switchover] started —
D, [2021-03-17 12:39:16+0100 #19752] DEBUG – : Running command foreman-rake katello:pulp3_migration with stdin nil
D, [2021-03-17 12:41:06+0100 #19752] DEBUG – : output of the command:
enabled
Starting task.
2021-03-17 12:39:36 +0100: Content migration starting. ^M ^M2021-03-17 12:39:46 +0100: Initial Migration steps complete.^M ^M2021-03-17 12:39:56 +0100: Initial Migration steps complete.^M ^M2021-03-17 12:40:06 +0100: Repo version creation 0/-27^M ^M2021-03-17 12:40:16 +0100: Starting katello import phase.^M ^M2021-03-17 12:40:26 +0100: Importing migrated yum repositories: 21/610^M ^M2021-03-17 12:40:36 +0100: Importing migrated yum repositories: 81/610^M ^M2021-03-17 12:40:46 +0100: Importing migrated yum repositories: 181/610^M ^M2021-03-17 12:40:56 +0100: Importing migrated yum repositories: 361/610Content Migration completed successfully
D, [2021-03-17 12:41:06+0100 #19752] DEBUG – : Running command foreman-rake katello:pulp3_post_migration_check with stdin nil
D, [2021-03-17 12:41:26+0100 #19752] DEBUG – : output of the command:

D, [2021-03-17 12:41:26+0100 #19752] DEBUG – : Running command foreman-rake katello:pulp3_content_switchover with stdin nil
D, [2021-03-17 12:41:44+0100 #19752] DEBUG – : output of the command:
ERROR: at least one katello_yum_metadata_files record has migrated_pulp3_href NULL value
E, [2021-03-17 12:41:44+0100 #19752] ERROR – : Failed executing foreman-rake katello:pulp3_content_switchover, exit status 1:
ERROR: at least one katello_yum_metadata_files record has migrated_pulp3_href NULL value (ForemanMaintain::Error::ExecutionError)
/usr/share/gems/gems/foreman_maintain-0.7.5/lib/foreman_maintain/utils/command_runner.rb:54:in execution_error' /usr/share/gems/gems/foreman_maintain-0.7.5/lib/foreman_maintain/concerns/system_helpers.rb:68:in execute!’
/usr/share/gems/gems/foreman_maintain-0.7.5/definitions/procedures/content/switchover.rb:19:in run' /usr/share/gems/gems/foreman_maintain-0.7.5/lib/foreman_maintain/executable.rb:139:in run
/usr/share/gems/gems/foreman_maintain-0.7.5/lib/foreman_maintain/runner/execution.rb:83:in block (2 levels) in run' /usr/share/gems/gems/foreman_maintain-0.7.5/lib/foreman_maintain/runner/execution.rb:107:in capture_errors’
/usr/share/gems/gems/foreman_maintain-0.7.5/lib/foreman_maintain/runner/execution.rb:82:in block in run' /usr/share/gems/gems/foreman_maintain-0.7.5/lib/foreman_maintain/runner/execution.rb:101:in with_metadata_calculation’
/usr/share/gems/gems/foreman_maintain-0.7.5/lib/foreman_maintain/runner/execution.rb:81:in run' /usr/share/gems/gems/foreman_maintain-0.7.5/lib/foreman_maintain/runner.rb:125:in run_step’
/usr/share/gems/gems/foreman_maintain-0.7.5/lib/foreman_maintain/runner.rb:114:in run_steps' /usr/share/gems/gems/foreman_maintain-0.7.5/lib/foreman_maintain/runner.rb:107:in execute_scenario_steps’
/usr/share/gems/gems/foreman_maintain-0.7.5/lib/foreman_maintain/runner.rb:52:in run_scenario' /usr/share/gems/gems/foreman_maintain-0.7.5/lib/foreman_maintain/runner.rb:34:in block in run’
/usr/share/gems/gems/foreman_maintain-0.7.5/lib/foreman_maintain/runner.rb:33:in each' /usr/share/gems/gems/foreman_maintain-0.7.5/lib/foreman_maintain/runner.rb:33:in run’
/usr/share/gems/gems/foreman_maintain-0.7.5/lib/foreman_maintain/cli/base.rb:64:in run_scenario' /usr/share/gems/gems/foreman_maintain-0.7.5/lib/foreman_maintain/cli/base.rb:68:in run_scenarios_and_exit’
/usr/share/gems/gems/foreman_maintain-0.7.5/lib/foreman_maintain/cli/content_command.rb:14:in execute' /usr/share/gems/gems/clamp-1.1.2/lib/clamp/command.rb:63:in run’
/usr/share/gems/gems/clamp-1.1.2/lib/clamp/subcommand/execution.rb:11:in execute' /usr/share/gems/gems/clamp-1.1.2/lib/clamp/command.rb:63:in run’
/usr/share/gems/gems/clamp-1.1.2/lib/clamp/subcommand/execution.rb:11:in execute' /usr/share/gems/gems/clamp-1.1.2/lib/clamp/command.rb:63:in run’
/usr/share/gems/gems/foreman_maintain-0.7.5/lib/foreman_maintain/cli.rb:42:in run' /usr/share/gems/gems/clamp-1.1.2/lib/clamp/command.rb:132:in run’
/usr/share/gems/gems/foreman_maintain-0.7.5/bin/foreman-maintain:15:in <top (required)>' /usr/bin/foreman-maintain:23:in load’
/usr/bin/foreman-maintain:23:in `’
I, [2021-03-17 12:41:44+0100 #19752] INFO – : — Execution step ‘Switch support for certain content from Pulp 2 to Pulp 3’ finished —
I, [2021-03-17 12:41:44+0100 #19752] INFO – : === Scenario ‘Switch support for certain content from Pulp 2 to Pulp 3’ finished ===
I, [2021-03-17 12:41:44+0100 #19752] INFO – : foreman-maintain command finished with

this under Foreman 2.3.3 and Katello 3.18.1

Thanks in advance for providing any feedback or solution.

@iballou ,
No more help is needed, I found the solution in https://wiki.theforeman.org/issues/31919#change-146134
I applied this fix in our migration.rb file (version 3.18.1) and the process is currently going further in the process. Hope it will end well till the end.

2 Likes

@iballou , @Justin_Sherrill ,
Before the migration to pulp 3 I was still able to create new products/repo’s or adding el8 repo’s to existing products.

However, after the pulp3 migration this results in an error for both sitiations.

This morning I created the product “postgres 11” followed by the el8 repo for it (Index of /pub/repos/yum/11/redhat/rhel-8-x86_64/).
I configured Verify SSL to “no” and did not configured the user/password.
Once I saved the config, it resulted in an error, complaining that the user/password was required!
As said, before the pulp3 migration this worked well this way.

I did a force unlock and verified the config which seems correct as I did before the migration to pulp3.
But once I start the repo-sync it fails.

In the foreman log I see following:

2021-03-22T11:52:29 [I|app|04353405] Started POST “/katello/api/v2/repositories/977/sync?organization_id=3” for 127.0.0.1 at 2021-03-22 11:52:29 +0100
2021-03-22T11:52:29 [I|app|04353405] Processing by Katello::Api::V2::RepositoriesController#sync as HTML
2021-03-22T11:52:29 [I|app|04353405] Parameters: {“id”=>“977”, “organization_id”=>“3”, “api_version”=>“v2”, “repository”=>{“id”=>“977”}}
2021-03-22T11:52:29 [I|bac|04353405] Task {label: Actions::Katello::Repository::Sync, id: 4e19f4e0-bc9a-4339-aa8c-565035d6dac0, execution_plan_id: 48f4c8e0-1eb8-42fa-8965-675d704b0206} state changed: planning
2021-03-22T11:52:29 [I|bac|] Task {label: Actions::Katello::Repository::Sync, id: 4e19f4e0-bc9a-4339-aa8c-565035d6dac0, execution_plan_id: 48f4c8e0-1eb8-42fa-8965-675d704b0206} state changed: planned
2021-03-22T11:52:29 [I|app|] Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/views/katello/api/v2/repositories/sync.json.rabl within katello/api/v2/layouts/resource
2021-03-22T11:52:29 [I|app|] Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/views/katello/api/v2/repositories/sync.json.rabl within katello/api/v2/layouts/resource (Duration: 52.0ms | Allocations: 36701)
2021-03-22T11:52:29 [I|app|] Completed 202 Accepted in 433ms (Views: 53.9ms | ActiveRecord: 38.9ms | Allocations: 165544)
2021-03-22T11:52:29 [I|bac|] Task {label: Actions::Katello::Repository::Sync, id: 4e19f4e0-bc9a-4339-aa8c-565035d6dac0, execution_plan_id: 48f4c8e0-1eb8-42fa-8965-675d704b0206} state changed: running
2021-03-22T11:52:29 [I|app|8b84a43b] Started GET “/tasks/views/task-details.html” for 127.0.0.1 at 2021-03-22 11:52:29 +0100
2021-03-22T11:52:29 [E|bac|] Error in progress calculation
2021-03-22T11:52:29 [E|bac|] private method select' called for nil:NilClass (NoMethodError) | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/pulp3/repository/presenters/abstract_sync_presenter.rb:19:in sync_task’
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/pulp3/repository/presenters/content_unit_presenter.rb:32:in total_units' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/pulp3/repository/presenters/content_unit_presenter.rb:7:in progress’
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/pulp3/repository/sync.rb:35:in run_progress' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:18:in block in run’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:34:in ensure in with_progress_calculation' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:33:in with_progress_calculation’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:17:in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_request_id.rb:15:in block in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_request_id.rb:49:in restore_current_request_id’
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_request_id.rb:15:in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_timezone.rb:15:in block in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_timezone.rb:44:in restore_curent_timezone’
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_timezone.rb:15:in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_user.rb:15:in block in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_user.rb:44:in restore_curent_user’
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_user.rb:15:in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in block in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in restore_current_taxonomies’
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/world.rb:31:in execute' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:570:in block (2 levels) in execute_run’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in catch' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in block in execute_run’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in block in with_error_handling' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in catch’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in with_error_handling' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:564:in execute_run’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:285:in execute' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in block (2 levels) in execute’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in block (2 levels) in execute' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract.rb:167:in with_meta_calculation’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in block in execute' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in open_action’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in execute' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/director.rb:68:in execute’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in block (2 levels) in perform' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors.rb:18:in run_user_code’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in block in perform' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in with_telemetry’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in perform' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/serialization.rb:27:in perform’
| [ sidekiq ]
| [ concurrent-ruby ]
2021-03-22T11:52:29 [E|bac|] Error message: the server returns an error
| HTTP status code: 400
| Response headers: {“date”=>“Mon, 22 Mar 2021 10:52:29 GMT”, “server”=>“gunicorn/20.0.4”, “content-type”=>“application/json”, “vary”=>“Accept,Cookie”, “allow”=>“POST, OPTIONS”, “x-frame-options”=>“SAMEORIGIN”, “content-length”=>“82”, “via”=>“1.1 foreman-server”, “connection”=>“close”}
| Response body: {“remote”:[“This field is required since a remote is not set on the repository.”]} (PulpRpmClient::ApiError)
| /opt/theforeman/tfm/root/usr/share/gems/gems/pulp_rpm_client-3.7.0/lib/pulp_rpm_client/api_client.rb:81:in call_api' | /opt/theforeman/tfm/root/usr/share/gems/gems/pulp_rpm_client-3.7.0/lib/pulp_rpm_client/api/repositories_rpm_api.rb:496:in sync_with_http_info’
| /opt/theforeman/tfm/root/usr/share/gems/gems/pulp_rpm_client-3.7.0/lib/pulp_rpm_client/api/repositories_rpm_api.rb:440:in sync' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/services/katello/pulp3/repository.rb:198:in sync’
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/pulp3/repository/sync.rb:13:in invoke_external_task' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:84:in initiate_external_action’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:19:in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/cancellable.rb:14:in run’
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/pulp3/abstract_async_task.rb:10:in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:571:in block (3 levels) in execute_run’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/middleware/remote_action.rb:16:in block in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/middleware/remote_action.rb:40:in block in as_remote_user’
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/models/katello/concerns/user_extensions.rb:21:in cp_config' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/middleware/remote_action.rb:27:in as_cp_user’
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/middleware/remote_action.rb:39:in as_remote_user' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/middleware/remote_action.rb:16:in run’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/rails_executor_wrap.rb:14:in block in run’
| /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/execution_wrapper.rb:88:in wrap' | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/rails_executor_wrap.rb:13:in run’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:31:in with_progress_calculation’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:17:in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_request_id.rb:15:in block in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_request_id.rb:49:in restore_current_request_id’
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_request_id.rb:15:in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_timezone.rb:15:in block in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_timezone.rb:44:in restore_curent_timezone’
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_timezone.rb:15:in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_user.rb:15:in block in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_user.rb:44:in restore_curent_user’
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_user.rb:15:in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in block in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in restore_current_taxonomies’
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/world.rb:31:in execute' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:570:in block (2 levels) in execute_run’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in catch' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in block in execute_run’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in block in with_error_handling' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in catch’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in with_error_handling' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:564:in execute_run’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:285:in execute' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in block (2 levels) in execute’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract.rb:167:in with_meta_calculation' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in block in execute’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in open_action' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in execute’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/director.rb:68:in execute' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in block (2 levels) in perform’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors.rb:18:in run_user_code' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/director.rb:68:in execute’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in block (2 levels) in perform' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors.rb:18:in run_user_code’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in block in perform' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in with_telemetry’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in perform' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/serialization.rb:27:in perform’
| [ sidekiq ]
| [ concurrent-ruby ]

If I try to modify any configuration in the repo-config it popups the error “An error occurred saving the Repository: undefined method `label’ for nil:NilClass”

This happens with Foreman 2.3.3/Katello 3.18.1 (with modified migration.rb which will become available in version 3.18.2)

Any help is welcome.
Thnx in advance

@iballou , @Justin_Sherrill ,
Despite the content migrate went well without an error, is it possible that this fixes could be necessary:

In version 3.18.1 it still contain:
task :pulp3_content_switchover => :environment do

If so, what does that mean for the migrate I have done last week?

Hi @langesmalle,

You shouldn’t have to worry about your switchover with that issue you mentioned since yours completed fine. That fix that @Justin_Sherrill mentioned is related to the The Dynflow world was not initialized yet. bug.

As for your remote issue, let’s take a look at your repository configuration from the backend.

  1. Find your repository’s numeric ID from the URL when on that repository’s page
  2. Open up the foreman console: foreman-rake console
  3. Note the output of ::Katello::Repository.find(<id here>).remote_href
  4. If the remote_href is nil, can you show me the Dynflow console for your repository create command? You could copy/paste, or upload the html.

I’m guessing the force unlock skipped the remote creation step. It may look fine in the UI, but the backend probably didn’t save properly.

Your original issue with the user/password is probably the root cause. Did your error look like what’s described in this issue? Bug #31949: Unable to create repositories in Pulp 3 mode with Firefox as the browser - username : This field may not be blank. - Katello - Foreman

This is a bug where, if you have the username and password saved for Foreman in your browser, it bugs out the browser. One workaround could be to just remove the saved username and password from your browser (if that is the case). Alternatively, you could apply this patch: Fixes #31949 - ignore repo user/pass if blank by jlsherrill · Pull Request #9192 · Katello/katello · GitHub

Let me know if you have any further questions.

@iballou ,

The remote_href returned “nil” for the 2 repo’s I have an issue with.
However, I applied the change Fixes #31949 - ignore repo user/pass if blank by jlsherrill · Pull Request #9192 · Katello/katello · GitHub in the repository.rb file and started the sync again for the Postgres 11 el8 repo.
This time it started well, but seems to hang at 75%.
It is still in pending state but reported already 2 errors:

The full errors from Tasks/Errors:

PulpRpmClient::ApiError: Error message: the server returns an error
HTTP status code: 400
Response headers: {“date”=>“Tue, 23 Mar 2021 07:46:39 GMT”, “server”=>“gunicorn/20.0.4”, “content-type”=>“application/json”, “vary”=>“Accept,Cookie”, “allow”=>“POST, OPTIONS”, “x-frame-options”=>“SAMEORIGIN”, “content-length”=>“82”, “via”=>“1.1 foreman-server”, “connection”=>“close”}
Response body: {“remote”:[“This field is required since a remote is not set on the repository.”]}

PulpRpmClient::ApiError: Error message: the server returns an error
HTTP status code: 400
Response headers: {“date”=>“Tue, 23 Mar 2021 07:46:39 GMT”, “server”=>“gunicorn/20.0.4”, “content-type”=>“application/json”, “vary”=>“Accept,Cookie”, “allow”=>“GET, POST, HEAD, OPTIONS”, “x-frame-options”=>“SAMEORIGIN”, “content-length”=>“44”, “via”=>“1.1 foreman-server”, “connection”=>“close”}
Response body: {“base_path”:[“This field must be unique.”]}

For the other repo (foreman client for el8) I tried to first remove the repo in order to execute the creation/sync steps again from scratch.
However, despite it gave a popup-screen that the repo was removed successfull, it still exists, and in the task-overview I see that a sub-task failed:

with following backtrace-log:

/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/with_sub_plans.rb:231:in check_for_errors!' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/with_sub_plans.rb:137:in try_to_finish’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/with_sub_plans.rb:22:in block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:74:in block in assigns’
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in tap' /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in assigns’
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:56:in match_value' /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:36:in block in match?’
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in each' /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in match?’
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:23:in match' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/with_sub_plans.rb:12:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/with_bulk_sub_plans.rb:24:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/bulk_action.rb:21:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:571:in block (3 levels) in execute_run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/rails_executor_wrap.rb:14:in block in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/execution_wrapper.rb:88:in wrap' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/rails_executor_wrap.rb:13:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:31:in with_progress_calculation’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:17:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_request_id.rb:15:in block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_request_id.rb:49:in restore_current_request_id’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_request_id.rb:15:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_timezone.rb:15:in block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_timezone.rb:44:in restore_curent_timezone’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_timezone.rb:15:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_user.rb:15:in block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_user.rb:44:in restore_curent_user’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_user.rb:15:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in restore_current_taxonomies’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/world.rb:31:in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:570:in block (2 levels) in execute_run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in catch' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in block in execute_run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in block in with_error_handling' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in catch’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in with_error_handling' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:564:in execute_run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:285:in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in block (2 levels) in execute’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract.rb:167:in with_meta_calculation' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in block in execute’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in open_action' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in execute’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/director.rb:93:in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in block (2 levels) in perform’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors.rb:18:in run_user_code' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in block in perform’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in with_telemetry' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in perform’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/serialization.rb:27:in perform' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in execute_job’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in block (2 levels) in process' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in block in invoke’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in invoke' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in block in process’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in block (6 levels) in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in local’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in block (5 levels) in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq.rb:37:in block in module:Sidekiq
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in block (4 levels) in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in stats’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in block (3 levels) in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in block (2 levels) in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in global’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in block in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in with_context’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in with_job_hash_context' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in dispatch’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in process' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in process_one’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in watchdog’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in block in safe_thread' /opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in block in create_with_logging_context’

Any sugestion?