Pulp 2 to Pulp3 upgrade fails

Support
, ,
22

I have now managed the Pulp migration work as expected and to pull containers from Foreman to Docker. The first thing that whilst obvious when you know is that you must always use the Foreman server’s FQDN when making queries and no the IP-Address as this won’t work e.g.

# curl https://$(hostname)/pulp/repos/ --insecure

This also applies if you enter this URL into the browser.

Install the Pulp administration utility:

# pip3 install pulp-cli[pygments]

Check the status of the Pulp service:

# pulp-admin status
# pulp container repository list

Install Docker on the Foreman server in order to test if the containers are now available:

# yum install docker -y
# systemctl enable --now docker

Add Foreman to the list of insecure registries:

# cat /etc/docker/daemon.json
{
"insecure-registries" : ["foreman.test.com","localhost:5000"]
}

Login to Docker

# docker login $(hostname) --username admin --password admin

If this is working you should be able to see your Foreman containers e.g.

# docker search $(hostname)/fedora

Add your containers to a Content-View, then ensure that you Publish and Promote it. If successful you can now select Content->Container Image Tags and view the URL by selecting a tag and then Lifecycle Environments.

Now all you have to do is to copy the string/URL for your lifecycle-env and use it in your “Docker Pull” command.

23

Same here with Foreman 2.3.3, Katello 3.18.1and pulp 2.21.5 while running ‘foreman-maintain content prepare’.

And we only have centos7 repos/packages!

Is it safe to go further with the last migration step ‘foreman-maintain content switchover’?

24

The table ‘katello_debs’ does not exist in the postgres database pulpcore!

25

I tried: Bug #28966: Add ‘foreman-maintain content switchover’ command - Foreman Maintain - Foreman (theforeman.org) and that doesn’t work either

26

I have also tried the manual foreman-rake tasks and get as far as:

# foreman-rake katello:pulp3_migration
# foreman-rake katello:pulp3_post_migration_check
# foreman-rake katello:pulp3_content_switchover

then it fails with can’t build task.

27

If you are not using the deb plugin, a workaround will be to do the following:

foreman-installer --katello-use-pulp-2-for-deb=false --katello-enable-deb=false
28

This helps one step further, thanks for it!

foreman-installer --katello-use-pulp-2-for-deb=false --katello-enable-deb=false
…
output: Success!

  • Foreman is running at …

next step:
foreman-maintain content prepare
output: 021-03-10 19:04:28 +0100: Content migration starting. Content Migration completed successfully

                                                                  ***[OK]***

next step:
foreman-maintain content switchover

output: Switching specified content over to pulp 3 [FAIL]
Failed executing foreman-rake katello:pulp3_content_switchover, exit status 1:
rake aborted!
The Dynflow world was not initialized yet. If your plugin uses it, make sure to call Rails.application.dynflow.require! in some initializer
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/rails.rb:75:in world' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/lib/foreman_tasks.rb:23:in trigger’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/lib/foreman_tasks.rb:29:in block in trigger_task' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/lib/foreman_tasks.rb:49:in block in rails_safe_trigger_task’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/dependencies/interlock.rb:48:in block in permit_concurrent_loads' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/share_lock.rb:187:in yield_shares’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/dependencies/interlock.rb:47:in permit_concurrent_loads' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/lib/foreman_tasks.rb:48:in rails_safe_trigger_task’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/lib/foreman_tasks.rb:27:in trigger_task' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/lib/foreman_tasks.rb:58:in sync_task’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/lib/katello/tasks/pulp3_content_switchover.rake:19:in block (2 levels) in <top (required)>' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in <top (required)>’
Tasks: TOP => katello:pulp3_content_switchover
(See full trace by running task with --trace)

Scenario [Switch support for certain content from Pulp 2 to Pulp 3] failed.

The following steps ended up in failing state:

[content-switchover]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist=“content-switchover”

May somebody have any ideas?

thanks

1 Like
29

Hi @tbec ,

The fix for that issue will likely be coming soon with Katello 3.18.2.

Here’s another workaround that should fix things:

  1. Open up the following file (correct the dynflow version):
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/rails/configuration.rb
  1. Find the line that looks like:
self.rake_tasks_with_executor = %w(db:migrate db:seed)
  1. Add katello:pulp3_content_switchover like so:
self.rake_tasks_with_executor = %w(db:migrate db:seed katello:pulp3_content_switchover)

Then, to be safe, restart all your services.

Let me know if you’re still having issues.

2 Likes
30

Hi @iballou,

the workaround did it for me.

  1. changed the configuration.rb
  2. foreman-maintain service restart
  3. foreman-maintain content prepare
  4. foreman-maintain content switchover
  5. foreman-maintain service restart
    And all is working well.
    Thanks a lot for your help!
3 Likes
31

Hi @iballou ,

In my case it resulted in following error after applying the workaround:

[root@foreman-server ~]# foreman-maintain content switchover
Running Switch support for certain content from Pulp 2 to Pulp 3

Switch support for certain content from Pulp 2 to Pulp 3:
Performing final content migration before switching content
enabled
Starting task.
2021-03-17 12:40:56 +0100: Importing migrated yum repositories: 361/610Content Migration completed successfully
Performing a check to verify everything that is needed has been migrated

Switching specified content over to pulp 3 [FAIL]
Failed executing foreman-rake katello:pulp3_content_switchover, exit status 1:
ERROR: at least one katello_yum_metadata_files record has migrated_pulp3_href NULL value

Scenario [Switch support for certain content from Pulp 2 to Pulp 3] failed.

The following steps ended up in failing state:

[content-switchover]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist=“content-switchover”

From the logfile:

I, [2021-03-17 12:39:16+0100 #19752] INFO – : === Scenario ‘Switch support for certain content from Pulp 2 to Pulp 3’ started ===
I, [2021-03-17 12:39:16+0100 #19752] INFO – : — Execution step ‘Switch support for certain content from Pulp 2 to Pulp 3’ [content-switchover] started —
D, [2021-03-17 12:39:16+0100 #19752] DEBUG – : Running command foreman-rake katello:pulp3_migration with stdin nil
D, [2021-03-17 12:41:06+0100 #19752] DEBUG – : output of the command:
enabled
Starting task.
2021-03-17 12:39:36 +0100: Content migration starting. ^M ^M2021-03-17 12:39:46 +0100: Initial Migration steps complete.^M ^M2021-03-17 12:39:56 +0100: Initial Migration steps complete.^M ^M2021-03-17 12:40:06 +0100: Repo version creation 0/-27^M ^M2021-03-17 12:40:16 +0100: Starting katello import phase.^M ^M2021-03-17 12:40:26 +0100: Importing migrated yum repositories: 21/610^M ^M2021-03-17 12:40:36 +0100: Importing migrated yum repositories: 81/610^M ^M2021-03-17 12:40:46 +0100: Importing migrated yum repositories: 181/610^M ^M2021-03-17 12:40:56 +0100: Importing migrated yum repositories: 361/610Content Migration completed successfully
D, [2021-03-17 12:41:06+0100 #19752] DEBUG – : Running command foreman-rake katello:pulp3_post_migration_check with stdin nil
D, [2021-03-17 12:41:26+0100 #19752] DEBUG – : output of the command:

D, [2021-03-17 12:41:26+0100 #19752] DEBUG – : Running command foreman-rake katello:pulp3_content_switchover with stdin nil
D, [2021-03-17 12:41:44+0100 #19752] DEBUG – : output of the command:
ERROR: at least one katello_yum_metadata_files record has migrated_pulp3_href NULL value
E, [2021-03-17 12:41:44+0100 #19752] ERROR – : Failed executing foreman-rake katello:pulp3_content_switchover, exit status 1:
ERROR: at least one katello_yum_metadata_files record has migrated_pulp3_href NULL value (ForemanMaintain::Error::ExecutionError)
/usr/share/gems/gems/foreman_maintain-0.7.5/lib/foreman_maintain/utils/command_runner.rb:54:in execution_error' /usr/share/gems/gems/foreman_maintain-0.7.5/lib/foreman_maintain/concerns/system_helpers.rb:68:in execute!’
/usr/share/gems/gems/foreman_maintain-0.7.5/definitions/procedures/content/switchover.rb:19:in run' /usr/share/gems/gems/foreman_maintain-0.7.5/lib/foreman_maintain/executable.rb:139:in run’
/usr/share/gems/gems/foreman_maintain-0.7.5/lib/foreman_maintain/runner/execution.rb:83:in block (2 levels) in run' /usr/share/gems/gems/foreman_maintain-0.7.5/lib/foreman_maintain/runner/execution.rb:107:in capture_errors’
/usr/share/gems/gems/foreman_maintain-0.7.5/lib/foreman_maintain/runner/execution.rb:82:in block in run' /usr/share/gems/gems/foreman_maintain-0.7.5/lib/foreman_maintain/runner/execution.rb:101:in with_metadata_calculation’
/usr/share/gems/gems/foreman_maintain-0.7.5/lib/foreman_maintain/runner/execution.rb:81:in run' /usr/share/gems/gems/foreman_maintain-0.7.5/lib/foreman_maintain/runner.rb:125:in run_step’
/usr/share/gems/gems/foreman_maintain-0.7.5/lib/foreman_maintain/runner.rb:114:in run_steps' /usr/share/gems/gems/foreman_maintain-0.7.5/lib/foreman_maintain/runner.rb:107:in execute_scenario_steps’
/usr/share/gems/gems/foreman_maintain-0.7.5/lib/foreman_maintain/runner.rb:52:in run_scenario' /usr/share/gems/gems/foreman_maintain-0.7.5/lib/foreman_maintain/runner.rb:34:in block in run’
/usr/share/gems/gems/foreman_maintain-0.7.5/lib/foreman_maintain/runner.rb:33:in each' /usr/share/gems/gems/foreman_maintain-0.7.5/lib/foreman_maintain/runner.rb:33:in run’
/usr/share/gems/gems/foreman_maintain-0.7.5/lib/foreman_maintain/cli/base.rb:64:in run_scenario' /usr/share/gems/gems/foreman_maintain-0.7.5/lib/foreman_maintain/cli/base.rb:68:in run_scenarios_and_exit’
/usr/share/gems/gems/foreman_maintain-0.7.5/lib/foreman_maintain/cli/content_command.rb:14:in execute' /usr/share/gems/gems/clamp-1.1.2/lib/clamp/command.rb:63:in run’
/usr/share/gems/gems/clamp-1.1.2/lib/clamp/subcommand/execution.rb:11:in execute' /usr/share/gems/gems/clamp-1.1.2/lib/clamp/command.rb:63:in run’
/usr/share/gems/gems/clamp-1.1.2/lib/clamp/subcommand/execution.rb:11:in execute' /usr/share/gems/gems/clamp-1.1.2/lib/clamp/command.rb:63:in run’
/usr/share/gems/gems/foreman_maintain-0.7.5/lib/foreman_maintain/cli.rb:42:in run' /usr/share/gems/gems/clamp-1.1.2/lib/clamp/command.rb:132:in run’
/usr/share/gems/gems/foreman_maintain-0.7.5/bin/foreman-maintain:15:in <top (required)>' /usr/bin/foreman-maintain:23:in load’
/usr/bin/foreman-maintain:23:in `’
I, [2021-03-17 12:41:44+0100 #19752] INFO – : — Execution step ‘Switch support for certain content from Pulp 2 to Pulp 3’ finished —
I, [2021-03-17 12:41:44+0100 #19752] INFO – : === Scenario ‘Switch support for certain content from Pulp 2 to Pulp 3’ finished ===
I, [2021-03-17 12:41:44+0100 #19752] INFO – : foreman-maintain command finished with

this under Foreman 2.3.3 and Katello 3.18.1

Thanks in advance for providing any feedback or solution.

32

@iballou ,
No more help is needed, I found the solution in https://wiki.theforeman.org/issues/31919#change-146134
I applied this fix in our migration.rb file (version 3.18.1) and the process is currently going further in the process. Hope it will end well till the end.

2 Likes
33

@iballou , @Justin_Sherrill ,
Before the migration to pulp 3 I was still able to create new products/repo’s or adding el8 repo’s to existing products.

However, after the pulp3 migration this results in an error for both sitiations.

This morning I created the product “postgres 11” followed by the el8 repo for it (Index of /pub/repos/yum/11/redhat/rhel-8-x86_64/).
I configured Verify SSL to “no” and did not configured the user/password.
Once I saved the config, it resulted in an error, complaining that the user/password was required!
As said, before the pulp3 migration this worked well this way.

I did a force unlock and verified the config which seems correct as I did before the migration to pulp3.
But once I start the repo-sync it fails.

In the foreman log I see following:

2021-03-22T11:52:29 [I|app|04353405] Started POST “/katello/api/v2/repositories/977/sync?organization_id=3” for 127.0.0.1 at 2021-03-22 11:52:29 +0100
2021-03-22T11:52:29 [I|app|04353405] Processing by Katello::Api::V2::RepositoriesController#sync as HTML
2021-03-22T11:52:29 [I|app|04353405] Parameters: {“id”=>“977”, “organization_id”=>“3”, “api_version”=>“v2”, “repository”=>{“id”=>“977”}}
2021-03-22T11:52:29 [I|bac|04353405] Task {label: Actions::Katello::Repository::Sync, id: 4e19f4e0-bc9a-4339-aa8c-565035d6dac0, execution_plan_id: 48f4c8e0-1eb8-42fa-8965-675d704b0206} state changed: planning
2021-03-22T11:52:29 [I|bac|] Task {label: Actions::Katello::Repository::Sync, id: 4e19f4e0-bc9a-4339-aa8c-565035d6dac0, execution_plan_id: 48f4c8e0-1eb8-42fa-8965-675d704b0206} state changed: planned
2021-03-22T11:52:29 [I|app|] Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/views/katello/api/v2/repositories/sync.json.rabl within katello/api/v2/layouts/resource
2021-03-22T11:52:29 [I|app|] Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/views/katello/api/v2/repositories/sync.json.rabl within katello/api/v2/layouts/resource (Duration: 52.0ms | Allocations: 36701)
2021-03-22T11:52:29 [I|app|] Completed 202 Accepted in 433ms (Views: 53.9ms | ActiveRecord: 38.9ms | Allocations: 165544)
2021-03-22T11:52:29 [I|bac|] Task {label: Actions::Katello::Repository::Sync, id: 4e19f4e0-bc9a-4339-aa8c-565035d6dac0, execution_plan_id: 48f4c8e0-1eb8-42fa-8965-675d704b0206} state changed: running
2021-03-22T11:52:29 [I|app|8b84a43b] Started GET “/tasks/views/task-details.html” for 127.0.0.1 at 2021-03-22 11:52:29 +0100
2021-03-22T11:52:29 [E|bac|] Error in progress calculation
2021-03-22T11:52:29 [E|bac|] private method select' called for nil:NilClass (NoMethodError) | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/pulp3/repository/presenters/abstract_sync_presenter.rb:19:in sync_task’
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/pulp3/repository/presenters/content_unit_presenter.rb:32:in total_units' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/pulp3/repository/presenters/content_unit_presenter.rb:7:in progress’
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/pulp3/repository/sync.rb:35:in run_progress' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:18:in block in run’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:34:in ensure in with_progress_calculation' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:33:in with_progress_calculation’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:17:in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_request_id.rb:15:in block in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_request_id.rb:49:in restore_current_request_id’
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_request_id.rb:15:in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_timezone.rb:15:in block in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_timezone.rb:44:in restore_curent_timezone’
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_timezone.rb:15:in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_user.rb:15:in block in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_user.rb:44:in restore_curent_user’
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_user.rb:15:in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in block in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in restore_current_taxonomies’
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/world.rb:31:in execute' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:570:in block (2 levels) in execute_run’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in catch' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in block in execute_run’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in block in with_error_handling' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in catch’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in with_error_handling' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:564:in execute_run’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:285:in execute' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in block (2 levels) in execute’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in block (2 levels) in execute' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract.rb:167:in with_meta_calculation’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in block in execute' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in open_action’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in execute' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/director.rb:68:in execute’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in block (2 levels) in perform' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors.rb:18:in run_user_code’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in block in perform' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in with_telemetry’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in perform' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/serialization.rb:27:in perform’
| [ sidekiq ]
| [ concurrent-ruby ]
2021-03-22T11:52:29 [E|bac|] Error message: the server returns an error
| HTTP status code: 400
| Response headers: {“date”=>“Mon, 22 Mar 2021 10:52:29 GMT”, “server”=>“gunicorn/20.0.4”, “content-type”=>“application/json”, “vary”=>“Accept,Cookie”, “allow”=>“POST, OPTIONS”, “x-frame-options”=>“SAMEORIGIN”, “content-length”=>“82”, “via”=>“1.1 foreman-server”, “connection”=>“close”}
| Response body: {“remote”:[“This field is required since a remote is not set on the repository.”]} (PulpRpmClient::ApiError)
| /opt/theforeman/tfm/root/usr/share/gems/gems/pulp_rpm_client-3.7.0/lib/pulp_rpm_client/api_client.rb:81:in call_api' | /opt/theforeman/tfm/root/usr/share/gems/gems/pulp_rpm_client-3.7.0/lib/pulp_rpm_client/api/repositories_rpm_api.rb:496:in sync_with_http_info’
| /opt/theforeman/tfm/root/usr/share/gems/gems/pulp_rpm_client-3.7.0/lib/pulp_rpm_client/api/repositories_rpm_api.rb:440:in sync' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/services/katello/pulp3/repository.rb:198:in sync’
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/pulp3/repository/sync.rb:13:in invoke_external_task' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:84:in initiate_external_action’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:19:in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/cancellable.rb:14:in run’
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/pulp3/abstract_async_task.rb:10:in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:571:in block (3 levels) in execute_run’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/middleware/remote_action.rb:16:in block in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/middleware/remote_action.rb:40:in block in as_remote_user’
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/models/katello/concerns/user_extensions.rb:21:in cp_config' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/middleware/remote_action.rb:27:in as_cp_user’
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/middleware/remote_action.rb:39:in as_remote_user' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/middleware/remote_action.rb:16:in run’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/rails_executor_wrap.rb:14:in block in run’
| /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/execution_wrapper.rb:88:in wrap' | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/rails_executor_wrap.rb:13:in run’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:31:in with_progress_calculation’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:17:in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_request_id.rb:15:in block in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_request_id.rb:49:in restore_current_request_id’
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_request_id.rb:15:in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_timezone.rb:15:in block in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_timezone.rb:44:in restore_curent_timezone’
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_timezone.rb:15:in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_user.rb:15:in block in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_user.rb:44:in restore_curent_user’
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_user.rb:15:in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in block in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in restore_current_taxonomies’
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/world.rb:31:in execute' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:570:in block (2 levels) in execute_run’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in catch' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in block in execute_run’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in block in with_error_handling' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in catch’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in with_error_handling' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:564:in execute_run’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:285:in execute' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in block (2 levels) in execute’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract.rb:167:in with_meta_calculation' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in block in execute’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in open_action' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in execute’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/director.rb:68:in execute' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in block (2 levels) in perform’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors.rb:18:in run_user_code' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/director.rb:68:in execute’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in block (2 levels) in perform' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors.rb:18:in run_user_code’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in block in perform' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in with_telemetry’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in perform' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/serialization.rb:27:in perform’
| [ sidekiq ]
| [ concurrent-ruby ]

If I try to modify any configuration in the repo-config it popups the error “An error occurred saving the Repository: undefined method `label’ for nil:NilClass”

This happens with Foreman 2.3.3/Katello 3.18.1 (with modified migration.rb which will become available in version 3.18.2)

Any help is welcome.
Thnx in advance

34

@iballou , @Justin_Sherrill ,
Despite the content migrate went well without an error, is it possible that this fixes could be necessary:

In version 3.18.1 it still contain:
task :pulp3_content_switchover => :environment do

If so, what does that mean for the migrate I have done last week?

35

Hi @langesmalle,

You shouldn’t have to worry about your switchover with that issue you mentioned since yours completed fine. That fix that @Justin_Sherrill mentioned is related to the The Dynflow world was not initialized yet. bug.

As for your remote issue, let’s take a look at your repository configuration from the backend.

  1. Find your repository’s numeric ID from the URL when on that repository’s page
  2. Open up the foreman console: foreman-rake console
  3. Note the output of ::Katello::Repository.find(<id here>).remote_href
  4. If the remote_href is nil, can you show me the Dynflow console for your repository create command? You could copy/paste, or upload the html.

I’m guessing the force unlock skipped the remote creation step. It may look fine in the UI, but the backend probably didn’t save properly.

Your original issue with the user/password is probably the root cause. Did your error look like what’s described in this issue? Bug #31949: Unable to create repositories in Pulp 3 mode with Firefox as the browser - username : This field may not be blank. - Katello - Foreman

This is a bug where, if you have the username and password saved for Foreman in your browser, it bugs out the browser. One workaround could be to just remove the saved username and password from your browser (if that is the case). Alternatively, you could apply this patch: Fixes #31949 - ignore repo user/pass if blank by jlsherrill · Pull Request #9192 · Katello/katello · GitHub

Let me know if you have any further questions.

36

@iballou ,

The remote_href returned “nil” for the 2 repo’s I have an issue with.
However, I applied the change Fixes #31949 - ignore repo user/pass if blank by jlsherrill · Pull Request #9192 · Katello/katello · GitHub in the repository.rb file and started the sync again for the Postgres 11 el8 repo.
This time it started well, but seems to hang at 75%.
It is still in pending state but reported already 2 errors:

image
image1416×585 98.3 KB

The full errors from Tasks/Errors:

PulpRpmClient::ApiError: Error message: the server returns an error
HTTP status code: 400
Response headers: {“date”=>“Tue, 23 Mar 2021 07:46:39 GMT”, “server”=>“gunicorn/20.0.4”, “content-type”=>“application/json”, “vary”=>“Accept,Cookie”, “allow”=>“POST, OPTIONS”, “x-frame-options”=>“SAMEORIGIN”, “content-length”=>“82”, “via”=>“1.1 foreman-server”, “connection”=>“close”}
Response body: {“remote”:[“This field is required since a remote is not set on the repository.”]}

PulpRpmClient::ApiError: Error message: the server returns an error
HTTP status code: 400
Response headers: {“date”=>“Tue, 23 Mar 2021 07:46:39 GMT”, “server”=>“gunicorn/20.0.4”, “content-type”=>“application/json”, “vary”=>“Accept,Cookie”, “allow”=>“GET, POST, HEAD, OPTIONS”, “x-frame-options”=>“SAMEORIGIN”, “content-length”=>“44”, “via”=>“1.1 foreman-server”, “connection”=>“close”}
Response body: {“base_path”:[“This field must be unique.”]}

For the other repo (foreman client for el8) I tried to first remove the repo in order to execute the creation/sync steps again from scratch.
However, despite it gave a popup-screen that the repo was removed successfull, it still exists, and in the task-overview I see that a sub-task failed:

image
image1302×704 104 KB

with following backtrace-log:

/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/with_sub_plans.rb:231:in check_for_errors!' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/with_sub_plans.rb:137:in try_to_finish’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/with_sub_plans.rb:22:in block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:74:in block in assigns’
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in tap' /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in assigns’
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:56:in match_value' /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:36:in block in match?’
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in each' /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in match?’
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:23:in match' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/with_sub_plans.rb:12:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/with_bulk_sub_plans.rb:24:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/bulk_action.rb:21:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:571:in block (3 levels) in execute_run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/rails_executor_wrap.rb:14:in block in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/execution_wrapper.rb:88:in wrap' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/rails_executor_wrap.rb:13:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:31:in with_progress_calculation’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:17:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_request_id.rb:15:in block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_request_id.rb:49:in restore_current_request_id’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_request_id.rb:15:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_timezone.rb:15:in block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_timezone.rb:44:in restore_curent_timezone’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_timezone.rb:15:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_user.rb:15:in block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_user.rb:44:in restore_curent_user’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_user.rb:15:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in restore_current_taxonomies’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/world.rb:31:in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:570:in block (2 levels) in execute_run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in catch' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in block in execute_run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in block in with_error_handling' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in catch’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in with_error_handling' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:564:in execute_run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:285:in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in block (2 levels) in execute’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract.rb:167:in with_meta_calculation' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in block in execute’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in open_action' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in execute’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/director.rb:93:in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in block (2 levels) in perform’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors.rb:18:in run_user_code' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in block in perform’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in with_telemetry' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in perform’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/serialization.rb:27:in perform' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in execute_job’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in block (2 levels) in process' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in block in invoke’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in invoke' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in block in process’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in block (6 levels) in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in local’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in block (5 levels) in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq.rb:37:in block in module:Sidekiq’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in block (4 levels) in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in stats’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in block (3 levels) in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in block (2 levels) in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in global’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in block in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in with_context’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in with_job_hash_context' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in dispatch’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in process' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in process_one’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in watchdog’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in block in safe_thread' /opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in block in create_with_logging_context’

Any sugestion?

37

@iballou ,

I managed to get it work well.
First I did a foreman-maintain service restart.
Then I removed the 2 repo’s which resulted in errors.
In order to get this done I had to force-unlock the task, had to skip the action in the dynflow console, resume the task, and when it resulted in another error/warning I needed to redo those actions.

Anyway, afterwards I was able to recreate the repo’s as well as syncing them whithout any issues

So, the fix in Bug #31949: Unable to create repositories in Pulp 3 mode with Firefox as the browser - username : This field may not be blank. - Katello - Foreman did it, however after the restart of the foreman-services.

38

@iballou , @Justin_Sherrill ,

I’m facing a new issue :unamused:.

Now that I was able to sync the new product/repos I cant see them in the folder /var/www/pub/yum/< http or https >/repos/< organization >/< Library or environment >/…

For the Foreman_Client EL8 I added the repo to the existing content view as well promoted it to an environment, but as said can not find it.

Before, with pulp2, after publishing the product/repo, it was visible in /var/www/pub/yum/< http or https >/repos/< organization >/< Library >/custom/< product >/

Where repo-name is a symlink to /var/lib/pulp/published/yum/master/yum_distributor/< uuid >/< subfolder >

Here an overview of the latest success tasks:

image
image1197×346 125 KB

All help is welcome to fix this new issue, or what is changed since the migration to pulp3?

Thnx in advance

39

Hi @langesmalle ,

I’m glad you got past your repository issues. Apologies for not mentioning that the username/password patch only fixes the issue for newly created repositories.

That is new to Pulp 3. You will no longer see content units on the filesystem like you used to with Pulp 2. The best way now to see the content directly from Pulp would be to browse the hosted repositories from your browser (http://hostname.com/pulp/repos/…) Trying to consume content from a subscribed host could be a good test too. Let me know if you’re having further troubles with consuming content from Pulp 3.

Also, if you’re ever looking to debug Pulp 3 itself, I would recommend giving pulp-cli a try.

40

@iballou ,

Thanks for your feedback.
when I do a wget on one of our hosts of for example the published Postgres11 (only in Library environment I get a 404.

For wget http://my_foreman_server/pulp/repos/DIDM/Library

I can not see the product Postgres_11 at all in the Library file.

Neither in the wget http://my_foreman_server/pulp/repos/DIDM/Library/custom

So I guess it looks like something is still wrong in my enviromment.

This is the dynflow result of the publish I did for the Postgres_11/Postgres_11_EL8.

image
image634×853 181 KB

Thanks for providing me any advice.

41

@langesmalle I get 404s as well if I try only /pulp/repos/Default_Organization/Library/custom/. Looks like you have to navigate to the repository itself to not get the 404, i.e. /pulp/repos/Default_Organization/Library/custom/Product/Repository/

If you really can’t see that content, can I see the output of:

sudo foreman-installer --full-help | grep "\-\-foreman-proxy-content-proxy-pulp-yum-to-pulpcore"