Pulp 2 to Pulp3 upgrade fails

Support
, ,
29

Hi @tbec ,

The fix for that issue will likely be coming soon with Katello 3.18.2.

Here’s another workaround that should fix things:

  1. Open up the following file (correct the dynflow version):
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/rails/configuration.rb
  1. Find the line that looks like:
self.rake_tasks_with_executor = %w(db:migrate db:seed)
  1. Add katello:pulp3_content_switchover like so:
self.rake_tasks_with_executor = %w(db:migrate db:seed katello:pulp3_content_switchover)

Then, to be safe, restart all your services.

Let me know if you’re still having issues.

2 Likes
30

Hi @iballou,

the workaround did it for me.

  1. changed the configuration.rb
  2. foreman-maintain service restart
  3. foreman-maintain content prepare
  4. foreman-maintain content switchover
  5. foreman-maintain service restart
    And all is working well.
    Thanks a lot for your help!
3 Likes
31

Hi @iballou ,

In my case it resulted in following error after applying the workaround:

[root@foreman-server ~]# foreman-maintain content switchover
Running Switch support for certain content from Pulp 2 to Pulp 3

Switch support for certain content from Pulp 2 to Pulp 3:
Performing final content migration before switching content
enabled
Starting task.
2021-03-17 12:40:56 +0100: Importing migrated yum repositories: 361/610Content Migration completed successfully
Performing a check to verify everything that is needed has been migrated

Switching specified content over to pulp 3 [FAIL]
Failed executing foreman-rake katello:pulp3_content_switchover, exit status 1:
ERROR: at least one katello_yum_metadata_files record has migrated_pulp3_href NULL value

Scenario [Switch support for certain content from Pulp 2 to Pulp 3] failed.

The following steps ended up in failing state:

[content-switchover]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist=“content-switchover”

From the logfile:

I, [2021-03-17 12:39:16+0100 #19752] INFO – : === Scenario ‘Switch support for certain content from Pulp 2 to Pulp 3’ started ===
I, [2021-03-17 12:39:16+0100 #19752] INFO – : — Execution step ‘Switch support for certain content from Pulp 2 to Pulp 3’ [content-switchover] started —
D, [2021-03-17 12:39:16+0100 #19752] DEBUG – : Running command foreman-rake katello:pulp3_migration with stdin nil
D, [2021-03-17 12:41:06+0100 #19752] DEBUG – : output of the command:
enabled
Starting task.
2021-03-17 12:39:36 +0100: Content migration starting. ^M ^M2021-03-17 12:39:46 +0100: Initial Migration steps complete.^M ^M2021-03-17 12:39:56 +0100: Initial Migration steps complete.^M ^M2021-03-17 12:40:06 +0100: Repo version creation 0/-27^M ^M2021-03-17 12:40:16 +0100: Starting katello import phase.^M ^M2021-03-17 12:40:26 +0100: Importing migrated yum repositories: 21/610^M ^M2021-03-17 12:40:36 +0100: Importing migrated yum repositories: 81/610^M ^M2021-03-17 12:40:46 +0100: Importing migrated yum repositories: 181/610^M ^M2021-03-17 12:40:56 +0100: Importing migrated yum repositories: 361/610Content Migration completed successfully
D, [2021-03-17 12:41:06+0100 #19752] DEBUG – : Running command foreman-rake katello:pulp3_post_migration_check with stdin nil
D, [2021-03-17 12:41:26+0100 #19752] DEBUG – : output of the command:

D, [2021-03-17 12:41:26+0100 #19752] DEBUG – : Running command foreman-rake katello:pulp3_content_switchover with stdin nil
D, [2021-03-17 12:41:44+0100 #19752] DEBUG – : output of the command:
ERROR: at least one katello_yum_metadata_files record has migrated_pulp3_href NULL value
E, [2021-03-17 12:41:44+0100 #19752] ERROR – : Failed executing foreman-rake katello:pulp3_content_switchover, exit status 1:
ERROR: at least one katello_yum_metadata_files record has migrated_pulp3_href NULL value (ForemanMaintain::Error::ExecutionError)
/usr/share/gems/gems/foreman_maintain-0.7.5/lib/foreman_maintain/utils/command_runner.rb:54:in execution_error' /usr/share/gems/gems/foreman_maintain-0.7.5/lib/foreman_maintain/concerns/system_helpers.rb:68:in execute!’
/usr/share/gems/gems/foreman_maintain-0.7.5/definitions/procedures/content/switchover.rb:19:in run' /usr/share/gems/gems/foreman_maintain-0.7.5/lib/foreman_maintain/executable.rb:139:in run’
/usr/share/gems/gems/foreman_maintain-0.7.5/lib/foreman_maintain/runner/execution.rb:83:in block (2 levels) in run' /usr/share/gems/gems/foreman_maintain-0.7.5/lib/foreman_maintain/runner/execution.rb:107:in capture_errors’
/usr/share/gems/gems/foreman_maintain-0.7.5/lib/foreman_maintain/runner/execution.rb:82:in block in run' /usr/share/gems/gems/foreman_maintain-0.7.5/lib/foreman_maintain/runner/execution.rb:101:in with_metadata_calculation’
/usr/share/gems/gems/foreman_maintain-0.7.5/lib/foreman_maintain/runner/execution.rb:81:in run' /usr/share/gems/gems/foreman_maintain-0.7.5/lib/foreman_maintain/runner.rb:125:in run_step’
/usr/share/gems/gems/foreman_maintain-0.7.5/lib/foreman_maintain/runner.rb:114:in run_steps' /usr/share/gems/gems/foreman_maintain-0.7.5/lib/foreman_maintain/runner.rb:107:in execute_scenario_steps’
/usr/share/gems/gems/foreman_maintain-0.7.5/lib/foreman_maintain/runner.rb:52:in run_scenario' /usr/share/gems/gems/foreman_maintain-0.7.5/lib/foreman_maintain/runner.rb:34:in block in run’
/usr/share/gems/gems/foreman_maintain-0.7.5/lib/foreman_maintain/runner.rb:33:in each' /usr/share/gems/gems/foreman_maintain-0.7.5/lib/foreman_maintain/runner.rb:33:in run’
/usr/share/gems/gems/foreman_maintain-0.7.5/lib/foreman_maintain/cli/base.rb:64:in run_scenario' /usr/share/gems/gems/foreman_maintain-0.7.5/lib/foreman_maintain/cli/base.rb:68:in run_scenarios_and_exit’
/usr/share/gems/gems/foreman_maintain-0.7.5/lib/foreman_maintain/cli/content_command.rb:14:in execute' /usr/share/gems/gems/clamp-1.1.2/lib/clamp/command.rb:63:in run’
/usr/share/gems/gems/clamp-1.1.2/lib/clamp/subcommand/execution.rb:11:in execute' /usr/share/gems/gems/clamp-1.1.2/lib/clamp/command.rb:63:in run’
/usr/share/gems/gems/clamp-1.1.2/lib/clamp/subcommand/execution.rb:11:in execute' /usr/share/gems/gems/clamp-1.1.2/lib/clamp/command.rb:63:in run’
/usr/share/gems/gems/foreman_maintain-0.7.5/lib/foreman_maintain/cli.rb:42:in run' /usr/share/gems/gems/clamp-1.1.2/lib/clamp/command.rb:132:in run’
/usr/share/gems/gems/foreman_maintain-0.7.5/bin/foreman-maintain:15:in <top (required)>' /usr/bin/foreman-maintain:23:in load’
/usr/bin/foreman-maintain:23:in `’
I, [2021-03-17 12:41:44+0100 #19752] INFO – : — Execution step ‘Switch support for certain content from Pulp 2 to Pulp 3’ finished —
I, [2021-03-17 12:41:44+0100 #19752] INFO – : === Scenario ‘Switch support for certain content from Pulp 2 to Pulp 3’ finished ===
I, [2021-03-17 12:41:44+0100 #19752] INFO – : foreman-maintain command finished with

this under Foreman 2.3.3 and Katello 3.18.1

Thanks in advance for providing any feedback or solution.

32

@iballou ,
No more help is needed, I found the solution in https://wiki.theforeman.org/issues/31919#change-146134
I applied this fix in our migration.rb file (version 3.18.1) and the process is currently going further in the process. Hope it will end well till the end.

2 Likes
33

@iballou , @Justin_Sherrill ,
Before the migration to pulp 3 I was still able to create new products/repo’s or adding el8 repo’s to existing products.

However, after the pulp3 migration this results in an error for both sitiations.

This morning I created the product “postgres 11” followed by the el8 repo for it (Index of /pub/repos/yum/11/redhat/rhel-8-x86_64/).
I configured Verify SSL to “no” and did not configured the user/password.
Once I saved the config, it resulted in an error, complaining that the user/password was required!
As said, before the pulp3 migration this worked well this way.

I did a force unlock and verified the config which seems correct as I did before the migration to pulp3.
But once I start the repo-sync it fails.

In the foreman log I see following:

2021-03-22T11:52:29 [I|app|04353405] Started POST “/katello/api/v2/repositories/977/sync?organization_id=3” for 127.0.0.1 at 2021-03-22 11:52:29 +0100
2021-03-22T11:52:29 [I|app|04353405] Processing by Katello::Api::V2::RepositoriesController#sync as HTML
2021-03-22T11:52:29 [I|app|04353405] Parameters: {“id”=>“977”, “organization_id”=>“3”, “api_version”=>“v2”, “repository”=>{“id”=>“977”}}
2021-03-22T11:52:29 [I|bac|04353405] Task {label: Actions::Katello::Repository::Sync, id: 4e19f4e0-bc9a-4339-aa8c-565035d6dac0, execution_plan_id: 48f4c8e0-1eb8-42fa-8965-675d704b0206} state changed: planning
2021-03-22T11:52:29 [I|bac|] Task {label: Actions::Katello::Repository::Sync, id: 4e19f4e0-bc9a-4339-aa8c-565035d6dac0, execution_plan_id: 48f4c8e0-1eb8-42fa-8965-675d704b0206} state changed: planned
2021-03-22T11:52:29 [I|app|] Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/views/katello/api/v2/repositories/sync.json.rabl within katello/api/v2/layouts/resource
2021-03-22T11:52:29 [I|app|] Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/views/katello/api/v2/repositories/sync.json.rabl within katello/api/v2/layouts/resource (Duration: 52.0ms | Allocations: 36701)
2021-03-22T11:52:29 [I|app|] Completed 202 Accepted in 433ms (Views: 53.9ms | ActiveRecord: 38.9ms | Allocations: 165544)
2021-03-22T11:52:29 [I|bac|] Task {label: Actions::Katello::Repository::Sync, id: 4e19f4e0-bc9a-4339-aa8c-565035d6dac0, execution_plan_id: 48f4c8e0-1eb8-42fa-8965-675d704b0206} state changed: running
2021-03-22T11:52:29 [I|app|8b84a43b] Started GET “/tasks/views/task-details.html” for 127.0.0.1 at 2021-03-22 11:52:29 +0100
2021-03-22T11:52:29 [E|bac|] Error in progress calculation
2021-03-22T11:52:29 [E|bac|] private method select' called for nil:NilClass (NoMethodError) | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/pulp3/repository/presenters/abstract_sync_presenter.rb:19:in sync_task’
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/pulp3/repository/presenters/content_unit_presenter.rb:32:in total_units' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/pulp3/repository/presenters/content_unit_presenter.rb:7:in progress’
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/pulp3/repository/sync.rb:35:in run_progress' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:18:in block in run’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:34:in ensure in with_progress_calculation' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:33:in with_progress_calculation’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:17:in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_request_id.rb:15:in block in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_request_id.rb:49:in restore_current_request_id’
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_request_id.rb:15:in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_timezone.rb:15:in block in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_timezone.rb:44:in restore_curent_timezone’
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_timezone.rb:15:in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_user.rb:15:in block in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_user.rb:44:in restore_curent_user’
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_user.rb:15:in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in block in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in restore_current_taxonomies’
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/world.rb:31:in execute' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:570:in block (2 levels) in execute_run’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in catch' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in block in execute_run’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in block in with_error_handling' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in catch’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in with_error_handling' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:564:in execute_run’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:285:in execute' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in block (2 levels) in execute’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in block (2 levels) in execute' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract.rb:167:in with_meta_calculation’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in block in execute' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in open_action’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in execute' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/director.rb:68:in execute’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in block (2 levels) in perform' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors.rb:18:in run_user_code’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in block in perform' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in with_telemetry’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in perform' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/serialization.rb:27:in perform’
| [ sidekiq ]
| [ concurrent-ruby ]
2021-03-22T11:52:29 [E|bac|] Error message: the server returns an error
| HTTP status code: 400
| Response headers: {“date”=>“Mon, 22 Mar 2021 10:52:29 GMT”, “server”=>“gunicorn/20.0.4”, “content-type”=>“application/json”, “vary”=>“Accept,Cookie”, “allow”=>“POST, OPTIONS”, “x-frame-options”=>“SAMEORIGIN”, “content-length”=>“82”, “via”=>“1.1 foreman-server”, “connection”=>“close”}
| Response body: {“remote”:[“This field is required since a remote is not set on the repository.”]} (PulpRpmClient::ApiError)
| /opt/theforeman/tfm/root/usr/share/gems/gems/pulp_rpm_client-3.7.0/lib/pulp_rpm_client/api_client.rb:81:in call_api' | /opt/theforeman/tfm/root/usr/share/gems/gems/pulp_rpm_client-3.7.0/lib/pulp_rpm_client/api/repositories_rpm_api.rb:496:in sync_with_http_info’
| /opt/theforeman/tfm/root/usr/share/gems/gems/pulp_rpm_client-3.7.0/lib/pulp_rpm_client/api/repositories_rpm_api.rb:440:in sync' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/services/katello/pulp3/repository.rb:198:in sync’
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/pulp3/repository/sync.rb:13:in invoke_external_task' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:84:in initiate_external_action’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:19:in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/cancellable.rb:14:in run’
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/pulp3/abstract_async_task.rb:10:in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:571:in block (3 levels) in execute_run’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/middleware/remote_action.rb:16:in block in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/middleware/remote_action.rb:40:in block in as_remote_user’
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/models/katello/concerns/user_extensions.rb:21:in cp_config' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/middleware/remote_action.rb:27:in as_cp_user’
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/middleware/remote_action.rb:39:in as_remote_user' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/middleware/remote_action.rb:16:in run’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/rails_executor_wrap.rb:14:in block in run’
| /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/execution_wrapper.rb:88:in wrap' | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/rails_executor_wrap.rb:13:in run’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:31:in with_progress_calculation’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:17:in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_request_id.rb:15:in block in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_request_id.rb:49:in restore_current_request_id’
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_request_id.rb:15:in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_timezone.rb:15:in block in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_timezone.rb:44:in restore_curent_timezone’
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_timezone.rb:15:in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_user.rb:15:in block in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_user.rb:44:in restore_curent_user’
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_user.rb:15:in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in block in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in restore_current_taxonomies’
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/world.rb:31:in execute' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:570:in block (2 levels) in execute_run’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in catch' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in block in execute_run’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in block in with_error_handling' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in catch’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in with_error_handling' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:564:in execute_run’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:285:in execute' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in block (2 levels) in execute’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract.rb:167:in with_meta_calculation' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in block in execute’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in open_action' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in execute’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/director.rb:68:in execute' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in block (2 levels) in perform’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors.rb:18:in run_user_code' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/director.rb:68:in execute’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in block (2 levels) in perform' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors.rb:18:in run_user_code’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in block in perform' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in with_telemetry’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in perform' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/serialization.rb:27:in perform’
| [ sidekiq ]
| [ concurrent-ruby ]

If I try to modify any configuration in the repo-config it popups the error “An error occurred saving the Repository: undefined method `label’ for nil:NilClass”

This happens with Foreman 2.3.3/Katello 3.18.1 (with modified migration.rb which will become available in version 3.18.2)

Any help is welcome.
Thnx in advance

34

@iballou , @Justin_Sherrill ,
Despite the content migrate went well without an error, is it possible that this fixes could be necessary:

In version 3.18.1 it still contain:
task :pulp3_content_switchover => :environment do

If so, what does that mean for the migrate I have done last week?

35

Hi @langesmalle,

You shouldn’t have to worry about your switchover with that issue you mentioned since yours completed fine. That fix that @Justin_Sherrill mentioned is related to the The Dynflow world was not initialized yet. bug.

As for your remote issue, let’s take a look at your repository configuration from the backend.

  1. Find your repository’s numeric ID from the URL when on that repository’s page
  2. Open up the foreman console: foreman-rake console
  3. Note the output of ::Katello::Repository.find(<id here>).remote_href
  4. If the remote_href is nil, can you show me the Dynflow console for your repository create command? You could copy/paste, or upload the html.

I’m guessing the force unlock skipped the remote creation step. It may look fine in the UI, but the backend probably didn’t save properly.

Your original issue with the user/password is probably the root cause. Did your error look like what’s described in this issue? Bug #31949: Unable to create repositories in Pulp 3 mode with Firefox as the browser - username : This field may not be blank. - Katello - Foreman

This is a bug where, if you have the username and password saved for Foreman in your browser, it bugs out the browser. One workaround could be to just remove the saved username and password from your browser (if that is the case). Alternatively, you could apply this patch: Fixes #31949 - ignore repo user/pass if blank by jlsherrill · Pull Request #9192 · Katello/katello · GitHub

Let me know if you have any further questions.

36

@iballou ,

The remote_href returned “nil” for the 2 repo’s I have an issue with.
However, I applied the change Fixes #31949 - ignore repo user/pass if blank by jlsherrill · Pull Request #9192 · Katello/katello · GitHub in the repository.rb file and started the sync again for the Postgres 11 el8 repo.
This time it started well, but seems to hang at 75%.
It is still in pending state but reported already 2 errors:

image
image1416×585 98.3 KB

The full errors from Tasks/Errors:

PulpRpmClient::ApiError: Error message: the server returns an error
HTTP status code: 400
Response headers: {“date”=>“Tue, 23 Mar 2021 07:46:39 GMT”, “server”=>“gunicorn/20.0.4”, “content-type”=>“application/json”, “vary”=>“Accept,Cookie”, “allow”=>“POST, OPTIONS”, “x-frame-options”=>“SAMEORIGIN”, “content-length”=>“82”, “via”=>“1.1 foreman-server”, “connection”=>“close”}
Response body: {“remote”:[“This field is required since a remote is not set on the repository.”]}

PulpRpmClient::ApiError: Error message: the server returns an error
HTTP status code: 400
Response headers: {“date”=>“Tue, 23 Mar 2021 07:46:39 GMT”, “server”=>“gunicorn/20.0.4”, “content-type”=>“application/json”, “vary”=>“Accept,Cookie”, “allow”=>“GET, POST, HEAD, OPTIONS”, “x-frame-options”=>“SAMEORIGIN”, “content-length”=>“44”, “via”=>“1.1 foreman-server”, “connection”=>“close”}
Response body: {“base_path”:[“This field must be unique.”]}

For the other repo (foreman client for el8) I tried to first remove the repo in order to execute the creation/sync steps again from scratch.
However, despite it gave a popup-screen that the repo was removed successfull, it still exists, and in the task-overview I see that a sub-task failed:

image
image1302×704 104 KB

with following backtrace-log:

/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/with_sub_plans.rb:231:in check_for_errors!' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/with_sub_plans.rb:137:in try_to_finish’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/with_sub_plans.rb:22:in block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:74:in block in assigns’
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in tap' /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in assigns’
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:56:in match_value' /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:36:in block in match?’
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in each' /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in match?’
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:23:in match' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/with_sub_plans.rb:12:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/with_bulk_sub_plans.rb:24:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/bulk_action.rb:21:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:571:in block (3 levels) in execute_run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/rails_executor_wrap.rb:14:in block in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/execution_wrapper.rb:88:in wrap' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/rails_executor_wrap.rb:13:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:31:in with_progress_calculation’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:17:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_request_id.rb:15:in block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_request_id.rb:49:in restore_current_request_id’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_request_id.rb:15:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_timezone.rb:15:in block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_timezone.rb:44:in restore_curent_timezone’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_timezone.rb:15:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_user.rb:15:in block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_user.rb:44:in restore_curent_user’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_user.rb:15:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in restore_current_taxonomies’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.3/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/world.rb:31:in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:570:in block (2 levels) in execute_run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in catch' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in block in execute_run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in block in with_error_handling' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in catch’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in with_error_handling' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:564:in execute_run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:285:in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in block (2 levels) in execute’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract.rb:167:in with_meta_calculation' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in block in execute’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in open_action' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in execute’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/director.rb:93:in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in block (2 levels) in perform’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors.rb:18:in run_user_code' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in block in perform’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in with_telemetry' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in perform’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/serialization.rb:27:in perform' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in execute_job’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in block (2 levels) in process' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in block in invoke’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in invoke' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in block in process’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in block (6 levels) in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in local’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in block (5 levels) in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq.rb:37:in block in module:Sidekiq’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in block (4 levels) in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in stats’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in block (3 levels) in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in block (2 levels) in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in global’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in block in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in with_context’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in with_job_hash_context' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in dispatch’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in process' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in process_one’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in watchdog’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in block in safe_thread' /opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in block in create_with_logging_context’

Any sugestion?

37

@iballou ,

I managed to get it work well.
First I did a foreman-maintain service restart.
Then I removed the 2 repo’s which resulted in errors.
In order to get this done I had to force-unlock the task, had to skip the action in the dynflow console, resume the task, and when it resulted in another error/warning I needed to redo those actions.

Anyway, afterwards I was able to recreate the repo’s as well as syncing them whithout any issues

So, the fix in Bug #31949: Unable to create repositories in Pulp 3 mode with Firefox as the browser - username : This field may not be blank. - Katello - Foreman did it, however after the restart of the foreman-services.

38

@iballou , @Justin_Sherrill ,

I’m facing a new issue :unamused:.

Now that I was able to sync the new product/repos I cant see them in the folder /var/www/pub/yum/< http or https >/repos/< organization >/< Library or environment >/…

For the Foreman_Client EL8 I added the repo to the existing content view as well promoted it to an environment, but as said can not find it.

Before, with pulp2, after publishing the product/repo, it was visible in /var/www/pub/yum/< http or https >/repos/< organization >/< Library >/custom/< product >/

Where repo-name is a symlink to /var/lib/pulp/published/yum/master/yum_distributor/< uuid >/< subfolder >

Here an overview of the latest success tasks:

image
image1197×346 125 KB

All help is welcome to fix this new issue, or what is changed since the migration to pulp3?

Thnx in advance

39

Hi @langesmalle ,

I’m glad you got past your repository issues. Apologies for not mentioning that the username/password patch only fixes the issue for newly created repositories.

That is new to Pulp 3. You will no longer see content units on the filesystem like you used to with Pulp 2. The best way now to see the content directly from Pulp would be to browse the hosted repositories from your browser (http://hostname.com/pulp/repos/…) Trying to consume content from a subscribed host could be a good test too. Let me know if you’re having further troubles with consuming content from Pulp 3.

Also, if you’re ever looking to debug Pulp 3 itself, I would recommend giving pulp-cli a try.

40

@iballou ,

Thanks for your feedback.
when I do a wget on one of our hosts of for example the published Postgres11 (only in Library environment I get a 404.

For wget http://my_foreman_server/pulp/repos/DIDM/Library

I can not see the product Postgres_11 at all in the Library file.

Neither in the wget http://my_foreman_server/pulp/repos/DIDM/Library/custom

So I guess it looks like something is still wrong in my enviromment.

This is the dynflow result of the publish I did for the Postgres_11/Postgres_11_EL8.

image
image634×853 181 KB

Thanks for providing me any advice.

41

@langesmalle I get 404s as well if I try only /pulp/repos/Default_Organization/Library/custom/. Looks like you have to navigate to the repository itself to not get the 404, i.e. /pulp/repos/Default_Organization/Library/custom/Product/Repository/

If you really can’t see that content, can I see the output of:

sudo foreman-installer --full-help | grep "\-\-foreman-proxy-content-proxy-pulp-yum-to-pulpcore"
42

@iballou ,

Thnx for your feedback.

My first test was with ‘the product/repo’ included in the url which resulted in a 404.
Only a wget without ‘product/repo’ went well. And for a wget for ‘Library/custom’ I could see that the content for the custom file did not contained the new product/repo, neither the new repo I added into an existing product.

wget http://my-foreman-server/pulp/repos/DIDM/Library/custom/Postgres_11/Postgres_11_EL8/
–2021-03-24 08:29:25-- http://my-foreman-server/pulp/repos/DIDM/Library/custom/Postgres_11/Postgres_11_EL8/
Resolving my-foreman-server (my-foreman-server)… 10.16.x.y
Connecting to my-foreman-server (my-foreman-server)|10.16.x.y|:80… connected.
HTTP request sent, awaiting response… 404 Not Found

So regarding the requested output, it has the state ‘false’:

[root@my-foreman-server ~]# foreman-installer --full-help | grep “–foreman-proxy-content-proxy-pulp-yum-to-pulpcore”
–foreman-proxy-content-proxy-pulp-yum-to-pulpcore Proxy /pulp/yum to Pulpcore at /pulp/content (current: false)

Is this required since the migration to pulp3, because before the migration it worked well with pulp2?
Before the migration I created a new product for centos8 and repos as well added new repos to existing products. And those were immediately available.

As far as I can remember, this was not mentioned in the documentation that it had to be activated.

So in case you confirm I need to activate this proxy, how should I proceed in order to make these new product and repo’s available? Do I need to remove them en create them again, or is there an easy way of workaround to do that?

Thnx in advance for your help.

43

Hi @langesmalle,

You didn’t miss any docs, it’s something we need to add to foreman maintain. I’ll be doing that soon.

To fix this, you simply need to run the following to update Apache:

foreman-installer --foreman-proxy-content-proxy-pulp-yum-to-pulpcore true --foreman-proxy-content-proxy-pulp-isos-to-pulpcore true --foreman-proxy-content-proxy-pulp-deb-to-pulpcore true

After that, all of your content should be hosted.

This is a one-time step that should be run after the switchover. You won’t need to recreate any products or repos. For anyone else reading this, foreman maintain should be doing this for you during the switchover command. That is the fix that will be taken care of soon.

44

@iballou ,
Thnx for you confirmation that i did not missed any action from the docs.

The provided forman-installer command is currently busy.
Will keep you informed about the result or when i discover other issues.

45

@iballou ,

Yeah, I can confirm that after applying this command, the repo’s are available on the hosts, a wget works as expected.

However, I just notices that the new product and/or repos I added after the pulp3 migration are not visible in the folders:

  • /var/www/pub/yum/http/repos/< organization >/Library/custom< product >/< repo >
  • /var/www/pub/yum/http/repos/< organization >/Library/< content view >/custom< product >/< repo >
  • /var/www/pub/yum/http/repos/< organization >/< contentview >/custom/< product >/< repo >

This was the case before the pulp3 migration.
Is this normal?

46

@langesmalle,

Glad the installer fixed it, apologies for the mix-up there.

That’s normal. You won’t be seeing hosted content on the filesystem. I just double checked, and with new Pulp 3 only installs, that /var/www/pub/ folder doesn’t even exist. You’re seeing leftover files from Pulp 2.

47

@iballou ,

OK thnx for this feedback.
Will this mean that the folder /var/www/pub (pub is the symlink) is no more needed?
Is it safe to remove it, and is this some thing that also needs to be implemented during the foreman-maintain content switchover process?

48

@langesmalle /var/www/pub/ can be deleted.

Pulp 2 cleanup is going to happen during the upgrade to 4.0 (foreman_maintain/remove.rb at master · theforeman/foreman_maintain · GitHub). At the moment I don’t think that symlink is cleaned up though, I’ll add an issue to make that happen.