ActionsPulpRepositoryCreate fails, publish task remains paused at 90%

Problem:

There are two issues here.

First

The publish task fails. Actions::Pulp::Repository::Create is the action with “skip”

**Input:**
---
content_type: yum
pulp_id: 1-Extras-v78_0-67c3263a-2685-47fe-8e9b-ef71639970d7
name: zabbix 4.0
docker_upstream_name: 
docker_tags_whitelist: 
feed: http://repo.zabbix.com/zabbix/4.0/rhel/7/x86_64/
ssl_ca_cert: 
ssl_client_cert: 
ssl_client_key: 
unprotected: true
checksum_type: sha256
path: pmcc/content_views/Extras/78.0/custom/Zabbix_Official_Repository/zabbix_4_0
download_policy: immediate
ostree_upstream_sync_depth: 0
ostree_publish_depth: 0
deb_releases: 
deb_components: 
deb_architectures: 
with_importer: true
mirror_on_sync: true
ssl_validation: true
upstream_username: ''
upstream_password: XXXXXXX
repo_registry_id: 
proxy_host: 
ignorable_content:
- srpm
remote_user: admin
remote_cp_user: admin
current_user_id: 3
current_organization_id: 1
current_location_id: 

**Output:**
--- {}

**Error:**

RestClient::BadRequest

400 Bad Request 
---
- "/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/abstract_response.rb:223:in
  `exception_with_response'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/abstract_response.rb:103:in
  `return!'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.9.0/lib/runcible/base.rb:95:in
  `block in get_response'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:807:in
  `process_result'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:725:in
  `block in transmit'"
- "/opt/rh/rh-ruby25/root/usr/share/ruby/net/http.rb:910:in `start'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:715:in
  `transmit'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:145:in
  `execute'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:52:in
  `execute'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/resource.rb:67:in
  `post'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.9.0/lib/runcible/base.rb:94:in
  `get_response'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.9.0/lib/runcible/base.rb:74:in
  `call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.9.0/lib/runcible/resources/repository.rb:22:in
  `create'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.9.0/lib/runcible/extensions/repository.rb:69:in
  `create_with_importer_and_distributors'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.9.0/app/lib/actions/pulp/repository/create.rb:37:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/action.rb:539:in
  `block (3 levels) in execute_run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/stack.rb:26:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware.rb:18:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware.rb:31:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/stack.rb:22:in
  `call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/stack.rb:26:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware.rb:18:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.9.0/app/lib/actions/middleware/remote_action.rb:16:in
  `block in run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.9.0/app/lib/actions/middleware/remote_action.rb:40:in
  `block in as_remote_user'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.9.0/app/models/katello/concerns/user_extensions.rb:21:in
  `cp_config'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.9.0/app/lib/actions/middleware/remote_action.rb:27:in
  `as_cp_user'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.9.0/app/lib/actions/middleware/remote_action.rb:39:in
  `as_remote_user'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.9.0/app/lib/actions/middleware/remote_action.rb:16:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/stack.rb:22:in
  `call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/stack.rb:26:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware.rb:18:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.3/app/lib/actions/middleware/rails_executor_wrap.rb:14:in
  `block in run'"
- "/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/execution_wrapper.rb:87:in
  `wrap'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.3/app/lib/actions/middleware/rails_executor_wrap.rb:13:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/stack.rb:22:in
  `call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/stack.rb:26:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware.rb:18:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/action/progress.rb:30:in
  `with_progress_calculation'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/action/progress.rb:16:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/stack.rb:22:in
  `call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/stack.rb:26:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware.rb:18:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.9.0/app/lib/actions/middleware/keep_locale.rb:11:in
  `block in run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.9.0/app/lib/actions/middleware/keep_locale.rb:22:in
  `with_locale'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.9.0/app/lib/actions/middleware/keep_locale.rb:11:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/stack.rb:22:in
  `call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/stack.rb:26:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware.rb:18:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.3/app/lib/actions/middleware/keep_current_user.rb:15:in
  `block in run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.3/app/lib/actions/middleware/keep_current_user.rb:43:in
  `restore_curent_user'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.3/app/lib/actions/middleware/keep_current_user.rb:15:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/stack.rb:22:in
  `call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/stack.rb:26:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware.rb:18:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.3/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in
  `block in run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.3/app/lib/actions/middleware/keep_current_taxonomies.rb:44:in
  `restore_current_taxonomies'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.3/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/stack.rb:22:in
  `call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/stack.rb:26:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware.rb:18:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware.rb:31:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/stack.rb:22:in
  `call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/world.rb:30:in
  `execute'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/action.rb:538:in
  `block (2 levels) in execute_run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/action.rb:537:in
  `catch'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/action.rb:537:in
  `block in execute_run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/action.rb:451:in
  `block in with_error_handling'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/action.rb:451:in
  `catch'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/action.rb:451:in
  `with_error_handling'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/action.rb:532:in
  `execute_run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/action.rb:278:in
  `execute'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in
  `block (2 levels) in execute'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/execution_plan/steps/abstract.rb:162:in
  `with_meta_calculation'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in
  `block in execute'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:30:in
  `open_action'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:15:in
  `execute'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/director.rb:43:in
  `execute'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/executors/parallel/worker.rb:14:in
  `block in on_message'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/executors.rb:12:in
  `run_user_code'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/executors/parallel/worker.rb:13:in
  `on_message'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/context.rb:46:in
  `on_envelope'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/executes_context.rb:7:in
  `on_envelope'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/actor.rb:26:in
  `on_envelope'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/awaits.rb:15:in
  `on_envelope'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/sets_results.rb:14:in
  `on_envelope'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/buffer.rb:38:in
  `process_envelope'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/buffer.rb:31:in
  `process_envelopes?'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/buffer.rb:20:in
  `on_envelope'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/termination.rb:55:in
  `on_envelope'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/removes_child.rb:10:in
  `on_envelope'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/sets_results.rb:14:in
  `on_envelope'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/core.rb:161:in
  `process_envelope'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/core.rb:95:in
  `block in on_envelope'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/core.rb:118:in
  `block (2 levels) in schedule_execution'"
- "/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/synchronization/mri_lockable_object.rb:38:in
  `block in synchronize'"
- "/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/synchronization/mri_lockable_object.rb:38:in
  `synchronize'"
- "/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/synchronization/mri_lockable_object.rb:38:in
  `synchronize'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/core.rb:115:in
  `block in schedule_execution'"
- "/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/serialized_execution.rb:18:in
  `call'"
- "/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/serialized_execution.rb:96:in
  `work'"
- "/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/serialized_execution.rb:77:in
  `block in call_job'"
- "/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:348:in
  `run_task'"
- "/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:337:in
  `block (3 levels) in create_worker'"
- "/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in
  `loop'"
- "/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in
  `block (2 levels) in create_worker'"
- "/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:in
  `catch'"
- "/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:in
  `block in create_worker'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in
  `block in create_with_logging_context'"

Second

When Actions::Pulp::Repository::Create fails there’s no obvious “out”. So using the UI, I go to Dynflow Console and then I need to do this:

  1. Find paused action, click skip
  2. Back to the top of the page, click resume
  3. Refresh page
  4. Find next paused action, click skip.
  5. Return to step two until task completed with warning

In this case, there are two necessary changes:

  1. Steps 1, 2, 3 for preference - but possibly just 2,3 - shouldn’t be necessary. (and step 5 by extrapolation)

  2. clicking skip on Actions::Pulp::Repository::Create should cascade through the next actions that require it’s success:

    183: Actions::Katello::Repository::Create (pending)
    186: Actions::Pulp::Repository::CopySrpm (pending)
    188: Actions::Pulp::Repository::CopyRpm (pending)
    190: Actions::Pulp::Repository::CopyErrata (pending)
    192: Actions::Pulp::Repository::CopyPackageGroup (pending)
    194: Actions::Pulp::Repository::CopyYumMetadataFile (pending)
    196: Actions::Pulp::Repository::CopyDistribution (pending)
    198: Actions::Pulp::Repository::CopyModuleStream (pending)
    200: Actions::Pulp::Repository::CopyModuleDefault (pending)
    202: Actions::Katello::Repository::IndexContent (pending)
    204: Actions::Pulp::Repository::PurgeEmptyErrata (pending)
    206: Actions::Pulp::Repository::PurgeEmptyPackageGroups (pending)
    208: Actions::Katello::Repository::IndexErrata (pending)
    210: Actions::Katello::Repository::IndexPackageGroups (pending)
    213: Actions::Pulp::Repository::DistributorPublish (pending)
    217: Actions::Katello::Repository::CheckMatchingContent (pending)
    219: Actions::Katello::Repository::IndexContent (pending)
    222: Actions::Pulp::Repository::DistributorPublish (pending
    

Foreman and Proxy versions:
CentOS 7.5, up to date
Foreman 1.20
Katello 3.9
candlepin-2.5.7
pulp-server-2.17.1-1

I’ve re-sync’d the Zabbix 4.0 repo successfully, so that seems to be ok.

I successfully published this CV yesterday. The two differences:

  • foreman/kattelo upgrade and
  • I’d added a new product/repo to the CV.

If I go into the Zabbix 4.0 product, and then click the link shown at the “published at:” field, I get a repo looking end point - packages and repodata directories.