PLP0000: Pulp exception occurred: PulpExecutionException and pulp-manage-db issue

Support
1

Hi all,

I am trying to publish a new version of a content view that has RHEL
content, but it is stuck at 84% with the error "PLP0000: Pulp exception
occurred: PulpExecutionException"

If I go to the dynaflow console, it is stuck at "966:
Actions::Pulp::Repository::CopyYumMetadataFile (error)"

Started at: 2016-08-17 21:11:17 UTC

Ended at: 2016-08-17 21:11:18 UTC

Real time: 1.30s

Execution time (excluding suspended state): 0.65s

Input:

··· --- source_pulp_id: NIWA-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_7_Server_-_Supplementary_RPMs_x86_64_7Server target_pulp_id: NIWA-RHEL7_Server-37_0-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_7_Server_-_Supplementary_RPMs_x86_64_7Server clauses: remote_user: admin remote_cp_user: admin

Output:

pulp_tasks:- exception:
task_type: pulp.server.managers.repo.unit_association.associate_from_repo
_href: "/pulp/api/v2/tasks/05d9f48e-dcc6-4ddd-8cd8-28efb7ab676a/"
task_id: 05d9f48e-dcc6-4ddd-8cd8-28efb7ab676a
tags:

  • pulp:repository:NIWA-RHEL7_Server-37_0-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_7_Server_-_Supplementary_RPMs_x86_64_7Server
  • pulp:repository:NIWA-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_7_Server_-_Supplementary_RPMs_x86_64_7Server
  • pulp:action:associate
    finish_time: ‘2016-08-17T21:11:18Z’
    _ns: task_status
    start_time: '2016-08-17T21:11:18Z’
    traceback: |
    Traceback (most recent call last):
    File “/usr/lib/python2.7/site-packages/celery/app/trace.py”, line 240, in trace_task
    R = retval = fun(*args, **kwargs)
    File “/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py”, line 473, in call
    return super(Task, self).call(*args, **kwargs)
    File “/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py”, line 103, in call
    return super(PulpTask, self).call(*args, **kwargs)
    File “/usr/lib/python2.7/site-packages/celery/app/trace.py”, line 437, in protected_call
    return self.run(*args, **kwargs)
    File “/usr/lib/python2.7/site-packages/pulp/server/managers/repo/unit_association.py”, line 273, in associate_from_repo
    units=transfer_units)
    File “/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/importer.py”, line 57, in import_units
    return associate.associate(source_repo, dest_repo, import_conduit, config, units)
    File “/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/associate.py”, line 53, in associate
    associated_units = set([_associate_unit(dest_repo, unit) for unit in units])
    File “/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/associate.py”, line 333, in _associate_unit
    return associate_copy_for_repo(unit, dest_repo, True)
    File “/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/associate.py”, line 369, in associate_copy_for_repo
    new_unit.safe_import_content(unit._storage_path)
    File “/usr/lib/python2.7/site-packages/pulp/server/db/model/init.py”, line 905, in safe_import_content
    self.import_content(path, location)
    File “/usr/lib/python2.7/site-packages/pulp/server/db/model/init.py”, line 877, in import_content
    raise exceptions.PulpCodedException(error_code=error_codes.PLP0037, path=path)
    PulpExecutionException: Pulp exception occurred: PulpExecutionException
    spawned_tasks: []
    progress_report: {}
    queue: reserved_resource_worker-4@wellkatello.dq
    state: error
    worker_name: reserved_resource_worker-4@wellkatello
    error:
    code: PLP0000
    data: {}
    description: 'Pulp exception occurred: PulpExecutionException’
    sub_errors: []
    _id:
    "$oid": 57b4d2f513f5c6fec4e80642
    id: 57b4d2f513f5c6fec4e80642
    poll_attempts:
    total: 1
    failed: 1

Error:

Katello::Errors::PulpError

PLP0000: Pulp exception occurred: PulpExecutionException

---- “/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/actions/pulp/abstract_async_task.rb:121:in
block in external_task='"- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/actions/pulp/abstract_async_task.rb:119:ineach’”- “/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/actions/pulp/abstract_async_task.rb:119:in
external_task='"- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/actions/pulp/repository/abstract_copy_content.rb:40:inexternal_task=’”- “/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action/polling.rb:98:in
poll_external_task_with_rescue'"- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action/polling.rb:21:inrun’”- “/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action/cancellable.rb:9:in
run'"- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/actions/pulp/abstract_async_task.rb:45:inrun’”- “/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:506:in
block (3 levels) in execute_run'"- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:incall’”- “/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in
pass'"- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:inpass’”- “/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:30:in
run'"- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:incall’”- “/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in
pass'"- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:inpass’”- “/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/actions/middleware/remote_action.rb:16:in
block in run'"- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/actions/middleware/remote_action.rb:40:inblock in as_remote_user’”- “/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/models/katello/concerns/user_extensions.rb:20:in
cp_config'"- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/actions/middleware/remote_action.rb:27:inas_cp_user’”- “/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/actions/middleware/remote_action.rb:39:in
as_remote_user'"- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/actions/middleware/remote_action.rb:16:inrun’”- “/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:in
call'"- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:inpass’”- “/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:in
pass'"- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action/progress.rb:30:inwith_progress_calculation’”- “/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action/progress.rb:16:in
run'"- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:incall’”- “/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in
pass'"- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:inpass’”- “/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/actions/middleware/keep_locale.rb:11:in
block in run'"- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/actions/middleware/keep_locale.rb:22:inwith_locale’”- “/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/actions/middleware/keep_locale.rb:11:in
run'"- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:incall’”- “/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in
pass'"- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:inpass’”- “/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:30:in
run'"- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:incall’”- “/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/world.rb:30:in
execute'"- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:505:inblock (2 levels) in execute_run’”- “/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:504:in
catch'"- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:504:inblock in execute_run’”- “/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:419:in
call'"- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:419:inblock in with_error_handling’”- “/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:419:in
catch'"- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:419:inwith_error_handling’”- “/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:499:in
execute_run'"- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:260:inexecute’”- “/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:9:in
block (2 levels) in execute'"- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/abstract.rb:155:incall’”- “/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/abstract.rb:155:in
with_meta_calculation'"- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:8:inblock in execute’”- “/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:22:in
open_action'"- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:7:inexecute’”- “/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/executors/parallel/worker.rb:15:in
block in on_message'"- "/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:74:inblock in assigns’”- “/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in
tap'"- "/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:inassigns’”- “/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:56:in
match_value'"- "/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:36:inblock in match?’”- “/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in
each'"- "/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:inmatch?’”- “/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:23:in
match'"- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/executors/parallel/worker.rb:12:inon_message’”- “/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/context.rb:46:in
on_envelope'"- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/executes_context.rb:7:inon_envelope’”- “/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in
pass'"- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/actor.rb:26:inon_envelope’”- “/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in
pass'"- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/awaits.rb:15:inon_envelope’”- “/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in
pass'"- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/sets_results.rb:14:inon_envelope’”- “/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in
pass'"- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:38:inprocess_envelope’”- “/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:31:in
process_envelopes?'"- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:20:inon_envelope’”- “/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in
pass'"- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/termination.rb:55:inon_envelope’”- “/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in
pass'"- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/removes_child.rb:10:inon_envelope’”- “/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in
pass'"- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/sets_results.rb:14:inon_envelope’”- “/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:161:in
process_envelope'"- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:95:inblock in on_envelope’”- “/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:118:in
block (2 levels) in schedule_execution'"- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:inblock in synchronize’”- “/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:in
synchronize'"- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:insynchronize’”- “/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:115:in
block in schedule_execution'"- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:18:incall’”- “/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:18:in
call'"- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:96:inwork’”- “/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:77:in
block in call_job'"- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:333:incall’”- “/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:333:in
run_task'"- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:322:inblock (3 levels) in create_worker’”- “/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:305:in
loop'"- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:305:inblock (2 levels) in create_worker’”- “/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:304:in
catch'"- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:304:inblock in create_worker’”- “/opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in
call'"- "/opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:inblock in create_with_logging_context’”

If I resume the job while monitoring /var/log/messages for pulp messages, I
was seeing

Aug 17 16:19:36 wellkatello.niwa.local pulp: celery.worker.strategy:INFO:
Received task:
pulp.server.async.tasks._release_resource[0c85402f-a9ca-4571-a1c7-7f2371282f3b]
Aug 17 16:19:36 wellkatello.niwa.local pulp: celery.worker.job:INFO: Task
pulp.server.async.tasks._queue_reserved_task[e1bebe64-7f44-417b-8ca7-564e9e6b0538]
succeeded in 0.303873108s: None
Aug 17 16:19:36 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4695-73376) Exception
from importer [yum_importer] while importing units into repository
[NIWA-RHEL7_Server-36_0-Oracle_Java__for_RHEL_Serve$
Aug 17 16:19:36 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4695-73376) Traceback
(most recent call last):
Aug 17 16:19:36 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4695-73376) File
"/usr/lib/python2.7/site-packages/pulp/server/managers/repo/unit_association.py",
line 273, in associate_from_repo
Aug 17 16:19:36 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4695-73376)
units=transfer_units)
Aug 17 16:19:36 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4695-73376) File
"/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/importer.py",
line 57, in import_units
Aug 17 16:19:36 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4695-73376) return
associate.associate(source_repo, dest_repo, import_conduit, config, units)
Aug 17 16:19:36 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4695-73376) File
"/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/associate.py",
line 53, in associate
Aug 17 16:19:36 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4695-73376)
associated_units = set([_associate_unit(dest_repo, unit) for unit in units])
Aug 17 16:19:36 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4695-73376) File
"/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/associate.py",
line 333, in _associate_unit
Aug 17 16:19:36 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4695-73376) return
associate_copy_for_repo(unit, dest_repo, True)
Aug 17 16:19:36 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4695-73376) File
"/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/associate.py",
line 369, in associate_copy_for_repo
Aug 17 16:19:36 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4695-73376)
new_unit.safe_import_content(unit._storage_path)
Aug 17 16:19:36 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4695-73376) File
"/usr/lib/python2.7/site-packages/pulp/server/db/model/init.py", line

2

Ok, so I changed the user apache from /sbin/nologin to /bin/bash in
/etc/passwd and "su - apache" and ran pulp-manage-db - it ran successfully
by the looks and pulp is now running again

-bash-4.2$ pulp-manage-db
Attempting to connect to localhost:27017
Attempting to connect to localhost:27017
Write concern for Mongo connection: {}
Loading content types.
Loading type descriptors []
Parsing type descriptors
Validating type descriptor syntactic integrity
Validating type descriptor semantic integrity
Loading unit model: erratum = pulp_rpm.plugins.db.models:Errata
Loading unit model: distribution = pulp_rpm.plugins.db.models:Distribution
Loading unit model: package_group = pulp_rpm.plugins.db.models:PackageGroup
Loading unit model: package_category =
pulp_rpm.plugins.db.models:PackageCategory
Loading unit model: iso = pulp_rpm.plugins.db.models:ISO
Loading unit model: package_environment =
pulp_rpm.plugins.db.models:PackageEnvironment
Loading unit model: drpm = pulp_rpm.plugins.db.models:DRPM
Loading unit model: srpm = pulp_rpm.plugins.db.models:SRPM
Loading unit model: rpm = pulp_rpm.plugins.db.models:RPM
Loading unit model: yum_repo_metadata_file =
pulp_rpm.plugins.db.models:YumMetadataFile
Loading unit model: puppet_module = pulp_puppet.plugins.db.models:Module
Loading unit model: docker_blob = pulp_docker.plugins.models:Blob
Loading unit model: docker_manifest = pulp_docker.plugins.models:Manifest
Loading unit model: docker_image = pulp_docker.plugins.models:Image
Loading unit model: docker_tag = pulp_docker.plugins.models:Tag
Updating the database with types []
Found the following type definitions that were not present in the update
collection [puppet_module, drpm, docker_manifest, docker_blob, erratum,
srpm, package_group, package_category, iso, package_environment,
docker_tag, distribution, rpm, yum_repo_metadata_file, docker_image]
Updating the database with types [puppet_module, docker_tag, erratum,
docker_blob, docker_manifest, yum_repo_metadata_file, package_group,
package_category, iso, package_environment, drpm, distribution, rpm, srpm,
docker_image]
Content types loaded.
Ensuring the admin role and user are in place.
Admin role and user are in place.
Beginning database migrations.
Migration package pulp.server.db.migrations is up to date at version 23
Migration package pulp_docker.plugins.migrations is up to date at version 2
Applying pulp_puppet.plugins.migrations version 5
***Note. You may want to re-publish the list of repos found in
/var/lib/pulp/0005_puppet_module_name_change.txt.
This migration fixed an issue with modules installation related to wrong
puppet_module name.
/usr/lib/python2.7/site-packages/mongoengine/document.py:367:
DeprecationWarning: update is deprecated. Use replace_one, update_one or
update_many instead.
upsert=upsert, **write_concern)

Migration to pulp_puppet.plugins.migrations version 5 complete.
Applying pulp_rpm.plugins.migrations version 31
Migration to pulp_rpm.plugins.migrations version 31 complete.
Applying pulp_rpm.plugins.migrations version 32
/usr/lib/python2.7/site-packages/pulp_rpm/plugins/migrations/0032_ensure_variant_field.py:25:
DeprecationWarning: update is deprecated. Use replace_one, update_one or
update_many instead.
{'$set': {variant: ''}}, multi=True)

Migration to pulp_rpm.plugins.migrations version 32 complete.
Loading unit model: erratum = pulp_rpm.plugins.db.models:Errata
Loading unit model: distribution = pulp_rpm.plugins.db.models:Distribution
Loading unit model: package_group = pulp_rpm.plugins.db.models:PackageGroup
Loading unit model: package_category =
pulp_rpm.plugins.db.models:PackageCategory
Loading unit model: iso = pulp_rpm.plugins.db.models:ISO
Loading unit model: package_environment =
pulp_rpm.plugins.db.models:PackageEnvironment
Loading unit model: drpm = pulp_rpm.plugins.db.models:DRPM
Loading unit model: srpm = pulp_rpm.plugins.db.models:SRPM
Loading unit model: rpm = pulp_rpm.plugins.db.models:RPM
Loading unit model: yum_repo_metadata_file =
pulp_rpm.plugins.db.models:YumMetadataFile
Loading unit model: puppet_module = pulp_puppet.plugins.db.models:Module
Loading unit model: docker_blob = pulp_docker.plugins.models:Blob
Loading unit model: docker_manifest = pulp_docker.plugins.models:Manifest
Loading unit model: docker_image = pulp_docker.plugins.models:Image
Loading unit model: docker_tag = pulp_docker.plugins.models:Tag
Database migrations complete.

[root@wellkatello tmp]# hammer ping
candlepin:
Status: ok
Server Response: Duration: 49ms
candlepin_auth:
Status: ok
Server Response: Duration: 50ms
pulp:
Status: ok
Server Response: Duration: 5726ms
foreman_tasks:
Status: ok
Server Response: Duration: 37ms

[root@wellkatello tmp]#

not sure what's up with the response time for pulp tho?

After I got pulp working again, I started the failed content view
publish…and it finished successfully! Sorry for posting - it seems
everything is OK now!

3

The PulpExecutionException error has reoccurred and is affecting all our content views now, any help greatly appreciated!

4

Are you getting the PLP0037 error?

··· On 08/24/2016 03:44 AM, Dylan Baars wrote: > The PulpExecutionException error has reoccurred and is affecting all our content views now, any help greatly appreciated! >
5

Hi Chris,

yes, the PLP0037 error - here's what happens if I try and resume the task

Note: I got the wrong end of the stick w.r.t the issue, it is still only
affecting a content view with RHEL repos in it only - a content view with
CentOS repos is working fine

Aug 25 06:58:57 wellkatello.niwa.local pulp: celery.worker.strategy:INFO:
Received task:
pulp.server.async.tasks._queue_reserved_task[4be9c103-57f6-48af-bb50-146416ec6a52]
Aug 25 06:58:57 wellkatello.niwa.local pulp: celery.worker.strategy:INFO:
Received task:
pulp.server.async.tasks._queue_reserved_task[a04c8468-6c92-41bc-9d39-1bface5b4726]
Aug 25 06:58:57 wellkatello.niwa.local pulp: celery.worker.strategy:INFO:
Received task:
pulp.server.async.tasks._queue_reserved_task[a2dbb186-59f7-49f3-9780-78973e15c9d7]
Aug 25 06:58:57 wellkatello.niwa.local pulp: celery.worker.strategy:INFO:
Received task:
pulp.server.managers.repo.unit_association.associate_from_repo[76b6d820-da07-48c1-b326-4ced76816eba]
Aug 25 06:58:57 wellkatello.niwa.local pulp: celery.worker.strategy:INFO:
Received task:
pulp.server.async.tasks._release_resource[d8630f69-6b5d-4c65-bc72-73b3ae721385]
Aug 25 06:58:57 wellkatello.niwa.local pulp: celery.worker.job:INFO: Task
pulp.server.async.tasks._queue_reserved_task[4be9c103-57f6-48af-bb50-146416ec6a52]
succeeded in 0.0682786369871s: None
Aug 25 06:58:57 wellkatello.niwa.local pulp: celery.worker.strategy:INFO:
Received task:
pulp.server.managers.repo.unit_association.associate_from_repo[1c36c42d-1e91-4ec1-aaa5-7c92beb36dbf]
Aug 25 06:58:57 wellkatello.niwa.local pulp: celery.worker.job:INFO: Task
pulp.server.async.tasks._queue_reserved_task[a04c8468-6c92-41bc-9d39-1bface5b4726]
succeeded in 0.0472956059966s: None
Aug 25 06:58:57 wellkatello.niwa.local pulp: celery.worker.strategy:INFO:
Received task:
pulp.server.async.tasks._release_resource[142f9ad1-1a21-436b-94f4-94808c89e5dc]
Aug 25 06:58:57 wellkatello.niwa.local pulp: celery.worker.strategy:INFO:
Received task:
pulp.server.managers.repo.unit_association.associate_from_repo[fd34f15b-f672-48e4-924b-9367302e69c7]
Aug 25 06:58:57 wellkatello.niwa.local pulp: celery.worker.job:INFO: Task
pulp.server.async.tasks.queue_reserved_task[a2dbb186-59f7-49f3-9780-78973e15c9d7]
succeeded in 0.546405074012s: None
Aug 25 06:58:57 wellkatello.niwa.local pulp: celery.worker.strategy:INFO:
Received task:
pulp.server.async.tasks.release_resource[0a8a3405-dc56-4956-8341-66fcc07e4d07]
Aug 25 06:58:58 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4160-00704) Exception
from importer [yum_importer] while importing units into repository
[NIWA-Prod-Server-RHEL7_Server-Oracle_Java__for_RHEL_Server-Red_Hat_Enterprise_Linux_7_Server-_Oracle_Java_RPMs_x86_64_7Server]
Aug 25 06:58:58 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4160-00704) Traceback
(most recent call last):
Aug 25 06:58:58 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4160-00704) File
"/usr/lib/python2.7/site-packages/pulp/server/managers/repo/unit_association.py",
line 273, in associate_from_repo
Aug 25 06:58:58 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4160-00704)
units=transfer_units)
Aug 25 06:58:58 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4160-00704) File
"/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/importer.py",
line 57, in import_units
Aug 25 06:58:58 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4160-00704) return
associate.associate(source_repo, dest_repo, import_conduit, config, units)
Aug 25 06:58:58 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4160-00704) File
"/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/associate.py",
line 53, in associate
Aug 25 06:58:58 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4160-00704)
associated_units = set([_associate_unit(dest_repo, unit) for unit in units])
Aug 25 06:58:58 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4160-00704) File
"/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/associate.py",
line 333, in _associate_unit
Aug 25 06:58:58 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4160-00704) return
associate_copy_for_repo(unit, dest_repo, True)
Aug 25 06:58:58 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4160-00704) File
"/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/associate.py",
line 373, in associate_copy_for_repo
Aug 25 06:58:58 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4160-00704)
new_unit.safe_import_content(unit._storage_path)
Aug 25 06:58:58 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4160-00704) File
"/usr/lib/python2.7/site-packages/pulp/server/db/model/init.py", line
905, in safe_import_content
Aug 25 06:58:58 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4160-00704)
self.import_content(path, location)
Aug 25 06:58:58 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4160-00704) File
"/usr/lib/python2.7/site-packages/pulp/server/db/model/init.py", line
877, in import_content
Aug 25 06:58:58 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4160-00704) raise
exceptions.PulpCodedException(error_code=error_codes.PLP0037, path=path)
Aug 25 06:58:58 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4160-00704)
PulpCodedException: Content import of
/var/lib/pulp/content/units/yum_repo_metadata_file/3f/07e460b4eaf0910dc60e6c183373c7c61fa7f95626c6cd0888f64ae62d2050/productid.gz
failed - must be an existing file.
Aug 25 06:58:58 wellkatello.niwa.local pulp: pulp.server.async.tasks:INFO:
Task failed : [1c36c42d-1e91-4ec1-aaa5-7c92beb36dbf]
Aug 25 06:58:58 wellkatello.niwa.local pulp: celery.worker.job:ERROR:
(1860-00704) Task
pulp.server.managers.repo.unit_association.associate_from_repo[1c36c42d-1e91-4ec1-aaa5-7c92beb36dbf]
raised unexpected: PulpExecutionException()
Aug 25 06:58:58 wellkatello.niwa.local pulp: celery.worker.job:ERROR:
(1860-00704) Traceback (most recent call last):
Aug 25 06:58:58 wellkatello.niwa.local pulp: celery.worker.job:ERROR:
(1860-00704) File "/usr/lib/python2.7/site-packages/celery/app/trace.py",
line 240, in trace_task
Aug 25 06:58:58 wellkatello.niwa.local pulp: celery.worker.job:ERROR:
(1860-00704) R = retval = fun(*args, **kwargs)
Aug 25 06:58:58 wellkatello.niwa.local pulp: celery.worker.job:ERROR:
(1860-00704) File
"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 473, in
call
Aug 25 06:58:58 wellkatello.niwa.local pulp: celery.worker.job:ERROR:
(1860-00704) return super(Task, self).call(*args, **kwargs)
Aug 25 06:58:58 wellkatello.niwa.local pulp: celery.worker.job:ERROR:
(1860-00704) File
"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 103, in
call
Aug 25 06:58:58 wellkatello.niwa.local pulp: celery.worker.job:ERROR:
(1860-00704) return super(PulpTask, self).call(*args, **kwargs)
Aug 25 06:58:58 wellkatello.niwa.local pulp: celery.worker.job:ERROR:
(1860-00704) File "/usr/lib/python2.7/site-packages/celery/app/trace.py",
line 437, in protected_call
Aug 25 06:58:58 wellkatello.niwa.local pulp: celery.worker.job:ERROR:
(1860-00704) return self.run(*args, **kwargs)
Aug 25 06:58:58 wellkatello.niwa.local pulp: celery.worker.job:ERROR:
(1860-00704) File
"/usr/lib/python2.7/site-packages/pulp/server/managers/repo/unit_association.py",
line 273, in associate_from_repo
Aug 25 06:58:58 wellkatello.niwa.local pulp: celery.worker.job:ERROR:
(1860-00704) units=transfer_units)
Aug 25 06:58:58 wellkatello.niwa.local pulp: celery.worker.job:ERROR:
(1860-00704) File
"/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/importer.py",
line 57, in import_units
Aug 25 06:58:58 wellkatello.niwa.local pulp: celery.worker.job:ERROR:
(1860-00704) return associate.associate(source_repo, dest_repo,
import_conduit, config, units)
Aug 25 06:58:58 wellkatello.niwa.local pulp: celery.worker.job:ERROR:
(1860-00704) File
"/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/associate.py",
line 53, in associate
Aug 25 06:58:58 wellkatello.niwa.local pulp: celery.worker.job:ERROR:
(1860-00704) associated_units = set([_associate_unit(dest_repo, unit)
for unit in units])
Aug 25 06:58:58 wellkatello.niwa.local pulp: celery.worker.job:ERROR:
(1860-00704) File
"/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/associate.py",
line 333, in _associate_unit
Aug 25 06:58:58 wellkatello.niwa.local pulp: celery.worker.job:ERROR:
(1860-00704) return associate_copy_for_repo(unit, dest_repo, True)
Aug 25 06:58:58 wellkatello.niwa.local pulp: celery.worker.job:ERROR:
(1860-00704) File
"/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/associate.py",
line 373, in associate_copy_for_repo
Aug 25 06:58:58 wellkatello.niwa.local pulp: celery.worker.job:ERROR:
(1860-00704) new_unit.safe_import_content(unit._storage_path)
Aug 25 06:58:58 wellkatello.niwa.local pulp: celery.worker.job:ERROR:
(1860-00704) File
"/usr/lib/python2.7/site-packages/pulp/server/db/model/init.py", line
905, in safe_import_content
Aug 25 06:58:58 wellkatello.niwa.local pulp: celery.worker.job:ERROR:
(1860-00704) self.import_content(path, location)
Aug 25 06:58:58 wellkatello.niwa.local pulp: celery.worker.job:ERROR:
(1860-00704) File
"/usr/lib/python2.7/site-packages/pulp/server/db/model/init.py", line
877, in import_content
Aug 25 06:58:58 wellkatello.niwa.local pulp: celery.worker.job:ERROR:
(1860-00704) raise
exceptions.PulpCodedException(error_code=error_codes.PLP0037, path=path)
Aug 25 06:58:58 wellkatello.niwa.local pulp: celery.worker.job:ERROR:
(1860-00704) PulpExecutionException: Pulp exception occurred:
PulpExecutionException
Aug 25 06:58:58 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4193-14144) Exception
from importer [yum_importer] while importing units into repository
[NIWA-Prod-Server-RHEL7_Server-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_7_Server_RPMs_x86_64_7Server]
Aug 25 06:58:58 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4193-14144) Traceback
(most recent call last):
Aug 25 06:58:58 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4193-14144) File
"/usr/lib/python2.7/site-packages/pulp/server/managers/repo/unit_association.py",
line 273, in associate_from_repo
Aug 25 06:58:58 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4193-14144)
units=transfer_units)
Aug 25 06:58:58 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4193-14144) File
"/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/importer.py",
line 57, in import_units
Aug 25 06:58:58 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4193-14144) return
associate.associate(source_repo, dest_repo, import_conduit, config, units)
Aug 25 06:58:58 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4193-14144) File
"/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/associate.py",
line 53, in associate
Aug 25 06:58:58 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4193-14144)
associated_units = set([_associate_unit(dest_repo, unit) for unit in units])
Aug 25 06:58:58 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4193-14144) File
"/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/associate.py",
line 333, in _associate_unit
Aug 25 06:58:58 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4193-14144) return
associate_copy_for_repo(unit, dest_repo, True)
Aug 25 06:58:58 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4193-14144) File
"/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/associate.py",
line 373, in associate_copy_for_repo
Aug 25 06:58:58 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4193-14144)
new_unit.safe_import_content(unit._storage_path)
Aug 25 06:58:58 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4193-14144) File
"/usr/lib/python2.7/site-packages/pulp/server/db/model/init.py", line
905, in safe_import_content
Aug 25 06:58:58 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4193-14144)
self.import_content(path, location)
Aug 25 06:58:58 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4193-14144) File
"/usr/lib/python2.7/site-packages/pulp/server/db/model/init.py", line
877, in import_content
Aug 25 06:58:58 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4193-14144) raise
exceptions.PulpCodedException(error_code=error_codes.PLP0037, path=path)
Aug 25 06:58:58 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4193-14144)
PulpCodedException: Content import of
/var/lib/pulp/content/units/yum_repo_metadata_file/b6/86a3f3969741abeff744699c9fdd663406ec955470b70649508652cb82a903/productid.gz
failed - must be an existing file.
Aug 25 06:58:58 wellkatello.niwa.local pulp: pulp.server.async.tasks:INFO:
Task failed : [76b6d820-da07-48c1-b326-4ced76816eba]
Aug 25 06:58:58 wellkatello.niwa.local pulp: celery.worker.job:INFO: Task
pulp.server.async.tasks.release_resource[142f9ad1-1a21-436b-94f4-94808c89e5dc]
succeeded in 0.0551061260048s: None
Aug 25 06:58:58 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4152-95680) Exception
from importer [yum_importer] while importing units into repository
[NIWA-Prod-Server-RHEL7_Server-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_7_Server-_Supplementary_RPMs_x86_64_7Server]
Aug 25 06:58:58 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4152-95680) Traceback
(most recent call last):
Aug 25 06:58:58 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4152-95680) File
"/usr/lib/python2.7/site-packages/pulp/server/managers/repo/unit_association.py",
line 273, in associate_from_repo
Aug 25 06:58:58 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4152-95680)
units=transfer_units)
Aug 25 06:58:58 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4152-95680) File
"/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/importer.py",
line 57, in import_units
Aug 25 06:58:58 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4152-95680) return
associate.associate(source_repo, dest_repo, import_conduit, config, units)
Aug 25 06:58:58 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4152-95680) File
"/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/associate.py",
line 53, in associate
Aug 25 06:58:58 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4152-95680)
associated_units = set([_associate_unit(dest_repo, unit) for unit in units])
Aug 25 06:58:58 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4152-95680) File
"/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/associate.py",
line 333, in _associate_unit
Aug 25 06:58:58 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4152-95680) return
associate_copy_for_repo(unit, dest_repo, True)
Aug 25 06:58:58 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4152-95680) File
"/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/associate.py",
line 373, in associate_copy_for_repo
Aug 25 06:58:58 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4152-95680)
new_unit.safe_import_content(unit._storage_path)
Aug 25 06:58:58 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4152-95680) File
"/usr/lib/python2.7/site-packages/pulp/server/db/model/init.py", line
905, in safe_import_content
Aug 25 06:58:58 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4152-95680)
self.import_content(path, location)
Aug 25 06:58:58 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4152-95680) File
"/usr/lib/python2.7/site-packages/pulp/server/db/model/init.py", line
877, in import_content
Aug 25 06:58:58 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4152-95680) raise
exceptions.PulpCodedException(error_code=error_codes.PLP0037, path=path)
Aug 25 06:58:58 wellkatello.niwa.local pulp:
pulp.server.managers.repo.unit_association:ERROR: (4152-95680)
PulpCodedException: Content import of
/var/lib/pulp/content/units/yum_repo_metadata_file/63/383e07b79cd0848c5f598cec27537810ec122229724820bf53bc951a0aebab/productid.gz
failed - must be an existing file.
Aug 25 06:58:58 wellkatello.niwa.local pulp: pulp.server.async.tasks:INFO:
Task failed : [fd34f15b-f672-48e4-924b-9367302e69c7]
Aug 25 06:58:58 wellkatello.niwa.local pulp: celery.worker.job:ERROR:
(1945-14144) Task
pulp.server.managers.repo.unit_association.associate_from_repo[76b6d820-da07-48c1-b326-4ced76816eba]
raised unexpected: PulpExecutionException()
Aug 25 06:58:58 wellkatello.niwa.local pulp: celery.worker.job:ERROR:
(1945-14144) Traceback (most recent call last):
Aug 25 06:58:58 wellkatello.niwa.local pulp: celery.worker.job:ERROR:
(1945-14144) File "/usr/lib/python2.7/site-packages/celery/app/trace.py",
line 240, in trace_task
Aug 25 06:58:58 wellkatello.niwa.local pulp: celery.worker.job:ERROR:
(1945-14144) R = retval = fun(*args, **kwargs)
Aug 25 06:58:58 wellkatello.niwa.local pulp: celery.worker.job:ERROR:
(1945-14144) File
"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 473, in
call
Aug 25 06:58:58 wellkatello.niwa.local pulp: celery.worker.job:ERROR:
(1945-14144) return super(Task, self).call(*args, **kwargs)
Aug 25 06:58:58 wellkatello.niwa.local pulp: celery.worker.job:ERROR:
(1945-14144) File
"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 103, in
call
Aug 25 06:58:58 wellkatello.niwa.local pulp: celery.worker.job:ERROR:
(1945-14144) return super(PulpTask, self).call(*args, **kwargs)
Aug 25 06:58:58 wellkatello.niwa.local pulp: celery.worker.job:ERROR:
(1945-14144) File "/usr/lib/python2.7/site-packages/celery/app/trace.py",
line 437, in protected_call
Aug 25 06:58:58 wellkatello.niwa.local pulp: celery.worker.job:ERROR:
(1945-14144) return self.run(*args, **kwargs)
Aug 25 06:58:58 wellkatello.niwa.local pulp: celery.worker.job:ERROR:
(1

6

Hi all

Anyone have any ideas? We can't promote any red hat repos at the moment

Thanks!

7

The issue is likely:

pulp.server.managers.repo.unit_association:ERROR: (4160-00704)
PulpCodedException: Content import of
/var/lib/pulp/content/units/yum_repo_metadata_file/3f/07e460b4eaf0910dc60e6c183373c7c61fa7f95626c6cd0888f64ae62d2050/productid.gz
failed - must be an existing file.

I remember this error but not the solution, I reached out to the Pulp
team via #pulp to see if they know more.

··· On 08/29/2016 04:28 AM, Dylan Baars wrote: > Hi all > > Anyone have any ideas? We can't promote any red hat repos at the moment > > Thanks! >
8

It looks like you hit https://pulp.plan.io/issues/1944, which should be
fixed in Pulp 2.8.5 and newer.

Your pulp-manage-db output says "Migration to
pulp_rpm.plugins.migrations version 32 complete." which would include
https://github.com/pulp/pulp_rpm/pull/900/files, which should have fixed
this issue.

Did you install a beta version of Pulp at any point on this system?

··· On 08/29/2016 07:50 AM, Chris Duryee wrote: > > > On 08/29/2016 04:28 AM, Dylan Baars wrote: >> Hi all >> >> Anyone have any ideas? We can't promote any red hat repos at the moment >> >> Thanks! >> > > The issue is likely: > > pulp.server.managers.repo.unit_association:ERROR: (4160-00704) > PulpCodedException: Content import of > /var/lib/pulp/content/units/yum_repo_metadata_file/3f/07e460b4eaf0910dc60e6c183373c7c61fa7f95626c6cd0888f64ae62d2050/productid.gz > failed - must be an existing file. > > I remember this error but not the solution, I reached out to the Pulp > team via #pulp to see if they know more. >
9

Hi Chis,

we've never had pulp beta (or katello beta/RCs etc) installed on this
server - it is purely production. I can confirm the file you linked to (
https://github.com/pulp/pulp_rpm/pull/900/files) does seem to be what my
system has. The system is also now at pulp 2.8.6. Is there any way for me
to confirm that the database is at the correct version?

Thanks :slight_smile:
Dylan

··· On Tuesday, August 30, 2016 at 12:57:36 AM UTC+12, Chris Duryee wrote: > > > > On 08/29/2016 07:50 AM, Chris Duryee wrote: > > > > > > On 08/29/2016 04:28 AM, Dylan Baars wrote: > >> Hi all > >> > >> Anyone have any ideas? We can't promote any red hat repos at the moment > >> > >> Thanks! > >> > > > > The issue is likely: > > > > pulp.server.managers.repo.unit_association:ERROR: (4160-00704) > > PulpCodedException: Content import of > > > /var/lib/pulp/content/units/yum_repo_metadata_file/3f/07e460b4eaf0910dc60e6c183373c7c61fa7f95626c6cd0888f64ae62d2050/productid.gz > > > failed - must be an existing file. > > > > I remember this error but not the solution, I reached out to the Pulp > > team via #pulp to see if they know more. > > > > It looks like you hit https://pulp.plan.io/issues/1944 > , > which should be > fixed in Pulp 2.8.5 and newer. > > Your pulp-manage-db output says "Migration to > pulp_rpm.plugins.migrations version 32 complete." which would include > https://github.com/pulp/pulp_rpm/pull/900/files, which should have fixed > this issue. > > Did you install a beta version of Pulp at any point on this system? >
10

FYI I also tried creating a new content view with the same repos as the
failing one, it stops/fails with the same error :frowning:

11

> Hi Chis,
>
> we've never had pulp beta (or katello beta/RCs etc) installed on this
> server - it is purely production. I can confirm the file you linked to (
> https://github.com/pulp/pulp_rpm/pull/900/files) does seem to be what my
> system has. The system is also now at pulp 2.8.6. Is there any way for me
> to confirm that the database is at the correct version?

This should print the version # of the latest migration for each plugin:

mongo pulp_database --eval 'db.migration_trackers.find().forEach(printjson)'

··· On 08/30/2016 02:48 AM, Dylan Baars wrote:

Thanks :slight_smile:
Dylan

On Tuesday, August 30, 2016 at 12:57:36 AM UTC+12, Chris Duryee wrote:

On 08/29/2016 07:50 AM, Chris Duryee wrote:

On 08/29/2016 04:28 AM, Dylan Baars wrote:

Hi all

Anyone have any ideas? We can’t promote any red hat repos at the moment

Thanks!

The issue is likely:

pulp.server.managers.repo.unit_association:ERROR: (4160-00704)
PulpCodedException: Content import of

/var/lib/pulp/content/units/yum_repo_metadata_file/3f/07e460b4eaf0910dc60e6c183373c7c61fa7f95626c6cd0888f64ae62d2050/productid.gz

failed - must be an existing file.

I remember this error but not the solution, I reached out to the Pulp
team via #pulp to see if they know more.

It looks like you hit https://pulp.plan.io/issues/1944
https://www.google.com/url?q=https%3A%2F%2Fpulp.plan.io%2Fissues%2F1944&sa=D&sntz=1&usg=AFQjCNHi3k7hRBn1K75M1qzFhHgwZJFUDg,
which should be
fixed in Pulp 2.8.5 and newer.

Your pulp-manage-db output says “Migration to
pulp_rpm.plugins.migrations version 32 complete.” which would include
https://github.com/pulp/pulp_rpm/pull/900/files, which should have fixed
this issue.

Did you install a beta version of Pulp at any point on this system?

12

Hi Chris,

here's the output, is this what I should be expecting?

[root@wellkatello ~]# mongo pulp_database --eval
'db.migration_trackers.find().forEach(printjson)'
MongoDB shell version: 2.6.11
connecting to: pulp_database
{
"_id" : ObjectId("56662ddf244b0e226d6293fb"),
"version" : 23,
"name" : "pulp.server.db.migrations",
"_ns" : "migration_trackers"
}
{
"_id" : ObjectId("56662ddf244b0e226d6293fc"),
"version" : 32,
"name" : "pulp_rpm.plugins.migrations",
"_ns" : "migration_trackers"
}
{
"_id" : ObjectId("56662de0244b0e226d6293fd"),
"version" : 5,
"name" : "pulp_puppet.plugins.migrations",
"_ns" : "migration_trackers"
}
{
"_id" : ObjectId("5776d9b49444a66530307d22"),
"name" : "pulp_docker.plugins.migrations",
"version" : 2,
"_ns" : "migration_trackers"
}

13

Had exactly the same issue - Pulp 2.8.6 did not fix it. Finally have a
workaround.

My understanding is that while 2.8.6 fixes the bug, if there have been no
upstream changes in the repo since you updated Pulp to 2.8.6, the repo will
remain in a broken state. I think this is because metadata does not get
regenerated if there are no upstream changes.

So I forced a content change by deleting a file from the Katello repo.
Katello will not let you remove packages from Red Hat repos so I used
pulp-admin, something like:

pulp-admin -u admin -p <password> rpm repo remove rpm
–match="filename=<something.rpm>"
–repo-id=NIWA-Prod-Server-RHEL7_Server-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_7_Server_RPMs_x86_64_7Server

Removing any one .rpm file from the repo does the job.
Then I ran sync with:

pulp-admin -u admin -p <password> rpm repo sync run
–repo-id=NIWA-Prod-Server-RHEL7_Server-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_7_Server_RPMs_x86_64_7Server

And the sync completed successfully. I was now able to sync the same repo
from Katello or using hammer and it completes successfully.

Abir

··· On Wednesday, August 31, 2016 at 7:01:55 AM UTC+10, Dylan Baars wrote: > > Hi Chris, > > here's the output, is this what I should be expecting? > > [root@wellkatello ~]# mongo pulp_database --eval > 'db.migration_trackers.find().forEach(printjson)' > MongoDB shell version: 2.6.11 > connecting to: pulp_database > { > "_id" : ObjectId("56662ddf244b0e226d6293fb"), > "version" : 23, > "name" : "pulp.server.db.migrations", > "_ns" : "migration_trackers" > } > { > "_id" : ObjectId("56662ddf244b0e226d6293fc"), > "version" : 32, > "name" : "pulp_rpm.plugins.migrations", > "_ns" : "migration_trackers" > } > { > "_id" : ObjectId("56662de0244b0e226d6293fd"), > "version" : 5, > "name" : "pulp_puppet.plugins.migrations", > "_ns" : "migration_trackers" > } > { > "_id" : ObjectId("5776d9b49444a66530307d22"), > "name" : "pulp_docker.plugins.migrations", > "version" : 2, > "_ns" : "migration_trackers" > } > >