Metadata not found when syncing RHEL repositories

Hi all,

we have been syncing RHEL repos from cdn.redhat.com without issue for a
while, but now when I attempt to sync any I get

RPM1004: Error retrieving metadata: Not found

in the GUI. If I try a sync and monitor /var/log/messages I see

Sep 13 14:30:58 wellkatello.niwa.local pulp: kombu.transport.qpid:INFO:
Connected to qpid with SASL mechanism ANONYMOUS
Sep 13 14:31:00 wellkatello.niwa.local pulp: celery.worker.strategy:INFO:
Received task:
pulp.server.async.tasks._queue_reserved_task[f284f6e7-215a-493e-9bc6-bf3d1a433ec8]
Sep 13 14:31:00 wellkatello.niwa.local pulp: celery.worker.strategy:INFO:
Received task:
pulp.server.managers.repo.sync.sync[981a01c6-fd95-42d4-8d6f-e60470bf0fe9]
Sep 13 14:31:00 wellkatello.niwa.local pulp: celery.worker.strategy:INFO:
Received task:
pulp.server.async.tasks._release_resource[9dd3b103-26d7-4c8f-9ced-9dc785ca8f03]
Sep 13 14:31:00 wellkatello.niwa.local pulp: celery.worker.job:INFO: Task
pulp.server.async.tasks._queue_reserved_task[f284f6e7-215a-493e-9bc6-bf3d1a433ec8]
succeeded in 0.125369669s: None
Sep 13 14:31:01 wellkatello.niwa.local pulp:
pulp_rpm.plugins.importers.yum.sync:INFO: Downloading metadata from
https://cdn.redhat.com/content/fastrack/rhel/server/7/x86_64/optional/os/.
Sep 13 14:31:01 wellkatello.niwa.local pulp:
requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS
connection (1): cdn.redhat.com
Sep 13 14:31:03 wellkatello.niwa.local pulp:
requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS
connection (1): cdn.redhat.com
Sep 13 14:31:03 wellkatello.niwa.local pulp: pulp.server.async.tasks:INFO:
Task failed : [981a01c6-fd95-42d4-8d6f-e60470bf0fe9] : Error retrieving
metadata: Not found
Sep 13 14:31:04 wellkatello.niwa.local pulp: celery.worker.job:INFO: Task
pulp.server.managers.repo.sync.sync[981a01c6-fd95-42d4-8d6f-e60470bf0fe9]
raised expected: PulpCodedException()
Sep 13 14:31:04 wellkatello.niwa.local pulp: celery.worker.job:INFO: Task
pulp.server.async.tasks._release_resource[9dd3b103-26d7-4c8f-9ced-9dc785ca8f03]
succeeded in 0.0117141319997s: None

in /var/log/foreman/production.log I see

2016-09-13 14:37:12 [app] [I] Processing by
Katello::SyncManagementController#sync as JS
2016-09-13 14:37:12 [app] [I] Parameters: {"utf8"=>"✓",
"sync_status_url"=>"", "repoids"=>["69"], "commit"=>"Synchronize Now"}
2016-09-13 14:37:15 [app] [I] Completed 200 OK in 2797ms (Views: 0.8ms |
ActiveRecord: 1701.8ms)
2016-09-13 14:37:16 [app] [I] Started GET
"/katello/sync_management/sync_status?repoids%5B%5D=69&=1473732915912" for
192.168.222.132 at 2016-09-13 14:37:16 +1200
2016-09-13 14:37:16 [app] [I] Processing by
Katello::SyncManagementController#sync_status as JSON
2016-09-13 14:37:16 [app] [I] Parameters: {"repoids"=>["69"],
"
"=>"1473732915912"}
2016-09-13 14:37:16 [app] [I] Completed 200 OK in 94ms (Views: 0.7ms |
ActiveRecord: 12.9ms)
2016-09-13 14:37:17 [app] [I] Started GET
"/katello/sync_management/sync_status?repoids%5B%5D=69&=1473732915913" for
192.168.222.132 at 2016-09-13 14:37:17 +1200
2016-09-13 14:37:17 [app] [I] Processing by
Katello::SyncManagementController#sync_status as JSON
2016-09-13 14:37:17 [app] [I] Parameters: {"repoids"=>["69"],
"
"=>"1473732915913"}
2016-09-13 14:37:18 [app] [I] Completed 200 OK in 102ms (Views: 0.4ms |
ActiveRecord: 12.0ms)
2016-09-13 14:37:19 [app] [I] Started GET
"/katello/sync_management/sync_status?repoids%5B%5D=69&=1473732915914" for
192.168.222.132 at 2016-09-13 14:37:19 +1200
2016-09-13 14:37:19 [foreman-tasks/action] [E] RPM1004: Error retrieving
metadata: Not found (Katello::Errors::PulpError)
>
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/actions/pulp/abstract_async_task.rb:121:in
block in external_task=' > /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/actions/pulp/abstract_async_task.rb:119:ineach'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/actions/pulp/abstract_async_task.rb:119:in
external_task=' > /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/actions/pulp/repository/sync.rb:46:inexternal_task='
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action/polling.rb:98:in
poll_external_task_with_rescue' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action/polling.rb:21:inrun'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action/cancellable.rb:9:in
run' > /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/actions/pulp/abstract_async_task.rb:45:inrun'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:506:in
block (3 levels) in execute_run' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:incall'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in
pass' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:inpass'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:30:in
run' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:incall'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in
pass' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:inpass'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/actions/middleware/remote_action.rb:16:in
block in run' > /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/actions/middleware/remote_action.rb:40:inblock in as_remote_user'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/models/katello/concerns/user_extensions.rb:20:in
cp_config' > /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/actions/middleware/remote_action.rb:27:inas_cp_user'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/actions/middleware/remote_action.rb:39:in
as_remote_user' > /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/actions/middleware/remote_action.rb:16:inrun'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:in
call' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:inpass'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:in
pass' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action/progress.rb:30:inwith_progress_calculation'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action/progress.rb:16:in
run' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:incall'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in
pass' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:inpass'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/actions/middleware/keep_locale.rb:11:in
block in run' > /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/actions/middleware/keep_locale.rb:22:inwith_locale'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/actions/middleware/keep_locale.rb:11:in
run' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:incall'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in
pass' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:inpass'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:30:in
run' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:incall'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/world.rb:30:in
execute' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:505:inblock (2 levels) in execute_run'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:504:in
catch' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:504:inblock in execute_run'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:419:in
call' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:419:inblock in with_error_handling'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:419:in
catch' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:419:inwith_error_handling'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:499:in
execute_run' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:260:inexecute'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:9:in
block (2 levels) in execute' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/abstract.rb:155:incall'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/abstract.rb:155:in
with_meta_calculation' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:8:inblock in execute'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:22:in
open_action' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:7:inexecute'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/executors/parallel/worker.rb:15:in
block in on_message' > /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:74:inblock in assigns'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in
tap' > /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:inassigns'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:56:in
match_value' > /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:36:inblock in match?'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in
each' > /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:inmatch?'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:23:in
match' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/executors/parallel/worker.rb:12:inon_message'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/context.rb:46:in
on_envelope' > /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/executes_context.rb:7:inon_envelope'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in
pass' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/actor.rb:26:inon_envelope'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in
pass' > /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/awaits.rb:15:inon_envelope'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in
pass' > /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/sets_results.rb:14:inon_envelope'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in
pass' > /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:38:inprocess_envelope'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:31:in
process_envelopes?' > /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:20:inon_envelope'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in
pass' > /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/termination.rb:55:inon_envelope'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in
pass' > /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/removes_child.rb:10:inon_envelope'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in
pass' > /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/sets_results.rb:14:inon_envelope'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:161:in
process_envelope' > /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:95:inblock in on_envelope'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:118:in
block (2 levels) in schedule_execution' > /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:inblock in synchronize'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:in
synchronize' > /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:insynchronize'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:115:in
block in schedule_execution' > /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:18:incall'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:18:in
call' > /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:96:inwork'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:77:in
block in call_job' > /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:333:incall'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:333:in
run_task' > /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:322:inblock (3 levels) in create_worker'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:305:in
loop' > /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:305:inblock (2 levels) in create_worker'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:304:in
catch' > /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:304:inblock in create_worker'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in
call' > /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:inblock in create_with_logging_context'
2016-09-13 14:37:19 [app] [I] Processing by
Katello::SyncManagementController#sync_status as JSON
2016-09-13 14:37:19 [app] [I] Parameters: {"repoids"=>["69"],
"
"=>"1473732915914"}
2016-09-13 14:37:19 [app] [I] Completed 200 OK in 78ms (Views: 0.3ms |
ActiveRecord: 10.8ms)
2016-09-13 14:37:19 [app] [I] Started GET
"/katello/sync_management/sync_status?repoids%5B%5D=1697&repoids%5B%5D=1698&=1473730874920"
for 192.168.56.124 at 2016-09-13 14:37:19 +1200
2016-09-13 14:37:19 [app] [I] Processing by
Katello::SyncManagementController#sync_status as JSON
2016-09-13 14:37:19 [app] [I] Parameters: {"repoids"=>["1697", "1698"],
"
"=>"1473730874920"}
2016-09-13 14:37:20 [app] [I] Completed 200 OK in 213ms (Views: 0.6ms |
ActiveRecord: 60.7ms)
2016-09-13 14:37:21 [app] [I] Started GET
"/katello/sync_management/sync_status?repoids%5B%5D=69&=1473732915915" for
192.168.222.132 at 2016-09-13 14:37:21 +1200
2016-09-13 14:37:21 [app] [I] Processing by
Katello::SyncManagementController#sync_status as JSON
2016-09-13 14:37:21 [app] [I] Parameters: {"repoids"=>["69"],
"
"=>"1473732915915"}
2016-09-13 14:37:21 [app] [I] Completed 200 OK in 116ms (Views: 0.3ms |
ActiveRecord: 13.7ms)

I've seen various posts/issues about this being related to either being
behind a proxy (we're not) or re-importing of a manifest file (we haven't)
so not sure what to do. I have recently had a problem with content view
promotion/publishing of RHEL repositories, as
per Bug #16236: Cannot publish new content view version, pulp error - Katello - Foreman where a metadata file copy
results in an error (related pulp issue: https://pulp.plan.io/issues/2248)

Any help very much appreciated :slight_smile:

Thanks!
Dylan

Dylan,

Looks like you are running into a known issue of pulp saying "not found"
when it means "forbidden"

Can you try refreshing your manifest?

Thanks,

John Mitsch
Red Hat Engineering
(860)-967-7285
irc: jomitsch

··· On Mon, Sep 12, 2016 at 10:38 PM, Dylan Baars wrote:

Hi all,

we have been syncing RHEL repos from cdn.redhat.com without issue for a
while, but now when I attempt to sync any I get

RPM1004: Error retrieving metadata: Not found

in the GUI. If I try a sync and monitor /var/log/messages I see

Sep 13 14:30:58 wellkatello.niwa.local pulp: kombu.transport.qpid:INFO:
Connected to qpid with SASL mechanism ANONYMOUS
Sep 13 14:31:00 wellkatello.niwa.local pulp: celery.worker.strategy:INFO:
Received task: pulp.server.async.tasks._queue_reserved_task[f284f6e7-
215a-493e-9bc6-bf3d1a433ec8]
Sep 13 14:31:00 wellkatello.niwa.local pulp: celery.worker.strategy:INFO:
Received task: pulp.server.managers.repo.sync.sync[981a01c6-fd95-42d4-
8d6f-e60470bf0fe9]
Sep 13 14:31:00 wellkatello.niwa.local pulp: celery.worker.strategy:INFO:
Received task: pulp.server.async.tasks._release_resource[9dd3b103-
26d7-4c8f-9ced-9dc785ca8f03]
Sep 13 14:31:00 wellkatello.niwa.local pulp: celery.worker.job:INFO: Task
pulp.server.async.tasks._queue_reserved_task[f284f6e7-215a-493e-9bc6-bf3d1a433ec8]
succeeded in 0.125369669s: None
Sep 13 14:31:01 wellkatello.niwa.local pulp: pulp_rpm.plugins.importers.yum.sync:INFO:
Downloading metadata from https://cdn.redhat.com/
content/fastrack/rhel/server/7/x86_64/optional/os/.
Sep 13 14:31:01 wellkatello.niwa.local pulp: requests.packages.urllib3.connectionpool:INFO:
Starting new HTTPS connection (1): cdn.redhat.com
Sep 13 14:31:03 wellkatello.niwa.local pulp: requests.packages.urllib3.connectionpool:INFO:
Starting new HTTPS connection (1): cdn.redhat.com
Sep 13 14:31:03 wellkatello.niwa.local pulp: pulp.server.async.tasks:INFO:
Task failed : [981a01c6-fd95-42d4-8d6f-e60470bf0fe9] : Error retrieving
metadata: Not found
Sep 13 14:31:04 wellkatello.niwa.local pulp: celery.worker.job:INFO: Task
pulp.server.managers.repo.sync.sync[981a01c6-fd95-42d4-8d6f-e60470bf0fe9]
raised expected: PulpCodedException()
Sep 13 14:31:04 wellkatello.niwa.local pulp: celery.worker.job:INFO: Task
pulp.server.async.tasks._release_resource[9dd3b103-26d7-4c8f-9ced-9dc785ca8f03]
succeeded in 0.0117141319997s: None

in /var/log/foreman/production.log I see

2016-09-13 14:37:12 [app] [I] Processing by Katello::SyncManagementController#sync
as JS
2016-09-13 14:37:12 [app] [I] Parameters: {“utf8”=>“✓”,
“sync_status_url”=>"", “repoids”=>[“69”], “commit”=>“Synchronize Now”}
2016-09-13 14:37:15 [app] [I] Completed 200 OK in 2797ms (Views: 0.8ms |
ActiveRecord: 1701.8ms)
2016-09-13 14:37:16 [app] [I] Started GET “/katello/sync_management/
sync_status?repoids%5B%5D=69&=1473732915912" for 192.168.222.132 at
2016-09-13 14:37:16 +1200
2016-09-13 14:37:16 [app] [I] Processing by Katello::
SyncManagementController#sync_status as JSON
2016-09-13 14:37:16 [app] [I] Parameters: {“repoids”=>[“69”],
"
”=>“1473732915912”}
2016-09-13 14:37:16 [app] [I] Completed 200 OK in 94ms (Views: 0.7ms |
ActiveRecord: 12.9ms)
2016-09-13 14:37:17 [app] [I] Started GET “/katello/sync_management/
sync_status?repoids%5B%5D=69&=1473732915913" for 192.168.222.132 at
2016-09-13 14:37:17 +1200
2016-09-13 14:37:17 [app] [I] Processing by Katello::
SyncManagementController#sync_status as JSON
2016-09-13 14:37:17 [app] [I] Parameters: {“repoids”=>[“69”],
"
”=>“1473732915913”}
2016-09-13 14:37:18 [app] [I] Completed 200 OK in 102ms (Views: 0.4ms |
ActiveRecord: 12.0ms)
2016-09-13 14:37:19 [app] [I] Started GET “/katello/sync_management/
sync_status?repoids%5B%5D=69&_=1473732915914” for 192.168.222.132 at
2016-09-13 14:37:19 +1200
2016-09-13 14:37:19 [foreman-tasks/action] [E] RPM1004: Error retrieving
metadata: Not found (Katello::Errors::PulpError)

/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/
app/lib/actions/pulp/abstract_async_task.rb:121:in block in external_task=' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/ app/lib/actions/pulp/abstract_async_task.rb:119:ineach’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/
app/lib/actions/pulp/abstract_async_task.rb:119:in external_task=' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/ app/lib/actions/pulp/repository/sync.rb:46:inexternal_task=’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.
11/lib/dynflow/action/polling.rb:98:in poll_external_task_with_rescue' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. 11/lib/dynflow/action/polling.rb:21:inrun’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.
11/lib/dynflow/action/cancellable.rb:9:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/ app/lib/actions/pulp/abstract_async_task.rb:45:inrun’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.
11/lib/dynflow/action.rb:506:in block (3 levels) in execute_run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. 11/lib/dynflow/middleware/stack.rb:26:incall’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.
11/lib/dynflow/middleware/stack.rb:26:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. 11/lib/dynflow/middleware.rb:17:inpass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.
11/lib/dynflow/middleware.rb:30:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. 11/lib/dynflow/middleware/stack.rb:22:incall’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.
11/lib/dynflow/middleware/stack.rb:26:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. 11/lib/dynflow/middleware.rb:17:inpass’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/
app/lib/actions/middleware/remote_action.rb:16:in block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/ app/lib/actions/middleware/remote_action.rb:40:inblock in
as_remote_user’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/
app/models/katello/concerns/user_extensions.rb:20:in cp_config' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/ app/lib/actions/middleware/remote_action.rb:27:inas_cp_user’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/
app/lib/actions/middleware/remote_action.rb:39:in as_remote_user' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/ app/lib/actions/middleware/remote_action.rb:16:inrun’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.
11/lib/dynflow/middleware/stack.rb:22:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. 11/lib/dynflow/middleware/stack.rb:26:inpass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.
11/lib/dynflow/middleware.rb:17:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. 11/lib/dynflow/action/progress.rb:30:inwith_progress_calculation’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.
11/lib/dynflow/action/progress.rb:16:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. 11/lib/dynflow/middleware/stack.rb:22:incall’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.
11/lib/dynflow/middleware/stack.rb:26:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. 11/lib/dynflow/middleware.rb:17:inpass’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/
app/lib/actions/middleware/keep_locale.rb:11:in block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/ app/lib/actions/middleware/keep_locale.rb:22:inwith_locale’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/
app/lib/actions/middleware/keep_locale.rb:11:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. 11/lib/dynflow/middleware/stack.rb:22:incall’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.
11/lib/dynflow/middleware/stack.rb:26:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. 11/lib/dynflow/middleware.rb:17:inpass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.
11/lib/dynflow/middleware.rb:30:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. 11/lib/dynflow/middleware/stack.rb:22:incall’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.
11/lib/dynflow/middleware/world.rb:30:in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. 11/lib/dynflow/action.rb:505:inblock (2 levels) in execute_run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.
11/lib/dynflow/action.rb:504:in catch' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. 11/lib/dynflow/action.rb:504:inblock in execute_run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.
11/lib/dynflow/action.rb:419:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. 11/lib/dynflow/action.rb:419:inblock in with_error_handling’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.
11/lib/dynflow/action.rb:419:in catch' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. 11/lib/dynflow/action.rb:419:inwith_error_handling’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.
11/lib/dynflow/action.rb:499:in execute_run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. 11/lib/dynflow/action.rb:260:inexecute’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.
11/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:9:in block (2 levels) in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. 11/lib/dynflow/execution_plan/steps/abstract.rb:155:incall’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.
11/lib/dynflow/execution_plan/steps/abstract.rb:155:in
with_meta_calculation' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. 11/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:8:inblock in
execute’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.
11/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:22:in
open_action' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. 11/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:7:inexecute’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.
11/lib/dynflow/executors/parallel/worker.rb:15:in block in on_message' /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7. 3/lib/algebrick/matchers/abstract.rb:74:inblock in assigns’
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.
3/lib/algebrick/matchers/abstract.rb:73:in tap' /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7. 3/lib/algebrick/matchers/abstract.rb:73:inassigns’
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.
3/lib/algebrick/matching.rb:56:in match_value' /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7. 3/lib/algebrick/matching.rb:36:inblock in match?’
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.
3/lib/algebrick/matching.rb:35:in each' /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7. 3/lib/algebrick/matching.rb:35:inmatch?’
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.
3/lib/algebrick/matching.rb:23:in match' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. 11/lib/dynflow/executors/parallel/worker.rb:12:inon_message’
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-
ruby-edge-0.2.0/lib/concurrent/actor/context.rb:46:in on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- ruby-edge-0.2.0/lib/concurrent/actor/behaviour/executes_context.rb:7:inon_envelope’
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-
ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/actor.rb:26:inon_envelope’
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-
ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- ruby-edge-0.2.0/lib/concurrent/actor/behaviour/awaits.rb:15:inon_envelope’
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-
ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- ruby-edge-0.2.0/lib/concurrent/actor/behaviour/sets_results.rb:14:inon_envelope’
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-
ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:38:inprocess_envelope’
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-
ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:31:in
process_envelopes?' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:20:inon_envelope’
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-
ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- ruby-edge-0.2.0/lib/concurrent/actor/behaviour/termination.rb:55:inon_envelope’
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-
ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- ruby-edge-0.2.0/lib/concurrent/actor/behaviour/removes_child.rb:10:inon_envelope’
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-
ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- ruby-edge-0.2.0/lib/concurrent/actor/behaviour/sets_results.rb:14:inon_envelope’
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-
ruby-edge-0.2.0/lib/concurrent/actor/core.rb:161:in process_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- ruby-edge-0.2.0/lib/concurrent/actor/core.rb:95:inblock in on_envelope’
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-
ruby-edge-0.2.0/lib/concurrent/actor/core.rb:118:in block (2 levels) in schedule_execution' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:inblock in synchronize’
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-
ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:in
synchronize' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:insynchronize’
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-
ruby-edge-0.2.0/lib/concurrent/actor/core.rb:115:in block in schedule_execution' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:18:incall’
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-
ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:18:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:96:inwork’
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-
ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:77:in block in call_job' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:333:incall’
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-
ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:333:in
run_task' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:322:inblock (3 levels) in create_worker’
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-
ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:305:in
loop' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:305:inblock (2 levels) in create_worker’
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-
ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:304:in
catch' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:304:inblock in create_worker’
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/
lib/logging/diagnostic_context.rb:323:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/ lib/logging/diagnostic_context.rb:323:inblock in
create_with_logging_context’
2016-09-13 14:37:19 [app] [I] Processing by Katello::
SyncManagementController#sync_status as JSON
2016-09-13 14:37:19 [app] [I] Parameters: {“repoids”=>[“69”],
""=>“1473732915914”}
2016-09-13 14:37:19 [app] [I] Completed 200 OK in 78ms (Views: 0.3ms |
ActiveRecord: 10.8ms)
2016-09-13 14:37:19 [app] [I] Started GET "/katello/sync_management/
sync_status?repoids%5B%5D=1697&repoids%5B%5D=1698&
=1473730874920" for
192.168.56.124 at 2016-09-13 14:37:19 +1200
2016-09-13 14:37:19 [app] [I] Processing by Katello::
SyncManagementController#sync_status as JSON
2016-09-13 14:37:19 [app] [I] Para

> Dylan,
>
> Looks like you are running into a known issue of pulp saying "not found"
> when it means "forbidden"
>
> Can you try refreshing your manifest?

issue number?

– bk

··· On 09/13/2016 06:02 PM, John Mitsch wrote:

Thanks,

John Mitsch
Red Hat Engineering
(860)-967-7285
irc: jomitsch

On Mon, Sep 12, 2016 at 10:38 PM, Dylan Baars <baarsd@gmail.com > mailto:baarsd@gmail.com> wrote:

Hi all,

we have been syncing RHEL repos from cdn.redhat.com
<http://cdn.redhat.com> without issue for a while, but now when I
attempt to sync any I get

RPM1004: Error retrieving metadata: Not found

in the GUI. If I try a sync and monitor /var/log/messages I see

Sep 13 14:30:58 wellkatello.niwa.local pulp:
kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism
ANONYMOUS
Sep 13 14:31:00 wellkatello.niwa.local pulp:
celery.worker.strategy:INFO: Received task:
pulp.server.async.tasks._queue_reserved_task[f284f6e7-215a-493e-9bc6-bf3d1a433ec8]
Sep 13 14:31:00 wellkatello.niwa.local pulp:
celery.worker.strategy:INFO: Received task:
pulp.server.managers.repo.sync.sync[981a01c6-fd95-42d4-8d6f-e60470bf0fe9]
Sep 13 14:31:00 wellkatello.niwa.local pulp:
celery.worker.strategy:INFO: Received task:
pulp.server.async.tasks._release_resource[9dd3b103-26d7-4c8f-9ced-9dc785ca8f03]
Sep 13 14:31:00 wellkatello.niwa.local pulp: celery.worker.job:INFO:
Task
pulp.server.async.tasks._queue_reserved_task[f284f6e7-215a-493e-9bc6-bf3d1a433ec8]
succeeded in 0.125369669s: None
Sep 13 14:31:01 wellkatello.niwa.local pulp:
pulp_rpm.plugins.importers.yum.sync:INFO: Downloading metadata from
https://cdn.redhat.com/content/fastrack/rhel/server/7/x86_64/optional/os/
<https://cdn.redhat.com/content/fastrack/rhel/server/7/x86_64/optional/os/>.
Sep 13 14:31:01 wellkatello.niwa.local pulp:
requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS
connection (1): cdn.redhat.com <http://cdn.redhat.com>
Sep 13 14:31:03 wellkatello.niwa.local pulp:
requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS
connection (1): cdn.redhat.com <http://cdn.redhat.com>
Sep 13 14:31:03 wellkatello.niwa.local pulp:
pulp.server.async.tasks:INFO: Task failed :
[981a01c6-fd95-42d4-8d6f-e60470bf0fe9] : Error retrieving metadata:
Not found
Sep 13 14:31:04 wellkatello.niwa.local pulp: celery.worker.job:INFO:
Task
pulp.server.managers.repo.sync.sync[981a01c6-fd95-42d4-8d6f-e60470bf0fe9]
raised expected: PulpCodedException()
Sep 13 14:31:04 wellkatello.niwa.local pulp: celery.worker.job:INFO:
Task
pulp.server.async.tasks._release_resource[9dd3b103-26d7-4c8f-9ced-9dc785ca8f03]
succeeded in 0.0117141319997s: None

in /var/log/foreman/production.log I see

2016-09-13 14:37:12 [app] [I] Processing by
Katello::SyncManagementController#sync as JS
2016-09-13 14:37:12 [app] [I]   Parameters: {"utf8"=>"✓",
"sync_status_url"=>"", "repoids"=>["69"], "commit"=>"Synchronize Now"}
2016-09-13 14:37:15 [app] [I] Completed 200 OK in 2797ms (Views:
0.8ms | ActiveRecord: 1701.8ms)
2016-09-13 14:37:16 [app] [I] Started GET
"/katello/sync_management/sync_status?repoids%5B%5D=69&_=1473732915912"
for 192.168.222.132 at 2016-09-13 14:37:16 +1200
2016-09-13 14:37:16 [app] [I] Processing by
Katello::SyncManagementController#sync_status as JSON
2016-09-13 14:37:16 [app] [I]   Parameters: {"repoids"=>["69"],
"_"=>"1473732915912"}
2016-09-13 14:37:16 [app] [I] Completed 200 OK in 94ms (Views: 0.7ms
> ActiveRecord: 12.9ms)
2016-09-13 14:37:17 [app] [I] Started GET
"/katello/sync_management/sync_status?repoids%5B%5D=69&_=1473732915913"
for 192.168.222.132 at 2016-09-13 14:37:17 +1200
2016-09-13 14:37:17 [app] [I] Processing by
Katello::SyncManagementController#sync_status as JSON
2016-09-13 14:37:17 [app] [I]   Parameters: {"repoids"=>["69"],
"_"=>"1473732915913"}
2016-09-13 14:37:18 [app] [I] Completed 200 OK in 102ms (Views:
0.4ms | ActiveRecord: 12.0ms)
2016-09-13 14:37:19 [app] [I] Started GET
"/katello/sync_management/sync_status?repoids%5B%5D=69&_=1473732915914"
for 192.168.222.132 at 2016-09-13 14:37:19 +1200
2016-09-13 14:37:19 [foreman-tasks/action] [E] RPM1004: Error
retrieving metadata: Not found (Katello::Errors::PulpError)
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/actions/pulp/abstract_async_task.rb:121:in
`block in external_task='
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/actions/pulp/abstract_async_task.rb:119:in
`each'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/actions/pulp/abstract_async_task.rb:119:in
`external_task='
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/actions/pulp/repository/sync.rb:46:in
`external_task='
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action/polling.rb:98:in
`poll_external_task_with_rescue'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action/polling.rb:21:in
`run'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action/cancellable.rb:9:in
`run'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/actions/pulp/abstract_async_task.rb:45:in
`run'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:506:in
`block (3 levels) in execute_run'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in
`call'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in
`pass'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:in
`pass'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:30:in
`run'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:in
`call'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in
`pass'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:in
`pass'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/actions/middleware/remote_action.rb:16:in
`block in run'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/actions/middleware/remote_action.rb:40:in
`block in as_remote_user'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/models/katello/concerns/user_extensions.rb:20:in
`cp_config'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/actions/middleware/remote_action.rb:27:in
`as_cp_user'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/actions/middleware/remote_action.rb:39:in
`as_remote_user'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/actions/middleware/remote_action.rb:16:in
`run'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:in
`call'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in
`pass'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:in
`pass'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action/progress.rb:30:in
`with_progress_calculation'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action/progress.rb:16:in
`run'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:in
`call'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in
`pass'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:in
`pass'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/actions/middleware/keep_locale.rb:11:in
`block in run'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/actions/middleware/keep_locale.rb:22:in
`with_locale'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/actions/middleware/keep_locale.rb:11:in
`run'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:in
`call'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in
`pass'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:in
`pass'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:30:in
`run'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:in
`call'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/world.rb:30:in
`execute'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:505:in
`block (2 levels) in execute_run'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:504:in
`catch'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:504:in
`block in execute_run'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:419:in
`call'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:419:in
`block in with_error_handling'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:419:in
`catch'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:419:in
`with_error_handling'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:499:in
`execute_run'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:260:in
`execute'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:9:in
`block (2 levels) in execute'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/abstract.rb:155:in
`call'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/abstract.rb:155:in
`with_meta_calculation'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:8:in
`block in execute'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:22:in
`open_action'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:7:in
`execute'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/executors/parallel/worker.rb:15:in
`block in on_message'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:74:in
`block in assigns'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in
`tap'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in
`assigns'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:56:in
`match_value'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:36:in
`block in match?'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in
`each'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in
`match?'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:23:in
`match'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/executors/parallel/worker.rb:12:in
`on_message'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/context.rb:46:in
`on_envelope'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/executes_context.rb:7:in
`on_envelope'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in
`pass'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/actor.rb:26:in
`on_envelope'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in
`pass'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/awaits.rb:15:in
`on_envelope'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in
`pass'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/sets_results.rb:14:in
`on_envelope'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in
`pass'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:38:in
`process_envelope'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:31:in
`process_envelopes?'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:20:in
`on_envelope'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in
`pass'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/termination.rb:55:in
`on_envelope'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in
`pass'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/removes_child.rb:10:in
`on_envelope'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in
`pass'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/sets_results.rb:14:in
`on_envelope'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:161:in
`process_envelope'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:95:in
`block in on_envelope'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:118:in
`block (2 levels) in schedule_execution'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:in
`block in synchronize'
  >
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:in

Hi John,

that looks to have sorted it - in Katello, Content > Redhat Subscriptions >
Manage Manifest. Clicked "Refresh manifest"

Then Content > Sync Status
Selected one of the RH repos > Synchronise now, and it completed
successfully. Doing the others and they are all grabbing new files
successfully - thanks!

Dylan

··· On Wednesday, September 14, 2016 at 10:02:05 AM UTC+12, John Mitsch wrote: > > Dylan, > > Looks like you are running into a known issue of pulp saying "not found" > when it means "forbidden" > > Can you try refreshing your manifest? > > Thanks, > > John Mitsch > Red Hat Engineering > (860)-967-7285 > irc: jomitsch > >

Issue is fixed upstream in pulp already, here is the BZ
https://bugzilla.redhat.com/show_bug.cgi?id=1373662

John Mitsch
Red Hat Engineering
(860)-967-7285
irc: jomitsch

··· On Tue, Sep 13, 2016 at 6:16 PM, Bryan Kearney wrote:

On 09/13/2016 06:02 PM, John Mitsch wrote:

Dylan,

Looks like you are running into a known issue of pulp saying "not found"
when it means “forbidden”

Can you try refreshing your manifest?

issue number?

– bk

Thanks,

John Mitsch
Red Hat Engineering
(860)-967-7285
irc: jomitsch

On Mon, Sep 12, 2016 at 10:38 PM, Dylan Baars <baarsd@gmail.com >> mailto:baarsd@gmail.com> wrote:

Hi all,

we have been syncing RHEL repos from cdn.redhat.com
<http://cdn.redhat.com> without issue for a while, but now when I
attempt to sync any I get

RPM1004: Error retrieving metadata: Not found

in the GUI. If I try a sync and monitor /var/log/messages I see

Sep 13 14:30:58 wellkatello.niwa.local pulp:
kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism
ANONYMOUS
Sep 13 14:31:00 wellkatello.niwa.local pulp:
celery.worker.strategy:INFO: Received task:
pulp.server.async.tasks._queue_reserved_task[f284f6e7-215a-

493e-9bc6-bf3d1a433ec8]
Sep 13 14:31:00 wellkatello.niwa.local pulp:
celery.worker.strategy:INFO: Received task:
pulp.server.managers.repo.sync.sync[981a01c6-fd95-42d4-8d6f-
e60470bf0fe9]
Sep 13 14:31:00 wellkatello.niwa.local pulp:
celery.worker.strategy:INFO: Received task:
pulp.server.async.tasks.release_resource[9dd3b103-26d7-
4c8f-9ced-9dc785ca8f03]
Sep 13 14:31:00 wellkatello.niwa.local pulp: celery.worker.job:INFO:
Task
pulp.server.async.tasks.queue_reserved_task[f284f6e7-215a-
493e-9bc6-bf3d1a433ec8]
succeeded in 0.125369669s: None
Sep 13 14:31:01 wellkatello.niwa.local pulp:
pulp_rpm.plugins.importers.yum.sync:INFO: Downloading metadata from
https://cdn.redhat.com/content/fastrack/rhel/server/7/x86

64/optional/os/
<https://cdn.redhat.com/content/fastrack/rhel/server/7/x86

64/optional/os/>.
Sep 13 14:31:01 wellkatello.niwa.local pulp:
requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS
connection (1): cdn.redhat.com http://cdn.redhat.com
Sep 13 14:31:03 wellkatello.niwa.local pulp:
requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS
connection (1): cdn.redhat.com http://cdn.redhat.com

Sep 13 14:31:03 wellkatello.niwa.local pulp:
pulp.server.async.tasks:INFO: Task failed :
[981a01c6-fd95-42d4-8d6f-e60470bf0fe9] : Error retrieving metadata:
Not found
Sep 13 14:31:04 wellkatello.niwa.local pulp: celery.worker.job:INFO:
Task
pulp.server.managers.repo.sync.sync[981a01c6-fd95-42d4-8d6f-

e60470bf0fe9]
raised expected: PulpCodedException()
Sep 13 14:31:04 wellkatello.niwa.local pulp: celery.worker.job:INFO:
Task
pulp.server.async.tasks._release_resource[9dd3b103-26d7-
4c8f-9ced-9dc785ca8f03]
succeeded in 0.0117141319997s: None

in /var/log/foreman/production.log I see

2016-09-13 14:37:12 [app] [I] Processing by
Katello::SyncManagementController#sync as JS
2016-09-13 14:37:12 [app] [I]   Parameters: {"utf8"=>"✓",
"sync_status_url"=>"", "repoids"=>["69"], "commit"=>"Synchronize Now"}
2016-09-13 14:37:15 [app] [I] Completed 200 OK in 2797ms (Views:
0.8ms | ActiveRecord: 1701.8ms)
2016-09-13 14:37:16 [app] [I] Started GET
"/katello/sync_management/sync_status?repoids%5B%5D=69&_=

1473732915912"
for 192.168.222.132 at 2016-09-13 14:37:16 +1200
2016-09-13 14:37:16 [app] [I] Processing by
Katello::SyncManagementController#sync_status as JSON
2016-09-13 14:37:16 [app] [I] Parameters: {“repoids”=>[“69”],
""=>“1473732915912”}
2016-09-13 14:37:16 [app] [I] Completed 200 OK in 94ms (Views: 0.7ms
> ActiveRecord: 12.9ms)
2016-09-13 14:37:17 [app] [I] Started GET
"/katello/sync_management/sync_status?repoids%5B%5D=69&
=
1473732915913"
for 192.168.222.132 at 2016-09-13 14:37:17 +1200
2016-09-13 14:37:17 [app] [I] Processing by
Katello::SyncManagementController#sync_status as JSON
2016-09-13 14:37:17 [app] [I] Parameters: {“repoids”=>[“69”],
""=>“1473732915913”}
2016-09-13 14:37:18 [app] [I] Completed 200 OK in 102ms (Views:
0.4ms | ActiveRecord: 12.0ms)
2016-09-13 14:37:19 [app] [I] Started GET
"/katello/sync_management/sync_status?repoids%5B%5D=69&
=
1473732915914"
for 192.168.222.132 at 2016-09-13 14:37:19 +1200
2016-09-13 14:37:19 [foreman-tasks/action] [E] RPM1004: Error
retrieving metadata: Not found (Katello::Errors::PulpError)
>
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/a
pp/lib/actions/pulp/abstract_async_task.rb:121:in
block in external_task=' > /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/a pp/lib/actions/pulp/abstract_async_task.rb:119:ineach’
>
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/a
pp/lib/actions/pulp/abstract_async_task.rb:119:in
external_task=' > /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/a pp/lib/actions/pulp/repository/sync.rb:46:inexternal_task=’
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/
lib/dynflow/action/polling.rb:98:in
poll_external_task_with_rescue' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/ lib/dynflow/action/polling.rb:21:inrun’
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/
lib/dynflow/action/cancellable.rb:9:in
run' > /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/a pp/lib/actions/pulp/abstract_async_task.rb:45:inrun’
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/
lib/dynflow/action.rb:506:in
block (3 levels) in execute_run' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/ lib/dynflow/middleware/stack.rb:26:incall’
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/
lib/dynflow/middleware/stack.rb:26:in
pass' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/ lib/dynflow/middleware.rb:17:inpass’
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/
lib/dynflow/middleware.rb:30:in
run' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/ lib/dynflow/middleware/stack.rb:22:incall’
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/
lib/dynflow/middleware/stack.rb:26:in
pass' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/ lib/dynflow/middleware.rb:17:inpass’
>
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/a
pp/lib/actions/middleware/remote_action.rb:16:in
block in run' > /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/a pp/lib/actions/middleware/remote_action.rb:40:inblock in as_remote_user’
>
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/a
pp/models/katello/concerns/user_extensions.rb:20:in
cp_config' > /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/a pp/lib/actions/middleware/remote_action.rb:27:inas_cp_user’
>
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/a
pp/lib/actions/middleware/remote_action.rb:39:in
as_remote_user' > /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/a pp/lib/actions/middleware/remote_action.rb:16:inrun’
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/
lib/dynflow/middleware/stack.rb:22:in
call' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/ lib/dynflow/middleware/stack.rb:26:inpass’
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/
lib/dynflow/middleware.rb:17:in
pass' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/ lib/dynflow/action/progress.rb:30:inwith_progress_calculation’
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/
lib/dynflow/action/progress.rb:16:in
run' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/ lib/dynflow/middleware/stack.rb:22:incall’
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/
lib/dynflow/middleware/stack.rb:26:in
pass' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/ lib/dynflow/middleware.rb:17:inpass’
>
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/a
pp/lib/actions/middleware/keep_locale.rb:11:in
block in run' > /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/a pp/lib/actions/middleware/keep_locale.rb:22:inwith_locale’
>
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/a
pp/lib/actions/middleware/keep_locale.rb:11:in
run' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/ lib/dynflow/middleware/stack.rb:22:incall’
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/
lib/dynflow/middleware/stack.rb:26:in
pass' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/ lib/dynflow/middleware.rb:17:inpass’
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/
lib/dynflow/middleware.rb:30:in
run' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/ lib/dynflow/middleware/stack.rb:22:incall’
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/
lib/dynflow/middleware/world.rb:30:in
execute' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/ lib/dynflow/action.rb:505:inblock (2 levels) in execute_run’
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/
lib/dynflow/action.rb:504:in
catch' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/ lib/dynflow/action.rb:504:inblock in execute_run’
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/
lib/dynflow/action.rb:419:in
call' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/ lib/dynflow/action.rb:419:inblock in with_error_handling’
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/
lib/dynflow/action.rb:419:in
catch' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/ lib/dynflow/action.rb:419:inwith_error_handling’
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/
lib/dynflow/action.rb:499:in
execute_run' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/ lib/dynflow/action.rb:260:inexecute’
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/
lib/dynflow/execution_plan/steps/abstract_flow_step.rb:9:in
block (2 levels) in execute' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/ lib/dynflow/execution_plan/steps/abstract.rb:155:incall’
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/
lib/dynflow/execution_plan/steps/abstract.rb:155:in
with_meta_calculation' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/ lib/dynflow/execution_plan/steps/abstract_flow_step.rb:8:inblock in execute’
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/
lib/dynflow/execution_plan/steps/abstract_flow_step.rb:22:in
open_action' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/ lib/dynflow/execution_plan/steps/abstract_flow_step.rb:7:inexecute’
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/
lib/dynflow/executors/parallel/worker.rb:15:in
block in on_message' > /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3 /lib/algebrick/matchers/abstract.rb:74:inblock in assigns’
>
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3
/lib/algebrick/matchers/abstract.rb:73:in
tap' > /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3 /lib/algebrick/matchers/abstract.rb:73:inassigns’
>
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3
/lib/algebrick/matching.rb:56:in
match_value' > /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3 /lib/algebrick/matching.rb:36:inblock in match?’
>
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3
/lib/algebrick/matching.rb:35:in
each' > /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3 /lib/algebrick/matching.rb:35:inmatch?’
>
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3
/lib/algebrick/matching.rb:23:in
match' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/ lib/dynflow/executors/parallel/worker.rb:12:inon_message’
>
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby
-edge-0.2.0/lib/concurrent/actor/context.rb:46:in
on_envelope' > /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby -edge-0.2.0/lib/concurrent/actor/behaviour/executes_context.rb:7:inon_envelope’
>
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby
-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in
pass' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/ lib/dynflow/actor.rb:26:inon_envelope’
>
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby
-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in
pass' > /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby -edge-0.2.0/lib/concurrent/actor/behaviour/awaits.rb:15:inon_envelope’
>
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby
-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in
pass' > /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby -edge-0.2.0/lib/concurrent/actor/behaviour/sets_results.rb:14:inon_envelope’
>
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby
-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in
pass' > /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby -edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:38:inprocess_envelope’
>
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby
-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:31:in
process_envelopes?' > /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby -edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:20:inon_envelope’
>
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby
-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in
pass' > /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby -edge-0.2.0/lib/concurrent/actor/behaviour/termination.rb:55:inon_envelope’
>
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby
-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in
pass' > /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby -edge-0.2.0/lib/concurrent/actor/behaviour/removes_child.rb:10:inon_envelope’
>
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby
-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in
pass' > /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby -edge-0.2.0/lib/concurrent/actor/behaviour/sets_results.rb:14:inon_envelope’
>
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby
-edge-0.2.0/lib/concurrent/actor/core.rb:161:in
process_envelope' > /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby -edge-0.2.0/lib/concurrent/actor/core.rb:95:inblock in on_envelope’
>
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby
-edge-0.2.0/lib/concurrent/actor/core.rb:118:in
block (2 levels) in schedule_execution' > /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby -1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:inblock in synchronize’

Dylan,

Good to hear! Glad it was sorted out.

John

John Mitsch
Red Hat Engineering
(860)-967-7285
irc: jomitsch

··· On Wed, Sep 14, 2016 at 1:00 AM, Dylan Baars wrote:

Hi John,

that looks to have sorted it - in Katello, Content > Redhat Subscriptions

Manage Manifest. Clicked “Refresh manifest”

Then Content > Sync Status
Selected one of the RH repos > Synchronise now, and it completed
successfully. Doing the others and they are all grabbing new files
successfully - thanks!

Dylan

On Wednesday, September 14, 2016 at 10:02:05 AM UTC+12, John Mitsch wrote:

Dylan,

Looks like you are running into a known issue of pulp saying "not found"
when it means “forbidden”

Can you try refreshing your manifest?

Thanks,

John Mitsch
Red Hat Engineering
(860)-967-7285
irc: jomitsch


You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.