Katello 3.16 RC4 repository synchronisation issues

Support
1

Hello,

I tried a new Foreman/Katello 3.16 RC4 installation on CentOS 7 and it seems like repo synchronization doesn’t work anymore but I have some trouble finding why.
I did an almost similar installation in 3.16 RC3 last week and repo synchronization did work perfectly fine (and still works in my 3.16 RC3 VM)

The installation is quite straightforward once the usual repos are configured:

yum install foreman-release-scl
yum install katello
foreman-installer --scenario katello --foreman-initial-organization “XXX” --foreman-initial-location “XXX” --enable-foreman-plugin-remote-execution --enable-foreman-proxy-plugin-remote-execution-ssh --enable-foreman-plugin-openscap --enable-foreman-proxy-plugin-openscap --enable-foreman-plugin-ansible --enable-foreman-proxy-plugin-ansible

The download policy for repositories is of course set by default to “immediate”.

Now these are the logs after the initial repository synchronization, and the associated filesystem view. All RPMs were downloaded and are stuck in /var/lib/pulp/docroot:

2020-07-14T10:53:04 [I|app|f46956dc] Completed 200 OK in 79ms (Views: 56.8ms | ActiveRecord: 4.9ms | Allocations: 52127)
2020-07-14T10:53:06 [I|aud|] Katello::Repository (1) update event on distribution_version , 8
2020-07-14T10:53:06 [I|aud|] Katello::Repository (1) update event on distribution_arch , x86_64
2020-07-14T10:53:06 [I|aud|] Katello::Repository (1) update event on distribution_family , CentOS Linux
2020-07-14T10:53:06 [I|aud|] Katello::Repository (1) update event on distribution_uuid , /pulp/api/v3/content/rpm/distribution_trees/5349349e-0b04-4eef-808c-f4b4e09d7907/
2020-07-14T10:53:06 [I|aud|] Katello::Repository (1) update event on distribution_bootable , true
2020-07-14T10:53:06 [I|aud|] Katello::Repository (1) update event on distribution_variant , AppStream
2020-07-14T10:53:06 [I|bac|] Task {label: Actions::Katello::Repository::CapsuleSync, id: efc84080-31c1-4ba4-837d-16c546a0f1e2, execution_plan_id: 8ee7b75f-44ab-439f-9a26-cdc3a146ca40} state changed: planning
2020-07-14T10:53:06 [I|bac|] Task {label: Actions::Katello::Repository::CapsuleSync, id: efc84080-31c1-4ba4-837d-16c546a0f1e2, execution_plan_id: 8ee7b75f-44ab-439f-9a26-cdc3a146ca40} state changed: planned
2020-07-14T10:53:06 [I|bac|] Task {label: Actions::Katello::Repository::CapsuleSync, id: efc84080-31c1-4ba4-837d-16c546a0f1e2, execution_plan_id: 8ee7b75f-44ab-439f-9a26-cdc3a146ca40} state changed: running
2020-07-14T10:53:06 [I|bac|] Task {label: Actions::Katello::Repository::CapsuleSync, id: efc84080-31c1-4ba4-837d-16c546a0f1e2, execution_plan_id: 8ee7b75f-44ab-439f-9a26-cdc3a146ca40} state changed: stopped result: success
2020-07-14T10:53:06 [I|bac|] Task {label: Actions::Katello::Repository::CapsuleSync, id: efc84080-31c1-4ba4-837d-16c546a0f1e2, execution_plan_id: 8ee7b75f-44ab-439f-9a26-cdc3a146ca40} state changed: stopped result: success
2020-07-14T10:53:07 [I|aud|] Operatingsystem (2) create event on major 8
2020-07-14T10:53:07 [I|aud|] Operatingsystem (2) create event on name CentOS_Linux
2020-07-14T10:53:07 [I|aud|] Operatingsystem (2) create event on minor
2020-07-14T10:53:07 [I|aud|] Operatingsystem (2) create event on nameindicator
2020-07-14T10:53:07 [I|aud|] Operatingsystem (2) create event on release_name
2020-07-14T10:53:07 [I|aud|] Operatingsystem (2) create event on description
2020-07-14T10:53:07 [I|aud|] Operatingsystem (2) create event on password_hash SHA256
2020-07-14T10:53:07 [I|aud|] Operatingsystem (2) create event on title CentOS_Linux 8
2020-07-14T10:53:07 [I|bac|] Task {label: Actions::Katello::Repository::Sync, id: f7edb01c-eeea-4f6e-a264-3e66d479ed0f, execution_plan_id: 5c206f2e-7d48-44ef-9984-38f8a6107df5} state changed: stopped result: success
2020-07-14T10:53:07 [I|bac|] Task {label: Actions::Katello::Repository::Sync, id: f7edb01c-eeea-4f6e-a264-3e66d479ed0f, execution_plan_id: 5c206f2e-7d48-44ef-9984-38f8a6107df5} state changed: stopped result: success
2020-07-14T10:53:08 [I|bac|] Task {label: Actions::BulkAction, id: 4dcf587d-85cd-4867-af34-bff7828ea45f, execution_plan_id: d9775b28-5ad2-4213-bed2-bb170a4095b4} state changed: stopped result: success
2020-07-14T10:53:08 [I|bac|] Task {label: Actions::BulkAction, id: 4dcf587d-85cd-4867-af34-bff7828ea45f, execution_plan_id: d9775b28-5ad2-4213-bed2-bb170a4095b4} state changed: stopped result: success

[root@foreman pulp]# du -hs *
4.0K 0005_puppet_module_name_change.txt
6.4M assets
8.4G docroot
0 katello-export
0 packages
0 published
0 static
0 tmp
0 uploads

And as you can see the repository statistics seem to be OK:

Screenshot 2020-07-14 at 10.54.32
Screenshot 2020-07-14 at 10.54.321256×172 14.9 KB

When I run a new syn I get these logs (which seems fine) and the filesystem also doesn’t change:

2020-07-14T10:54:48 [I|app|08efbb4d] Started GET “/notification_recipients” for 127.0.0.1 at 2020-07-14 10:54:48 +0200
2020-07-14T10:54:48 [I|app|08efbb4d] Processing by NotificationRecipientsController#index as JSON
2020-07-14T10:54:48 [I|app|08efbb4d] Completed 200 OK in 18ms (Views: 0.2ms | ActiveRecord: 2.3ms | Allocations: 2178)
2020-07-14T10:54:49 [I|app|85b63fa7] Started POST “/katello/api/v2/repositories/bulk/sync?organization_id=1” for 127.0.0.1 at 2020-07-14 10:54:49 +0200
2020-07-14T10:54:49 [I|app|85b63fa7] Processing by Katello::Api::V2::RepositoriesBulkActionsController#sync_repositories as HTML
2020-07-14T10:54:49 [I|app|85b63fa7] Parameters: {“ids”=>[1], “organization_id”=>“1”, “api_version”=>“v2”, “repositories_bulk_action”=>{“ids”=>[1]}}
2020-07-14T10:54:49 [I|bac|85b63fa7] Task {label: Actions::BulkAction, id: 92058f84-9e7b-4d5f-9605-203828cb1727, execution_plan_id: 163d3110-1235-4b92-ac29-4eb6d06d940f} state changed: planning
2020-07-14T10:54:49 [I|bac|] Task {label: Actions::BulkAction, id: 92058f84-9e7b-4d5f-9605-203828cb1727, execution_plan_id: 163d3110-1235-4b92-ac29-4eb6d06d940f} state changed: planned
2020-07-14T10:54:49 [I|app|] Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.rc4/app/views/katello/api/v2/common/async.json.rabl within katello/api/v2/layouts/resource
2020-07-14T10:54:49 [I|app|] Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.rc4/app/views/katello/api/v2/common/async.json.rabl within katello/api/v2/layouts/resource (Duration: 43.0ms | Allocations: 6876)
2020-07-14T10:54:49 [I|app|] Completed 202 Accepted in 220ms (Views: 51.2ms | ActiveRecord: 28.2ms | Allocations: 33323)
2020-07-14T10:54:49 [I|app|5f5ae6e1] Started GET “/tasks/views/task-details.html” for 127.0.0.1 at 2020-07-14 10:54:49 +0200
2020-07-14T10:54:49 [I|bac|] Task {label: Actions::BulkAction, id: 92058f84-9e7b-4d5f-9605-203828cb1727, execution_plan_id: 163d3110-1235-4b92-ac29-4eb6d06d940f} state changed: running
2020-07-14T10:54:49 [I|app|a83d9c1b] Started POST “/foreman_tasks/api/tasks/bulk_search” for 127.0.0.1 at 2020-07-14 10:54:49 +0200
2020-07-14T10:54:49 [I|app|a83d9c1b] Processing by ForemanTasks::Api::TasksController#bulk_search as HTML
2020-07-14T10:54:49 [I|app|a83d9c1b] Parameters: {“searches”=>[{“type”=>“task”, “task_id”=>“92058f84-9e7b-4d5f-9605-203828cb1727”, “search_id”=>“1”}], “task”=>{}}
2020-07-14T10:54:49 [I|bac|] Task {label: Actions::Katello::Repository::Sync, id: 376c23b9-4400-4798-9b42-baa06f76551f, execution_plan_id: 5e0ecb07-4378-4db2-9084-a328f92f21b2} state changed: planning
2020-07-14T10:54:49 [I|app|a83d9c1b] Completed 200 OK in 80ms (Views: 1.6ms | ActiveRecord: 10.0ms | Allocations: 14268)
2020-07-14T10:54:50 [I|bac|] Task {label: Actions::Katello::Repository::Sync, id: 376c23b9-4400-4798-9b42-baa06f76551f, execution_plan_id: 5e0ecb07-4378-4db2-9084-a328f92f21b2} state changed: planned
2020-07-14T10:54:50 [I|bac|] Task {label: Actions::Katello::Repository::Sync, id: 376c23b9-4400-4798-9b42-baa06f76551f, execution_plan_id: 5e0ecb07-4378-4db2-9084-a328f92f21b2} state changed: running
2020-07-14T10:54:50 [I|app|0220c9ad] Started POST “/foreman_tasks/api/tasks/bulk_search” for 127.0.0.1 at 2020-07-14 10:54:50 +0200
2020-07-14T10:54:50 [I|app|0220c9ad] Processing by ForemanTasks::Api::TasksController#bulk_search as HTML
2020-07-14T10:54:50 [I|app|0220c9ad] Parameters: {“searches”=>[{“type”=>“task”, “task_id”=>“92058f84-9e7b-4d5f-9605-203828cb1727”, “search_id”=>“1”}], “task”=>{}}
2020-07-14T10:54:50 [I|app|0220c9ad] Completed 200 OK in 163ms (Views: 0.8ms | ActiveRecord: 9.3ms | Allocations: 75551)
2020-07-14T10:54:52 [I|app|5d2d6896] Started POST “/foreman_tasks/api/tasks/bulk_search” for 127.0.0.1 at 2020-07-14 10:54:52 +0200
2020-07-14T10:54:52 [I|app|5d2d6896] Processing by ForemanTasks::Api::TasksController#bulk_search as HTML
2020-07-14T10:54:52 [I|app|5d2d6896] Parameters: {“searches”=>[{“type”=>“task”, “task_id”=>“92058f84-9e7b-4d5f-9605-203828cb1727”, “search_id”=>“1”}], “task”=>{}}
2020-07-14T10:54:52 [I|app|5d2d6896] Completed 200 OK in 197ms (Views: 0.9ms | ActiveRecord: 8.9ms | Allocations: 77694)
2020-07-14T10:54:55 [I|app|200ec98a] Started POST “/foreman_tasks/api/tasks/bulk_search” for 127.0.0.1 at 2020-07-14 10:54:55 +0200
2020-07-14T10:54:55 [I|app|200ec98a] Processing by ForemanTasks::Api::TasksController#bulk_search as HTML
2020-07-14T10:54:55 [I|app|200ec98a] Parameters: {“searches”=>[{“type”=>“task”, “task_id”=>“92058f84-9e7b-4d5f-9605-203828cb1727”, “search_id”=>“1”}], “task”=>{}}
2020-07-14T10:54:56 [I|app|200ec98a] Completed 200 OK in 139ms (Views: 0.9ms | ActiveRecord: 7.1ms | Allocations: 77839)
2020-07-14T10:54:58 [I|app|0688e662] Started GET “/notification_recipients” for 127.0.0.1 at 2020-07-14 10:54:58 +0200
2020-07-14T10:54:58 [I|app|0688e662] Processing by NotificationRecipientsController#index as JSON
2020-07-14T10:54:58 [I|app|0688e662] Completed 200 OK in 10ms (Views: 0.1ms | ActiveRecord: 0.8ms | Allocations: 2178)
2020-07-14T10:55:00 [I|app|59840b4b] Started POST “/foreman_tasks/api/tasks/bulk_search” for 127.0.0.1 at 2020-07-14 10:55:00 +0200
2020-07-14T10:55:00 [I|app|59840b4b] Processing by ForemanTasks::Api::TasksController#bulk_search as HTML
2020-07-14T10:55:00 [I|app|59840b4b] Parameters: {“searches”=>[{“type”=>“task”, “task_id”=>“92058f84-9e7b-4d5f-9605-203828cb1727”, “search_id”=>“1”}], “task”=>{}}
2020-07-14T10:55:00 [I|app|59840b4b] Completed 200 OK in 172ms (Views: 0.9ms | ActiveRecord: 7.1ms | Allocations: 77485)
2020-07-14T10:55:05 [I|app|d4f48590] Started POST “/foreman_tasks/api/tasks/bulk_search” for 127.0.0.1 at 2020-07-14 10:55:05 +0200
2020-07-14T10:55:05 [I|app|d4f48590] Processing by ForemanTasks::Api::TasksController#bulk_search as HTML
2020-07-14T10:55:05 [I|app|d4f48590] Parameters: {“searches”=>[{“type”=>“task”, “task_id”=>“92058f84-9e7b-4d5f-9605-203828cb1727”, “search_id”=>“1”}], “task”=>{}}
2020-07-14T10:55:05 [I|app|d4f48590] Completed 200 OK in 103ms (Views: 0.9ms | ActiveRecord: 4.2ms | Allocations: 77480)
2020-07-14T10:55:08 [I|app|a431bd43] Started GET “/notification_recipients” for 127.0.0.1 at 2020-07-14 10:55:08 +0200
2020-07-14T10:55:08 [I|app|a431bd43] Processing by NotificationRecipientsController#index as JSON
2020-07-14T10:55:08 [I|app|a431bd43] Completed 200 OK in 9ms (Views: 0.2ms | ActiveRecord: 0.9ms | Allocations: 2178)
2020-07-14T10:55:11 [I|app|f1f4e780] Started POST “/foreman_tasks/api/tasks/bulk_search” for 127.0.0.1 at 2020-07-14 10:55:11 +0200
2020-07-14T10:55:11 [I|app|f1f4e780] Processing by ForemanTasks::Api::TasksController#bulk_search as HTML
2020-07-14T10:55:11 [I|app|f1f4e780] Parameters: {“searches”=>[{“type”=>“task”, “task_id”=>“92058f84-9e7b-4d5f-9605-203828cb1727”, “search_id”=>“1”}], “task”=>{}}
2020-07-14T10:55:11 [I|app|f1f4e780] Completed 200 OK in 97ms (Views: 0.9ms | ActiveRecord: 4.0ms | Allocations: 77480)
2020-07-14T10:55:18 [I|app|5c16629a] Started POST “/foreman_tasks/api/tasks/bulk_search” for 127.0.0.1 at 2020-07-14 10:55:18 +0200
2020-07-14T10:55:18 [I|app|5c16629a] Processing by ForemanTasks::Api::TasksController#bulk_search as HTML
2020-07-14T10:55:18 [I|app|5c16629a] Parameters: {“searches”=>[{“type”=>“task”, “task_id”=>“92058f84-9e7b-4d5f-9605-203828cb1727”, “search_id”=>“1”}], “task”=>{}}
2020-07-14T10:55:18 [I|app|5c16629a] Completed 200 OK in 100ms (Views: 0.8ms | ActiveRecord: 4.0ms | Allocations: 77868)
2020-07-14T10:55:18 [I|app|8fa54e4b] Started GET “/notification_recipients” for 127.0.0.1 at 2020-07-14 10:55:18 +0200
2020-07-14T10:55:18 [I|app|8fa54e4b] Processing by NotificationRecipientsController#index as JSON
2020-07-14T10:55:18 [I|app|8fa54e4b] Completed 200 OK in 8ms (Views: 0.1ms | ActiveRecord: 0.8ms | Allocations: 2178)
2020-07-14T10:55:26 [I|app|e436438c] Started POST “/foreman_tasks/api/tasks/bulk_search” for 127.0.0.1 at 2020-07-14 10:55:26 +0200
2020-07-14T10:55:26 [I|app|e436438c] Processing by ForemanTasks::Api::TasksController#bulk_search as HTML
2020-07-14T10:55:26 [I|app|e436438c] Parameters: {“searches”=>[{“type”=>“task”, “task_id”=>“92058f84-9e7b-4d5f-9605-203828cb1727”, “search_id”=>“1”}], “task”=>{}}
2020-07-14T10:55:26 [I|app|e436438c] Completed 200 OK in 97ms (Views: 0.8ms | ActiveRecord: 3.9ms | Allocations: 77496)
2020-07-14T10:55:28 [I|app|0a5ca34b] Started GET “/notification_recipients” for 127.0.0.1 at 2020-07-14 10:55:28 +0200
2020-07-14T10:55:28 [I|app|0a5ca34b] Processing by NotificationRecipientsController#index as JSON
2020-07-14T10:55:28 [I|app|0a5ca34b] Completed 200 OK in 8ms (Views: 0.2ms | ActiveRecord: 0.9ms | Allocations: 2178)
2020-07-14T10:55:35 [I|app|5ebbff30] Started POST “/foreman_tasks/api/tasks/bulk_search” for 127.0.0.1 at 2020-07-14 10:55:35 +0200
2020-07-14T10:55:35 [I|app|5ebbff30] Processing by ForemanTasks::Api::TasksController#bulk_search as HTML
2020-07-14T10:55:35 [I|app|5ebbff30] Parameters: {“searches”=>[{“type”=>“task”, “task_id”=>“92058f84-9e7b-4d5f-9605-203828cb1727”, “search_id”=>“1”}], “task”=>{}}
2020-07-14T10:55:35 [I|app|5ebbff30] Completed 200 OK in 159ms (Views: 1.8ms | ActiveRecord: 4.1ms | Allocations: 77893)
2020-07-14T10:55:38 [I|app|50412fae] Started GET “/notification_recipients” for 127.0.0.1 at 2020-07-14 10:55:38 +0200
2020-07-14T10:55:38 [I|app|50412fae] Processing by NotificationRecipientsController#index as JSON
2020-07-14T10:55:38 [I|app|50412fae] Completed 200 OK in 8ms (Views: 0.1ms | ActiveRecord: 0.8ms | Allocations: 2178)
2020-07-14T10:55:45 [I|app|3f15854a] Started POST “/foreman_tasks/api/tasks/bulk_search” for 127.0.0.1 at 2020-07-14 10:55:45 +0200
2020-07-14T10:55:45 [I|app|3f15854a] Processing by ForemanTasks::Api::TasksController#bulk_search as HTML
2020-07-14T10:55:45 [I|app|3f15854a] Parameters: {“searches”=>[{“type”=>“task”, “task_id”=>“92058f84-9e7b-4d5f-9605-203828cb1727”, “search_id”=>“1”}], “task”=>{}}
2020-07-14T10:55:46 [I|app|3f15854a] Completed 200 OK in 97ms (Views: 0.9ms | ActiveRecord: 3.9ms | Allocations: 77433)
2020-07-14T10:55:47 [I|aud|] Katello::Repository (1) update event on version_href /pulp/api/v3/repositories/rpm/rpm/0f8bce63-45e8-4894-a750-df29d9eb73d3/versions/1/, /pulp/api/v3/repositories/rpm/rpm/0f8bce63-45e8-4894-a750-df29d9eb73d3/versions/2/
2020-07-14T10:55:48 [I|aud|] Katello::Repository (1) update event on publication_href /pulp/api/v3/publications/rpm/rpm/a18d6310-d32d-428e-86cb-44749a0da8f4/, /pulp/api/v3/publications/rpm/rpm/2237af4c-e8f6-4d83-bf42-a3231bf4948c/
2020-07-14T10:55:48 [I|app|e0538d1b] Started GET “/notification_recipients” for 127.0.0.1 at 2020-07-14 10:55:48 +0200
2020-07-14T10:55:48 [I|app|e0538d1b] Processing by NotificationRecipientsController#index as JSON
2020-07-14T10:55:48 [I|app|e0538d1b] Completed 200 OK in 8ms (Views: 0.1ms | ActiveRecord: 1.0ms | Allocations: 2178)
2020-07-14T10:55:50 [I|bac|] Task {label: Actions::Katello::Repository::CapsuleSync, id: 5a7e6de1-d8c2-4840-b73f-7b97ea24e68c, execution_plan_id: 6ac72f77-9a88-4df5-8e9a-c1a00dbd319a} state changed: planning
2020-07-14T10:55:50 [I|bac|] Task {label: Actions::Katello::Repository::CapsuleSync, id: 5a7e6de1-d8c2-4840-b73f-7b97ea24e68c, execution_plan_id: 6ac72f77-9a88-4df5-8e9a-c1a00dbd319a} state changed: planned
2020-07-14T10:55:50 [I|bac|] Task {label: Actions::Katello::Repository::CapsuleSync, id: 5a7e6de1-d8c2-4840-b73f-7b97ea24e68c, execution_plan_id: 6ac72f77-9a88-4df5-8e9a-c1a00dbd319a} state changed: running
2020-07-14T10:55:50 [I|bac|] Task {label: Actions::Katello::Repository::CapsuleSync, id: 5a7e6de1-d8c2-4840-b73f-7b97ea24e68c, execution_plan_id: 6ac72f77-9a88-4df5-8e9a-c1a00dbd319a} state changed: stopped result: success
2020-07-14T10:55:50 [I|bac|] Task {label: Actions::Katello::Repository::CapsuleSync, id: 5a7e6de1-d8c2-4840-b73f-7b97ea24e68c, execution_plan_id: 6ac72f77-9a88-4df5-8e9a-c1a00dbd319a} state changed: stopped result: success
2020-07-14T10:55:51 [I|bac|] Task {label: Actions::Katello::Repository::Sync, id: 376c23b9-4400-4798-9b42-baa06f76551f, execution_plan_id: 5e0ecb07-4378-4db2-9084-a328f92f21b2} state changed: stopped result: success
2020-07-14T10:55:51 [I|bac|] Task {label: Actions::Katello::Repository::Sync, id: 376c23b9-4400-4798-9b42-baa06f76551f, execution_plan_id: 5e0ecb07-4378-4db2-9084-a328f92f21b2} state changed: stopped result: success
2020-07-14T10:55:51 [I|bac|] Task {label: Actions::BulkAction, id: 92058f84-9e7b-4d5f-9605-203828cb1727, execution_plan_id: 163d3110-1235-4b92-ac29-4eb6d06d940f} state changed: stopped result: success
2020-07-14T10:55:51 [I|bac|] Task {label: Actions::BulkAction, id: 92058f84-9e7b-4d5f-9605-203828cb1727, execution_plan_id: 163d3110-1235-4b92-ac29-4eb6d06d940f} state changed: stopped result: success
2020-07-14T10:55:56 [I|app|4053a3ef] Started POST “/foreman_tasks/api/tasks/bulk_search” for 127.0.0.1 at 2020-07-14 10:55:56 +0200
2020-07-14T10:55:56 [I|app|4053a3ef] Processing by ForemanTasks::Api::TasksController#bulk_search as HTML
2020-07-14T10:55:56 [I|app|4053a3ef] Parameters: {“searches”=>[{“type”=>“task”, “task_id”=>“92058f84-9e7b-4d5f-9605-203828cb1727”, “search_id”=>“1”}], “task”=>{}}
2020-07-14T10:55:56 [I|app|4053a3ef] Completed 200 OK in 203ms (Views: 0.9ms | ActiveRecord: 8.2ms | Allocations: 85390)

[root@foreman pulp]# du -hs *
4.0K 0005_puppet_module_name_change.txt
6.4M assets
8.4G docroot
0 katello-export
0 packages
0 published
0 static
0 tmp
0 uploads

The problem is that the repository statistics (as seen in the previous screenshot) go back to 0 packages. …

Some more info:

[root@foreman]# foreman-rake plugin:list
Collecting plugin information
Foreman plugin: foreman-tasks, 2.0.1, Ivan Nečas, The goal of this plugin is to unify the way of showing task statuses across the Foreman instance.
It defines Task model for keeping the information about the tasks and Lock for assigning the tasks
to resources. The locking allows dealing with preventing multiple colliding tasks to be run on the
same resource. It also optionally provides Dynflow infrastructure for using it for managing the tasks.
Foreman plugin: foreman_ansible, 5.1.1, Daniel Lobato Garcia, Ansible integration with Foreman
Foreman plugin: foreman_openscap, 3.0.0, slukasik@redhat.com, Foreman plug-in for managing security compliance reports
Foreman plugin: foreman_remote_execution, 3.3.2, Foreman Remote Execution team, A plugin bringing remote execution to the Foreman, completing the config management functionality with remote management functionality.
Foreman plugin: katello, 3.16.0.rc4, N/A, Katello adds Content and Subscription Management to Foreman. For this it relies on Candlepin and Pulp.

[root@foreman]# hammer repository list
—|---------------|----------|--------------|------------------------------------------------------

ID NAME PRODUCT CONTENT TYPE URL
1 CentOS 8 Base CentOS 8 yum http://mirror.centos.org/centos-8/8/BaseOS/x86_64/os/
— --------------- ---------- -------------- ------------------------------------------------------

[root@foreman pulp]# hammer repository synchronize --validate-contents=true --id=1
[…] [100%]
Total tasks: : 5540/5540

Associating Content: 0/0
Associating Content: 0/0
Downloading Artifacts: 0/0
Downloading Artifacts: 0/0
Downloading Metadata Files: 1/1
Downloading Metadata Files: 4/4
Optimizing Sync: 1/1
Parsed Comps: 67/67
Parsed Modulemd: 66/66
Parsed Modulemd-defaults: 42/42
Parsed Packages: 5359/5359
Un-Associating Content: 0/0
Error: undefined method `remove’ for Katello::Pulp3::YumMetadataFile:Class
404 Not Found

Now what I see on my 3.16 RC3 VM is that there is a /var/lib/pulp/content directory, which contains the RPMs instead of them being stuck in /var/lib/pulp/docroot
So I tried to create them manually with the same structure and right, but it doesn’t seem to change anything.

mkdir -p /var/lib/pulp/content/units/distribution
mkdir -p /var/lib/pulp/content/units/drpm
mkdir -p /var/lib/pulp/content/units/modulemd
mkdir -p /var/lib/pulp/content/units/modulemd_defaults
mkdir -p /var/lib/pulp/content/units/rpm
mkdir -p /var/lib/pulp/content/units/yum_repo_metadata_file
chown -R apache:apache /var/lib/pulp/content/

Any clue?

2

…and that is the repo view after the second synchronization, 0 packages:

Screenshot 2020-07-14 at 10.59.22
Screenshot 2020-07-14 at 10.59.221256×172 14.8 KB

3

After some further testing:
Fresh 3.16 RC4: breaks
Fresh 3.15 to 3.16RC4 upgrade: works

4

This sounds like it could somehow be related to Bug #30389: Setting katello_applicability to true empties repos on subsequent sync - Katello - Foreman where we also saw the counts going to zero. @iballou just tagging you in here as 3.16 release owner.

I’ll add this discussion to the above issue so it can be tracked

5

Oh, you or someone else has already added this discussion there. Nice!

6

I’ve updated the issue with my findings. The second sync is causing the repo version to increment to version 2 (for example), but that version 2 has nothing in it. The repo index then indexes nothing. Definitely a bug!

1 Like