Hi folks,
Post Katello/Foreman upgrade I’m having issues syncing repositories.
On Friday I upgraded to Katello 3.9/Foreman 1.20 having started out at Katello 3.6/Foreman 1.20.
Since the upgrade it appears none of the daily syncs had started.
Anyway today I kicked off a manual sync but none of the tasks have got going, they’re mostly all sitting at 0%.
oreman=# select id, label, started_at, state, result from foreman_tasks_tasks where state != ‘stopped’;
id | label | started_at | state | result
--------------------------------------±------------------------------------------------±--------------------±----------±--------
a0a7be11-93db-4110-a631-c2049ff0b906 | CreatePulpDiskSpaceNotifications | | scheduled | pending
9f60d644-5116-40ab-9b3c-cf4b66383569 | Actions::Katello::Repository::Sync | 2019-06-03 06:34:45 | running | pending
caf484cd-95a9-40a2-b41e-a8c5884a7624 | Actions::Katello::Repository::Sync | 2019-06-03 06:35:03 | running | pending
7a898089-bfcd-4704-aafb-ca9050fc63ff | Actions::Katello::Repository::Sync | 2019-06-03 06:34:53 | running | pending
5a30543c-f383-49a2-974a-b5476c624b20 | Actions::Katello::Repository::Sync | 2019-06-03 06:35:08 | running | pending
5f399c59-ced7-431c-b0bb-ee20223a742e | Actions::Katello::Repository::Sync | 2019-06-03 06:35:09 | running | pending
e3ba6db7-1a02-43dd-bff2-c8b674257d9d | Actions::Katello::Repository::Sync | 2019-06-03 06:35:20 | running | pending
63049575-ce8f-4222-87be-8ec5f7612210 | Actions::Katello::Repository::Sync | 2019-06-03 06:34:31 | running | pending
9be6ff0c-f804-4427-b79b-6f2ca85f71de | Actions::Katello::Repository::Sync | 2019-06-03 06:34:32 | running | pending
3347196c-fae5-4338-ad7e-26077491509b | Actions::Katello::Repository::Sync | 2019-06-03 06:35:13 | running | pending
a8dc4070-5033-4307-ba2e-46b4cc0a3b5b | Actions::Katello::Repository::Sync | 2019-06-03 06:34:25 | running | pending
1c60eefb-b4ea-4ee2-bd34-b8d756c84efe | Actions::Katello::Repository::Sync | 2019-06-03 06:37:19 | running | pending
68fede1d-1253-4681-88cc-3d16d099c2c4 | Actions::Katello::Repository::Sync | 2019-06-03 06:36:17 | running | pending
e17dfbf8-dce2-4b9e-8120-c1cb3a5c72f8 | Actions::Katello::Repository::Sync | 2019-06-03 06:35:59 | running | pending
8c73e4a2-14b8-45dc-9e34-5ab1e8b07a5c | Actions::Katello::Repository::Sync | 2019-06-03 06:36:06 | running | pending
170c3d65-9b50-451f-b748-ef7e4aec976a | Actions::Katello::Repository::Sync | 2019-06-03 06:35:34 | running | pending
69ffadf7-3996-4e7e-8ae8-eb10e6f2d9a1 | Actions::Katello::Repository::Sync | 2019-06-03 06:35:19 | running | pending
eafaa3e1-e56d-4439-9e30-2f5e8f26453f | Actions::Katello::Repository::Sync | 2019-06-03 06:35:38 | running | pending
11e10442-5fa8-4d7f-b2d2-ce8722dc6e7d | Actions::Katello::Repository::Sync | 2019-06-03 06:35:48 | running | pending
84f01317-14d0-4563-b3b5-694e934d9a9f | Actions::Katello::Repository::Sync | 2019-06-03 06:35:25 | running | pending
fbfe69e8-3747-4167-8129-685f409a9c3a | Actions::Katello::Repository::Sync | 2019-06-03 06:37:34 | running | pending
4452497b-db5a-4dad-b9e9-dd08262e65b5 | Actions::Katello::Repository::Sync | 2019-06-03 06:37:11 | running | pending
c6113c0c-8e41-4ba1-a33c-304cc88ff79d | Actions::Katello::Repository::Sync | 2019-06-03 06:36:21 | running | pending
01f62e49-4d86-4bce-b574-e62942e3cd4f | Actions::Katello::Repository::Sync | 2019-06-03 06:36:11 | running | pending
c1fb4c2e-702a-4d32-897d-464167c765ac | Actions::Katello::Repository::Sync | 2019-06-03 06:35:28 | running | pending
23995ea9-93df-4e31-970b-6b23ec11a720 | Actions::Katello::Repository::Sync | 2019-06-03 06:37:02 | running | pending
9c6c1270-3bf2-4136-89ca-1815ccfdc524 | Actions::Katello::Repository::Sync | 2019-06-03 06:36:29 | running | pending
61bc8100-e150-4190-93d2-f60f56c54f8f | Actions::Katello::Repository::Sync | 2019-06-03 06:36:33 | running | pending
a09e38ac-822f-4d24-abf8-05994234cf9f | Actions::Katello::Repository::Sync | 2019-06-03 06:36:55 | running | pending
7d5040a5-6301-44a9-ba21-d07caddb75ca | Actions::Katello::Repository::Sync | 2019-06-03 06:37:08 | running | pending
6875fc47-efc6-477f-b07e-ecffc81ba869 | Actions::Katello::Repository::Sync | 2019-06-03 06:35:36 | running | pending
482f4bf7-4825-40a2-b8a1-6a35a5f5c704 | Actions::Katello::Repository::Sync | 2019-06-03 06:35:31 | running | pending
07bed047-921d-4165-a159-9790b8d2a764 | Actions::Katello::Repository::Sync | 2019-06-03 06:35:41 | running | pending
e861090f-e564-44df-8edc-138ff9a66e56 | Actions::Katello::Repository::Sync | 2019-06-03 06:35:44 | running | pending
d6a7fda8-ec19-4ce9-865d-23b2d407762f | Actions::Katello::Repository::Sync | 2019-06-03 06:35:54 | running | pending
5bd7e5ca-4b78-42b1-8138-0bcc0ebfe00d | Actions::Katello::Repository::Sync | 2019-06-03 06:36:25 | running | pending
2d53a3de-b31d-438b-ade2-7506748cc069 | Actions::Katello::Repository::Sync | 2019-06-03 06:36:39 | running | pending
b3b41e7d-c1d2-4ba9-abbf-31c6656bfb74 | Actions::Katello::Repository::Sync | 2019-06-03 06:36:43 | running | pending
cb9de7e0-e911-4168-8e77-7a001e705664 | Actions::Katello::Repository::Sync | 2019-06-03 06:37:38 | running | pending
4f0d5e3f-25ce-469d-b189-b7f31314693b | Actions::Katello::Repository::Sync | 2019-06-03 06:36:49 | running | pending
c1536e5d-9164-4a0d-b734-37d5fd3e6936 | Actions::Katello::Repository::Sync | 2019-06-03 06:36:58 | running | pending
674a518d-548a-4c46-9ade-db843f1e47e7 | Actions::Katello::Repository::Sync | 2019-06-03 06:37:16 | running | pending
8390be98-2c64-4b90-b7a7-2947dce7daf2 | Actions::Katello::Repository::Sync | 2019-06-03 06:37:28 | running | pending
a1bdaceb-2cbf-4d24-95b3-09bc329f7c0d | Actions::Katello::Repository::Sync | 2019-06-03 06:35:50 | running | pending
e8d29dc5-63a9-481e-9273-2b7535c6f2f1 | Actions::Candlepin::ListenOnCandlepinEvents | 2019-05-31 14:09:43 | running | pending
df6a92da-f038-47e2-a558-828900f973bf | Actions::Katello::EventQueue::Monitor | 2019-05-31 14:09:44 | running | pending
3815d45e-712b-4701-84b8-9daa6b15834e | Actions::Katello::Repository::Sync | 2019-06-03 06:34:59 | running | pending
c9055874-cce1-492b-b037-59f0fdf50070 | Actions::Katello::Repository::Sync | 2019-06-03 06:34:47 | running | pending
0a5ca21e-3fb3-4faf-a042-e0956138d323 | Actions::Katello::Repository::Sync | 2019-06-03 06:35:57 | running | pending
aa4ebc48-bc69-43a0-8c85-b9dc1b79b422 | Actions::Katello::Repository::Sync | 2019-06-03 06:36:03 | running | pending
9d7906ab-565e-4782-be23-bf83cbe57219 | Actions::Katello::Organization::ManifestRefresh | 2019-06-03 06:56:42 | running | warning
(51 rows)
Based on something I read earlier i tried to refresh the manifest but that task hung at 16%.
I’ve observed the following errors in the production log
2019-06-03T07:33:38 [F|app|] ActionController::RoutingError (No route matches [POST] “/katello/api/v2/products/sync”):
2019-06-03T07:36:00 [F|app|] ActionController::RoutingError (No route matches [POST] “/katello/api/v2/repositories/sync_complete”):
2019-06-03T07:36:16 [F|app|] ActionController::RoutingError (No route matches [POST] “/katello/api/v2/repositories/sync_complete”):
2019-06-03T07:37:47 [F|app|] ActionController::RoutingError (No route matches [POST] “/katello/api/v2/repositories/sync_complete”):
2019-06-03T07:39:29 [F|app|] ActionController::RoutingError (No route matches [POST] “/katello/api/v2/repositories/sync_complete”):
2019-06-03T07:41:35 [F|app|] ActionController::RoutingError (No route matches [POST] “/katello/api/v2/repositories/sync_complete”):
2019-06-03T07:44:07 [F|app|] ActionController::RoutingError (No route matches [POST] “/katello/api/v2/repositories/sync_complete”):
2019-06-03T07:44:16 [F|app|] ActionController::RoutingError (No route matches [POST] “/katello/api/v2/repositories/sync_complete”):
2019-06-03T07:45:11 [F|app|] ActionController::RoutingError (No route matches [POST] “/katello/api/v2/repositories/sync_complete”):
2019-06-03T07:45:17 [F|app|] ActionController::RoutingError (No route matches [POST] “/katello/api/v2/repositories/sync_complete”):
2019-06-03T07:46:28 [F|app|] ActionController::RoutingError (No route matches [POST] “/katello/api/v2/repositories/sync_complete”):
2019-06-03T07:50:19 [F|app|] ActionController::RoutingError (No route matches [POST] “/katello/api/v2/repositories/sync_complete”):
2019-06-03T07:51:21 [F|app|] ActionController::RoutingError (No route matches [POST] “/katello/api/v2/repositories/sync_complete”):
2019-06-03T07:52:08 [F|app|] ActionController::RoutingError (No route matches [POST] “/katello/api/v2/repositories/sync_complete”):
2019-06-03T07:53:15 [F|app|] ActionController::RoutingError (No route matches [POST] “/katello/api/v2/repositories/sync_complete”):
2019-06-03T07:54:28 [F|app|] ActionController::RoutingError (No route matches [POST] “/katello/api/v2/repositories/sync_complete”):
2019-06-03T07:55:50 [F|app|] ActionController::RoutingError (No route matches [POST] “/katello/api/v2/repositories/sync_complete”):
2019-06-03T07:55:55 [F|app|] ActionController::RoutingError (No route matches [GET] “/assets/spinner.gif”):
2019-06-03T08:00:23 [F|app|] ActionController::RoutingError (No route matches [POST] “/katello/api/v2/repositories/sync_complete”):
2019-06-03T08:19:29 [F|app|] ActionController::RoutingError (No route matches [POST] “/katello/api/v2/repositories/sync_complete”):
2019-06-03T08:25:25 [F|app|] ActionController::RoutingError (No route matches [POST] “/katello/api/v2/repositories/sync_complete”):
2019-06-03T08:27:19 [F|app|] ActionController::RoutingError (No route matches [POST] “/katello/api/v2/repositories/sync_complete”):
2019-06-03T08:32:01 [F|app|] ActionController::RoutingError (No route matches [POST] “/katello/api/v2/repositories/sync_complete”):
2019-06-03T08:35:47 [F|app|] ActionController::RoutingError (No route matches [POST] “/katello/api/v2/repositories/sync_complete”):
2019-06-03T08:39:37 [F|app|] ActionController::RoutingError (No route matches [POST] “/katello/api/v2/repositories/sync_complete”):
2019-06-03T08:41:16 [F|app|] ActionController::RoutingError (No route matches [POST] “/katello/api/v2/repositories/sync_complete”):
2019-06-03T08:42:16 [F|app|] ActionController::RoutingError (No route matches [POST] “/katello/api/v2/repositories/sync_complete”):
2019-06-03T08:59:07 [F|app|] ActionController::RoutingError (No route matches [POST] “/katello/api/v2/repositories/sync_complete”):
2019-06-03T09:04:57 [F|app|] ActionController::RoutingError (No route matches [POST] “/katello/api/v2/repositories/sync_complete”):
2019-06-03T09:06:04 [F|app|] ActionController::RoutingError (No route matches [POST] “/katello/api/v2/repositories/sync_complete”):
2019-06-03T09:07:15 [F|app|] ActionController::RoutingError (No route matches [POST] “/katello/api/v2/repositories/sync_complete”):
2019-06-03T09:23:51 [F|app|] ActionController::RoutingError (No route matches [POST] “/katello/api/v2/repositories/sync_complete”):
2019-06-03T09:24:55 [F|app|] ActionController::RoutingError (No route matches [POST] “/katello/api/v2/repositories/sync_complete”):
2019-06-03T09:26:07 [F|app|] ActionController::RoutingError (No route matches [POST] “/katello/api/v2/repositories/sync_complete”):
2019-06-03T09:30:58 [F|app|] ActionController::RoutingError (No route matches [POST] “/katello/api/v2/repositories/sync_complete”):
2019-06-03T09:34:28 [F|app|] ActionController::RoutingError (No route matches [POST] “/katello/api/v2/repositories/sync_complete”):
2019-06-03T09:35:53 [F|app|] ActionController::RoutingError (No route matches [POST] “/katello/api/v2/repositories/sync_complete”):
2019-06-03T09:45:17 [F|app|] ActionController::RoutingError (No route matches [POST] “/katello/api/v2/repositories/sync_complete”):
2019-06-03T09:46:17 [F|app|] ActionController::RoutingError (No route matches [POST] “/katello/api/v2/repositories/sync_complete”):
2019-06-03T09:47:13 [F|app|] ActionController::RoutingError (No route matches [POST] “/katello/api/v2/repositories/sync_complete”):
2019-06-03T09:50:21 [F|app|] ActionController::RoutingError (No route matches [POST] “/katello/api/v2/repositories/sync_complete”):
2019-06-03T09:51:18 [F|app|] ActionController::RoutingError (No route matches [POST] “/katello/api/v2/repositories/sync_complete”):
2019-06-03T09:54:39 [F|app|] ActionController::RoutingError (No route matches [POST] “/katello/api/v2/repositories/sync_complete”):
2019-06-03T09:57:55 [F|app|] ActionController::RoutingError (No route matches [POST] “/katello/api/v2/repositories/sync_complete”):
2019-06-03T09:59:26 [F|app|] ActionController::RoutingError (No route matches [POST] “/katello/api/v2/repositories/sync_complete”):
2019-06-03T10:08:31 [F|app|] ActionController::RoutingError (No route matches [POST] “/katello/api/v2/repositories/sync_complete”):
2019-06-03T10:11:16 [F|app|] ActionController::RoutingError (No route matches [POST] “/katello/api/v2/repositories/sync_complete”):
2019-06-03T10:11:19 [F|app|] ActionController::RoutingError (No route matches [POST] “/katello/api/v2/repositories/sync_complete”):
2019-06-03T10:14:00 [F|app|] ActionController::RoutingError (No route matches [POST] “/katello/api/v2/repositories/sync_complete”):
2019-06-03T10:14:57 [F|app|] ActionController::RoutingError (No route matches [POST] “/katello/api/v2/repositories/sync_complete”):
2019-06-03T10:16:36 [F|app|] ActionController::RoutingError (No route matches [POST] “/katello/api/v2/repositories/sync_complete”):
2019-06-03T10:27:40 [F|app|] ActionController::RoutingError (No route matches [POST] “/katello/api/v2/repositories/sync_complete”):
