All repositories stopped syncing

Support
,
1

Problem:
Good afternoon. Today I noticed that all repositories that live on the server have stopped syncing successfully. below is a screenshot. This applies to both ubuntu and Centos

image
image589×726 44.7 KB

There are no errors in this case. foreman-maintan indicates that all services are running. Synchronization has not stopped for a long time

Expected outcome:
Successful completion of synchronization

Foreman and Proxy versions:
2.4
Foreman and Proxy plugin versions:

Distribution and version:

Other relevant data:

2

A suspended task doesn’t necessarily mean anything is wrong.

What does the Sync Status page say?

Does hammer ping show all services OK?

3

The task is not suspended, it is in the process of execution, but the process has not been completed for several hours, while there is no load on the server at all.

image
image1000×71 3.54 KB

4

You can look at the Pulp logs with

cat /var/log/messages | grep pulpcore

or

journalctl -u pulpcore-*

Any errors showing in there?

5

in logs such

Jun 03 14:54:13 foreman pulpcore-api[1382]: pulp [None]: django_guid:INFO: Header Correlation-ID was not found in the incoming request. Generated new GUID: 864fc9f7710546dabb6722d4d7a270d3
Jun 03 14:54:13 foreman pulpcore-api[1382]: - - [03/Jun/2022:11:54:13 +0000] “GET /pulp/api/v3/tasks/d65e240b-dae2-4318-aa79-195e4e751f97/ HTTP/1.1” 200 550 “-” “OpenAPI-Generator/3.9.0/ruby”
Jun 03 14:54:16 foreman pulpcore-api[1382]: pulp [None]: django_guid:INFO: Header Correlation-ID was not found in the incoming request. Generated new GUID: cb3c999cff12440296f3a28e43a5b21d
Jun 03 14:54:16 foreman pulpcore-api[1382]: - - [03/Jun/2022:11:54:16 +0000] “GET /pulp/api/v3/tasks/da429bda-f70c-40f2-80cb-1a95e384bc7b/ HTTP/1.1” 200 506 “-” “OpenAPI-Generator/3.9.0/ruby”
Jun 03 14:54:29 foreman pulpcore-api[1382]: pulp [None]: django_guid:INFO: Header Correlation-ID was not found in the incoming request. Generated new GUID: fd1c41fe025e47fb9f222f385fba0a39
Jun 03 14:54:29 foreman pulpcore-api[1382]: - - [03/Jun/2022:11:54:29 +0000] “GET /pulp/api/v3/tasks/d65e240b-dae2-4318-aa79-195e4e751f97/ HTTP/1.1” 200 550 “-” “OpenAPI-Generator/3.9.0/ruby”
Jun 03 14:54:32 foreman pulpcore-api[1382]: pulp [None]: django_guid:INFO: Header Correlation-ID was not found in the incoming request. Generated new GUID: 61f0d439eba2405b8b5acea4ec2f0b11
Jun 03 14:54:33 foreman pulpcore-api[1382]: - - [03/Jun/2022:11:54:33 +0000] “GET /pulp/api/v3/tasks/da429bda-f70c-40f2-80cb-1a95e384bc7b/ HTTP/1.1” 200 506 “-” “OpenAPI-Generator/3.9.0/ruby”

6

Does anyone have any ideas on this problem?

7

hammer doesn’t work either, he says:
Could not load the API description from the server:
403 “Forbidden”

  • is the server down?
  • was ‘foreman-rake apipie:cache’ run on the server when using apipie cache? (typical production settings)
    Warning: An error occured while loading module hammer_cli_foreman_bootdisk.
8

Is that the output for hammer ping? Does it help to restart the server?

9

Hello, no, restarting the server did not help, hammer ping said that everything was ok. At the moment, the problem has been solved by adding the heartbeat=120 value to the /etc/gofer/plugins/katello.conf config. However, the logs still contain errors like:gofer.messaging.adapter.proton.reliability:54 - connection ampqs://address:5647 disconnected: Condition(ampq:resource-limit-extendet, local-idle-timeout expired)

1 Like
10

Are you able to sync repos now? I am not sure how the heartbeat value for gofer plugin would help with repo sync. Not sure what would have caused the long sync tasks but could have to do with some other task taking up a lot of memory. Not really sure what happened here.

The ampq:resource-limit-extendet, local-idle-timeout expired is not of particular concern when it is followed by a successful connection. You can safely ignore that. (Goferd prints connection warning amqp:resource-limit-exceeded while installing or updating a bigger package - Red Hat Customer Portal)

11

already checked, everything works. All repositories are synchronized

1 Like