"The repository's publication is missing" after manually stopping repository sync jobs

Support
1

Problem:
After stopping/cancelling repository sync im no longer able to publish/promote content views with those repos as that results in the error message:

The repository's publication is missing. Please run a 'complete sync' on Ubuntu 20.04 security.The repository's publication is missing. Please run a 'complete sync' on Ubuntu 20.04 updates.

Additionally i have a “Remove orphans” job also errored out, i suspect it is related as it showed up at the same time.

Error message: the server returns an error
HTTP status code: 400
Response headers: {"date"=>"Mon, 10 Feb 2025 14:07:00 GMT", "server"=>"gunicorn", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"GET, DELETE, HEAD, OPTIONS", "x-frame-options"=>"DENY", "content-length"=>"174", "x-content-type-options"=>"nosniff", "referrer-policy"=>"same-origin", "cross-origin-opener-policy"=>"same-origin", "correlation-id"=>"4b449c3a505b42ebbbfb9a5c34079aad", "access-control-expose-headers"=>"Correlation-ID", "via"=>"1.1 puppet.ncop.net", "connection"=>"close"}
Response body: ["The repository version cannot be deleted because it (or its publications) are currently being used to distribute content. Please update the necessary distributions first."]

I have tried running an advanced sync->Complete sync, but this did not help.

When googling the error i come across this support article for Sattelite, but we do not have said rake task… (Note the error message on satellite 6.11 is the same as mine)
Getting ‘{“publication”:[“Invalid hyperlink - Object does not exist.”]}’ error when publishing a content-view on connected Satellite or when importing contents to a disconnected Satellite - Red Hat Customer Portal

Expected outcome:
Repos shouldnt be broken when cancelling sync. I should be able to repair this somehow.
Content views should not be blocked from publish/promote.
Foreman and Proxy versions:
3.12.1, Katello 4.14.1
Foreman and Proxy plugin versions:

Distribution and version:
Rhel 9.5
Other relevant data:
The reason the sync was stopped was due to upgrading from 3.10 and because repo sync took a long time for the ubuntu repos on 3.10 (10-14 hours) I manually stopped the two remaining jobs.
On the bright side, after upgrading from 3.10 to 3.11 (and EL from 8 to 9) and then 3.12.1 i immediately saw an improvement as now its down to around 30 minutes.

2

Just adding that the content views im having problems with are composite content views. Im not able to edit my original post.

I tried deleting the composite view for Ubuntu 20.04 but im not able to do so because of the task that is trying to publish the same content view is locking it.
I could possibly unlock/force unlock it and cancelling, but i dont want to make matters worse…

3

Ok, i think i have solved the problem.

I ran:
foreman-rake katello:regenerate_repo_metadata
This seems to have worked. It generated a lot of tasks, many of which were seemingly identical.
I think it created new metadata and re-published content views but im not sure…
Tried reading the sourcecode but found it kinda hard to trace out what was going on.

4

Hey,
you could also try to repair the repos:
pulp rpm repository list —> Copy the name of the repo
pulp --force rpm repository version repair --repository RepoName

5

This happens when you get a partial sync. (The Pulp sync completes successfully and the resulting repository version is saved to Katello, but the Pulp publish fails/is not saved to Katello).

There is also a version of the same issue if this happens on a re-sync rather than on the initial sync.

In either case a normal re-sync of the affected repo should fix it, at least if you are on a Katello version with the following fix: Bug #36859: A re-sync should always recover from a previous syncs failed publication - Katello - Foreman

6

Hello,

Yeah it sounds about right that it was a partial sync. It was going when i stopped it and then cancelled the two jobs. This was in fact not the initial sync of the repos if that matters, it was the daily scheduled re-sync of all repos.

There was no clear way of fixing this through any GUI. This was a real struggle for me to figure out as i couldnt find much information when searching. If you have a better way than running foreman-rake katello:regenerate_repo_metadata then please be specific in case someone else happens across this problem in the future, then at least they will have this thread :slight_smile:

As i wrote:

Any attempt at re-syncing the repos would result in the same error and as mentioned i was on Katello 4.14.1 so your answer makes me think i came a cross a bug? But i dont know enough about the inner workings of this to classify it as such.

7

Thanks for the alternatives. Since i fixed this issue i cant test your solution, but if anyone else comes across this problem in the future then they can try :slight_smile:

8

This does sound like the fix I added either no longer works, or the case you hit is subtly different from the one we “fixed”. The idea behind our fix was that both a complte sync, but also a normal optimized sync should heal a situation like yours.

But even if this had worked in your case, I feel like it is still sub optimal. A better fix would be to prevent a partial sync from saving a partial result to the DB in the first place. The underlying issue is that the Katello repository model saves both a Pulp version_href and a Pulp publication_href during sync. If it only updates one of the two (for example because the sync was cancelled after the new Pulp repo version was created, but before the new Pulp publication was created), then that leaves the Katello repo in an inconsistent state. What should happen is that both values are updated during sync at the same time, and then saved to the DB on a single call. That way they are either both updated or both not updated for any given Katello sync. I may open a new issue.

1 Like