Katello deb repo out of sync with upstream

Problem:

A recent build failed the FreeIPA enrollment, digging into why, I found that the apt package table was listing a package that had been removed from the upstream Ubuntu repo. Hence the freeipa-client install failed.

The local Ubuntu product repo is configured as ‘on demand’. Looking at the logs I found the host was trying to download a removed package. It must have been recently updated on the upstream, so I figured a sync might be in order. However, nothing I’ve tried has worked yet.

  • Advanced sync >> complete sync
  • Verify Content Checksum
  • Republish Repo Metadata (still running)

I was hoping to avoid a full sync of the Ubuntu repos (to save some disk space), but I’m starting to think this may not be a good idea for deb repos?

Expected outcome:

‘On demand’ repo works, or will work again after a sync.

Foreman and Proxy versions:

v3.13.0

Foreman and Proxy plugin versions:

Foreman: 3.13.0
Katello: 4.15.0

Distribution and version:

Foreman: CentOS Stream 9
Client/host: Ubuntu 24.04

Other relevant data:
Pulpcore-content logs:

Feb 20 16:39:34 fm01.some.domain.com pulpcore-content[3050261]: pulp [None]: backoff:ERROR: Giving up download_wrapper(...) after 1 tries (aiohttp.client_exceptions.ClientResponseError: 404, message='Not Found', url='http://archive.ubuntu.com/ubuntu/pool/main/v/vim/xxd_9.1.0016-1ubuntu7.5_amd64.deb')
Feb 20 16:39:34 fm01.some.domain.com pulpcore-content[3050261]: pulp [None]: pulpcore.content.handler:WARNING: Could not download remote artifact at 'http://archive.ubuntu.com/ubuntu/pool/main/v/vim/xxd_9.1.0016-1ubuntu7.5_amd64.deb': 404, message='Not Found', url='http://archive.ubuntu.com/ubuntu/pool/main/v/vim/xxd_9.1.0016-1ubuntu7.5_amd64.deb'
Feb 20 16:39:34 fm01.some.domain.com pulpcore-content[3050261]: Giving up download_wrapper(...) after 1 tries (aiohttp.client_exceptions.ClientResponseError: 404, message='Not Found', url='http://archive.ubuntu.com/ubuntu/pool/main/v/vim/xxd_9.1.0016-1ubuntu7.5_amd64.deb')

Hi @dmgeurts ,

It sounds like Pulp might be hanging on to remote artifacts that no longer exist. @quba42 who works on the pulp-deb plugin might know about a bug in this area?

In any case, I think an issue here )GitHub · Where software is built) might be helpful.

I did talk to a user who was experiencing the same issue recently, but only for very large amounts of Debian content.

Thank you; Pulp offers outdated content · Issue #1240 · pulp/pulp_deb · GitHub Raised.