Apt update errors on ubuntu 20.04 servers, "File name too long"

Problem:

Apt update is not working anymore on Ubuntu20.04 servers. Every repo tells “W: Problem unlinking the file /var/lib/apt/lists/partial/_pulp_deb_Default%5fOrganization_Library_Affichage%5fUbuntu%5f20-04_custom_Produit%5fUbuntu%5f20%5f04__%253Fcomp%253Dmain%252Cmultiverse%252Crestricted%252Cuniverse%2526rel%253D_dists__main_i18n_Translation-en.lz4 - PrepareFiles (36: File name too long)”

then:

E: Failed to fetch katello:///pulp/deb/Default_Organization/Library/Affichage_Ubuntu_20-04/custom/Produit_Ubuntu_20_04//%3Fcomp%3Dmain%2Cmultiverse%2Crestricted%2Cuniverse%26rel%3D/dists/default/all/binary-amd64/Packages 404 Not Found

I have the same error with every repos (focal-security, apt.atix.de, packages microsoft 20.04, openvox8-ubuntu20.04, focal, docker_ubuntu2004, focal-backport)

It was working well previously

I tried resyncing the repos and publishing the content-view.

Expected outcome:

Foreman and Proxy versions:

Foreman 3.17

Katello 4.19

Pulpcore 3.85

pulp-deb 3.7

Foreman and Proxy plugin versions:

Distribution and version:

Ubuntu 20.04 - focal

Other relevant data:

My repos are configured as this in rhsm.sources with Suites: default Components: all
Those parameters are properly defined with the working distributions: Suites: jammy-updates Components: main multiverse restricted universe
I tried to edit the file manually with Suites: focal-security Components: main multiverse restricted universe, but I had the same errors

# cat /etc/apt/sources.list.d/rhsm.sources

name: focal
baseurl: https:///pulp/deb/Default_Organization/Library/Affichage_Ubuntu_20-04/custom/Produit_Ubuntu_20_04/focal/%3Fcomp%3Dmain%2Cmultiverse%2Crestricted%2Cuniverse%26rel%3Dfocal
enabled: 1
gpgcheck: 1
gpgkey: https:///katello/api/v2/repositories/2978/gpg_key_content
sslverify: 1
sslcacert: /etc/rhsm/ca/katello-server-ca.pem
sslclientkey: /etc/pki/entitlement/2855718853301657961-key.pem
sslclientcert: /etc/pki/entitlement/2855718853301657961.pem
metadata_expire: 1
enabled_metadata: 0
arches: amd64
Types: deb
URIs: katello://2855718853301657961@/pulp/deb/Default_Organization/Library/Affichage_Ubuntu_20-04/custom/Produit_Ubuntu_20_04/focal/%3Fcomp%3Dmain%2Cmultiverse%2Crestricted%2Cuniverse%26rel%3Dfocal
Suites: default
Components: all
Trusted: yes
Architectures: amd64
id: Default_Organization_Produit_Ubuntu_20_04_focal

I tried RFC: Transitioning Katello to "structured APT (deb) content"
But:

#foreman-rake katello:enable_structured_content_for_deb[true]

rake aborted!
Don’t know how to build task ‘katello:enable_structured_content_for_deb’ (See the list of available tasks with rake --tasks)
Did you mean? katello:refresh_alternate_content_sources
/usr/share/gems/gems/rake-13.0.3/exe/rake:27:in `<top (required)>’
(See full trace by running task with --trace)

I also saw someone having a similar problem in Ubuntu Suites and Components - #2 by quba42

I also tried to add
$ cat /etc/pulp/server/plugins.conf.d/deb_distributor.json
{
“publish_default_release”: true
}
On the foreman server, and republish the content view but it did’t worked

Unfortunately, we are already aware of this issue.

Can you reduce the characters in Content View, Repository Name, etc.?

I renamed the CV to CVU20, and the focal repos to f, fs, and fbp. For the product, I would need to recreate everything, which would be very long.
I edited the source file like this:
name: fbp
baseurl: https:///pulp/deb/Default_Organization/Library/CVU20/custom/Produit_Ubuntu_20_04/focal-backport/%3Fcomp%3Dmain%2Cmultiverse%2Crestricted%2Cuniverse%26rel%3Dfocal-backports
enabled: 1
gpgcheck: 1
gpgkey: https:///katello/api/v2/repositories/2975/gpg_key_content
sslverify: 1
sslcacert: /etc/rhsm/ca/katello-server-ca.pem
sslclientkey: /etc/pki/entitlement/2855718853301657961-key.pem
sslclientcert: /etc/pki/entitlement/2855718853301657961.pem
metadata_expire: 1
enabled_metadata: 0
arches: amd64
Types: deb
URIs: katello://2855718853301657961@/pulp/deb/Default_Organization/Library/CVU20/custom/Produit_Ubuntu_20_04/fbp/%3Fcomp%3Dmain%2Cmultiverse%2Crestricted%2Cuniverse%26rel%3Dfocal-backports
Suites: default
Components: all
Trusted: yes
Architectures: amd64
id: Default_Organization_Produit_Ubuntu_20_04_focal-backport

I dont have “file name too long” errors, but the 404 are still here:
Ign:57 katello:///pulp/deb/Default_Organization/Library/Affichage_Ubuntu_20-04/custom/Produit_Ubuntu_20_04/openvox8-ubuntu20_04/%3Fcomp%3Dopenvox8%26rel%3Dubuntu20.04 default/all amd64 c-n-f Metadata
Err:13 katello:///pulp/deb/Default_Organization/Library/CVU20/custom/Produit_Ubuntu_20_04/fbp/%3Fcomp%3Dmain%2Cmultiverse%2Crestricted%2Cuniverse%26rel%3Dfocal-backports default/all amd64 Packages
404 Not Found
Ign:14 katello:///pulp/deb/Default_Organization/Library/CVU20/custom/Produit_Ubuntu_20_04/fbp/%3Fcomp%3Dmain%2Cmultiverse%2Crestricted%2Cuniverse%26rel%3Dfocal-backports default/all all Packages
Ign:18 katello:///pulp/deb/Default_Organization/Library/CVU20/custom/Produit_Ubuntu_20_04/f/%3Fcomp%3Dmain%2Cmultiverse%2Crestricted%2Cuniverse%26rel%3Dfocal default/all all Packages
Err:19 katello:///pulp/deb/Default_Organization/Library/CVU20/custom/Produit_Ubuntu_20_04/f/%3Fcomp%3Dmain%2Cmultiverse%2Crestricted%2Cuniverse%26rel%3Dfocal default/all amd64 Packages
404 Not Found
Ign:20 katello:///pulp/deb/Default_Organization/Library/CVU20/custom/Produit_Ubuntu_20_04/f/%3Fcomp%3Dmain%2Cmultiverse%2Crestricted%2Cuniverse%26rel%3Dfocal default/all Translation-en
Ign:23 katello:///pulp/deb/Default_Organization/Library/Affichage_Ubuntu_20-04/custom/Produit_Ubuntu_20_04/apt_atix_de/%3Fcomp%3Dmain%26rel%3Dstable default/all all Packages
Err:24 katello:///pulp/deb/Default_Organization/Library/Affichage_Ubuntu_20-04/custom/Produit_Ubuntu_20_04/apt_atix_de/%3Fcomp%3Dmain%26rel%3Dstable default/all amd64 Packages
404 Not Found
Ign:25 katello:///pulp/deb/Default_Organization/Library/Affichage_Ubuntu_20-04/custom/Produit_Ubuntu_20_04/apt_atix_de/%3Fcomp%3Dmain%26rel%3Dstable default/all Translation-en
Err:28 katello:///pulp/deb/Default_Organization/Library/Affichage_Ubuntu_20-04/custom/Produit_Ubuntu_20_04/packages_microsoft_20_04/%3Fcomp%3Dmain%26rel%3Dfocal default/all amd64 Packages
404 Not Found
Ign:29 katello:///pulp/deb/Default_Organization/Library/Affichage_Ubuntu_20-04/custom/Produit_Ubuntu_20_04/packages_microsoft_20_04/%3Fcomp%3Dmain%26rel%3Dfocal default/all all Packages
Err:33 katello:///pulp/deb/Default_Organization/Library/CVU20/custom/Produit_Ubuntu_20_04/fs/%3Fcomp%3Dmain%2Cmultiverse%2Crestricted%2Cuniverse%26rel%3Dfocal-security focal-security/main amd64 Packages
404 Not Found
Ign:34 katello:///pulp/deb/Default_Organization/Library/CVU20/custom/Produit_Ubuntu_20_04/fs/%3Fcomp%3Dmain%2Cmultiverse%2Crestricted%2Cuniverse%26rel%3Dfocal-security focal-security/main all Packages
Ign:53 katello:///pulp/deb/Default_Organization/Library/Affichage_Ubuntu_20-04/custom/Produit_Ubuntu_20_04/openvox8-ubuntu20_04/%3Fcomp%3Dopenvox8%26rel%3Dubuntu20.04 default/all all Packages
Err:54 katello:///pulp/deb/Default_Organization/Library/Affichage_Ubuntu_20-04/custom/Produit_Ubuntu_20_04/openvox8-ubuntu20_04/%3Fcomp%3Dopenvox8%26rel%3Dubuntu20.04 default/all amd64 Packages
404 Not Found
Ign:55 katello:///pulp/deb/Default_Organization/Library/Affichage_Ubuntu_20-04/custom/Produit_Ubuntu_20_04/openvox8-ubuntu20_04/%3Fcomp%3Dopenvox8%26rel%3Dubuntu20.04 default/all Translation-en
Reading package lists… Done
E: Failed to fetch katello:///pulp/deb/Default_Organization/Library/CVU20/custom/Produit_Ubuntu_20_04/fbp/%3Fcomp%3Dmain%2Cmultiverse%2Crestricted%2Cuniverse%26rel%3Dfocal-backports/dists/default/all/binary-amd64/Packages 404 Not Found
E: Failed to fetch katello:///pulp/deb/Default_Organization/Library/CVU20/custom/Produit_Ubuntu_20_04/f/%3Fcomp%3Dmain%2Cmultiverse%2Crestricted%2Cuniverse%26rel%3Dfocal/dists/default/all/binary-amd64/Packages 404 Not Found
E: Failed to fetch katello:///pulp/deb/Default_Organization/Library/Affichage_Ubuntu_20-04/custom/Produit_Ubuntu_20_04/apt_atix_de/%3Fcomp%3Dmain%26rel%3Dstable/dists/default/all/binary-amd64/Packages 404 Not Found
E: Failed to fetch katello:///pulp/deb/Default_Organization/Library/Affichage_Ubuntu_20-04/custom/Produit_Ubuntu_20_04/packages_microsoft_20_04/%3Fcomp%3Dmain%26rel%3Dfocal/dists/default/all/binary-amd64/Packages 404 Not Found
E: Failed to fetch katello:///pulp/deb/Default_Organization/Library/CVU20/custom/Produit_Ubuntu_20_04/fs/%3Fcomp%3Dmain%2Cmultiverse%2Crestricted%2Cuniverse%26rel%3Dfocal-security/dists/focal-security/main/binary-amd64/Packages 404 Not Found
E: Failed to fetch katello:///pulp/deb/Default_Organization/Library/Affichage_Ubuntu_20-04/custom/Produit_Ubuntu_20_04/openvox8-ubuntu20_04/%3Fcomp%3Dopenvox8%26rel%3Dubuntu20.04/dists/default/all/binary-amd64/Packages 404 Not Found
E: Some index files failed to download. They have been ignored, or old ones used instead.

Actually, the issue in the end is, that there is a limit of 255 characters for a filename. Therefore, apt is not able to delete the file.

@iballou we discussed multiple ways to fix this issue. We are unsure if this can ever happen for other content types.

To fix this issue, we thought about having a “short” URL which is auto-generated - like a hash of certain fields in the URI.

There are 2 ways to implement this:

1. adapt the method “generate_repo_path”

generate_repo_path in app/models/katello/repository.rb defines the repository’s
canonical relative_path.

  • Add a “use short URL” setting
  • New repos get a different primary pulp path, not an alias.
  • Existing long URLs stop matching unless every distribution is recreated and every downstream consumer is updated.
  • Candlepin content URLs, host bindings, proxy sync state, and stored repository references still need coordinated updates.
  • No “safe toggle” behavior. Turning the setting to use short URLS off later becomes a migration problem, not a presentation choice.

Advantage: it would be possible to activate this feature and then everything is using this short URLs.

Drawback: a migration need to run for old content. It will not work to activate / deactive this feature a lot.

2. Use second pulp distribution with short URL

  • Add a “use short URL” setting
  • The normal repository path is still generated and stored as before in app/models/katello/repository.rb
  • the short URL is send to candlepin
  • no urgend update is needed on the consumer
  • if the consumer runs sub-man repos and received the new, short URL, it will “just work”

Advantage:

  • No migration needed
  • keep long URLs valid
  • can be activated / deactivated easily. Actually, it would also be possible to have this on a “per repo base (or if we detected the URL will be pretty long)

Drawback:

  • The solution is distributed throughout Katello and not located in a central location like generate_repo_path