Apt update errors on ubuntu 20.04 servers, "File name too long"

noe · March 12, 2026, 9:28am

Problem:

Apt update is not working anymore on Ubuntu20.04 servers. Every repo tells “W: Problem unlinking the file /var/lib/apt/lists/partial/_pulp_deb_Default%5fOrganization_Library_Affichage%5fUbuntu%5f20-04_custom_Produit%5fUbuntu%5f20%5f04__%253Fcomp%253Dmain%252Cmultiverse%252Crestricted%252Cuniverse%2526rel%253D_dists__main_i18n_Translation-en.lz4 - PrepareFiles (36: File name too long)”

then:

E: Failed to fetch katello:///pulp/deb/Default_Organization/Library/Affichage_Ubuntu_20-04/custom/Produit_Ubuntu_20_04//%3Fcomp%3Dmain%2Cmultiverse%2Crestricted%2Cuniverse%26rel%3D/dists/default/all/binary-amd64/Packages 404 Not Found

I have the same error with every repos (focal-security, apt.atix.de, packages microsoft 20.04, openvox8-ubuntu20.04, focal, docker_ubuntu2004, focal-backport)

It was working well previously

I tried resyncing the repos and publishing the content-view.

Expected outcome:

Foreman and Proxy versions:

Foreman 3.17

Katello 4.19

Pulpcore 3.85

pulp-deb 3.7

Foreman and Proxy plugin versions:

Distribution and version:

Ubuntu 20.04 - focal

Other relevant data:

My repos are configured as this in rhsm.sources with Suites: default Components: all
Those parameters are properly defined with the working distributions: Suites: jammy-updates Components: main multiverse restricted universe
I tried to edit the file manually with Suites: focal-security Components: main multiverse restricted universe, but I had the same errors

# cat /etc/apt/sources.list.d/rhsm.sources

name: focal
baseurl: https:///pulp/deb/Default_Organization/Library/Affichage_Ubuntu_20-04/custom/Produit_Ubuntu_20_04/focal/%3Fcomp%3Dmain%2Cmultiverse%2Crestricted%2Cuniverse%26rel%3Dfocal
enabled: 1
gpgcheck: 1
gpgkey: https:///katello/api/v2/repositories/2978/gpg_key_content
sslverify: 1
sslcacert: /etc/rhsm/ca/katello-server-ca.pem
sslclientkey: /etc/pki/entitlement/2855718853301657961-key.pem
sslclientcert: /etc/pki/entitlement/2855718853301657961.pem
metadata_expire: 1
enabled_metadata: 0
arches: amd64
Types: deb
URIs: katello://2855718853301657961@/pulp/deb/Default_Organization/Library/Affichage_Ubuntu_20-04/custom/Produit_Ubuntu_20_04/focal/%3Fcomp%3Dmain%2Cmultiverse%2Crestricted%2Cuniverse%26rel%3Dfocal
Suites: default
Components: all
Trusted: yes
Architectures: amd64
id: Default_Organization_Produit_Ubuntu_20_04_focal

I tried RFC: Transitioning Katello to "structured APT (deb) content"
But:

#foreman-rake katello:enable_structured_content_for_deb[true]

rake aborted!
Don’t know how to build task ‘katello:enable_structured_content_for_deb’ (See the list of available tasks with rake --tasks)
Did you mean? katello:refresh_alternate_content_sources
/usr/share/gems/gems/rake-13.0.3/exe/rake:27:in `<top (required)>’
(See full trace by running task with --trace)

I also saw someone having a similar problem in Ubuntu Suites and Components - #2 by quba42

noe · March 12, 2026, 12:25pm

I also tried to add
$ cat /etc/pulp/server/plugins.conf.d/deb_distributor.json
{
“publish_default_release”: true
}
On the foreman server, and republish the content view but it did’t worked

Bernhard_Suttner · March 12, 2026, 2:23pm

Unfortunately, we are already aware of this issue.

Can you reduce the characters in Content View, Repository Name, etc.?

noe · March 12, 2026, 4:01pm

I renamed the CV to CVU20, and the focal repos to f, fs, and fbp. For the product, I would need to recreate everything, which would be very long.
I edited the source file like this:
name: fbp
baseurl: https:///pulp/deb/Default_Organization/Library/CVU20/custom/Produit_Ubuntu_20_04/focal-backport/%3Fcomp%3Dmain%2Cmultiverse%2Crestricted%2Cuniverse%26rel%3Dfocal-backports
enabled: 1
gpgcheck: 1
gpgkey: https:///katello/api/v2/repositories/2975/gpg_key_content
sslverify: 1
sslcacert: /etc/rhsm/ca/katello-server-ca.pem
sslclientkey: /etc/pki/entitlement/2855718853301657961-key.pem
sslclientcert: /etc/pki/entitlement/2855718853301657961.pem
metadata_expire: 1
enabled_metadata: 0
arches: amd64
Types: deb
URIs: katello://2855718853301657961@/pulp/deb/Default_Organization/Library/CVU20/custom/Produit_Ubuntu_20_04/fbp/%3Fcomp%3Dmain%2Cmultiverse%2Crestricted%2Cuniverse%26rel%3Dfocal-backports
Suites: default
Components: all
Trusted: yes
Architectures: amd64
id: Default_Organization_Produit_Ubuntu_20_04_focal-backport

I dont have “file name too long” errors, but the 404 are still here:
Ign:57 katello:///pulp/deb/Default_Organization/Library/Affichage_Ubuntu_20-04/custom/Produit_Ubuntu_20_04/openvox8-ubuntu20_04/%3Fcomp%3Dopenvox8%26rel%3Dubuntu20.04 default/all amd64 c-n-f Metadata
Err:13 katello:///pulp/deb/Default_Organization/Library/CVU20/custom/Produit_Ubuntu_20_04/fbp/%3Fcomp%3Dmain%2Cmultiverse%2Crestricted%2Cuniverse%26rel%3Dfocal-backports default/all amd64 Packages
404 Not Found
Ign:14 katello:///pulp/deb/Default_Organization/Library/CVU20/custom/Produit_Ubuntu_20_04/fbp/%3Fcomp%3Dmain%2Cmultiverse%2Crestricted%2Cuniverse%26rel%3Dfocal-backports default/all all Packages
Ign:18 katello:///pulp/deb/Default_Organization/Library/CVU20/custom/Produit_Ubuntu_20_04/f/%3Fcomp%3Dmain%2Cmultiverse%2Crestricted%2Cuniverse%26rel%3Dfocal default/all all Packages
Err:19 katello:///pulp/deb/Default_Organization/Library/CVU20/custom/Produit_Ubuntu_20_04/f/%3Fcomp%3Dmain%2Cmultiverse%2Crestricted%2Cuniverse%26rel%3Dfocal default/all amd64 Packages
404 Not Found
Ign:20 katello:///pulp/deb/Default_Organization/Library/CVU20/custom/Produit_Ubuntu_20_04/f/%3Fcomp%3Dmain%2Cmultiverse%2Crestricted%2Cuniverse%26rel%3Dfocal default/all Translation-en
Ign:23 katello:///pulp/deb/Default_Organization/Library/Affichage_Ubuntu_20-04/custom/Produit_Ubuntu_20_04/apt_atix_de/%3Fcomp%3Dmain%26rel%3Dstable default/all all Packages
Err:24 katello:///pulp/deb/Default_Organization/Library/Affichage_Ubuntu_20-04/custom/Produit_Ubuntu_20_04/apt_atix_de/%3Fcomp%3Dmain%26rel%3Dstable default/all amd64 Packages
404 Not Found
Ign:25 katello:///pulp/deb/Default_Organization/Library/Affichage_Ubuntu_20-04/custom/Produit_Ubuntu_20_04/apt_atix_de/%3Fcomp%3Dmain%26rel%3Dstable default/all Translation-en
Err:28 katello:///pulp/deb/Default_Organization/Library/Affichage_Ubuntu_20-04/custom/Produit_Ubuntu_20_04/packages_microsoft_20_04/%3Fcomp%3Dmain%26rel%3Dfocal default/all amd64 Packages
404 Not Found
Ign:29 katello:///pulp/deb/Default_Organization/Library/Affichage_Ubuntu_20-04/custom/Produit_Ubuntu_20_04/packages_microsoft_20_04/%3Fcomp%3Dmain%26rel%3Dfocal default/all all Packages
Err:33 katello:///pulp/deb/Default_Organization/Library/CVU20/custom/Produit_Ubuntu_20_04/fs/%3Fcomp%3Dmain%2Cmultiverse%2Crestricted%2Cuniverse%26rel%3Dfocal-security focal-security/main amd64 Packages
404 Not Found
Ign:34 katello:///pulp/deb/Default_Organization/Library/CVU20/custom/Produit_Ubuntu_20_04/fs/%3Fcomp%3Dmain%2Cmultiverse%2Crestricted%2Cuniverse%26rel%3Dfocal-security focal-security/main all Packages
Ign:53 katello:///pulp/deb/Default_Organization/Library/Affichage_Ubuntu_20-04/custom/Produit_Ubuntu_20_04/openvox8-ubuntu20_04/%3Fcomp%3Dopenvox8%26rel%3Dubuntu20.04 default/all all Packages
Err:54 katello:///pulp/deb/Default_Organization/Library/Affichage_Ubuntu_20-04/custom/Produit_Ubuntu_20_04/openvox8-ubuntu20_04/%3Fcomp%3Dopenvox8%26rel%3Dubuntu20.04 default/all amd64 Packages
404 Not Found
Ign:55 katello:///pulp/deb/Default_Organization/Library/Affichage_Ubuntu_20-04/custom/Produit_Ubuntu_20_04/openvox8-ubuntu20_04/%3Fcomp%3Dopenvox8%26rel%3Dubuntu20.04 default/all Translation-en
Reading package lists… Done
E: Failed to fetch katello:///pulp/deb/Default_Organization/Library/CVU20/custom/Produit_Ubuntu_20_04/fbp/%3Fcomp%3Dmain%2Cmultiverse%2Crestricted%2Cuniverse%26rel%3Dfocal-backports/dists/default/all/binary-amd64/Packages 404 Not Found
E: Failed to fetch katello:///pulp/deb/Default_Organization/Library/CVU20/custom/Produit_Ubuntu_20_04/f/%3Fcomp%3Dmain%2Cmultiverse%2Crestricted%2Cuniverse%26rel%3Dfocal/dists/default/all/binary-amd64/Packages 404 Not Found
E: Failed to fetch katello:///pulp/deb/Default_Organization/Library/Affichage_Ubuntu_20-04/custom/Produit_Ubuntu_20_04/apt_atix_de/%3Fcomp%3Dmain%26rel%3Dstable/dists/default/all/binary-amd64/Packages 404 Not Found
E: Failed to fetch katello:///pulp/deb/Default_Organization/Library/Affichage_Ubuntu_20-04/custom/Produit_Ubuntu_20_04/packages_microsoft_20_04/%3Fcomp%3Dmain%26rel%3Dfocal/dists/default/all/binary-amd64/Packages 404 Not Found
E: Failed to fetch katello:///pulp/deb/Default_Organization/Library/CVU20/custom/Produit_Ubuntu_20_04/fs/%3Fcomp%3Dmain%2Cmultiverse%2Crestricted%2Cuniverse%26rel%3Dfocal-security/dists/focal-security/main/binary-amd64/Packages 404 Not Found
E: Failed to fetch katello:///pulp/deb/Default_Organization/Library/Affichage_Ubuntu_20-04/custom/Produit_Ubuntu_20_04/openvox8-ubuntu20_04/%3Fcomp%3Dopenvox8%26rel%3Dubuntu20.04/dists/default/all/binary-amd64/Packages 404 Not Found
E: Some index files failed to download. They have been ignored, or old ones used instead.

Bernhard_Suttner · March 12, 2026, 9:07pm

Actually, the issue in the end is, that there is a limit of 255 characters for a filename. Therefore, apt is not able to delete the file.

@iballou we discussed multiple ways to fix this issue. We are unsure if this can ever happen for other content types.

To fix this issue, we thought about having a “short” URL which is auto-generated - like a hash of certain fields in the URI.

There are 2 ways to implement this:

1. adapt the method “generate_repo_path”

generate_repo_path in app/models/katello/repository.rb defines the repository’s
canonical relative_path.

Add a “use short URL” setting
New repos get a different primary pulp path, not an alias.
Existing long URLs stop matching unless every distribution is recreated and every downstream consumer is updated.
Candlepin content URLs, host bindings, proxy sync state, and stored repository references still need coordinated updates.
No “safe toggle” behavior. Turning the setting to use short URLS off later becomes a migration problem, not a presentation choice.

Advantage: it would be possible to activate this feature and then everything is using this short URLs.

Drawback: a migration need to run for old content. It will not work to activate / deactive this feature a lot.

2. Use second pulp distribution with short URL

Add a “use short URL” setting
The normal repository path is still generated and stored as before in app/models/katello/repository.rb
the short URL is send to candlepin
no urgend update is needed on the consumer
if the consumer runs sub-man repos and received the new, short URL, it will “just work”

Advantage:

No migration needed
keep long URLs valid
can be activated / deactivated easily. Actually, it would also be possible to have this on a “per repo base (or if we detected the URL will be pretty long)

Drawback:

The solution is distributed throughout Katello and not located in a central location like generate_repo_path

iballou · March 13, 2026, 4:04pm

Hello folks,

The best workaround that I can see for now is to rename everything, as annoying as it is.

@Bernhard_Suttner ,

It looks like this was an issue in the past for even Red Hat content: 1417752

Some questions come to mind.

How is a file being created with greater than 255 chars but cannot be deleted? Should Pulp be throwing a warning at publication/distribution time?

Using a short URL to override the default path is an interesting idea. In fact, we use this concept today for containers. On the lifecycle environment details page, there’s a place to configure the registry naming pattern, which lets users shorten the naming that is super verbose by default.

Related to my questions above, I’d like to know what should alert a user that they need to use a short URL. Ideally it would be without going through painful debugging first. Is there a chance Pulp could detect this and throw an error?

Where should this new setting go, and should it be specific per repository type? The container setting is per-lifecycle because it needs to work for both Library/Default Org View repos and content view repositories.

Per LCE though may mean that we’d need a split for each repository type. I don’t think a user would want all path-based repositories to change.

We also need to be careful about distributions clashing - we have logic in the container implementation of this idea.

All of these considerations makes me wonder if the setting should be as close to a single repository as possible to not allow users to do something dangerous. It could also keep the logic simpler. The question is just then about how to deal with the path needing to change when a repo is published in a CV.

For implementing actually updating the path, one place to look to for inspiration could be the UpdateContentUrls action. Sometimes Red Hat repositories change names/paths so we need to propagate those changes through Katello.

I think I’m generally more a fan of (1) - update the distribution instead of having a second “working” one while letting the broken one sit around. But, let’s see how the discussion goes.

I’ll keep my thoughts at this for now

iballou · March 13, 2026, 4:20pm

New feature idea aside, I’m trying to figure out what the offending file was - want to make sure the shortened path idea would be really necessary.

/var/lib/apt/lists/partial/_pulp_deb_Default%5fOrganization_Library_Affichage%5fUbuntu%5f20-04_custom_Produit%5fUbuntu%5f20%5f04__%253Fcomp%253Dmain%252Cmultiverse%252Crestricted%252Cuniverse%2526rel%253D_dists__main_i18n_Translation-en.lz4

The path should allow for more than 255 chars, and in this case the file name is _pulp_deb_Default%5fOrganization_Library_Affichage%5fUbuntu%5f20-04_custom_Produit%5fUbuntu%5f20%5f04__%253Fcomp%253Dmain%252Cmultiverse%252Crestricted%252Cuniverse%2526rel%253D_dists__main_i18n_Translation-en.lz4. That name is less than 255 chars, so I think I’m missing something.

Does structured APT help here at all by utilizing paths more? (I know it was mentioned earlier)

Bernhard_Suttner · March 13, 2026, 4:32pm

The file is not created by pulp. apt is creating the file during “apt-get update”. Later on, it tries to update the file and tries to unlink the file first. then it throws this error.

File name length:

Or have a look at /usr/include/linux/limits.h

#define NAME_MAX         255	/* # chars in a file name */

For the mentioned second solution, this would mean:

The current idea is, that there is a global setting. So, if you have a pretty long hostname, organization name, location name and content views you can activate it.

There are 2 possible ways:

activate this setting only, if you recognize such issues - if you then activate it, and you publish a new content view version, it should auto-heal the next time the consumer grabs the repository information
activate this setting by default.

If the short URL feature is activated, it will generate short URL and human readable URLs at the same time but provide the short URLs to the consumers. In your browser you still can browse throw the human readable paths as you like it.

The first solution has one big drawback: you need to update all your repositories.

Dirk · March 13, 2026, 6:18pm

Should not be a problem I think. If using subscription-manager the rhsm.sources is recreated automatically. If configured manual you would not have used so long URLs (while possible, I have seen manual configuration most of the time only with near to no content views, and in environments with more structure there it is managed by configuration management). So I think this should not stop us using the first solution if we think it is the best.

And I am also more a fan of this solution.

iballou · March 13, 2026, 7:14pm

What’s confusing me here is how the too-long file ended up on the filesystem in the first place. The unlinking step is failing rather than the creation part.

But anyway, okay, so apt generates this file and it’s unrelated enough to what Pulp serves that it wouldn’t be able to tell if the file length limit is being reached. If that’s the case, then it would be hard for Pulp to trigger a warning.

I missed this bit before, I think a hash would work. It’s a bit similar to how Pulp stores artifacts now compared to Pulp 2. That is a lot simpler to implement than something user-specified.

Smart proxy syncing is a consideration too. Perhaps nothing to worry about, but worth a check early on.

Since cases of too-long filenames are rare (as far as I’m aware), introducing this as an opt-in feature seems less risky.

I know we have debian content in mind here because of the ubuntu flavor of the user issue, but are there special considerations we need to consider with the other repo types?

Container repositories probably shouldn’t have the feature applied to it.
RPM content could benefit, maybe with a caveat for Red Hat repos.
OSTree, Python, and Ansible I’m not so sure about
File is probably just fine with shortened URLs.

This is probably a good case for a quick proof of concept. My assumption is that subscription-manger would help handle this gracefully on clients after everything is updated in Katello, Pulp, and Candlepin.

iballou · March 13, 2026, 7:25pm

Is there a spec or some documentation about which files’ filenames apt sticks all of these attributes into? Maybe it’s only a problem because Katello is verbose in distribution paths, but that sounds like it could be generally problematic for some mirrors out there.
Have any other repo management tools solved this for deb by creating shorter distribution paths?

Bernhard_Suttner · March 13, 2026, 8:10pm

Katello provides the URL of the repository. subscription-manager recieves this URL and writes it to rhsm.sources. When apt-get update runs, it will convert the URI, including hostname, /pulp/content///custom//… to a file name:

https://salsa.debian.org/apt-team/apt/-/blob/main/apt-pkg/contrib/strutl.cc#L521

if you run apt-get update again it tries to update the files and therefore removes the file using:

https://salsa.debian.org/apt-team/apt/-/blob/main/apt-pkg/contrib/fileutl.cc#L216

Unfortunately, this is not something we can adapt. I would have prefered this path,too.

As we have learned in the past, the best documentation is often the source code see above.

If there is a apt sources like:

URIs: http://security.ubuntu.com/ubuntu
Suites: plucky-security
Components: main restricted universe multiverse

it does generate files:

security.ubuntu.com_ubuntu_dists_plucky-security_InRelease
security.ubuntu.com_ubuntu_dists_plucky-security_main_binary-amd64_Packages
security.ubuntu.com_ubuntu_dists_plucky-security_main_binary-i386_Packages

[_dists_<release/suite>_REPO_METADATA]

We are only able to reduce the length of the . Everything in […] is apt-managed and I don’t think we have any chance.

we are not the only running into this issue:

looks likes, this was not created as PR or was not merged.

iballou · March 19, 2026, 7:28pm

Thanks for the details, so this issue at the moment seems pretty apt-specific, but I think other content types could benefit from the shortening.

During implementation, I’d recommend adding a setting to repository types, a boolean maybe, to configure if a type is “distribution path shortening” capable. We can start with Debian shortening, and then do some testing on other content types like file, yum, python, ostree, and ansible.

Bernhard_Suttner · March 19, 2026, 7:35pm

Good idea.

Started to work on it already: Draft: Repo URL shortener by sbernhard · Pull Request #11678 · Katello/katello · GitHub

noe · March 27, 2026, 2:53pm

In the end, my problem wasn’t really about the file name to long, but the non structured apt. I tried modifying the rhsm.sources file from this structure:

name: packages microsoft 20.04
baseurl: https://.dit.cb/pulp/deb/Default_Organization/Library/Affichage_Ubuntu_20-04/custom/Produit_Ubuntu_20_04/packages_microsoft_20_04/%3Fcomp%3Dmain%26rel%3Dfocal
enabled: 1
gpgcheck: 1
gpgkey: https://.dit.cb/katello/api/v2/repositories/2973/gpg_key_content
sslverify: 1
sslcacert: /etc/rhsm/ca/katello-server-ca.pem
sslclientkey: /etc/pki/entitlement/2475434634231457002-key.pem
sslclientcert: /etc/pki/entitlement/2475434634231457002.pem
metadata_expire: 1
enabled_metadata: 0
arches: amd64
Types: deb
URIs: katello://2475434634231457002@.dit.cb/pulp/deb/Default_Organization/Library/Affichage_Ubuntu_20-04/custom/Produit_Ubuntu_20_04/packages_microsoft_20_04/%3Fcomp%3Dmain%26rel%3Dfocal
Suites: default
Components: all
Trusted: yes
Architectures: amd64
id: Default_Organization_Produit_Ubuntu_20_04_packages_microsoft_20_04

To this one:

name: packages microsoft 20.04
baseurl: https://.dit.cb/pulp/deb/Default_Organization/Library/Affichage_Ubuntu_20-04/custom/Produit_Ubuntu_20_04/packages_microsoft_20_04/%3Fcomp%3Dmain%26rel%3Dfocal
enabled: 1
gpgcheck: 1
gpgkey: https://.dit.cb/katello/api/v2/repositories/2973/gpg_key_content
sslverify: 1
sslcacert: /etc/rhsm/ca/katello-server-ca.pem
sslclientkey: /etc/pki/entitlement/2475434634231457002-key.pem
sslclientcert: /etc/pki/entitlement/2475434634231457002.pem
metadata_expire: 1
enabled_metadata: 0
arches: amd64
Types: deb
URIs: katello://2475434634231457002@.dit.cb/pulp/deb/Default_Organization/Library/Affichage_Ubuntu_20-04/custom/Produit_Ubuntu_20_04/packages_microsoft_20_04/
Suites: focal
Components: main
Trusted: yes
Architectures: amd64
id: Default_Organization_Produit_Ubuntu_20_04_packages_microsoft_20_04

so I just removed the last part of the URIs, and defined the Suites and Components. I made this for every repo and then the update is OK.

What I don’t get, is why don’t these options don’t change when I define the suites and components in my foreman GUI?

I Changed all my repos in the product like this:

Releases/Distributions :focal

Components : main multiverse restricted universe

Architectures : amd64

Then I resynced the repos, republish the CV and refreshed the subscription manager on my target host, but the suites and Components option stay the same. After a manual change, when I refresh the subscription-manager, these options stays like this and don’t come back to “default” “all”.

Could you tell if I’m doing something wrong?

Thanks

quba42 · March 30, 2026, 9:22am

You should not have to manually edit the rhsm.sources file. A sufficiently new version of subscription-manager should do this automatically. Try backing up your manually edited rhsm.sources file, and then run subscription-manager refresh to make sure everything is up to date the way your currently installed subscription-manager wants it. If this reverts your configuration to using the URIs: field with %3Fcomp%3Dmain%26rel%3Dfocal at the end of the URL, then I suspect you need a newer version of subscritpion-manager.

It is possible the subscritpion-manger for Ubuntu 20 at Index of /Ubuntu20LTS does not have the required feature. (I would need to investigate further)

I have tested subscription-manager for Ubuntu 22 at Index of /Ubuntu22LTS does have the reuquired feature.

What version of the subscription-manager package is installed on your host?