DEB repository synchronization results with Success but no content added

Problem:
I’ve successfully created a new Product and was able to add the Repository from http://us.archive.ubuntu.com/ubuntu bionic-updates main restricted universe multiverse. It is able to “synchronize” successfully (no errors generated Result reports back as success); however, in the Details section it reads the total number steps completed and shows No content added. What is preventing packages from being downloaded? Granted the steps vary here and there for yum packages, but I was able to get that synchronized successfully and have a repository ready and waiting to be used with content. deb seems to be the only one giving me issue.

Repository configuration is below:

Upstream URL: http://us.archive.ubuntu.com/ubuntu

Releases: bionic-updates/

Components: main restricted universe multiverse

Architecture: I’ve tried both No restriction and just leaving it blank.

Verify SSL: Yes/No; Tried both

And I’ve tried this both with and without the GPG key.

No SSL info specified.

Download Policy: Immediate

Mirroring Policy: Content Only

Expected outcome:
That content begins to download from the specified Components from within the specified Release directories.

Foreman and Proxy versions:
3.3.0-develop

Foreman and Proxy plugin versions:

Distribution and version:
Rocky Linux 8.5 is hosting the Foreman Server.

Other relevant data:
Sample output of the Details section after each synchronization:

No content added.
Total steps: 6/6
--------------------------------
Associating Content: 1/1
Downloading Artifacts: 3/3
Un-Associating Content: 1/1
Update PackageIndex units: 0/0
Update ReleaseFile units: 1/1

Tried same as above on brand new server using Stable 3.2 version. What is causing content to not be added even though it’s reaching everything as it’s supposed to? Documentation provides no explanation for this from what I can find. Would greatly appreciate some insight/input on this.

Repository configuration is below:

Upstream URL: http://us.archive.ubuntu.com/ubuntu/dists/

Releases: bionic-updates/

Components: main restricted universe multiverse

Architecture: Left blank.

Verify SSL: Yes

No GPG key.

No SSL info specified.

Download Policy: Immediate

Mirroring Policy: Content Only

Log Output

No content added.
Total steps: 13/13
--------------------------------
Associating Content: 9/9
Downloading Artifacts: 3/3
Update PackageIndex units: 0/0
Update ReleaseFile units: 1/1

URL should be http://us.archive.ubuntu.com/ubuntu and release with /. There was bug where you couldn’t specify multiple components, but not sure if it was fixed. You can leave it blank. Then sync, and it should work

I changed my Upsream URL as you suggested and updated the Releases field to dists/focal/. Otherwise, it wouldn’t know where to go to retrieve the Release file. And also as you suggested, I left the Components field blank. It’s still saying No content added even though it synchronizes properly without errors.

So, now I’m running into the same issue with Foreman that I was running into with Uyuni - incomplete documentation and unstated bugs. These types of pitfalls should be notated in the documentation along with mitigation and work arounds for those trying to implement this. I’ve checked both the Stable 3.2 documentation and the Nightly Build 3.3.0. Both fail in this regard.

Is there any way of implementing Ubuntu to work with Foreman+Katello setup? Am I supposed to be using something else to manage this specific type of distro such as Puppet or Ansible? Or am I spinning my wheels and wasting my time? To be clear, if my frustration is coming out through my post, it’s not with you. I just expect if a dev team in general has technical write-ups in place that what is documented should straight up work, especially when it’s being followed to the letter.

Keep the release to just focal, pulp_deb knows how to navigate the folder structure.

Before change

Basic Information
Name Ubuntu Focal
Label Ubuntu_Focal
Type deb
Sync Settings
Upstream URL http://us.archive.ubuntu.com/ubuntu
Releases dists/focal/

^ This setup states a successful synchronization, likely because it finds the Release file successfully, but fails to pull any content.

After suggested change

Basic Information
Name Ubuntu Focal
Label Ubuntu_Focal
Type deb
Sync Settings
Upstream URL http://us.archive.ubuntu.com/ubuntu
Releases focal

Output Details

No content added.
Total steps: 21/21
--------------------------------
Associating Content: 13/13
Downloading Artifacts: 7/7
Update PackageIndex units: 0/0
Update ReleaseFile units: 1/1

Errors:
No suitable Package index file found in 'dists/focal/main/binary-arm64'. If you are syncing from a partial mirror, try setting ignore_missing_package_indices to 'True' on your remote.

Upon searching for that setting, I came across this thread, where the solution by OP is stated as such,

The RHEL repos are working again all of a sudden without doing anything. :slight_smile:

For what is supposed to provide an enterprise-wide solution, this is unacceptable. And according to Pulp’s own bug tracking, this issue should have been fixed a year ago.

Deleted Repository Entirely and Followed [Foreman Documentation] Again to the Letter (https://docs.theforeman.org/nightly/Content_Management_Guide/index-katello.html#Adding_Custom_DEB_Repositories_content-management)

You can find the upstream URLs on Debian-based systems in /etc/apt/sources.list.

So I cat that very file.
http://us.archive.ubuntu.com/ubuntu is the upstream URL displayed; so, I paste that into the Upstream URL field.

  1. Optional: In the Releases field, set one or multiple releases.
    For official Debian repositories, set a codename in the Releases field, for example buster for Debian 10 or bullseye for Debian 11. Avoid using stable or testing because the codename they reference changes over time. This helps to avoid drastic changes once a new Debian version is released and the reference is changed. To keep things easy to manage and to avoid potential performance and network issues during synchronization, create one repository per release in Foreman. For official Ubuntu repositories, use the Ubuntu suite, for example focal or focal-updates.

CAUTION Ensure that you enter both Releases and Components exactly as they are in an /etc/apt/sources.list file.

Double-checked this and ensured the formatting is the same. I’ve tried both with and without Components, separating them with white space as instructed when creating the repository.

Output Details

No content added.
Total steps: 21/21
--------------------------------
Associating Content: 13/13
Downloading Artifacts: 7/7
Update PackageIndex units: 0/0
Update ReleaseFile units: 1/1

Errors:
No suitable Package index file found in 'dists/focal/main/binary-arm64'. If you are syncing from a partial mirror, try setting ignore_missing_package_indices to 'True' on your remote.

So, it would appear that Pulp does not in fact know how to navigate the folder structure. And this falls right in line with my other thread about the poor state of documentation and mitigations for errors regarding Ubuntu.

I performed another quoted search for that particular setting and came up with another page for Pulp Support Issue #9555. It appears this feature has been added with this issue being closed as of 3 months ago; however, if my assumption is correct, this feature has not been implemented when using Pulp with Foreman+Katello. I’m currently in the process of looking for this option so I can enable it and try again to see if the error persists.

Checking Pulp’s official website and finding their Workflow documentation, the following is revealed:

By default, syncs will fail if the upstream repo is missing package indices that are present in the Release file. This breaks synchronization from partial mirrors, and can be overriden by setting ignore_missing_package_indices=True on the remote. Alternatively, use FORCE_IGNORE_MISSING_PACKAGE_INDICES=True in your Pulp configuration file, to force this behaviour for all remotes.

Issuing the command pulp reveals the available sub-commands, one of which is config and then edit.

Issue the following commands pulp config edit brings up the Pulp configuration file in vi.
I append FORCE_IGNORE_MISSING_PACKAGE_INDICES=True to the Pulp configuration file, hit Esc, and issue the write function :w and receive the following result:
"/tmp/editor-6f3v2une.txt" 4L, 145C written

After issuing the quit :q function, I receive the following message:

Invalid date or number (line 4 column 1 char 103)
Retry [y/N]:

I’m currently scouring their documentation further to confirm whether or not that is indeed the correct configuration file to edit and what the proper syntax would be because taking it at face value is apparently not working. Why this seems to be par for the course for any sort of Linux documentation in general is beyond me.

Documenting my process here for posterity in case any other poor soul runs into this frustrating issue.

As stated in my previous post, I tried appending the FORCE_IGNORE_MISSING_PACKAGE_INDICES=True to the Pulp config file using the command pulp config edit. After further digging and hurdling over broken links in Pulp’s own documentation website, I found out the above variable is in all CAPS because of this caveat mentioned on Dynaconf’s homepage:

  • .py - Not Recommended but supported for backwards compatibility.
    settings.py
NAME = "Bruno"

:warning: on .py files dynaconf only read UPPERCASE variables.

If you’re wondering where I pulled Dynaconf from:

Pulpcore > Configuration > Applying Settings

Pulp uses dynaconf for its settings which allows you to configure Pulp in a few ways:

That link for dynaconf is the broken link I mentioned earlier.

Anyway, I’ve tried several changes, including chgrp and chmod as detailed here along with exporting variables as detailed on the same page. None of it has resolved this issue. I’ve gone as far as editing the /etc/pulp/settings.py file with both variations of the variable mentioned above. None of it made any difference.

The original error No suitable Package index file found in 'dists/focal/main/binary-arm64'. If you are syncing from a partial mirror, try setting ignore_missing_package_indices to 'True' on your remote. persists, making The Foreman practically useless to me. Would greatly appreciate some insight from the devs as to whether this is a bug I’ve stumbled upon or if there is better documentation I should be following.

Thank you.

Based on documentation and GUI info, Repository Configuration should be something like this:

Upstream URL: http://us.archive.ubuntu.com/ubuntu/

Releases: bionic-updates

Components: main restricted universe multiverse

And rest of the fields can remain blank/default.

Regarding ignore_missing_package_indices to 'True', you need to add it to /etc/pulp/settings.py as a root and add FORCE_IGNORE_MISSING_PACKAGE_INDICES=True and restart your pulp environment.

Entered the variable into /etc/pulp/settings.py already as shown below.
image

Restarted every single service associated with the word pulp and ensured they’re all active and enabled.
image

Sync Settings are already set to what you suggested and as specified in documentation.

Showing you my next step is Sync Now.
image

Starts off green showing Pending and then goes orange with warning as shown below. No content added. Fails synchronization.

Expanding the More Details reveals the same error that the variable above is supposed to prevent.

Looks like we may have found the issue after seeing your response on GitHub…

image

If these two services are crucial for this to work, then that begs the question as to why they’re not included in the Foreman+Katello installation in the first place.

The documentation on the services might be outdated IIRC. As long as you have the pulpcore-api , pulpcore-content and pulpcore-worker* services, you should have a working pulp.

Unfortunately, I don’t have anything helpful to offer on the original sync issue you’re seeing. Hopefully it will be resolved on the pulp_deb issue.

Since I am only joining this discussion now, I am not certain I know exactly what has and hasn’t been tried.

However, I am fairly certain the original issue was caused by the trailing slash in the Releases: bionic-updates/ entry. Supplying a Release/distribution with a trailing slash indicates to pulp_deb that the repository should be interpreted as using “flat repo format”. However, http://us.archive.ubuntu.com/ubuntu uses structured format. I am surprised this did not throw an error, since it should have resulted in the sync looking for Release files in http://us.archive.ubuntu.com/ubuntu/bionic-updates/ instead of http://us.archive.ubuntu.com/ubuntu/dists/bionic-updates/, and the former does not exist on that mirror.

I should note, that “flat repo format” handling has seen a lot of rework, so I will need to re-test with the current pulp_deb main branch to see if such a sync now throws an error. If not then that is a bug.

If you try syncing the repo without the trailing slash, perhaps create a new repo for good measure, are you getting content then?

One more unrelated warning: We recently discovered a bug in pulpcore, that causes extreme performance issues for large APT/deb repo syncs. You may want to wait for the fix for Implementation of does_batch in DeclarativeContent is wrong · Issue #2557 · pulp/pulpcore · GitHub to be released, or else patch your pulpcore with the fix, or the above sync will probably take many hours to complete.

1 Like

I’ve completely deleted the Product and am creating a brand new Product and Repository via the Foreman Web UI. Screenshots below of the process and outcomes. I should note that the reason I used a trailing slash in the first place was because of the tool-tip provided when clicking the ? on the Releases/Distributions field:

A “distribution” provides the path from the repository root to the “Release” file
you want to access. Each distribution in the list should use /etc/apt/sources.list
syntax. For most official Debian and Ubuntu repositories, the distribution is equal
to either the codename or the suite. When syncing a repo using flat repository format
specify exactly one distribution, which must end with a “/”.

I’m certain I’ve tried both of these methods without content synchronizing successfully, but for the sake of being thorough and posterity, I’ll try both methods here and document the results. The global variable FORCE_IGNORE_MISSING_PACKAGE_INDICES=True has been added to /etc/pulp/settings.py. Also, this is after following your suggestion on GitHub for pulp_deb and all services have been properly restarted using the commands you provided.

Traditional Method

image

Results

More Details

Flat Repo Format

image

Results
States success but also No content added.

More Details

There is no need to keep testing with the trailing slash/treating it as a flat repo fromat, this repo is not flat!

You should not be getting the error you are getting if the ignore missing package indices setting was taking effect. Can you check on the about page what version of the python3-pulp-deb package is installed?

Also, you may be able to work around the problem of the missing package indices by explicitly filtering for architectures that do exist on the mirror. Setting any of the following might work:

Architectures: amd64  # Sync only amd64
Architectures: i386   # Sync only i386
Architectures: i386 amd64  # Sync both available architectures
python38-pulp-deb-2.16.1-1.el8.noarch

Went ahead and included the rest of the pulp list below along with Foreman and Katello versions in case they’re needed.

pulpcore-selinux-1.3.0-1.el8.x86_64
python38-pulp-ansible-0.10.1-1.el8.noarch
python38-pulp-certguard-1.5.1-1.el8.noarch
python38-pulp-cli-0.14.0-1.el8.noarch
python38-pulp-container-2.9.2-1.el8.noarch
python38-pulp-file-1.10.1-1.el8.noarch
python38-pulp-python-3.5.2-1.el8.noarch
python38-pulp-rpm-3.17.3-2.el8.noarch
python38-pulpcore-3.16.6-1.el8.noarch
Katello 4.5.0
Foreman 3.3.0

Traditional Method

Specified amd64. Seems to be doing a lot more than last time. I know for certain I tried specifying the architecture last time without success; however, that was before knowing about and implementing the global variable for the indices. I’ll drop another update once it completes and let you know if it’s a success or failure. If you’d like for me to test the Flat Repo Method as well with architecture specified, let me know and I’ll post results for that, too.

No need, that can’t possibly work (though it should just throw an error rather than doing some random nonsense, I will look into that).

Results of Traditional Method

Good news is it seems to have actually started synchronizing and downloading the repo.
Bad news is I will apparently need more space to store what is needed. I suppose this is where a Smart Proxy comes in to play. Thank you everyone who has replied and for your assistance through this ordeal.

So I went back and had it synchronize only the main Components to ensure it was only a space problem. It synchronized and everything is there. Screenshot below. Thanks again for the solution to the repo problem, @quba42.