Create Ubuntu repository, like nagios ncpa

Support
1

Problem:
I want to add a Ubuntu repository for Nagios NCPA software.
Sync Settings
Upstream URL: https://repo.nagios.com/deb
Releases: focal
Components:
Architectures: amd64

When I start the sync, it fails.

Expected outcome:
I expected the repository to be synced

Foreman and Proxy versions:
2.5.1

Foreman and Proxy plugin versions:
|foreman-tasks|4.1.2|
|foreman_ansible|6.3.0|
|foreman_bootdisk|17.1.0|
|foreman_remote_execution|4.5.0|
|foreman_scc_manager||1.8.9|
|katello|4.1.0|

Distribution and version:
CentOS 7.9

Other relevant data:

Jul 7 20:49:26 foreman pulpcore-worker-2: pulp [c71c255b-7a7c-4344-b6e0-03e93c58abc7]: pulp_deb.app.tasks.synchronizing:INFO: Downloading Release file for distribution: “focal”
Jul 7 20:49:26 foreman pulpcore-api: pulp [c71c255b-7a7c-4344-b6e0-03e93c58abc7]: - - [07/Jul/2021:18:49:26 +0000] “GET /pulp/api/v3/tasks/343ae9fa-5ffe-4794-a1cf-d19ca727125f/ HTTP/1.1” 200 1267 “-” “OpenAPI-Generator/3.11.0/ruby”
Jul 7 20:49:26 foreman pulpcore-worker-2: Giving up _run(…) after 1 tries (aiohttp.client_exceptions.ClientResponseError: 404, message=‘Not Found’, url=URL(‘https://repo.nagios.com/deb/dists/focal/Release’))
Jul 7 20:49:26 foreman pulpcore-worker-2: pulp [c71c255b-7a7c-4344-b6e0-03e93c58abc7]: backoff:ERROR: Giving up _run(…) after 1 tries (aiohttp.client_exceptions.ClientResponseError: 404, message=‘Not Found’, url=URL(‘https://repo.nag
ios.com/deb/dists/focal/Release’))
Jul 7 20:49:26 foreman pulpcore-worker-2: pulp [c71c255b-7a7c-4344-b6e0-03e93c58abc7]: pulp_deb.app.tasks.synchronizing:INFO: Artifact not found. Ignored
Jul 7 20:49:26 foreman pulpcore-worker-2: Giving up _run(…) after 1 tries (aiohttp.client_exceptions.ClientResponseError: 404, message=‘Not Found’, url=URL(‘https://repo.nagios.com/deb/dists/focal/Release.gpg’))
Jul 7 20:49:26 foreman pulpcore-worker-2: pulp [c71c255b-7a7c-4344-b6e0-03e93c58abc7]: backoff:ERROR: Giving up _run(…) after 1 tries (aiohttp.client_exceptions.ClientResponseError: 404, message=‘Not Found’, url=URL(‘https://repo.nag
ios.com/deb/dists/focal/Release.gpg’))
Jul 7 20:49:26 foreman pulpcore-worker-2: pulp [c71c255b-7a7c-4344-b6e0-03e93c58abc7]: pulp_deb.app.tasks.synchronizing:INFO: Artifact not found. Ignored
Jul 7 20:49:26 foreman pulpcore-worker-2: Giving up _run(…) after 1 tries (aiohttp.client_exceptions.ClientResponseError: 404, message=‘Not Found’, url=URL(‘https://repo.nagios.com/deb/dists/focal/InRelease’))
Jul 7 20:49:26 foreman pulpcore-worker-2: pulp [c71c255b-7a7c-4344-b6e0-03e93c58abc7]: backoff:ERROR: Giving up _run(…) after 1 tries (aiohttp.client_exceptions.ClientResponseError: 404, message=‘Not Found’, url=URL(‘https://repo.nag
ios.com/deb/dists/focal/InRelease’))
Jul 7 20:49:26 foreman pulpcore-worker-2: pulp [c71c255b-7a7c-4344-b6e0-03e93c58abc7]: pulp_deb.app.tasks.synchronizing:INFO: Artifact not found. Ignored
Jul 7 20:49:27 foreman pulpcore-worker-2: pulp [c71c255b-7a7c-4344-b6e0-03e93c58abc7]: rq.worker:ERROR: Traceback (most recent call last):
Jul 7 20:49:27 foreman pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/rq/worker.py”, line 975, in perform_job
Jul 7 20:49:27 foreman pulpcore-worker-2: rv = job.perform()
Jul 7 20:49:27 foreman pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/rq/job.py”, line 696, in perform
Jul 7 20:49:27 foreman pulpcore-worker-2: self._result = self._execute()
Jul 7 20:49:27 foreman pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/rq/job.py”, line 719, in _execute
Jul 7 20:49:27 foreman pulpcore-worker-2: return self.func(*self.args, **self.kwargs)
Jul 7 20:49:27 foreman pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/pulp_deb/app/tasks/synchronizing.py”, line 122, in synchronize
Jul 7 20:49:27 foreman pulpcore-worker-2: DebDeclarativeVersion(first_stage, repository, mirror=mirror).create()
Jul 7 20:49:27 foreman pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py”, line 147, in create
Jul 7 20:49:27 foreman pulpcore-worker-2: loop.run_until_complete(pipeline)
Jul 7 20:49:27 foreman pulpcore-worker-2: File “/usr/lib64/python3.6/asyncio/base_events.py”, line 484, in run_until_complete
Jul 7 20:49:27 foreman pulpcore-worker-2: return future.result()
Jul 7 20:49:27 foreman pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py”, line 225, in create_pipeline
Jul 7 20:49:27 foreman pulpcore-worker-2: await asyncio.gather(*futures)
Jul 7 20:49:27 foreman pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py”, line 43, in call
Jul 7 20:49:27 foreman pulpcore-worker-2: await self.run()
Jul 7 20:49:27 foreman pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/pulp_deb/app/tasks/synchronizing.py”, line 269, in run
Jul 7 20:49:27 foreman pulpcore-worker-2: raise NoReleaseFile(distribution=release_file.distribution)
Jul 7 20:49:27 foreman pulpcore-worker-2: pulp_deb.app.tasks.synchronizing.NoReleaseFile: No valid Release file found for ‘focal’.
Jul 7 20:49:27 foreman pulpcore-worker-2: Traceback (most recent call last):
Jul 7 20:49:27 foreman pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/rq/worker.py”, line 975, in perform_job
Jul 7 20:49:27 foreman pulpcore-worker-2: rv = job.perform()
Jul 7 20:49:27 foreman pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/rq/job.py”, line 696, in perform
Jul 7 20:49:27 foreman pulpcore-worker-2: self._result = self._execute()
Jul 7 20:49:27 foreman pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/rq/job.py”, line 719, in _execute
Jul 7 20:49:27 foreman pulpcore-worker-2: return self.func(*self.args, **self.kwargs)
Jul 7 20:49:27 foreman pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/pulp_deb/app/tasks/synchronizing.py”, line 122, in synchronize
Jul 7 20:49:27 foreman pulpcore-worker-2: DebDeclarativeVersion(first_stage, repository, mirror=mirror).create()
Jul 7 20:49:27 foreman pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py”, line 147, in create
Jul 7 20:49:27 foreman pulpcore-worker-2: loop.run_until_complete(pipeline)
Jul 7 20:49:27 foreman pulpcore-worker-2: File “/usr/lib64/python3.6/asyncio/base_events.py”, line 484, in run_until_complete
Jul 7 20:49:27 foreman pulpcore-worker-2: return future.result()
Jul 7 20:49:27 foreman pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py”, line 225, in create_pipeline
Jul 7 20:49:27 foreman pulpcore-worker-2: await asyncio.gather(*futures)
Jul 7 20:49:27 foreman pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py”, line 43, in call
Jul 7 20:49:27 foreman pulpcore-worker-2: await self.run()
Jul 7 20:49:27 foreman pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/pulp_deb/app/tasks/synchronizing.py”, line 269, in run
Jul 7 20:49:27 foreman pulpcore-worker-2: raise NoReleaseFile(distribution=release_file.distribution)
Jul 7 20:49:27 foreman pulpcore-worker-2: pulp_deb.app.tasks.synchronizing.NoReleaseFile: No valid Release file found for ‘focal’.
Jul 7 20:49:27 foreman pulpcore-api: pulp [c71c255b-7a7c-4344-b6e0-03e93c58abc7]: - - [07/Jul/2021:18:49:27 +0000] “GET /pulp/api/v3/tasks/343ae9fa-5ffe-4794-a1cf-d19ca727125f/ HTTP/1.1” 200 2582 “-” “OpenAPI-Generator/3.11.0/ruby”
Jul 7 20:49:28 foreman pulpcore-api: pulp [c71c255b-7a7c-4344-b6e0-03e93c58abc7]: - - [07/Jul/2021:18:49:28 +0000] “GET /pulp/api/v3/distributions/deb/apt/?base_path=JPP%2FLibrary%2Fcustom%2FUbuntu%2FUbuntu_nagios HTTP/1.1” 200 486 "-
" “OpenAPI-Generator/2.11.1/ruby”
Jul 7 20:49:28 foreman pulpcore-resource-manager: pulp [None]: rq.worker:INFO: resource-manager: e9960788-5bb0-4bf5-a13e-202f05f90c37
Jul 7 20:49:28 foreman pulpcore-api: pulp [c71c255b-7a7c-4344-b6e0-03e93c58abc7]: - - [07/Jul/2021:18:49:28 +0000] “PATCH /pulp/api/v3/distributions/deb/apt/524c35b4-07c7-4424-aa90-b351e1308bd0/ HTTP/1.1” 202 67 “-” “OpenAPI-Generator
/2.11.1/ruby”
Jul 7 20:49:28 foreman pulpcore-api: pulp [c71c255b-7a7c-4344-b6e0-03e93c58abc7]: - - [07/Jul/2021:18:49:28 +0000] “GET /pulp/api/v3/tasks/3cd070eb-f298-4164-bd32-b1536e7c8e87/ HTTP/1.1” 200 454 “-” “OpenAPI-Generator/3.11.0/ruby”
Jul 7 20:49:28 foreman pulpcore-worker-1: pulp [None]: rq.worker:INFO: 1164@foreman.jpp.nl: 3cd070eb-f298-4164-bd32-b1536e7c8e87
Jul 7 20:49:28 foreman pulpcore-resource-manager: pulp [c71c255b-7a7c-4344-b6e0-03e93c58abc7]: rq.worker:INFO: resource-manager: Job OK (e9960788-5bb0-4bf5-a13e-202f05f90c37)
Jul 7 20:49:28 foreman pulpcore-worker-1: pulp [c71c255b-7a7c-4344-b6e0-03e93c58abc7]: rq.worker:INFO: 1164@foreman.jpp.nl: Job OK (3cd070eb-f298-4164-bd32-b1536e7c8e87)
Jul 7 20:49:28 foreman pulpcore-api: pulp [c71c255b-7a7c-4344-b6e0-03e93c58abc7]: - - [07/Jul/2021:18:49:28 +0000] “GET /pulp/api/v3/tasks/3cd070eb-f298-4164-bd32-b1536e7c8e87/ HTTP/1.1” 200 562 “-” “OpenAPI-Generator/3.11.0/ruby”
Jul 7 20:49:39 foreman systemd: Created slice User Slice of jurgen.
Jul 7 20:49:39 foreman systemd-logind: New session 685 of user jurgen.
Jul 7 20:49:39 foreman systemd: Started Session 685 of user jurgen.

2

Error message in task:
Errors:

No valid Release file found for ‘focal’.

3

Looking at the instructions here: Nagios Repositories, it looks like this is a repository using flat repository format.

For Ubuntu focal, you want Upstream URL: https://repo.nagios.com/deb/focal and Releases: /.
The syntax for the Upstream URL and Releases (distributions) fields is based on what your clients need to place in their /etc/apt/sources.list file. According to the instructions on the Nagios website this is the following for Ubuntu:

echo "deb https://repo.nagios.com/deb/$(lsb_release -cs) /" > /etc/apt/sources.list.d/nagios.list

You can see the URL, followed by a space and then /, which marks this repo as using flat repo format.

Note: The Flat repo format support for pulp_deb is in tech preview, and has some known issues, so the sync may or may not work. (Please report back). I have started working on imporving the flat repo implementation here: Make flat repo syncs more robust by quba42 · Pull Request #298 · pulp/pulp_deb · GitHub I will test this work with the nagios repo.

4

The issue is, in my opinion, that katello expect the following map structure:
Nagios Repositoriesdists/focal with to focus on ‘dists’ The online repo does not provide this mpa structure.

5

If your Releases: field ends in a / Katello will interpret your entry as a flat repo and omit the dists/ folder from it’s search path.

What you need is:

Upstream URL: <base_repo_url>
Releases: <path_to_release_file>/

In your case <base_repo_url> = https://repo.nagios.com/deb/focal and <path_to_release_file> is just the empty string, resulting in:

Upstream URL: https://repo.nagios.com/deb/focal
Releases: /

This should work if the flat repo in question meets certain assumptions. Once the work on the PR I linked is done, it should work for any flat repo.

The way you put it in your original post is:

Upstream URL: https://repo.nagios.com/deb
Releases: focal

This will not work because Katello will look for the Release file at https://repo.nagios.com/deb/dists/focal/, which is wrong for a flat repo.

6

Note: You can read up about “flat repo format” here: DebianRepository/Format - Debian Wiki

7

Hello,

Oke, now I know what you mean with flat repo. Thanks for the explantion. I’m more a RedHat man, but Ubuntu is coming on my way :slight_smile: