Alternate Content Source for fetching files?

Support
1

Problem:
I’m trying to use Alternate Content Source for fetching files from a webserver.

The log shows it’s trying to fetch PULP_MANIFEST, which isn’t at the location. So It’s highly likely I’m doing something wrong. Hence my first question is, can Foreman fetch deb/rpm packages from a web/file folder?

Expected outcome:
No errors where refreshing the source.

Foreman and Proxy versions:
Foreman v3.10.0
Katello v4.12.1

Distribution and version:
CentOS 8 stream

Other relevant data:

1 subtask(s) failed for task group /pulp/api/v3/task-groups/019123a0-2e4c-759e-8526-bc83e4556366/.
Errors:
 {"traceback"=>"  File \"/usr/lib/python3.11/site-packages/pulpcore/tasking/tasks.py\", line 61, in _execute_task\n    result = func(*args, **kwargs)\n             ^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib/python3.11/site-packages/pulp_file/app/tasks/synchronizing.py\", line 51, in synchronize\n    rv = dv.create()\n         ^^^^^^^^^^^\n  File \"/usr/lib/python3.11/site-packages/pulpcore/plugin/stages/declarative_version.py\", line 161, in create\n    loop.run_until_complete(pipeline)\n  File \"/usr/lib64/python3.11/asyncio/base_events.py\", line 653, in run_until_complete\n    return future.result()\n           ^^^^^^^^^^^^^^^\n  File \"/usr/lib/python3.11/site-packages/pulpcore/plugin/stages/api.py\", line 220, in create_pipeline\n    await asyncio.gather(*futures)\n  File \"/usr/lib/python3.11/site-packages/pulpcore/plugin/stages/api.py\", line 41, in __call__\n    await self.run()\n  File \"/usr/lib/python3.11/site-packages/asgiref/sync.py\", line 486, in thread_handler\n    raise exc_info[1]\n  File \"/usr/lib/python3.11/site-packages/pulp_file/app/tasks/synchronizing.py\", line 102, in run\n    result = await downloader.run()\n             ^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib/python3.11/site-packages/pulpcore/download/http.py\", line 269, in run\n    return await download_wrapper()\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib/python3.11/site-packages/backoff/_async.py\", line 151, in retry\n    ret = await target(*args, **kwargs)\n          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib/python3.11/site-packages/pulpcore/download/http.py\", line 254, in download_wrapper\n    return await self._run(extra_data=extra_data)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib/python3.11/site-packages/pulpcore/download/http.py\", line 290, in _run\n    self.raise_for_status(response)\n  File \"/usr/lib/python3.11/site-packages/pulpcore/download/http.py\", line 187, in raise_for_status\n    response.raise_for_status()\n  File \"/usr/lib64/python3.11/site-packages/aiohttp/client_reqrep.py\", line 1058, in raise_for_status\n    raise ClientResponseError(\n", "description"=>"404, message='Not Found', url=URL('https://jmp.domain.com/tech/sw/gnmic/PULP_MANIFEST')"}
2

Hi @dmgeurts ,

If your alternate content source is of type “File”, the repositories that you refer to need to all be proper Pulp File repositories with a PULP_MANIFEST file inside.

I’m curious about what you’re trying to achieve though since you mentioned debs and RPMs. ACSs only work with syncing repositories of the same type. So, if you’re using a File ACS to sync RPMs, the RPM repository will also need to be of type File for it to work as well.

The normal workflow for RPMs is to use the RPM ACS type, which expects the upstream repositories to be proper yum repos with repodata and all that. Debian does not have native ACS support yet, so you’d need to pack them into a File repository first.

For the PULP_MANIFEST file, you can use python3.11-pulp-manifest to help with generating it.

3

Hi @iballou ,

As you can tell I’m pretty new to this, thank you for your explanation.

As I’m scripting the check for new files (openconfig gnmic in this case), I can ensure that RPMs and debs are split into different folders. Adding the generation of a new PULP_MANIFEST file, when new files are downloaded, shouldn’t be too hard. I will look at what’s required to create a yum repo from downloaded RPMs to satisfy this requirement.

4

Is there a preferred path to use when using the foreman server to use a local file path (file://)?

I see that most folders under /var/lib/pulp are empty, so it’s tempting to use the imports folder:

user@foreman:~$ sudo du -hd1 /var/lib/pulp/
4.0K    /var/lib/pulp/tmp
443G    /var/lib/pulp/media
0       /var/lib/pulp/sync_imports
0       /var/lib/pulp/imports
0       /var/lib/pulp/exports
2.1M    /var/lib/pulp/assets
443G    /var/lib/pulp/
5

Reading the docs is leading me to think I should leave /var/lib/pulp alone and use a different path.

6

Hi @dmgeurts ,

I’d say the preferred import path is just anywhere that doesn’t conflict with system/software/library files. /var/lib/pulp is probably good to avoid like you mentioned since touching the artifacts within could cause Pulp issues.

Nice, so you could potentially use a yum/rpm ACS for the yum repositories and then a file type ACS for the debs and anything else.

By the way, if you need proper Debian ACS support, it would be a good RFE. Maybe @quba42 would be interested :slight_smile:

1 Like