Pulp deb repo sync between Locations two smart-proxy

Development
,
1

Hi, Foremansss.

Its possible, packets sync deb with smart-proxy between locations two smart-proxys?

I have two locations with same organization:

Location UAT
Location UAT2

I do sync content view in Location UAT and promote for UAT2, but not work sync between Locations

My smart-proxy UAT2 posssible “Packets 0”

image
image1316×534 49.4 KB

See log error in my UAT2 smart-proxy

Oct 23 21:20:08 uat2-l-srv-203 pulp: pulp_deb.plugins.importers.importer:INFO: [41ed68b2] Repo sync started.
Oct 23 21:20:08 uat2-l-srv-203 pulp: pulp_deb.plugins.importers.sync:INFO: [41ed68b2] Downloading https://uat-l-srv-203/pulp/deb/Default_Organization/UAT2/CV_UBUNTU16/custom/Canonical/xenial-backports/dists/stable/Release
Oct 23 21:20:08 uat2-l-srv-203 pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): uat-l-srv-203
Oct 23 21:20:08 uat2-l-srv-203 pulp: nectar.downloaders.threaded:INFO: Download failed: Download of https://uat-l-srv-203/pulp/deb/Default_Organization/UAT2/CV_UBUNTU16/custom/Canonical/xenial-backports/dists/stable/Release failed with code 404: Not Found
Oct 23 21:20:09 uat2-l-srv-203 pulp: pulp.server.async.tasks:INFO: [41ed68b2] Task failed : [41ed68b2-7c8d-4378-b9dd-e135cebdb54e]
Oct 23 21:20:09 uat2-l-srv-203 pulp: celery.app.trace:ERROR: [41ed68b2] (1748-88544) Task pulp.server.managers.repo.sync.sync[41ed68b2-7c8d-4378-b9dd-e135cebdb54e] raised unexpected: IOError(2, ‘No such file or directory’)
Oct 23 21:20:09 uat2-l-srv-203 pulp: celery.app.trace:ERROR: [41ed68b2] (1748-88544) Traceback (most recent call last):
Oct 23 21:20:09 uat2-l-srv-203 pulp: celery.app.trace:ERROR: [41ed68b2] (1748-88544) File “/usr/lib/python2.7/site-packages/celery/app/trace.py”, line 367, in trace_task
Oct 23 21:20:09 uat2-l-srv-203 pulp: celery.app.trace:ERROR: [41ed68b2] (1748-88544) R = retval = fun(*args, **kwargs)
Oct 23 21:20:09 uat2-l-srv-203 pulp: celery.app.trace:ERROR: [41ed68b2] (1748-88544) File “/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py”, line 529, in call
Oct 23 21:20:09 uat2-l-srv-203 pulp: celery.app.trace:ERROR: [41ed68b2] (1748-88544) return super(Task, self).call(*args, **kwargs)
Oct 23 21:20:09 uat2-l-srv-203 pulp: celery.app.trace:ERROR: [41ed68b2] (1748-88544) File “/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py”, line 107, in call
Oct 23 21:20:09 uat2-l-srv-203 pulp: celery.app.trace:ERROR: [41ed68b2] (1748-88544) return super(PulpTask, self).call(*args, **kwargs)
Oct 23 21:20:09 uat2-l-srv-203 pulp: celery.app.trace:ERROR: [41ed68b2] (1748-88544) File “/usr/lib/python2.7/site-packages/celery/app/trace.py”, line 622, in protected_call
Oct 23 21:20:09 uat2-l-srv-203 pulp: celery.app.trace:ERROR: [41ed68b2] (1748-88544) return self.run(*args, **kwargs)
Oct 23 21:20:09 uat2-l-srv-203 pulp: celery.app.trace:ERROR: [41ed68b2] (1748-88544) File “/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py”, line 769, in sync
Oct 23 21:20:09 uat2-l-srv-203 pulp: celery.app.trace:ERROR: [41ed68b2] (1748-88544) sync_report = sync_repo(transfer_repo, conduit, call_config)
Oct 23 21:20:09 uat2-l-srv-203 pulp: celery.app.trace:ERROR: [41ed68b2] (1748-88544) File “/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py”, line 737, in wrap_f
Oct 23 21:20:09 uat2-l-srv-203 pulp: celery.app.trace:ERROR: [41ed68b2] (1748-88544) return f(*args, **kwargs)
Oct 23 21:20:09 uat2-l-srv-203 pulp: celery.app.trace:ERROR: [41ed68b2] (1748-88544) File “/usr/lib/python2.7/site-packages/pulp_deb/plugins/importers/importer.py”, line 135, in sync_repo
Oct 23 21:20:09 uat2-l-srv-203 pulp: celery.app.trace:ERROR: [41ed68b2] (1748-88544) report = self._current_sync.process_lifecycle()
Oct 23 21:20:09 uat2-l-srv-203 pulp: celery.app.trace:ERROR: [41ed68b2] (1748-88544) File “/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py”, line 572, in process_lifecycle
Oct 23 21:20:09 uat2-l-srv-203 pulp: celery.app.trace:ERROR: [41ed68b2] (1748-88544) super(PluginStep, self).process_lifecycle()
Oct 23 21:20:09 uat2-l-srv-203 pulp: celery.app.trace:ERROR: [41ed68b2] (1748-88544) File “/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py”, line 163, in process_lifecycle
Oct 23 21:20:09 uat2-l-srv-203 pulp: celery.app.trace:ERROR: [41ed68b2] (1748-88544) step.process()
Oct 23 21:20:09 uat2-l-srv-203 pulp: celery.app.trace:ERROR: [41ed68b2] (1748-88544) File “/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py”, line 256, in process
Oct 23 21:20:09 uat2-l-srv-203 pulp: celery.app.trace:ERROR: [41ed68b2] (1748-88544) self._process_block()
Oct 23 21:20:09 uat2-l-srv-203 pulp: celery.app.trace:ERROR: [41ed68b2] (1748-88544) File “/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py”, line 303, in _process_block
Oct 23 21:20:09 uat2-l-srv-203 pulp: celery.app.trace:ERROR: [41ed68b2] (1748-88544) self.process_main()
Oct 23 21:20:09 uat2-l-srv-203 pulp: celery.app.trace:ERROR: [41ed68b2] (1748-88544) File “/usr/lib/python2.7/site-packages/pulp_deb/plugins/importers/sync.py”, line 133, in process_main
Oct 23 21:20:09 uat2-l-srv-203 pulp: celery.app.trace:ERROR: [41ed68b2] (1748-88544) release=open(self.parent.release_files[release], “rb”),
Oct 23 21:20:09 uat2-l-srv-203 pulp: celery.app.trace:ERROR: [41ed68b2] (1748-88544) IOError: [Errno 2] No such file or directory: u’/var/cache/pulp/reserved_resource_worker-0@uat2-l-srv-203/41ed68b2-7c8d-4378-b9dd-e135cebdb54e/stable/Release’
Oct 23 21:20:09 uat2-l-srv-203 pulp: celery.app.trace:INFO: [4aa17083] Task pulp.server.async.tasks._release_resource[4aa17083-e292-44eb-bc5c-e2d12f66b91d] succeeded in 0.00800271999833s: None

Tks

2

I cannot immediately see, why it does not work (and we have never tried tbh).
Considering your screenshot, i guess it will never pop up as packages, since that is really the model for rpms.
To see deb-Packages there, the view would need to be modified.

3

x9c4, only Content view I can count the packets.

it looks like smart-proxy can not download, but the url is ok.

Oct 24 15:55:07 uat2-l-srv-203 pulp: nectar.downloaders.threaded:INFO: Download failed: Download of https://uat-l-srv-203/pulp/deb/Default_Organization/UAT2/CV_UBUNTU16/custom/Canonical/ubuntu-xenial/dists/stable/Release failed with code 404: Not Found

should this function work? or can it be a bug?

4

Well, let’s make it a bug of the wishlist’s type, as this has never been implemented properly.

5

I think it’s not very complex.

See my task erro UAT

PLP0000: [Errno 2] No such file or directory: u’/var/cache/pulp/reserved_resource_worker-2@uat2-l-srv-203/c6bf5adb-9215-4987-b869-9ec1c005c8c8/stable/Release’

6

This usually means, that the feed-url of the downstream pulp is wrong.

8

Would you have any idea how I can try to circumvent this error?

10

Debug discovery

This may be an indication of the problem:

error: “[Errno 2] No such file or directory: u’/var/cache/pulp/reserved_resource_worker-2@uat2-l-srv-203/7b77fee7-b40b-4b9a-b015-e77e083868f8/stable/Release’”

URL OK -> http://uat-l-srv-203/pulp/deb/Default_Organization/UAT2/CV_UBUNTU16/custom/Canonical/ubuntu-xenial/dists/xenial/Release

Call sync smart-proxy URL Error -> https://uat-l-srv-203/pulp/deb/Default_Organization/UAT2/CV_UBUNTU16/custom/Canonical/ubuntu-xenial/dists/stable/Release

“stable/Release” not working feed-url

1 Like
11

Thank you, that might actually help.
This is about some confusion of codename vs. suite.
Can you try synching the upstream repository by codename (xenial) in the first place?
In an environment like katello it might not be desireable to sync it via ‘stable’ anyway, because that can change to the next release without warning.

12

I´m synching the upstream repository with codename xenial, but in synching between smart-proxy add stable in url.

I´ll try modify some lines in code.

13

I’m solved my problem by editing the file

/usr/lib/python2.7/site-packages/pulp_deb/plugins/importers/sync.py

Line: 56

self.releases = self.get_config (). get (‘releases’, ‘stable’). split (’,’)

to

self.releases = self.get_config (). get (‘releases’, ‘xenial’). split (’,’)

I believe that it is nescessario changes the “xenial” by a variable of the release of the repository configurable by the panel.

Is it possible to open a PR for this demand?

14

Ok, now i think the problem is, that the smartproxy does not see the “Releases, Architectures, Components” config from upstream Katello. This can be a little bit more involved.

15

you is correct, in my tests the variables receive “None”

  • self.releases = None
  • self.architectures = None
  • self.components = None
16

Solutions my env with support trusty,xenial,bionic.

Create repo product name “Canonical”

Add smart-proxy in file /usr/lib/python2.7/site-packages/pulp_deb/plugins/importers/sync.py

    url, release = self.get_config().get('feed').split('Canonical/')

    trusty = re.match(r"trusty*", release)
    xenial = re.match(r"xenial*", release)
    bionic = re.match(r"bionic*", release)

    if trusty != None:
     self.releases = self.get_config().get('releases', 'trusty').split(',')
    elif xenial != None:
     self.releases = self.get_config().get('releases', 'xenial').split(',')
    elif bionic != None:
     self.releases = self.get_config().get('releases', 'bionic').split(',')
    else:
     _LOG.info("no support releases  %s", releases)
17

Thanks for debugging this.
A proper solution must surely look different…
Do you mind to open a redmine ticket?

18

Send link this redmine.

19

https://projects.theforeman.org/projects/katello