Katello 4.3 - Repo Sync Error - Errno1 Operation Not Permitted

Support
6

The " Sequel::SQL::Blob" stuff seems to be Ruby-related, those are the only hits I get when I google it. Are you pulling task data out of the database directly?

Anyway… I think that’s unrelated to the pulp task failing. It’s hard to read due to the truncation though.

@JamesV21 or @tomzellner, could you try to fin an un-mutilated version of the traceback? Perhaps looking at the task record via pulp task details --href /pulp/api/v3/tasks/929b44b3-0b56-4254-a4ed-669348ac6c81/? If you have the pulp-cli package installed.

7

I can’t reproduce on the latest version but I haven’t tried the version shipped with 4.3 yet, FWIW.

8

I have just upgraded from 4.2 to 4.3 (again) and see the same error now when syncing the centos 7 os repo.

{"pulp_tasks"=>
  [{"pulp_href"=>"/pulp/api/v3/tasks/c77c47d7-c526-44b3-8ad0-f0e0320413fe/",
    "pulp_created"=>"2022-03-09T18:03:36.900+00:00",
    "state"=>"failed",
    "name"=>"pulp_rpm.app.tasks.synchronizing.synchronize",
    "logging_cid"=>"413226aa-898a-4e38-9a8e-ab48f40e6001",
    "started_at"=>"2022-03-09T18:03:37.296+00:00",
    "finished_at"=>"2022-03-09T18:09:04.085+00:00",
    "error"=>
     {"traceback"=>
       #<Sequel::SQL::Blob:0x36240 bytes=139 start="  File \"/o" end="form_task\n"> +
       #<Sequel::SQL::Blob:0x36254 bytes=35 start="    result" end="**kwargs)\n"> +
       #<Sequel::SQL::Blob:0x36268 bytes=137 start="  File \"/o" end="nchronize\n"> +
       #<Sequel::SQL::Blob:0x3627c bytes=56 start="    repo_v" end="version()\n"> +
       #<Sequel::SQL::Blob:0x36290 bytes=142 start="  File \"/o" end="in create\n"> +
       #<Sequel::SQL::Blob:0x362a4 bytes=38 start="    loop.r" end="pipeline)\n"> +
       #<Sequel::SQL::Blob:0x362b8 bytes=110 start="  File \"/o" end="_complete\n"> +
       #<Sequel::SQL::Blob:0x362cc bytes=27 start="    return" end=".result()\n"> +
       #<Sequel::SQL::Blob:0x362e0 bytes=135 start="  File \"/o" end="_pipeline\n"> +
       #<Sequel::SQL::Blob:0x362f4 bytes=35 start="    await " end="*futures)\n"> +
       #<Sequel::SQL::Blob:0x36308 bytes=127 start="  File \"/o" end=" __call__\n"> +
       #<Sequel::SQL::Blob:0x3631c bytes=21 start="    await " end="elf.run()\n"> +
       #<Sequel::SQL::Blob:0x36330 bytes=135 start="  File \"/o" end="9, in run\n"> +
       #<Sequel::SQL::Blob:0x36344 bytes=62 start="    await " end="_create)(\n"> +
       #<Sequel::SQL::Blob:0x36358 bytes=114 start="  File \"/o" end=" __call__\n"> +
       #<Sequel::SQL::Blob:0x3636c bytes=55 start="    ret = " end="out=None)\n"> +
       #<Sequel::SQL::Blob:0x36380 bytes=94 start="  File \"/o" end=" wait_for\n"> +
       #<Sequel::SQL::Blob:0x36394 bytes=21 start="    return" end="await fut\n"> +
       #<Sequel::SQL::Blob:0x363a8 bytes=100 start="  File \"/o" end="7, in run\n"> +
       #<Sequel::SQL::Blob:0x363bc bytes=48 start="    result" end="f.kwargs)\n"> +
       #<Sequel::SQL::Blob:0x363d0 bytes=120 start="  File \"/o" end="d_handler\n"> +
       #<Sequel::SQL::Blob:0x363e4 bytes=33 start="    return" end="**kwargs)\n"> +
       #<Sequel::SQL::Blob:0x363f8 bytes=138 start="  File \"/o" end="or_create\n"> +
       #<Sequel::SQL::Blob:0x3640c bytes=60 start="    return" end="tch_size)\n"> +
       #<Sequel::SQL::Blob:0x36420 bytes=131 start="  File \"/o" end="er_method\n"> +
       #<Sequel::SQL::Blob:0x36434 bytes=63 start="    return" end="**kwargs)\n"> +
       #<Sequel::SQL::Blob:0x36448 bytes=127 start="  File \"/o" end="lk_create\n"> +
       #<Sequel::SQL::Blob:0x3645c bytes=45 start="    return" end="d_insert(\n"> +
       #<Sequel::SQL::Blob:0x36470 bytes=132 start="  File \"/o" end="ed_insert\n"> +
       #<Sequel::SQL::Blob:0x36484 bytes=39 start="    insert" end="._insert(\n"> +
       #<Sequel::SQL::Blob:0x36498 bytes=124 start="  File \"/o" end="n _insert\n"> +
       #<Sequel::SQL::Blob:0x364ac bytes=73 start="    return" end="g_fields)\n"> +
       #<Sequel::SQL::Blob:0x364c0 bytes=135 start="  File \"/o" end="ecute_sql\n"> +
       #<Sequel::SQL::Blob:0x364d4 bytes=38 start="    for sq" end="as_sql():\n"> +
       #<Sequel::SQL::Blob:0x364e8 bytes=129 start="  File \"/o" end="in as_sql\n"> +
       #<Sequel::SQL::Blob:0x364fc bytes=58 start="    return" end=".as_sql()\n"> +
       #<Sequel::SQL::Blob:0x36510 bytes=130 start="  File \"/o" end="in as_sql\n"> +
       #<Sequel::SQL::Blob:0x36524 bytes=19 content="    value_rows = [\n"> +
       #<Sequel::SQL::Blob:0x36538 bytes=134 start="  File \"/o" end="listcomp>\n"> +
       #<Sequel::SQL::Blob:0x3654c bytes=83 start="    [self." end="n fields]\n"> +
       #<Sequel::SQL::Blob:0x36560 bytes=134 start="  File \"/o" end="listcomp>\n"> +
       #<Sequel::SQL::Blob:0x36574 bytes=83 start="    [self." end="n fields]\n"> +
       #<Sequel::SQL::Blob:0x36588 bytes=136 start="  File \"/o" end="_save_val\n"> +
       #<Sequel::SQL::Blob:0x3659c bytes=41 start="    return" end="add=True)\n"> +
       #<Sequel::SQL::Blob:0x365b0 bytes=127 start="  File \"/o" end=" pre_save\n"> +
       #<Sequel::SQL::Blob:0x365c4 bytes=49 start="    return" end="nce, add)\n"> +
       #<Sequel::SQL::Blob:0x365d8 bytes=131 start="  File \"/o" end=" pre_save\n"> +
       #<Sequel::SQL::Blob:0x365ec bytes=48 start="    file.s" end="ve=False)\n"> +
       #<Sequel::SQL::Blob:0x36600 bytes=126 start="  File \"/o" end=", in save\n"> +
       #<Sequel::SQL::Blob:0x36614 bytes=83 start="    self.n" end="x_length)\n"> +
       #<Sequel::SQL::Blob:0x36628 bytes=122 start="  File \"/o" end=", in save\n"> +
       #<Sequel::SQL::Blob:0x3663c bytes=37 start="    return" end=" content)\n"> +
       #<Sequel::SQL::Blob:0x36650 bytes=125 start="  File \"/o" end=" in _save\n"> +
       #<Sequel::SQL::Blob:0x36664 bytes=52 start="    os.chm" end="ons_mode)\n">,
      "description"=>
       "[Errno 1] Operation not permitted: '/var/lib/pulp/media/artifact/00/9f12969b8e86ea4f15b803c160c721f7f9426c8b3805deccdaf2d9c09bdc46'"},
    "worker"=>"/pulp/api/v3/workers/c1a4b937-d196-4505-a3d3-447a90223152/",
    "child_tasks"=>[],
    "progress_reports"=>
     [{"message"=>"Parsed Packages",
       "code"=>"sync.parsing.packages",
       "state"=>"canceled",
       "done"=>5140},
      {"message"=>"Downloading Metadata Files",
       "code"=>"sync.downloading.metadata",
       "state"=>"completed",
       "done"=>5},
      {"message"=>"Downloading Artifacts",
       "code"=>"sync.downloading.artifacts",
       "state"=>"canceled",
       "done"=>5},
      {"message"=>"Associating Content",
       "code"=>"associating.content",
       "state"=>"canceled",
       "done"=>0}],
    "created_resources"=>[],
    "reserved_resources_record"=>
     ["/pulp/api/v3/repositories/rpm/rpm/17e07bde-cdaf-485b-9f27-5316da1c20da/",
      "shared:/pulp/api/v3/remotes/rpm/rpm/eae954f5-3387-43c9-a58f-325ded40354c/"]}],
 "create_version"=>true,
 "task_groups"=>[],
 "poll_attempts"=>{"total"=>41, "failed"=>1}}

The file mentioned is the .treeinfo file:

# cat /var/lib/pulp/media/artifact/00/9f12969b8e86ea4f15b803c160c721f7f9426c8b3805deccdaf2d9c09bdc46
[general]
name = CentOS-7
family = CentOS
timestamp = 1603729576.26
variant = 
version = 7
packagedir = 
arch = x86_64

[stage2]
mainimage = LiveOS/squashfs.img

[images-x86_64]
kernel = images/pxeboot/vmlinuz
initrd = images/pxeboot/initrd.img
boot.iso = images/boot.iso

[images-xen]
kernel = images/pxeboot/vmlinuz
initrd = images/pxeboot/initrd.img

Pulp task details retrieved with curl:

# curl --cert /etc/pki/katello/certs/pulp-client.crt --key /etc/pki/katello/private/pulp-client.key https://foreman.example.com/pulp/api/v3/tasks/c77c47d7-c526-44b3-8ad0-f0e0320413fe/  | python -m json.tool
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  6189  100  6189    0     0  33413      0 --:--:-- --:--:-- --:--:-- 33635
{
    "child_tasks": [],
    "created_resources": [],
    "error": {
        "description": "[Errno 1] Operation not permitted: '/var/lib/pulp/media/artifact/00/9f12969b8e86ea4f15b803c160c721f7f9426c8b3805deccdaf2d9c09bdc46'",
        "traceback": "  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/tasking/pulpcore_worker.py\", line 377, in _perform_task\n    result = func(*args, **kwargs)\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulp_rpm/app/tasks/synchronizing.py\", line 544, in synchronize\n    repo_version = dv.create() or repo.latest_version()\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/plugin/stages/declarative_version.py\", line 161, in create\n    loop.run_until_complete(pipeline)\n  File \"/opt/rh/rh-python38/root/usr/lib64/python3.8/asyncio/base_events.py\", line 616, in run_until_complete\n    return future.result()\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/plugin/stages/api.py\", line 225, in create_pipeline\n    await asyncio.gather(*futures)\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/plugin/stages/api.py\", line 43, in __call__\n    await self.run()\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/plugin/stages/artifact_stages.py\", line 249, in run\n    await sync_to_async(Artifact.objects.bulk_get_or_create)(\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/asgiref/sync.py\", line 444, in __call__\n    ret = await asyncio.wait_for(future, timeout=None)\n  File \"/opt/rh/rh-python38/root/usr/lib64/python3.8/asyncio/tasks.py\", line 455, in wait_for\n    return await fut\n  File \"/opt/rh/rh-python38/root/usr/lib64/python3.8/concurrent/futures/thread.py\", line 57, in run\n    result = self.fn(*self.args, **self.kwargs)\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/asgiref/sync.py\", line 486, in thread_handler\n    return func(*args, **kwargs)\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/models/content.py\", line 86, in bulk_get_or_create\n    return super().bulk_create(objs, batch_size=batch_size)\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/manager.py\", line 85, in manager_method\n    return getattr(self.get_queryset(), name)(*args, **kwargs)\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/query.py\", line 502, in bulk_create\n    returned_columns = self._batched_insert(\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/query.py\", line 1287, in _batched_insert\n    inserted_rows.extend(self._insert(\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/query.py\", line 1270, in _insert\n    return query.get_compiler(using=using).execute_sql(returning_fields)\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/sql/compiler.py\", line 1415, in execute_sql\n    for sql, params in self.as_sql():\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django_readonly_field/compiler.py\", line 31, in as_sql\n    return super(ReadonlySQLCompilerMixin, self).as_sql()\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/sql/compiler.py\", line 1358, in as_sql\n    value_rows = [\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/sql/compiler.py\", line 1359, in <listcomp>\n    [self.prepare_value(field, self.pre_save_val(field, obj)) for field in fields]\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/sql/compiler.py\", line 1359, in <listcomp>\n    [self.prepare_value(field, self.pre_save_val(field, obj)) for field in fields]\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/sql/compiler.py\", line 1310, in pre_save_val\n    return field.pre_save(obj, add=True)\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/models/fields.py\", line 76, in pre_save\n    return super().pre_save(model_instance, add)\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/fields/files.py\", line 302, in pre_save\n    file.save(file.name, file.file, save=False)\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/fields/files.py\", line 89, in save\n    self.name = self.storage.save(name, content, max_length=self.field.max_length)\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/core/files/storage.py\", line 54, in save\n    return self._save(name, content)\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/models/storage.py\", line 94, in _save\n    os.chmod(full_path, self.file_permissions_mode)\n"
    },
    "finished_at": "2022-03-09T18:09:04.085427Z",
    "logging_cid": "413226aa-898a-4e38-9a8e-ab48f40e6001",
    "name": "pulp_rpm.app.tasks.synchronizing.synchronize",
    "parent_task": null,
    "progress_reports": [
        {
            "code": "sync.parsing.packages",
            "done": 5140,
            "message": "Parsed Packages",
            "state": "canceled",
            "suffix": null,
            "total": null
        },
        {
            "code": "sync.downloading.metadata",
            "done": 5,
            "message": "Downloading Metadata Files",
            "state": "completed",
            "suffix": null,
            "total": null
        },
        {
            "code": "sync.downloading.artifacts",
            "done": 5,
            "message": "Downloading Artifacts",
            "state": "canceled",
            "suffix": null,
            "total": null
        },
        {
            "code": "associating.content",
            "done": 0,
            "message": "Associating Content",
            "state": "canceled",
            "suffix": null,
            "total": null
        }
    ],
    "pulp_created": "2022-03-09T18:03:36.900012Z",
    "pulp_href": "/pulp/api/v3/tasks/c77c47d7-c526-44b3-8ad0-f0e0320413fe/",
    "reserved_resources_record": [
        "/pulp/api/v3/repositories/rpm/rpm/17e07bde-cdaf-485b-9f27-5316da1c20da/",
        "shared:/pulp/api/v3/remotes/rpm/rpm/eae954f5-3387-43c9-a58f-325ded40354c/"
    ],
    "started_at": "2022-03-09T18:03:37.296968Z",
    "state": "failed",
    "task_group": null,
    "worker": "/pulp/api/v3/workers/c1a4b937-d196-4505-a3d3-447a90223152/"
}

Traceback formatted:

  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/tasking/pulpcore_worker.py", line 377, in _perform_task
    result = func(*args, **kwargs)
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 544, in synchronize
    repo_version = dv.create() or repo.latest_version()
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/plugin/stages/declarative_version.py", line 161, in create
    loop.run_until_complete(pipeline)
  File "/opt/rh/rh-python38/root/usr/lib64/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
    await asyncio.gather(*futures)
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
    await self.run()
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 249, in run
    await sync_to_async(Artifact.objects.bulk_get_or_create)(
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/asgiref/sync.py", line 444, in __call__
    ret = await asyncio.wait_for(future, timeout=None)
  File "/opt/rh/rh-python38/root/usr/lib64/python3.8/asyncio/tasks.py", line 455, in wait_for
    return await fut
  File "/opt/rh/rh-python38/root/usr/lib64/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/asgiref/sync.py", line 486, in thread_handler
    return func(*args, **kwargs)
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/models/content.py", line 86, in bulk_get_or_create
    return super().bulk_create(objs, batch_size=batch_size)
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/query.py", line 502, in bulk_create
    returned_columns = self._batched_insert(
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/query.py", line 1287, in _batched_insert
    inserted_rows.extend(self._insert(
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/query.py", line 1270, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1415, in execute_sql
    for sql, params in self.as_sql():
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django_readonly_field/compiler.py", line 31, in as_sql
    return super(ReadonlySQLCompilerMixin, self).as_sql()
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1358, in as_sql
    value_rows = [
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1359, in <listcomp>
    [self.prepare_value(field, self.pre_save_val(field, obj)) for field in fields]
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1359, in <listcomp>
    [self.prepare_value(field, self.pre_save_val(field, obj)) for field in fields]
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1310, in pre_save_val
    return field.pre_save(obj, add=True)
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/models/fields.py", line 76, in pre_save
    return super().pre_save(model_instance, add)
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/fields/files.py", line 302, in pre_save
    file.save(file.name, file.file, save=False)
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/fields/files.py", line 89, in save
    self.name = self.storage.save(name, content, max_length=self.field.max_length)
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/core/files/storage.py", line 54, in save
    return self._save(name, content)
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/models/storage.py", line 94, in _save
    os.chmod(full_path, self.file_permissions_mode)

Permissions of the directory with that artifact and some others with similar beginning filename:

# ls -lad /var/lib/pulp/media/artifact/00/{,9f*} 
drwxr-xr-x. 2 pulp   pulp   94208 Mar  8 01:33 /var/lib/pulp/media/artifact/00/
-rw-rw-r--. 1 apache pulp     354 Nov 12  2020 /var/lib/pulp/media/artifact/00/9f12969b8e86ea4f15b803c160c721f7f9426c8b3805deccdaf2d9c09bdc46
-rw-------. 1 pulp   pulp 3095704 Feb  2 07:26 /var/lib/pulp/media/artifact/00/9f1e5fdfe0287c2d456390aacab52be52e77247292e285d5caa965c2330dab
-rw-r--r--. 1 pulp   pulp  129900 Jul 14  2021 /var/lib/pulp/media/artifact/00/9f2e85d869301681b61117ede92ac669c927746ec11e38fdb5beef212f81e3
-rw-rw-r--. 1 apache pulp 1206016 May 12  2020 /var/lib/pulp/media/artifact/00/9f498f5bbadf4103438c37fed783f3ed3f8fdcfd3e7e6de1d6554337da0760
-rw-rw-r--. 1 apache pulp  268204 May 22  2020 /var/lib/pulp/media/artifact/00/9f6dc27ef561a73c03e66fa06bf651205dfd51d51f5c7d34e25ee7411e0d1d

I hope this helps. I am not sure, yet, if I stick with 4.3 this time or I revert back.

9

@gvde This is a migrated system, yes? I’m pretty sure these permissions are supposed to be uniform, and probably the files owned by apache came from Pulp 2… could be a migration bug, which would explain why I can’t reproduce.

10

Yes. Started with Katello 3.15 I think, i.e. with pulp2 and migrated eventually to pulp3.

That could be. The migration steps in the 3.18 docs only set up uniform group and group permissions on everything in the pulp2 content directory.

The artifact causing the problem has owner apache, i.e. if the pulpcore worker running as user pulp tries to change permissions on that file it’ll fail with ‘Operation not permitted’.

Looking at the dates of the files listed it seems everything should be owner/group pulp.pulp and permission 0600?

11

On my fresh install (never 2to3 migrated), the permissions look like this:

# ls -al /var/lib/pulp/media/artifact/00/
total 10372
drwxr-xr-x.   2 pulp pulp    4096 Mar 10 12:04 .
drwxr-xr-x. 257 pulp pulp    8192 Mar 10 12:04 ..
-rw-------.   1 pulp pulp   13040 Mar 10 12:03 655717ec2494a953b1c3f477bc86b080760402b72a926057076144aa8b9c41
-rw-------.   1 pulp pulp 1381996 Mar 10 11:53 9767e4fcb269500c8b2ebe0d2cf7f818b9774858019cbed6c325d7c3e73596
-rw-------.   1 pulp pulp  686508 Mar 10 11:54 b9e63e287f45300d4a4f59b6b88e25918443c932ae3e5845d5761ae193c530
-rw-------.   1 pulp pulp 8515480 Mar 10 12:04 d0de456134668f41bd9ea308a076bc0a6a805180445af8a37209d433f41efe

So yes, I would say you are right.

12

Our system was migrated from pulp2 to pulp3 and the majority of files are owned by apache rather than pulp and have the wrong permissions as well

# ls -la /var/lib/pulp/media/artifact/00
total 1203204
drwxr-xr-x   2 pulp   pulp     32768 Feb 21 10:39 .
drwxr-xr-x 258 pulp   pulp      4096 Jun 22  2021 ..
-rw-rw-r--   1 apache pulp     42540 Mar 27  2020 0149f21cbf7579915c09ea2f659c357395c0da4b9a286c4de09df558098f7b
-rw-------   1 pulp   pulp    125729 Oct 26 08:18 0225aca00d646eb11e18344d3b8d97eb59eaa30fadef186d2bc11524570c1c
-rw-rw-r--   1 apache pulp     71284 Oct 21  2020 047c0995e145e37da62cc328a1566c494d3ef7298450494f23f9f1dfcaa543
-rw-rw-r--   1 apache pulp    343320 Nov 18  2020 063f470db329bdbf0defd2b9622d72db94830b9c0a108b7646f6acb3316563
-rw-rw-r--   1 apache pulp    364064 Mar 27  2020 0787002b6a9fa53a6818a92bbd82d75c0deec343540dc08a66ba630fa1e8e8
-rw-rw-r--   1 apache pulp     34432 Mar 27  2020 085cd8333fafa109179e0d672bb352b10dedd4eef3876fc0835833bafc8de0
-rw-rw-r--   1 apache pulp   5653596 Mar 27  2020 096c7131af7b7f1e7120bfb01651356a15d94f13e38494ccfa07e86af8fef6
-rw-rw-r--   1 apache pulp     13720 Mar 27  2020 0a434f3dc587fcad9a5e566391daedd4f78b38f48b37c59103d0dd51f7610b
-rw-rw-r--   1 apache pulp     46980 Dec 18  2020 0c783ad67a44fbc26df8ad81ecd49cdb80c72bac2e0468445fe1454d75a522
-rw-rw-r--   1 apache pulp     72616 Mar 27  2020 0c82d22234b0c7817153d85ba7a4a66e2b335173ab0fff97178f18a2684dca
-rw-rw-r--   1 apache pulp     76008 Oct 21  2020 0cb20f305398259ef2c1bef9919d6cd7abfae7492f3a75acf25c2ea26d266e
-rw-rw-r--   1 apache pulp     95224 Nov 12  2020 0ea54e38c1c408fcf9548fe7d830eb475e867dc698d6f2f5aa00850c4bd67b
-rw-rw-r--   1 apache pulp     98312 Apr 27  2020 0fd1719737652c9766ebfbb9d8acd63b4c854bb62f4738f3f8ee23eab188f7
-rw-rw-r--   1 apache pulp     57156 Mar 27  2020 0ff36975b410e23a092f6ab0b0b159dab993325d9a7b45565424c6333d326b
-rw-rw-r--   1 apache pulp    952356 Mar 27  2020 119100dc629a7357cf5847e231dc0523e24152e40ae7b9e85660d6e45bb650
-rw-rw-r--   1 apache pulp   2631124 Mar 27  2020 136ed06ddba50e2a66e0241fcfe9085634c5dda843984e2dc9ed87b3928472
-rw-rw-r--   1 apache pulp     56464 Mar 27  2020 164893c28d09401a9fd295082b4ee1e3d4201cede1b65c17fd08fe9322304c
-rw-rw-r--   1 apache pulp   7995824 Oct 21  2020 167451bfc93d622557ee62240a7f90d35e11e21fac2e953f83a9325f3bca28
-rw-r--r--   1 pulp   pulp    182840 Aug  4  2021 16a44310db36215e198957df42b0bdf3fc6098673e2347e4bb789e61b06e6e
-rw-rw-r--   1 apache pulp     65414 Feb 22  2021 1abafa991740446acf0ba79fa50c401b5555a081ce793f72c904f21e8ddddf
-rw-rw-r--   1 apache pulp    740492 Oct 21  2020 1db24815b09f65a0c60cceb4f436cdf71dc9e2742e0af358b378ce4b628780
-rw-rw-r--   1 apache pulp    289120 May  6  2020 21110620b0dce48b05ab0359cc22991cd701f058a1b8099e650c6272081873
-rw-rw-r--   1 apache pulp     23508 Apr 27  2020 2126fe84a7207f9ee184f2acd2618da10c2be4c49ab3b0248bfe2eddb90199
-rw-rw-r--   1 apache pulp     33788 Oct 21  2020 213fd7804aad49be9b2f9e0d949cc3a2c93a32b16ab958c4bf216eb78549c1
-rw-rw-r--   1 apache pulp 115758573 Apr 23  2020 21cb8ca0978f52a8f0ff2d759eb695001a8632cdd8cd4b782de167d66603c3
-rw-rw-r--   1 apache pulp    168484 Mar 27  2020 23356012efcb7eadd3be350868f8113ba41062ec20181d44a37d233e0e0001
-rw-rw-r--   1 apache pulp   2353520 Oct 21  2020 243e95aec094be1112a35536ad5d570d5d59423e2ad1de9cb5beeef0eba075
-rw-rw-r--   1 apache pulp     12756 Mar 27  2020 269ac5b12e96f874e9cee7f5090a80b97bdce1564a7a01819ae9f516559ab9
-rw-rw-r--   1 apache pulp     13730 Feb 27  2021 2a53b5c093bf8f171866dbd27aa667bf2407a400706ebd5365bc9b268d16ab
-rw-rw-r--   1 apache pulp     81540 Apr 29  2020 2b73881adce7c1097c6eacae3126bbc5a8291bcae3287bfd5c6ebd54d3df8a
-rw-rw-r--   1 apache pulp     30012 Mar 27  2020 2c83c87fbc85eea1404fbacb3a4afcea547a55933348b5d5d015dded765a9b
-rw-rw-r--   1 apache pulp     58720 Mar 27  2020 2e52327d7f3d35a74e802763af005e6f3e7c7296601da6c99feaa8234aff16
-rw-r--r--   1 pulp   pulp     99708 Feb 21 10:39 2ea2460b9e864c85bf1ed8a108e4203f562116643f19001914afc7b07ff4d5
13

So a recursive chown on the artifact folder should straighten out the problem with the owner which would allow pulpcore-workers to do chmod on files…

# chown -hR pulp.pulp /var/lib/pulp/media/artifact/

The permissions on files (600, 644, 664) shouldn’t be an issue here…

14

pulp-2to3-migration tries to make hardlinks for artifacts if it can, in order to not double the storage needed when migrating. That’s how you end up with pulp2 perms/users on pulp3 content post-migration, is my guess - your pulp2 and pulp3 storage live on the same volume. We prob need a post-migration step to do the suggested chown; you don’t want to do it until you’re ready to “turn off” pulp2, so as not to impact pulp2 Doing It’s Thing while you’re still preparing Pulp3.

1 Like
15

I am on 4.3 now, so migration was a long time ago. This error, though, is new. It does not happen with 4.2. So I guess it must be related with pulpcore 3.16 or something around that, that it tries to do a chmod on a file…

I have changed owners and the following sync just finished without problems.

16

Hm, good point. I am def surprised to see this being a problem “suddenly” - I would have expected us to hit this A Long Time Ago. “Just upgraded from core/3.15 to core/3.16” is def a big hint, though.

17

pulpcore 3.14 to 3.16 to be exact.

18

@dralley Is this now a pulpcore or a katello problem? How do we get this fixed? Should foreman-installer do a recursive chown on the pulp directory? Or is this something pulpcore should do as part of the migrations? Or how? I guess, a recursive chown can take a long time depending on the number of files in the directories and the speed of the disk, thus it’s probably not an option to do this during each upgrade…

19

We’ve been discussing that, the best place to put it is most probably foreman-maintain content switchover. So, only once, right at the end when Pulp 2 is being shut off.

It could also be done as a separate manual step, but it’s better to avoid making the migration process any more complicated than it is already

20

That’s useful for those you are on 3.18 running pulp2, but doesn’t help those you are running pulp3 and are on 4.0+. This chmod problem for instance only occurs on 4.3/pulpcore 3.16.

At a minimum, katello upgrade instructions needed to be updated, e.g. to include a simple find to see if there is a potential issue:

# find /var/lib/pulp/media/ \! -user pulp -ls

and then instruct the user to do the chown.

But I kind of don’t like these upgrade instructions where users have to fix things, that got broken somehow, when it could be easily scripted…

21

I think the best place for this would be in the Installer since it’s what maintains the overall state of the server’s configuration. To fix the issue a user would only need to re-run the foreman-installer. That would be better than a rake task since no documentation would be needed. Although, it makes me wonder why the other file permission steps from the Pulp 2 - Pulp 3 migration weren’t in the installer.

Has anyone tested updating the permissions to see if that does fix the issue?

22

@ekohl or @wbclark I’m curious what you would think about having the installer ensure that all permissions under /var/lib/pulp/media are owned by pulp. It would probably mean we’d just need another addition here? puppet-pulpcore/config.pp at master · theforeman/puppet-pulpcore · GitHub

23

This fixed the problem for me, many thanks!

1 Like
24

I have a PR out that should fix this: Fixes #34631 - Recurse into artifacts folder to ensure owner is proper by ianballou · Pull Request #250 · theforeman/puppet-pulpcore · GitHub

25

New PR: Fixes #34631 - add command & procedure to update pulpcore artifact ownership by ianballou · Pull Request #596 · theforeman/foreman_maintain · GitHub