Package/errata info disappears after a while on Content hosts page/Foreman 2.1.2

Support
30

@matemikulic,

The only way I was able to “reproduce” the errata issue was by installing the latest version of sed, but even then the erratum did show up with:
yum updateinfo info all --advisory=CEBA-2020:1041

Can you see the missing erratum by doing yum updateinfo info all ... instead of yum updateinfo info ... ? I’m going to guess not since that would just mean you have the latest sed, but it’s worth trying.

If it really is missing, then we can check if the advisory/erratum is actually in your private repository’s updateinfo file. You can check that by just downloading it here (find the hash value by browsing the “Published at” url in your repo details):

wget http://`hostname`/pulp/repos/<your organization>/<your lifecycle environment>/custom/<your product>/<your updates repo>/repodata/<some hash value>-updateinfo.xml.gz

If it’s missing, it might be worth doing a full re-sync of the repository. My erratum was in there, so if it’s not for you it’s either a transient issue or maybe a bug in your version of pulp-rpm.

31

Hey, tnx for checking this. You are absolutely correct. Errata is not there on foreman-proxy. How it gets lost on the way, not sure. But I’ve tried to do right now complete sync with content validation on repo with errata before promoting new version of content and it failed. Error output:

Actions::Pulp3::Repository::Repair
Input:
{"repository_id"=>5,
 "smart_proxy_id"=>1,
 "remote_user"=>"admin",
 "remote_cp_user"=>"admin",
 "current_request_id"=>nil,
 "current_timezone"=>"Europe/Zagreb",
 "current_user_id"=>5,
 "current_organization_id"=>3,
 "current_location_id"=>nil}
Output:
{"response"=>
  {"task"=>"/pulp/api/v3/tasks/76d39d7d-6c3f-4d54-8159-57c78986fe2f/"},
 "pulp_tasks"=>
  [{"pulp_href"=>"/pulp/api/v3/tasks/76d39d7d-6c3f-4d54-8159-57c78986fe2f/",
    "pulp_created"=>"2020-10-22T09:29:20.670+00:00",
    "state"=>"failed",
    "name"=>"pulpcore.app.tasks.repository.repair_version",
    "started_at"=>"2020-10-22T09:29:20.741+00:00",
    "finished_at"=>"2020-10-22T09:29:49.447+00:00",
    "error"=>
     {"traceback"=>
       "  File \"/usr/lib/python3.6/site-packages/rq/worker.py\", line 883, in perform_job\n" +
       "  File \"/usr/lib/python3.6/site-packages/rq/job.py\", line 645, in perform\n" +
       "  File \"/usr/lib/python3.6/site-packages/rq/job.py\", line 651, in _execute\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulpcore/app/tasks/repository.py\", line 118, in repair_version\n" +
       "  File \"/usr/lib64/python3.6/asyncio/base_events.py\", line 484, in run_until_complete\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulpcore/app/tasks/repository.py\", line 87, in _repair_repository_version\n" +
       "  File \"/usr/lib64/python3.6/concurrent/futures/thread.py\", line 56, in run\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulpcore/app/tasks/repository.py\", line 65, in _verify_ca\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulpcore/app/tasks/repository.py\", line 65, in <lambda>\n" +
       "  File \"/usr/lib/python3.6/site-packages/django/core/files/utils.py\", line 16, in <lambda>\n" +
       "  File \"/usr/lib/python3.6/site-packages/django/db/models/fields/files.py\", line 43, in _get_file\n" +
       "  File \"/usr/lib/python3.6/site-packages/django/core/files/storage.py\", line 36, in open\n" +
       "  File \"/usr/lib/python3.6/site-packages/django/core/files/storage.py\", line 224, in _open\n",
      "description"=>
       "[Errno 24] Too many open files: '/var/lib/pulp/docroot/artifact/12/3a4fe6f7ecc4ceebab59ee4d3bcfe6db3b48c5995fba0b0ad77b48dd4583c8'"},
    "worker"=>"/pulp/api/v3/workers/b0284a06-fa71-4814-8da0-231959b62ee5/",
    "child_tasks"=>[],
    "progress_reports"=>
     [{"message"=>"Identify corrupted units",
       "code"=>"repair.corrupted",
       "state"=>"failed",
       "done"=>0},
      {"message"=>"Repair corrupted units",
       "code"=>"repair.repaired",
       "state"=>"failed",
       "done"=>0}],
    "created_resources"=>[],
    "reserved_resources_record"=>
     ["/pulp/api/v3/repositories/rpm/rpm/7bac9f0a-05db-424a-b34f-1b8c6392815b/"]}],
 "task_groups"=>[],
 "poll_attempts"=>{"total"=>18, "failed"=>1}}
Exception:
Katello::Errors::Pulp3Error: [Errno 24] Too many open files: '/var/lib/pulp/docroot/artifact/12/3a4fe6f7ecc4ceebab59ee4d3bcfe6db3b48c5995fba0b0ad77b48dd4583c8'
Backtrace:
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.1.2/app/lib/actions/pulp3/abstract_async_task.rb:101:in `block in check_for_errors'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.1.2/app/lib/actions/pulp3/abstract_async_task.rb:99:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.1.2/app/lib/actions/pulp3/abstract_async_task.rb:99:in `check_for_errors'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.1.2/app/lib/actions/pulp3/abstract_async_task.rb:132:in `poll_external_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:100:in `poll_external_task_with_rescue'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:22:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/cancellable.rb:14:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.1.2/app/lib/actions/pulp3/abstract_async_task.rb:10:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:571:in `block (3 levels) in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.1.2/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.1.2/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.1.2/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.1.2/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.1.2/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.1.2/app/lib/actions/middleware/remote_action.rb:16:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/execution_wrapper.rb:88:in `wrap'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:17:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_request_id.rb:49:in `restore_current_request_id'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_user.rb:44:in `restore_curent_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/world.rb:31:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:570:in `block (2 levels) in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in `block in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `block in with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:564:in `execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:285:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/director.rb:93:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors.rb:18:in `run_user_code'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in `execute_job'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in `block (2 levels) in process'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in `invoke'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in `block in process'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in `block (6 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in `local'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in `block (5 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq.rb:37:in `block in <module:Sidekiq>'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in `block (4 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in `stats'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in `block (3 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in `block (2 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in `global'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in `block in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in `with_context'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in `with_job_hash_context'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in `dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in `process'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in `process_one'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in `watchdog'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in `block in safe_thread'
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
32

I just synced repo updates containing errata from internet to local repo, due to that bug with authentication. Local matches that one on the internet and contains the same files:

3ceff108624118bff78dd0800f6d36b716cce8ad5606653c446b834e76b3b915-primary.sqlite.bz2
48d986911145aa7e5dbcac07ad5b0faa7afdfc021a955ebe4f1ff422d0a71eb9-other.xml.gz
4c2d91df66c87a599e544cf53a3da280fdf59386e4ddf9027495602fa22ca508-filelists.xml.gz
737068c90d4c5e191c7b687188f562fab45fb1525c9c1a680a61a667a28cd4fb-other.sqlite.bz2
afe3eb5eba9fecfdc857c99fbd4574bc451ba16e2bf3d05de180a96e7a2f9d99-filelists.sqlite.bz2
f97bb3a553c10733d6984dfb16c75fc3d376acf119b39e0a9c57e43fc8bbfbe4-updateinfo.xml.gz
ff191ab1ffe6dfebfedb2aa7f6b81635f5c0877a8a229062ae65283e7939ea4c-primary.xml.gz
repomd.xml
repomd.xml.asc

Then I promoted new content version and on smart proxy I’m testing this on after oprimized sync went through did the complete sync. These are the files I found on the same but smart proxy updates repo:

1f5033c2476d2fc39c7347dfae1494c2060f9223478be95e58080c30577ce4be-filelists.xml.gz
783f1a372b71df69ccd2a636a7f8467e74c16ab81b56be78e830fbec5ea3bd41-updateinfo.xml.gz
7dedcf3aa2064dbbe9b33070f67a0fefcfb883434574899a7f1d689015c55e13-primary.xml.gz
95f929621bc5d36411be95f084a3d4f68f829f37c454ae78332ae865a97c249f-other.xml.gz
a27718cc28ec6d71432e0ef3e6da544b7f9d93f6bb7d0a55aacd592d03144b70-comps.xml
repomd.xml

Does katello/pulp run something like createrepo to generate new repodata?

33

Pulp does regenerate the new repodata itself, I’m not sure how exactly. Is your smart proxy missing any RPMs or errata? I’m not surprised that the repodata is structured differently, but it should have all the same content.

I think the bug you just found above with the repo sync + checksum checking is a Pulp issue, I will raise it with the Pulp team.

1 Like
34

I might recommend destroying the repository that’s missing the erratum and recreating/resyncing it. If your issue is reproducible, I will make another Pulp 3 issue for it.

Pulp issue for repo repair: https://pulp.plan.io/issues/7735

35

Hey, so I tried what you recommended. I’ve created new repo called updates-errata. I’ve synced that new repo from source where repodata looks like this:

3ceff108624118bff78dd0800f6d36b716cce8ad5606653c446b834e76b3b915-primary.sqlite.bz2
48d986911145aa7e5dbcac07ad5b0faa7afdfc021a955ebe4f1ff422d0a71eb9-other.xml.gz
4c2d91df66c87a599e544cf53a3da280fdf59386e4ddf9027495602fa22ca508-filelists.xml.gz
737068c90d4c5e191c7b687188f562fab45fb1525c9c1a680a61a667a28cd4fb-other.sqlite.bz2
afe3eb5eba9fecfdc857c99fbd4574bc451ba16e2bf3d05de180a96e7a2f9d99-filelists.sqlite.bz2
f97bb3a553c10733d6984dfb16c75fc3d376acf119b39e0a9c57e43fc8bbfbe4-updateinfo.xml.gz
ff191ab1ffe6dfebfedb2aa7f6b81635f5c0877a8a229062ae65283e7939ea4c-primary.xml.gz
repomd.xml
repomd.xml.asc

I’ve then in Content view replaced old repo called updates with new named updates-errata and published new version. Checking on master server that repo’s repodata looks like this:

16ae6b19f7cf6267ec8397437a16b14444ef4d5a7b1f21cc646a4cb0b94d9ccd-updateinfo.xml.gz
1c0e2a1afa47b56c9afc9bf8a5d40b94d9fa57ccd9edc103412b65498f52ba33-primary.xml.gz
1f6fd37cd09b0fe00b4f0a5a66bf7adf2620d59c708c9f33a8ce814999ec1a2c-filelists.sqlite.bz2
3263b4206bb57e382ffa8b0bc8b4d1bb1900d3162fb3fc49ebdbc1b33edeae47-primary.sqlite.bz2
6d20d179dda953a80005ff2915e8d8cd15ab7205f0fa6c2e42f5d1ac7a17e6f8-other.sqlite.bz2
c5b7e04df0454379ce33e162e6e35dff63efff87f9374f4c3fbcdc7a876070ab-filelists.xml.gz
f22c7871fe5ee1a1ea8822c6e2b4a2a7deee7952138ba01da874b31008807be4-other.xml.gz
repomd.xml

So it seems master has changed hashes and data inside.
I have 24 smart proxies connected to that master so I’ve went and checked around on a couple of them that same repo.
Smartproxy1

4d5b284bfb8e279e0d70756f585c3117307f9393b7a8af4763780b23da30e6e2-updateinfo.xml.gz
a27718cc28ec6d71432e0ef3e6da544b7f9d93f6bb7d0a55aacd592d03144b70-comps.xml
b1456bfc2d50d881bb5c8f6218a8d67bca76758b1e10740125311e5c68e9b545-primary.xml.gz
d03468c2194261d73b56bc4a9db8e56b11051723f924be483352c955e8fd97e8-other.xml.gz
d2dd49be984f54fceb37beb4aded7e34065069ccc13bd4fd002e4e26acc3e47a-filelists.xml.gz
repomd.xml

Smartproxy2

15ae9daa45709ea783c02d2be109f1fbb543bf08d869008d628ff3fc52f25eb8-other.xml.gz
718b264df61f770cb4c42e6899c54e806300a5283c08086d969c84d00d5c4344-primary.xml.gz
a27718cc28ec6d71432e0ef3e6da544b7f9d93f6bb7d0a55aacd592d03144b70-comps.xml
ba81fae9c8ff21ff0b71691bc3f5f9a056423185f7bb3ef8b0130d1316026bbd-filelists.xml.gz
cd167ac6e683c5e77555b4cf3a8b77cc546a329371856dd0507cb3c710b3ff6d-updateinfo.xml.gz
repomd.xml

Smartproxy3

117611b1e5191cf2025493e2f10884aaf9f736c9b25ec0ba212351ce6994b02d-other.xml.gz
33587a82c4ce441476acbc777e317eedfc6c793b9428de43148ba1ce3fbb1f17-primary.xml.gz
339000047b5867c1694b9d3d0aace12c1b33a2d3f96db1c71d43b2ff8dd8f09c-updateinfo.xml.gz
a27718cc28ec6d71432e0ef3e6da544b7f9d93f6bb7d0a55aacd592d03144b70-comps.xml
d2f8c0ac07801eaa44979a103ac26a551be197eb154f688c49a26f98ce021660-filelists.xml.gz
repomd.xml

Smartproxy4

003c6cc0d584c348a53712397f78cec5f5183850c005198c3c3df030d2726a1f-primary.xml.gz
059d2dc10978eccbf1755634c498455090befa4c88d1feec006462877d91caf3-other.xml.gz
5b535efd7fdffda1af1e8adc08661a1f74913f390ac983f4126faee1d474daa6-updateinfo.xml.gz
a27718cc28ec6d71432e0ef3e6da544b7f9d93f6bb7d0a55aacd592d03144b70-comps.xml
c08d3861193da0831c25f2209c14ba61e224cf56d25c51279adcc8c9c8fc9ae5-filelists.xml.gz
repomd.xml

And so on. Any smart proxy that I check I’ll find different hash prefixes. So it seems that pulp does regenerate repodata. And smart proxy also when syncing.

Errata that did not work in previous repo, doesn’t work in the new one as well.
I’ll send you latest errata file so you can do tests if necessary.

36

I’ve downloaded the version from orignal repo, version from master and smart proxies, unzipped them and numbered the lines.

Original repo

# cat f97bb3a553c10733d6984dfb16c75fc3d376acf119b39e0a9c57e43fc8bbfbe4-updateinfo.xml | wc -l
222611

Foreman master server

# cat 16ae6b19f7cf6267ec8397437a16b14444ef4d5a7b1f21cc646a4cb0b94d9ccd-updateinfo.xml | wc -l
250106

Smartproxy1

# cat 4d5b284bfb8e279e0d70756f585c3117307f9393b7a8af4763780b23da30e6e2-updateinfo.xml | wc -l
5552

Smartproxy2

# cat 5b535efd7fdffda1af1e8adc08661a1f74913f390ac983f4126faee1d474daa6-updateinfo.xml | wc -l
5552

All smart proxies updateinfo file has the same number of lines.
So it seems that from orignal repo hosted to foreman it appends a lot of stuff, and from foreman master to smart proxy it truncates a lot of stuff. :smiley:

37

Interesting about the smart proxies, are your smart proxies running Pulp 3 as well or Pulp 2? I’m going to guess Pulp 2 since we’re still early in Pulp 3 smart proxy development.

Got your updated errata file. I will try to reproduce it on Monday. I see you’re putting this repository in a content view, do you see the the missing erratum in the main repository you synced? There is a chance that the erratum was lost when your repository was copied to your content view version. I’d like to confirm that this erratum is also missing from the source, so to speak.

38

Hey, so here are the versions from the main server of services:

### Dynflow

**Version**

0.2.4

### HTTPBoot

**Version**

2.1.3

### Pulp

**Version**

2.1.0

**Pulp server version**

2.21.3

**Database connection**

**Messaging connection**

**Workers**

10

**Supported Content Types**

* deb
* puppet

### Pulpcore

**Version**

2.1.0

**Supported Content Types**

* docker
* file
* yum

### SSH

**Version**

0.3.0

### TFTP

**Version**

2.1.3

**TFTP server**

false

And here are for smart proxies:

### Dynflow

**Version**

0.2.4

### HTTPBoot

**Version**

2.1.2

### Pulp node

**Version**

2.1.0

**Pulp server version**

2.21.3

**Database connection**

**Messaging connection**

**Workers**

4

**Supported Content Types**

* deb
* docker
* file
* puppet
* yum

### SSH

**Version**

0.3.0

### TFTP

**Version**

2.1.2

**TFTP server**

false

### Templates

**Version**

2.1.2

Checking updateinfo file on main foreman server it seems to contain all errata that is should. That is why main foreman server calculates it as applicable, but when trying to apply it, server that is connected via smart proxy has no clue what to upgrade as it reads errata from smart proxy and there is a truncated file of errata.
If I were to connect servers directly to main server errata would most likely be applicable as it would get info. However due to complexity of my infrastructure, not using smart proxies would not be the most optimal solution do to network infrastructure.

39

Thanks, glad to know that the errata is working on the main server at least. I will try to reproduce the smart proxy issue.

1 Like
40

@matemikulic I’ve reproduced the issue. Maybe it’s an issue with On Demand vs Immediate? Are you using the On Demand download policy for your Smart Proxies? The errata should be there regardless for sure.

41

I have it set to Immediate so I don’t have to click sync 24 times on each smatproxy to get new packages/errata everywhere. On repo itself I had it on Immediate and then switched to on demand, no improvements. But I’ve tried running “Complete sync” on smartproxies afterwards it was first synced under Immediate policy. Didn’t help. I’ll try to switch to “on demand” all smart proxies, promote new version of content view and then manually click sync on each to see if that helps, and get back to you with results.

42

No luck. I’ve made new content version, set everything “On demand”, both repo and smart proxy. Initiated sync manually and:

# cat cc5266ad051eef71ecf4db75bf9700ab6937fb1938fd624526c86b97ad0d267a-updateinfo.xml | wc -l
5552

Same number of lines. Instead of 220k+.

43

I saw the issue with On Demand myself, but the fact that you were on Immediate before confirms it’s a separate issue. I got a tip that errata may not be copied if its packages are missing from the repository, I’ll have to check if that applies in this case.

1 Like
44

I doubt that could be the issue. Because for example for missing errata for sed that we used as example earlier.
I see that new package hosted on smart proxy inside os/base repo: sed-4.2.2-6.el7.x86_64.rpm. Errata is in updates repo. Only if everything needs to be in the same repo, which doesn’t make sense. As it works on main server.

I think the answer is somewhere what steps are made when you sync from main server to smart proxy. Something there what it does with repodata creates this issue. Maybe there is too much errata or there is a limit on file size or number of characters or something like that.

It regenerates repodata in some way that most of the updateinfo file is lost.

45

@matemikulic,

After talking with the Pulp team, I think I understand what’s going on. With Pulp 3, the errata in the updateinfo file is not filtered at all. Hence, you still see CEBA-2020:1041 on your Katello server. Pulp 2, on the other hand, actually filters out errata from the updateinfo file if the RPMs are missing from the repository. So it seems that, on your smart proxy with Pulp 2, CEBA-2020:1041 is being filtered out due to the missing sed RPM.

As a test, I manually uploaded the sed RPM to my CentOS 7 updates repository. After republishing the content view version and re-syncing my smart proxy, the CEBA-2020:1041 erratum reappeared.

In general it seems that Pulp 2 does not play well with repositories that don’t serve all of the errata RPMs. Since the errata filtering is a feature on Pulp 2 I do not think that will change. This confusion should be gone once Katello is fully on Pulp 3.

46

Ahaaa, so it is by design. It still doesn’t make sense to me why would this feature exist in pulp2. CentOS has always had separated os, updates, extras repos. But yes, on the other hand, CentOS has no official errata also. But why would pulp care what is in updateinfo. Server is the one who calculates it for itself, anything extra that does not have packages will simply not be applicable?

Do you perhaps know what is the target version for pulp3 being used on smartproxies as well? Is it the next major release Foreman 2.2/Katello 3.17?

If this is what pulp2 does, a possible workaround would then be cloning official mirror for os and extras locally, modifying that repo to include updateinfo file, and syncing foreman from there. That way when smart proxy syncs each repo, pulp2 will keep errata in os that has os related packages errata, same for extras and same for updates. That way the total of errata should be all there but distributed across 3 repos. Only thing I’m worried is will this cause a conflict on main Foreman/Katello server that is using pulp3, having duplicate errata in 3 repos?

47

I definitely agree there! I’m not totally sure why it exists. Perhaps Pulp 2 is trying to shield clients from “bad” errata metadata. I did find an old Pulp issue that might be related https://pulp.plan.io/issues/1548

Pulp 3 for Smart Proxies probably won’t come until Katello 4.0, which is when Pulp 3 will fully replace Pulp 2.

I think this is a good idea for a workaround. I’m not sure if Katello or Pulp 3 will mind the duplicate errata, but it’s definitely something to try. I’d be curious to hear what happens if you do.

One thing to look out for is a community solution to importing CentOS errata for Pulp 3. I know people made it work for Pulp 2, and I’m guessing someone is working on it for Pulp 3. Katello 3.16rc5 and pulp-admin wrt CentOS Errata

1 Like
48

I’ve tried the workaround and it works. It seems pulp3 knows this is duplicate errata and counts it as one. There is no tripling on total number. This is shown on content version page. In latest version each repo: os, updates, extra contain the same errata file, but pulp3 knows its all the same so the number is equal compared to first version.

Selection_039
Selection_0391496×170 26.1 KB

If I run on host

yum update-minimal --security --bugfix -y

It finds all errata and applies it. When checking back host on Content host page after this, I only see applicable errata for kernel. The bug we first discovered that is caused by multiple kernel versions being installed. Hopefully that one will be resolved in the next version.

But as fix for pulp2, this workaround is working awesome. Errata related to packages is distributed in os, updates, extras and in entirety is available to servers despite pulp2 on smart proxies which is most important.

Thank you very much for troubleshooting this with me. I really appreciate it. I think that’s it then. We’ve found and documented all the bugs that I found in this major version.

2 Likes
49

Glad the workaround works! I’ll bookmark this for future users who may run into this issue as well.

Happy to help, definitely uncovered some good bugs here. I’ll be keeping an eye on their progress :+1:

1 Like