Glad the disappearing errata was fixed!
I’ll take a look at this last errata issue you mentioned with CEBA-2020:1041. What do you mean by Katello doesn’t serve the errata? Are you seeing the applicable errata in the UI but it’s not making it to your synced repository’s updateinfo metadata?
Example of subscribed server, repos hosted on smart proxy.
I try the advisory that I have issues with and get no output.
[root@default-centos-7 vagrant]# yum updateinfo info --advisory=CEBA-2020:1041
Loaded plugins: enabled_repos_upload, fastestmirror, package_upload, product-id, search-disabled-repos, subscription-manager,
: tracer_upload
Loading mirror speeds from cached hostfile
organization_centos7_base | 2.1 kB 00:00:00
organization_centos7_epel | 2.1 kB 00:00:00
organization_centos7_extras | 2.1 kB 00:00:00
organization_centos7_ibrepo | 2.1 kB 00:00:00
organization_centos7_updates | 2.1 kB 00:00:00
updateinfo info done
Uploading Enabled Repositories Report
Loaded plugins: fastestmirror, product-id, subscription-manager
I try then advisory that does work and get info:
[root@default-centos-7 vagrant]# yum updateinfo info --advisory=CEBA-2020:3348
Loaded plugins: enabled_repos_upload, fastestmirror, package_upload, product-id, search-disabled-repos, subscription-manager,
: tracer_upload
Loading mirror speeds from cached hostfile
organization_centos7_base | 2.1 kB 00:00:00
organization_centos7_epel | 2.1 kB 00:00:00
organization_centos7_extras | 2.1 kB 00:00:00
organization_centos7_ibrepo | 2.1 kB 00:00:00
organization_centos7_updates | 2.1 kB 00:00:00
===============================================================================
CentOS curl BugFix Update
===============================================================================
Update ID : CEBA-2020:3348
Release : CentOS 7
Type : bugfix
Status : stable
Issued : 2020-08-07 13:25:11
Updated : 8443-06-18 07:00:00Description : CentOS curl BugFix Update
updateinfo info done
Uploading Enabled Repositories Report
Loaded plugins: fastestmirror, product-id, subscription-manager
Then I unsubscribe that same server and add repo containing errata directly on the server to make sure its not errata or server fault, and the issue is only present when server is subscribed and repo is hosted on Foreman Smart Proxy.
I try the both erratas again.
[root@default-centos-7 vagrant]# yum updateinfo info --advisory=CEBA-2020:3348
Loaded plugins: enabled_repos_upload, fastestmirror, package_upload, product-id, search-disabled-repos, subscription-manager,
: tracer_upload
This system is not registered with an entitlement server. You can use subscription-manager to register.
Loading mirror speeds from cached hostfile
===============================================================================
CentOS curl BugFix Update
===============================================================================
Update ID : CEBA-2020:3348
Release : CentOS 7
Type : bugfix
Status : stable
Issued : 2020-08-07 13:25:11
Description : CentOS curl BugFix Update
updateinfo info done
Uploading Enabled Repositories Report
Cannot upload enabled repos report, is this client registered?
[root@default-centos-7 vagrant]# yum updateinfo info --advisory=CEBA-2020:1041
Loaded plugins: enabled_repos_upload, fastestmirror, package_upload, product-id, search-disabled-repos, subscription-manager,
: tracer_upload
This system is not registered with an entitlement server. You can use subscription-manager to register.
Loading mirror speeds from cached hostfile
===============================================================================
CentOS sed BugFix Update
===============================================================================
Update ID : CEBA-2020:1041
Release : CentOS 7
Type : bugfix
Status : stable
Issued : 2020-04-08 19:18:42
Description : CentOS sed BugFix Update
updateinfo info done
Uploading Enabled Repositories Report
Cannot upload enabled repos report, is this client registered?
And both erratas are available for server to read and apply.
Interesting, thanks for the verbose example. I’ll look into it and probably make another redmine bug.
No problem. I know how much it helps when someone gives you exact steps how to reproduce the issue. 
I hope you will be able to find the issue. Thank you.
The only way I was able to “reproduce” the errata issue was by installing the latest version of sed, but even then the erratum did show up with:
yum updateinfo info all --advisory=CEBA-2020:1041
Can you see the missing erratum by doing yum updateinfo info all ... instead of yum updateinfo info ... ? I’m going to guess not since that would just mean you have the latest sed, but it’s worth trying.
If it really is missing, then we can check if the advisory/erratum is actually in your private repository’s updateinfo file. You can check that by just downloading it here (find the hash value by browsing the “Published at” url in your repo details):
wget http://`hostname`/pulp/repos/<your organization>/<your lifecycle environment>/custom/<your product>/<your updates repo>/repodata/<some hash value>-updateinfo.xml.gz
If it’s missing, it might be worth doing a full re-sync of the repository. My erratum was in there, so if it’s not for you it’s either a transient issue or maybe a bug in your version of pulp-rpm.
Hey, tnx for checking this. You are absolutely correct. Errata is not there on foreman-proxy. How it gets lost on the way, not sure. But I’ve tried to do right now complete sync with content validation on repo with errata before promoting new version of content and it failed. Error output:
Actions::Pulp3::Repository::Repair
Input:
{"repository_id"=>5,
"smart_proxy_id"=>1,
"remote_user"=>"admin",
"remote_cp_user"=>"admin",
"current_request_id"=>nil,
"current_timezone"=>"Europe/Zagreb",
"current_user_id"=>5,
"current_organization_id"=>3,
"current_location_id"=>nil}
Output:
{"response"=>
{"task"=>"/pulp/api/v3/tasks/76d39d7d-6c3f-4d54-8159-57c78986fe2f/"},
"pulp_tasks"=>
[{"pulp_href"=>"/pulp/api/v3/tasks/76d39d7d-6c3f-4d54-8159-57c78986fe2f/",
"pulp_created"=>"2020-10-22T09:29:20.670+00:00",
"state"=>"failed",
"name"=>"pulpcore.app.tasks.repository.repair_version",
"started_at"=>"2020-10-22T09:29:20.741+00:00",
"finished_at"=>"2020-10-22T09:29:49.447+00:00",
"error"=>
{"traceback"=>
" File \"/usr/lib/python3.6/site-packages/rq/worker.py\", line 883, in perform_job\n" +
" File \"/usr/lib/python3.6/site-packages/rq/job.py\", line 645, in perform\n" +
" File \"/usr/lib/python3.6/site-packages/rq/job.py\", line 651, in _execute\n" +
" File \"/usr/lib/python3.6/site-packages/pulpcore/app/tasks/repository.py\", line 118, in repair_version\n" +
" File \"/usr/lib64/python3.6/asyncio/base_events.py\", line 484, in run_until_complete\n" +
" File \"/usr/lib/python3.6/site-packages/pulpcore/app/tasks/repository.py\", line 87, in _repair_repository_version\n" +
" File \"/usr/lib64/python3.6/concurrent/futures/thread.py\", line 56, in run\n" +
" File \"/usr/lib/python3.6/site-packages/pulpcore/app/tasks/repository.py\", line 65, in _verify_ca\n" +
" File \"/usr/lib/python3.6/site-packages/pulpcore/app/tasks/repository.py\", line 65, in <lambda>\n" +
" File \"/usr/lib/python3.6/site-packages/django/core/files/utils.py\", line 16, in <lambda>\n" +
" File \"/usr/lib/python3.6/site-packages/django/db/models/fields/files.py\", line 43, in _get_file\n" +
" File \"/usr/lib/python3.6/site-packages/django/core/files/storage.py\", line 36, in open\n" +
" File \"/usr/lib/python3.6/site-packages/django/core/files/storage.py\", line 224, in _open\n",
"description"=>
"[Errno 24] Too many open files: '/var/lib/pulp/docroot/artifact/12/3a4fe6f7ecc4ceebab59ee4d3bcfe6db3b48c5995fba0b0ad77b48dd4583c8'"},
"worker"=>"/pulp/api/v3/workers/b0284a06-fa71-4814-8da0-231959b62ee5/",
"child_tasks"=>[],
"progress_reports"=>
[{"message"=>"Identify corrupted units",
"code"=>"repair.corrupted",
"state"=>"failed",
"done"=>0},
{"message"=>"Repair corrupted units",
"code"=>"repair.repaired",
"state"=>"failed",
"done"=>0}],
"created_resources"=>[],
"reserved_resources_record"=>
["/pulp/api/v3/repositories/rpm/rpm/7bac9f0a-05db-424a-b34f-1b8c6392815b/"]}],
"task_groups"=>[],
"poll_attempts"=>{"total"=>18, "failed"=>1}}
Exception:
Katello::Errors::Pulp3Error: [Errno 24] Too many open files: '/var/lib/pulp/docroot/artifact/12/3a4fe6f7ecc4ceebab59ee4d3bcfe6db3b48c5995fba0b0ad77b48dd4583c8'
Backtrace:
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.1.2/app/lib/actions/pulp3/abstract_async_task.rb:101:in `block in check_for_errors'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.1.2/app/lib/actions/pulp3/abstract_async_task.rb:99:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.1.2/app/lib/actions/pulp3/abstract_async_task.rb:99:in `check_for_errors'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.1.2/app/lib/actions/pulp3/abstract_async_task.rb:132:in `poll_external_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:100:in `poll_external_task_with_rescue'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:22:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/cancellable.rb:14:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.1.2/app/lib/actions/pulp3/abstract_async_task.rb:10:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:571:in `block (3 levels) in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.1.2/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.1.2/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.1.2/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.1.2/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.1.2/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.1.2/app/lib/actions/middleware/remote_action.rb:16:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/execution_wrapper.rb:88:in `wrap'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:17:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_request_id.rb:49:in `restore_current_request_id'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_user.rb:44:in `restore_curent_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/world.rb:31:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:570:in `block (2 levels) in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in `block in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `block in with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:564:in `execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:285:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/director.rb:93:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors.rb:18:in `run_user_code'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in `execute_job'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in `block (2 levels) in process'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in `invoke'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in `block in process'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in `block (6 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in `local'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in `block (5 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq.rb:37:in `block in <module:Sidekiq>'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in `block (4 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in `stats'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in `block (3 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in `block (2 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in `global'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in `block in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in `with_context'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in `with_job_hash_context'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in `dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in `process'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in `process_one'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in `watchdog'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in `block in safe_thread'
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
I just synced repo updates containing errata from internet to local repo, due to that bug with authentication. Local matches that one on the internet and contains the same files:
3ceff108624118bff78dd0800f6d36b716cce8ad5606653c446b834e76b3b915-primary.sqlite.bz2
48d986911145aa7e5dbcac07ad5b0faa7afdfc021a955ebe4f1ff422d0a71eb9-other.xml.gz
4c2d91df66c87a599e544cf53a3da280fdf59386e4ddf9027495602fa22ca508-filelists.xml.gz
737068c90d4c5e191c7b687188f562fab45fb1525c9c1a680a61a667a28cd4fb-other.sqlite.bz2
afe3eb5eba9fecfdc857c99fbd4574bc451ba16e2bf3d05de180a96e7a2f9d99-filelists.sqlite.bz2
f97bb3a553c10733d6984dfb16c75fc3d376acf119b39e0a9c57e43fc8bbfbe4-updateinfo.xml.gz
ff191ab1ffe6dfebfedb2aa7f6b81635f5c0877a8a229062ae65283e7939ea4c-primary.xml.gz
repomd.xml
repomd.xml.asc
Then I promoted new content version and on smart proxy I’m testing this on after oprimized sync went through did the complete sync. These are the files I found on the same but smart proxy updates repo:
1f5033c2476d2fc39c7347dfae1494c2060f9223478be95e58080c30577ce4be-filelists.xml.gz
783f1a372b71df69ccd2a636a7f8467e74c16ab81b56be78e830fbec5ea3bd41-updateinfo.xml.gz
7dedcf3aa2064dbbe9b33070f67a0fefcfb883434574899a7f1d689015c55e13-primary.xml.gz
95f929621bc5d36411be95f084a3d4f68f829f37c454ae78332ae865a97c249f-other.xml.gz
a27718cc28ec6d71432e0ef3e6da544b7f9d93f6bb7d0a55aacd592d03144b70-comps.xml
repomd.xml
Does katello/pulp run something like createrepo to generate new repodata?
Pulp does regenerate the new repodata itself, I’m not sure how exactly. Is your smart proxy missing any RPMs or errata? I’m not surprised that the repodata is structured differently, but it should have all the same content.
I think the bug you just found above with the repo sync + checksum checking is a Pulp issue, I will raise it with the Pulp team.
I might recommend destroying the repository that’s missing the erratum and recreating/resyncing it. If your issue is reproducible, I will make another Pulp 3 issue for it.
Pulp issue for repo repair: https://pulp.plan.io/issues/7735
Hey, so I tried what you recommended. I’ve created new repo called updates-errata. I’ve synced that new repo from source where repodata looks like this:
3ceff108624118bff78dd0800f6d36b716cce8ad5606653c446b834e76b3b915-primary.sqlite.bz2
48d986911145aa7e5dbcac07ad5b0faa7afdfc021a955ebe4f1ff422d0a71eb9-other.xml.gz
4c2d91df66c87a599e544cf53a3da280fdf59386e4ddf9027495602fa22ca508-filelists.xml.gz
737068c90d4c5e191c7b687188f562fab45fb1525c9c1a680a61a667a28cd4fb-other.sqlite.bz2
afe3eb5eba9fecfdc857c99fbd4574bc451ba16e2bf3d05de180a96e7a2f9d99-filelists.sqlite.bz2
f97bb3a553c10733d6984dfb16c75fc3d376acf119b39e0a9c57e43fc8bbfbe4-updateinfo.xml.gz
ff191ab1ffe6dfebfedb2aa7f6b81635f5c0877a8a229062ae65283e7939ea4c-primary.xml.gz
repomd.xml
repomd.xml.asc
I’ve then in Content view replaced old repo called updates with new named updates-errata and published new version. Checking on master server that repo’s repodata looks like this:
16ae6b19f7cf6267ec8397437a16b14444ef4d5a7b1f21cc646a4cb0b94d9ccd-updateinfo.xml.gz
1c0e2a1afa47b56c9afc9bf8a5d40b94d9fa57ccd9edc103412b65498f52ba33-primary.xml.gz
1f6fd37cd09b0fe00b4f0a5a66bf7adf2620d59c708c9f33a8ce814999ec1a2c-filelists.sqlite.bz2
3263b4206bb57e382ffa8b0bc8b4d1bb1900d3162fb3fc49ebdbc1b33edeae47-primary.sqlite.bz2
6d20d179dda953a80005ff2915e8d8cd15ab7205f0fa6c2e42f5d1ac7a17e6f8-other.sqlite.bz2
c5b7e04df0454379ce33e162e6e35dff63efff87f9374f4c3fbcdc7a876070ab-filelists.xml.gz
f22c7871fe5ee1a1ea8822c6e2b4a2a7deee7952138ba01da874b31008807be4-other.xml.gz
repomd.xml
So it seems master has changed hashes and data inside.
I have 24 smart proxies connected to that master so I’ve went and checked around on a couple of them that same repo.
Smartproxy1
4d5b284bfb8e279e0d70756f585c3117307f9393b7a8af4763780b23da30e6e2-updateinfo.xml.gz
a27718cc28ec6d71432e0ef3e6da544b7f9d93f6bb7d0a55aacd592d03144b70-comps.xml
b1456bfc2d50d881bb5c8f6218a8d67bca76758b1e10740125311e5c68e9b545-primary.xml.gz
d03468c2194261d73b56bc4a9db8e56b11051723f924be483352c955e8fd97e8-other.xml.gz
d2dd49be984f54fceb37beb4aded7e34065069ccc13bd4fd002e4e26acc3e47a-filelists.xml.gz
repomd.xml
Smartproxy2
15ae9daa45709ea783c02d2be109f1fbb543bf08d869008d628ff3fc52f25eb8-other.xml.gz
718b264df61f770cb4c42e6899c54e806300a5283c08086d969c84d00d5c4344-primary.xml.gz
a27718cc28ec6d71432e0ef3e6da544b7f9d93f6bb7d0a55aacd592d03144b70-comps.xml
ba81fae9c8ff21ff0b71691bc3f5f9a056423185f7bb3ef8b0130d1316026bbd-filelists.xml.gz
cd167ac6e683c5e77555b4cf3a8b77cc546a329371856dd0507cb3c710b3ff6d-updateinfo.xml.gz
repomd.xml
Smartproxy3
117611b1e5191cf2025493e2f10884aaf9f736c9b25ec0ba212351ce6994b02d-other.xml.gz
33587a82c4ce441476acbc777e317eedfc6c793b9428de43148ba1ce3fbb1f17-primary.xml.gz
339000047b5867c1694b9d3d0aace12c1b33a2d3f96db1c71d43b2ff8dd8f09c-updateinfo.xml.gz
a27718cc28ec6d71432e0ef3e6da544b7f9d93f6bb7d0a55aacd592d03144b70-comps.xml
d2f8c0ac07801eaa44979a103ac26a551be197eb154f688c49a26f98ce021660-filelists.xml.gz
repomd.xml
Smartproxy4
003c6cc0d584c348a53712397f78cec5f5183850c005198c3c3df030d2726a1f-primary.xml.gz
059d2dc10978eccbf1755634c498455090befa4c88d1feec006462877d91caf3-other.xml.gz
5b535efd7fdffda1af1e8adc08661a1f74913f390ac983f4126faee1d474daa6-updateinfo.xml.gz
a27718cc28ec6d71432e0ef3e6da544b7f9d93f6bb7d0a55aacd592d03144b70-comps.xml
c08d3861193da0831c25f2209c14ba61e224cf56d25c51279adcc8c9c8fc9ae5-filelists.xml.gz
repomd.xml
And so on. Any smart proxy that I check I’ll find different hash prefixes. So it seems that pulp does regenerate repodata. And smart proxy also when syncing.
Errata that did not work in previous repo, doesn’t work in the new one as well.
I’ll send you latest errata file so you can do tests if necessary.
I’ve downloaded the version from orignal repo, version from master and smart proxies, unzipped them and numbered the lines.
Original repo
# cat f97bb3a553c10733d6984dfb16c75fc3d376acf119b39e0a9c57e43fc8bbfbe4-updateinfo.xml | wc -l
222611
Foreman master server
# cat 16ae6b19f7cf6267ec8397437a16b14444ef4d5a7b1f21cc646a4cb0b94d9ccd-updateinfo.xml | wc -l
250106
Smartproxy1
# cat 4d5b284bfb8e279e0d70756f585c3117307f9393b7a8af4763780b23da30e6e2-updateinfo.xml | wc -l
5552
Smartproxy2
# cat 5b535efd7fdffda1af1e8adc08661a1f74913f390ac983f4126faee1d474daa6-updateinfo.xml | wc -l
5552
All smart proxies updateinfo file has the same number of lines.
So it seems that from orignal repo hosted to foreman it appends a lot of stuff, and from foreman master to smart proxy it truncates a lot of stuff. 
Interesting about the smart proxies, are your smart proxies running Pulp 3 as well or Pulp 2? I’m going to guess Pulp 2 since we’re still early in Pulp 3 smart proxy development.
Got your updated errata file. I will try to reproduce it on Monday. I see you’re putting this repository in a content view, do you see the the missing erratum in the main repository you synced? There is a chance that the erratum was lost when your repository was copied to your content view version. I’d like to confirm that this erratum is also missing from the source, so to speak.
Hey, so here are the versions from the main server of services:
### Dynflow
**Version**
0.2.4
### HTTPBoot
**Version**
2.1.3
### Pulp
**Version**
2.1.0
**Pulp server version**
2.21.3
**Database connection**
**Messaging connection**
**Workers**
10
**Supported Content Types**
* deb
* puppet
### Pulpcore
**Version**
2.1.0
**Supported Content Types**
* docker
* file
* yum
### SSH
**Version**
0.3.0
### TFTP
**Version**
2.1.3
**TFTP server**
false
And here are for smart proxies:
### Dynflow
**Version**
0.2.4
### HTTPBoot
**Version**
2.1.2
### Pulp node
**Version**
2.1.0
**Pulp server version**
2.21.3
**Database connection**
**Messaging connection**
**Workers**
4
**Supported Content Types**
* deb
* docker
* file
* puppet
* yum
### SSH
**Version**
0.3.0
### TFTP
**Version**
2.1.2
**TFTP server**
false
### Templates
**Version**
2.1.2
Checking updateinfo file on main foreman server it seems to contain all errata that is should. That is why main foreman server calculates it as applicable, but when trying to apply it, server that is connected via smart proxy has no clue what to upgrade as it reads errata from smart proxy and there is a truncated file of errata.
If I were to connect servers directly to main server errata would most likely be applicable as it would get info. However due to complexity of my infrastructure, not using smart proxies would not be the most optimal solution do to network infrastructure.
Thanks, glad to know that the errata is working on the main server at least. I will try to reproduce the smart proxy issue.
@matemikulic I’ve reproduced the issue. Maybe it’s an issue with On Demand vs Immediate? Are you using the On Demand download policy for your Smart Proxies? The errata should be there regardless for sure.
I have it set to Immediate so I don’t have to click sync 24 times on each smatproxy to get new packages/errata everywhere. On repo itself I had it on Immediate and then switched to on demand, no improvements. But I’ve tried running “Complete sync” on smartproxies afterwards it was first synced under Immediate policy. Didn’t help. I’ll try to switch to “on demand” all smart proxies, promote new version of content view and then manually click sync on each to see if that helps, and get back to you with results.
No luck. I’ve made new content version, set everything “On demand”, both repo and smart proxy. Initiated sync manually and:
# cat cc5266ad051eef71ecf4db75bf9700ab6937fb1938fd624526c86b97ad0d267a-updateinfo.xml | wc -l
5552
Same number of lines. Instead of 220k+.
I saw the issue with On Demand myself, but the fact that you were on Immediate before confirms it’s a separate issue. I got a tip that errata may not be copied if its packages are missing from the repository, I’ll have to check if that applies in this case.
I doubt that could be the issue. Because for example for missing errata for sed that we used as example earlier.
I see that new package hosted on smart proxy inside os/base repo: sed-4.2.2-6.el7.x86_64.rpm. Errata is in updates repo. Only if everything needs to be in the same repo, which doesn’t make sense. As it works on main server.
I think the answer is somewhere what steps are made when you sync from main server to smart proxy. Something there what it does with repodata creates this issue. Maybe there is too much errata or there is a limit on file size or number of characters or something like that.
It regenerates repodata in some way that most of the updateinfo file is lost.
After talking with the Pulp team, I think I understand what’s going on. With Pulp 3, the errata in the updateinfo file is not filtered at all. Hence, you still see CEBA-2020:1041 on your Katello server. Pulp 2, on the other hand, actually filters out errata from the updateinfo file if the RPMs are missing from the repository. So it seems that, on your smart proxy with Pulp 2, CEBA-2020:1041 is being filtered out due to the missing sed RPM.
As a test, I manually uploaded the sed RPM to my CentOS 7 updates repository. After republishing the content view version and re-syncing my smart proxy, the CEBA-2020:1041 erratum reappeared.
In general it seems that Pulp 2 does not play well with repositories that don’t serve all of the errata RPMs. Since the errata filtering is a feature on Pulp 2 I do not think that will change. This confusion should be gone once Katello is fully on Pulp 3.