Looks like I have reproduced the issue! Just have to dig into why.
1 Like
Applicability is wrong for the kernel because the older packages exist alongside the newer ones. The applicability algorithm assumes only one version of the package is installed, and that itās the oldest. If you removed the older kernel versions (not that you should have to), you would see the applicability correct itself. I will make a bug for this
@matemikulic have you seen applicability be incorrect for any packages that only have a single version installed?
I got so caught up in the applicability calculation issue that I forgot your original problem was something else. I will setup a machine and have it sit for some hours to see if the applicability drops to 0.
I havenāt yet reproduced the disappearing errata bug. Are there any tasks that run around the same time your errata disappears?
Aha, yeah, I have set kernel retention to 3 on all my servers. Comes in handy for troubleshooting.Tnx.
Iāve updated today Foreman/Katello to latest minor version. As far as I see data on Content hosts is still disappearing. But not 100% percent, will confirm by tomorrow.
Regarding other errata thatās showing applicability Iāve checked example of:
CEBA-2020:1041
Type
bugfix
Title
CentOS sed BugFix Update
Issued
2020-04-08
Updated
Reboot Suggested
false
Description
CentOS sed BugFix Update
Solution
Packages
sed-4.2.2-6.el7.src
sed-4.2.2-6.el7.x86_64
This seems to be showing correctly because on server:
# rpm -q sed
sed-4.2.2-5.el7.x86_64
Iāve promoted last this version that Iām testing on October 1st so this errata exists, however hosts cannot see it for some reason:
# yum updateinfo info --advisory=CEBA-2020:1041
Loaded plugins: enabled_repos_upload, fastestmirror, package_upload, product-id, search-disabled-repos, subscription-
: manager, tracer_upload
Loading mirror speeds from cached hostfile
organization_centos7_base | 2.1 kB 00:00:00
organization_centos7_epel | 2.1 kB 00:00:00
organization_centos7_extras | 2.1 kB 00:00:00
organization_centos7_ibrepo | 2.1 kB 00:00:00
organization_centos7_updates | 2.1 kB 00:00:00
updateinfo info done
Uploading Enabled Repositories Report
Loaded plugins: fastestmirror, product-id, subscription-manager
Last content view version was promoted on October 1st, so this errata from April should be visible.
Will try adding this repo with errata directly to host and see if it will be visible then. To see if errata has an issue or Katello. Will get back.
The issue with dissapearing errata has been resolved with update to Katello 3.16.1.2/Foreman 2.1.3. Now its showing constant. It took a bit to recalculate for all servers, but its now constantly showing after a couple of days of tests. Thank you very much for looking into this with me.
Regarding the other issue with wrongly calculated errata.
-
The issue where errata is showing for kernel to be applicable when multiple kernels are installed which you opened a bug for. Will wait for new release where that bug is resolved.
-
Foreman/Katello does not serve some errata but calculates it correctly. Example of CEBA-2020:1041 which I mentioned in previous post.
This is xml for that:
<id>CEBA-2020:1041</id>
<title>CentOS sed BugFix Update</title>
<release>CentOS 7</release>
<issued date="2020-04-08 19:18:42" />
<references>
<reference href="https://access.redhat.com/errata/RHBA-2020:1041" type="self" />
<reference href="https://lists.centos.org/pipermail/centos-cr-announce/2020-April/012599.html" type="self" />
</references>
<description>CentOS sed BugFix Update</description>
<pkglist>
<collection short="EL-7">
<name>CentOS 7</name>
<package arch="src" epoch="0" name="sed" release="6.el7" src="" version="4.2.2">
<filename>sed-4.2.2-6.el7.src.rpm</filename>
</package>
<package arch="x86_64" epoch="0" name="sed" release="6.el7" src="" version="4.2.2">
<filename>sed-4.2.2-6.el7.x86_64.rpm</filename>
</package>
</collection>
</pkglist>
</update>
<update from="cefs@steve-meier.de" status="stable" type="bugfix" version="1.4">
While it can clearly see for example:
<id>CEBA-2020:3348</id>
<title>CentOS curl BugFix Update</title>
<release>CentOS 7</release>
<issued date="2020-08-07 13:25:11" />
<references>
<reference href="https://access.redhat.com/errata/RHBA-2020:3348" type="self" />
<reference href="https://lists.centos.org/pipermail/centos-announce/2020-August/035799.html" type="self" />
</references>
<description>CentOS curl BugFix Update</description>
<pkglist>
<collection short="EL-7">
<name>CentOS 7</name>
<package arch="src" epoch="0" name="curl" release="57.el7_8.1" src="" version="7.29.0">
<filename>curl-7.29.0-57.el7_8.1.src.rpm</filename>
</package>
<package arch="x86_64" epoch="0" name="curl" release="57.el7_8.1" src="" version="7.29.0">
<filename>curl-7.29.0-57.el7_8.1.x86_64.rpm</filename>
</package>
<package arch="i686" epoch="0" name="libcurl" release="57.el7_8.1" src="" version="7.29.0">
<filename>libcurl-7.29.0-57.el7_8.1.i686.rpm</filename>
</package>
<package arch="x86_64" epoch="0" name="libcurl" release="57.el7_8.1" src="" version="7.29.0">
<filename>libcurl-7.29.0-57.el7_8.1.x86_64.rpm</filename>
</package>
<package arch="i686" epoch="0" name="libcurl-devel" release="57.el7_8.1" src="" version="7.29.0">
<filename>libcurl-devel-7.29.0-57.el7_8.1.i686.rpm</filename>
</package>
<package arch="x86_64" epoch="0" name="libcurl-devel" release="57.el7_8.1" src="" version="7.29.0">
<filename>libcurl-devel-7.29.0-57.el7_8.1.x86_64.rpm</filename>
</package>
</collection>
</pkglist>
</update>
<update from="cefs@steve-meier.de" status="stable" type="bugfix" version="1.4">
1 Like
Glad the disappearing errata was fixed!
Iāll take a look at this last errata issue you mentioned with CEBA-2020:1041. What do you mean by Katello doesnāt serve the errata? Are you seeing the applicable errata in the UI but itās not making it to your synced repositoryās updateinfo metadata?
Example of subscribed server, repos hosted on smart proxy.
I try the advisory that I have issues with and get no output.
[root@default-centos-7 vagrant]# yum updateinfo info --advisory=CEBA-2020:1041
Loaded plugins: enabled_repos_upload, fastestmirror, package_upload, product-id, search-disabled-repos, subscription-manager,
: tracer_upload
Loading mirror speeds from cached hostfile
organization_centos7_base | 2.1 kB 00:00:00
organization_centos7_epel | 2.1 kB 00:00:00
organization_centos7_extras | 2.1 kB 00:00:00
organization_centos7_ibrepo | 2.1 kB 00:00:00
organization_centos7_updates | 2.1 kB 00:00:00
updateinfo info done
Uploading Enabled Repositories Report
Loaded plugins: fastestmirror, product-id, subscription-manager
I try then advisory that does work and get info:
[root@default-centos-7 vagrant]# yum updateinfo info --advisory=CEBA-2020:3348
Loaded plugins: enabled_repos_upload, fastestmirror, package_upload, product-id, search-disabled-repos, subscription-manager,
: tracer_upload
Loading mirror speeds from cached hostfile
organization_centos7_base | 2.1 kB 00:00:00
organization_centos7_epel | 2.1 kB 00:00:00
organization_centos7_extras | 2.1 kB 00:00:00
organization_centos7_ibrepo | 2.1 kB 00:00:00
organization_centos7_updates | 2.1 kB 00:00:00
===============================================================================
CentOS curl BugFix Update
===============================================================================
Update ID : CEBA-2020:3348
Release : CentOS 7
Type : bugfix
Status : stable
Issued : 2020-08-07 13:25:11
Updated : 8443-06-18 07:00:00Description : CentOS curl BugFix Update
updateinfo info done
Uploading Enabled Repositories Report
Loaded plugins: fastestmirror, product-id, subscription-manager
Then I unsubscribe that same server and add repo containing errata directly on the server to make sure its not errata or server fault, and the issue is only present when server is subscribed and repo is hosted on Foreman Smart Proxy.
I try the both erratas again.
[root@default-centos-7 vagrant]# yum updateinfo info --advisory=CEBA-2020:3348
Loaded plugins: enabled_repos_upload, fastestmirror, package_upload, product-id, search-disabled-repos, subscription-manager,
: tracer_upload
This system is not registered with an entitlement server. You can use subscription-manager to register.
Loading mirror speeds from cached hostfile
===============================================================================
CentOS curl BugFix Update
===============================================================================
Update ID : CEBA-2020:3348
Release : CentOS 7
Type : bugfix
Status : stable
Issued : 2020-08-07 13:25:11
Description : CentOS curl BugFix Update
updateinfo info done
Uploading Enabled Repositories Report
Cannot upload enabled repos report, is this client registered?
[root@default-centos-7 vagrant]# yum updateinfo info --advisory=CEBA-2020:1041
Loaded plugins: enabled_repos_upload, fastestmirror, package_upload, product-id, search-disabled-repos, subscription-manager,
: tracer_upload
This system is not registered with an entitlement server. You can use subscription-manager to register.
Loading mirror speeds from cached hostfile
===============================================================================
CentOS sed BugFix Update
===============================================================================
Update ID : CEBA-2020:1041
Release : CentOS 7
Type : bugfix
Status : stable
Issued : 2020-04-08 19:18:42
Description : CentOS sed BugFix Update
updateinfo info done
Uploading Enabled Repositories Report
Cannot upload enabled repos report, is this client registered?
And both erratas are available for server to read and apply.
1 Like
Interesting, thanks for the verbose example. Iāll look into it and probably make another redmine bug.
No problem. I know how much it helps when someone gives you exact steps how to reproduce the issue. 
I hope you will be able to find the issue. Thank you.
The only way I was able to āreproduceā the errata issue was by installing the latest version of sed, but even then the erratum did show up with:
yum updateinfo info all --advisory=CEBA-2020:1041
Can you see the missing erratum by doing yum updateinfo info all ... instead of yum updateinfo info ... ? Iām going to guess not since that would just mean you have the latest sed, but itās worth trying.
If it really is missing, then we can check if the advisory/erratum is actually in your private repositoryās updateinfo file. You can check that by just downloading it here (find the hash value by browsing the āPublished atā url in your repo details):
wget http://`hostname`/pulp/repos/<your organization>/<your lifecycle environment>/custom/<your product>/<your updates repo>/repodata/<some hash value>-updateinfo.xml.gz
If itās missing, it might be worth doing a full re-sync of the repository. My erratum was in there, so if itās not for you itās either a transient issue or maybe a bug in your version of pulp-rpm.
Hey, tnx for checking this. You are absolutely correct. Errata is not there on foreman-proxy. How it gets lost on the way, not sure. But Iāve tried to do right now complete sync with content validation on repo with errata before promoting new version of content and it failed. Error output:
Actions::Pulp3::Repository::Repair
Input:
{"repository_id"=>5,
"smart_proxy_id"=>1,
"remote_user"=>"admin",
"remote_cp_user"=>"admin",
"current_request_id"=>nil,
"current_timezone"=>"Europe/Zagreb",
"current_user_id"=>5,
"current_organization_id"=>3,
"current_location_id"=>nil}
Output:
{"response"=>
{"task"=>"/pulp/api/v3/tasks/76d39d7d-6c3f-4d54-8159-57c78986fe2f/"},
"pulp_tasks"=>
[{"pulp_href"=>"/pulp/api/v3/tasks/76d39d7d-6c3f-4d54-8159-57c78986fe2f/",
"pulp_created"=>"2020-10-22T09:29:20.670+00:00",
"state"=>"failed",
"name"=>"pulpcore.app.tasks.repository.repair_version",
"started_at"=>"2020-10-22T09:29:20.741+00:00",
"finished_at"=>"2020-10-22T09:29:49.447+00:00",
"error"=>
{"traceback"=>
" File \"/usr/lib/python3.6/site-packages/rq/worker.py\", line 883, in perform_job\n" +
" File \"/usr/lib/python3.6/site-packages/rq/job.py\", line 645, in perform\n" +
" File \"/usr/lib/python3.6/site-packages/rq/job.py\", line 651, in _execute\n" +
" File \"/usr/lib/python3.6/site-packages/pulpcore/app/tasks/repository.py\", line 118, in repair_version\n" +
" File \"/usr/lib64/python3.6/asyncio/base_events.py\", line 484, in run_until_complete\n" +
" File \"/usr/lib/python3.6/site-packages/pulpcore/app/tasks/repository.py\", line 87, in _repair_repository_version\n" +
" File \"/usr/lib64/python3.6/concurrent/futures/thread.py\", line 56, in run\n" +
" File \"/usr/lib/python3.6/site-packages/pulpcore/app/tasks/repository.py\", line 65, in _verify_ca\n" +
" File \"/usr/lib/python3.6/site-packages/pulpcore/app/tasks/repository.py\", line 65, in <lambda>\n" +
" File \"/usr/lib/python3.6/site-packages/django/core/files/utils.py\", line 16, in <lambda>\n" +
" File \"/usr/lib/python3.6/site-packages/django/db/models/fields/files.py\", line 43, in _get_file\n" +
" File \"/usr/lib/python3.6/site-packages/django/core/files/storage.py\", line 36, in open\n" +
" File \"/usr/lib/python3.6/site-packages/django/core/files/storage.py\", line 224, in _open\n",
"description"=>
"[Errno 24] Too many open files: '/var/lib/pulp/docroot/artifact/12/3a4fe6f7ecc4ceebab59ee4d3bcfe6db3b48c5995fba0b0ad77b48dd4583c8'"},
"worker"=>"/pulp/api/v3/workers/b0284a06-fa71-4814-8da0-231959b62ee5/",
"child_tasks"=>[],
"progress_reports"=>
[{"message"=>"Identify corrupted units",
"code"=>"repair.corrupted",
"state"=>"failed",
"done"=>0},
{"message"=>"Repair corrupted units",
"code"=>"repair.repaired",
"state"=>"failed",
"done"=>0}],
"created_resources"=>[],
"reserved_resources_record"=>
["/pulp/api/v3/repositories/rpm/rpm/7bac9f0a-05db-424a-b34f-1b8c6392815b/"]}],
"task_groups"=>[],
"poll_attempts"=>{"total"=>18, "failed"=>1}}
Exception:
Katello::Errors::Pulp3Error: [Errno 24] Too many open files: '/var/lib/pulp/docroot/artifact/12/3a4fe6f7ecc4ceebab59ee4d3bcfe6db3b48c5995fba0b0ad77b48dd4583c8'
Backtrace:
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.1.2/app/lib/actions/pulp3/abstract_async_task.rb:101:in `block in check_for_errors'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.1.2/app/lib/actions/pulp3/abstract_async_task.rb:99:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.1.2/app/lib/actions/pulp3/abstract_async_task.rb:99:in `check_for_errors'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.1.2/app/lib/actions/pulp3/abstract_async_task.rb:132:in `poll_external_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:100:in `poll_external_task_with_rescue'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:22:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/cancellable.rb:14:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.1.2/app/lib/actions/pulp3/abstract_async_task.rb:10:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:571:in `block (3 levels) in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.1.2/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.1.2/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.1.2/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.1.2/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.1.2/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.1.2/app/lib/actions/middleware/remote_action.rb:16:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/execution_wrapper.rb:88:in `wrap'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:17:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_request_id.rb:49:in `restore_current_request_id'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_user.rb:44:in `restore_curent_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/world.rb:31:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:570:in `block (2 levels) in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in `block in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `block in with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:564:in `execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:285:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/director.rb:93:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors.rb:18:in `run_user_code'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in `execute_job'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in `block (2 levels) in process'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in `invoke'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in `block in process'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in `block (6 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in `local'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in `block (5 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq.rb:37:in `block in <module:Sidekiq>'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in `block (4 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in `stats'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in `block (3 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in `block (2 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in `global'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in `block in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in `with_context'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in `with_job_hash_context'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in `dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in `process'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in `process_one'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in `watchdog'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in `block in safe_thread'
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
I just synced repo updates containing errata from internet to local repo, due to that bug with authentication. Local matches that one on the internet and contains the same files:
3ceff108624118bff78dd0800f6d36b716cce8ad5606653c446b834e76b3b915-primary.sqlite.bz2
48d986911145aa7e5dbcac07ad5b0faa7afdfc021a955ebe4f1ff422d0a71eb9-other.xml.gz
4c2d91df66c87a599e544cf53a3da280fdf59386e4ddf9027495602fa22ca508-filelists.xml.gz
737068c90d4c5e191c7b687188f562fab45fb1525c9c1a680a61a667a28cd4fb-other.sqlite.bz2
afe3eb5eba9fecfdc857c99fbd4574bc451ba16e2bf3d05de180a96e7a2f9d99-filelists.sqlite.bz2
f97bb3a553c10733d6984dfb16c75fc3d376acf119b39e0a9c57e43fc8bbfbe4-updateinfo.xml.gz
ff191ab1ffe6dfebfedb2aa7f6b81635f5c0877a8a229062ae65283e7939ea4c-primary.xml.gz
repomd.xml
repomd.xml.asc
Then I promoted new content version and on smart proxy Iām testing this on after oprimized sync went through did the complete sync. These are the files I found on the same but smart proxy updates repo:
1f5033c2476d2fc39c7347dfae1494c2060f9223478be95e58080c30577ce4be-filelists.xml.gz
783f1a372b71df69ccd2a636a7f8467e74c16ab81b56be78e830fbec5ea3bd41-updateinfo.xml.gz
7dedcf3aa2064dbbe9b33070f67a0fefcfb883434574899a7f1d689015c55e13-primary.xml.gz
95f929621bc5d36411be95f084a3d4f68f829f37c454ae78332ae865a97c249f-other.xml.gz
a27718cc28ec6d71432e0ef3e6da544b7f9d93f6bb7d0a55aacd592d03144b70-comps.xml
repomd.xml
Does katello/pulp run something like createrepo to generate new repodata?
Pulp does regenerate the new repodata itself, Iām not sure how exactly. Is your smart proxy missing any RPMs or errata? Iām not surprised that the repodata is structured differently, but it should have all the same content.
I think the bug you just found above with the repo sync + checksum checking is a Pulp issue, I will raise it with the Pulp team.
1 Like
I might recommend destroying the repository thatās missing the erratum and recreating/resyncing it. If your issue is reproducible, I will make another Pulp 3 issue for it.
Pulp issue for repo repair: https://pulp.plan.io/issues/7735
Hey, so I tried what you recommended. Iāve created new repo called updates-errata. Iāve synced that new repo from source where repodata looks like this:
3ceff108624118bff78dd0800f6d36b716cce8ad5606653c446b834e76b3b915-primary.sqlite.bz2
48d986911145aa7e5dbcac07ad5b0faa7afdfc021a955ebe4f1ff422d0a71eb9-other.xml.gz
4c2d91df66c87a599e544cf53a3da280fdf59386e4ddf9027495602fa22ca508-filelists.xml.gz
737068c90d4c5e191c7b687188f562fab45fb1525c9c1a680a61a667a28cd4fb-other.sqlite.bz2
afe3eb5eba9fecfdc857c99fbd4574bc451ba16e2bf3d05de180a96e7a2f9d99-filelists.sqlite.bz2
f97bb3a553c10733d6984dfb16c75fc3d376acf119b39e0a9c57e43fc8bbfbe4-updateinfo.xml.gz
ff191ab1ffe6dfebfedb2aa7f6b81635f5c0877a8a229062ae65283e7939ea4c-primary.xml.gz
repomd.xml
repomd.xml.asc
Iāve then in Content view replaced old repo called updates with new named updates-errata and published new version. Checking on master server that repoās repodata looks like this:
16ae6b19f7cf6267ec8397437a16b14444ef4d5a7b1f21cc646a4cb0b94d9ccd-updateinfo.xml.gz
1c0e2a1afa47b56c9afc9bf8a5d40b94d9fa57ccd9edc103412b65498f52ba33-primary.xml.gz
1f6fd37cd09b0fe00b4f0a5a66bf7adf2620d59c708c9f33a8ce814999ec1a2c-filelists.sqlite.bz2
3263b4206bb57e382ffa8b0bc8b4d1bb1900d3162fb3fc49ebdbc1b33edeae47-primary.sqlite.bz2
6d20d179dda953a80005ff2915e8d8cd15ab7205f0fa6c2e42f5d1ac7a17e6f8-other.sqlite.bz2
c5b7e04df0454379ce33e162e6e35dff63efff87f9374f4c3fbcdc7a876070ab-filelists.xml.gz
f22c7871fe5ee1a1ea8822c6e2b4a2a7deee7952138ba01da874b31008807be4-other.xml.gz
repomd.xml
So it seems master has changed hashes and data inside.
I have 24 smart proxies connected to that master so Iāve went and checked around on a couple of them that same repo.
Smartproxy1
4d5b284bfb8e279e0d70756f585c3117307f9393b7a8af4763780b23da30e6e2-updateinfo.xml.gz
a27718cc28ec6d71432e0ef3e6da544b7f9d93f6bb7d0a55aacd592d03144b70-comps.xml
b1456bfc2d50d881bb5c8f6218a8d67bca76758b1e10740125311e5c68e9b545-primary.xml.gz
d03468c2194261d73b56bc4a9db8e56b11051723f924be483352c955e8fd97e8-other.xml.gz
d2dd49be984f54fceb37beb4aded7e34065069ccc13bd4fd002e4e26acc3e47a-filelists.xml.gz
repomd.xml
Smartproxy2
15ae9daa45709ea783c02d2be109f1fbb543bf08d869008d628ff3fc52f25eb8-other.xml.gz
718b264df61f770cb4c42e6899c54e806300a5283c08086d969c84d00d5c4344-primary.xml.gz
a27718cc28ec6d71432e0ef3e6da544b7f9d93f6bb7d0a55aacd592d03144b70-comps.xml
ba81fae9c8ff21ff0b71691bc3f5f9a056423185f7bb3ef8b0130d1316026bbd-filelists.xml.gz
cd167ac6e683c5e77555b4cf3a8b77cc546a329371856dd0507cb3c710b3ff6d-updateinfo.xml.gz
repomd.xml
Smartproxy3
117611b1e5191cf2025493e2f10884aaf9f736c9b25ec0ba212351ce6994b02d-other.xml.gz
33587a82c4ce441476acbc777e317eedfc6c793b9428de43148ba1ce3fbb1f17-primary.xml.gz
339000047b5867c1694b9d3d0aace12c1b33a2d3f96db1c71d43b2ff8dd8f09c-updateinfo.xml.gz
a27718cc28ec6d71432e0ef3e6da544b7f9d93f6bb7d0a55aacd592d03144b70-comps.xml
d2f8c0ac07801eaa44979a103ac26a551be197eb154f688c49a26f98ce021660-filelists.xml.gz
repomd.xml
Smartproxy4
003c6cc0d584c348a53712397f78cec5f5183850c005198c3c3df030d2726a1f-primary.xml.gz
059d2dc10978eccbf1755634c498455090befa4c88d1feec006462877d91caf3-other.xml.gz
5b535efd7fdffda1af1e8adc08661a1f74913f390ac983f4126faee1d474daa6-updateinfo.xml.gz
a27718cc28ec6d71432e0ef3e6da544b7f9d93f6bb7d0a55aacd592d03144b70-comps.xml
c08d3861193da0831c25f2209c14ba61e224cf56d25c51279adcc8c9c8fc9ae5-filelists.xml.gz
repomd.xml
And so on. Any smart proxy that I check Iāll find different hash prefixes. So it seems that pulp does regenerate repodata. And smart proxy also when syncing.
Errata that did not work in previous repo, doesnāt work in the new one as well.
Iāll send you latest errata file so you can do tests if necessary.
Iāve downloaded the version from orignal repo, version from master and smart proxies, unzipped them and numbered the lines.
Original repo
# cat f97bb3a553c10733d6984dfb16c75fc3d376acf119b39e0a9c57e43fc8bbfbe4-updateinfo.xml | wc -l
222611
Foreman master server
# cat 16ae6b19f7cf6267ec8397437a16b14444ef4d5a7b1f21cc646a4cb0b94d9ccd-updateinfo.xml | wc -l
250106
Smartproxy1
# cat 4d5b284bfb8e279e0d70756f585c3117307f9393b7a8af4763780b23da30e6e2-updateinfo.xml | wc -l
5552
Smartproxy2
# cat 5b535efd7fdffda1af1e8adc08661a1f74913f390ac983f4126faee1d474daa6-updateinfo.xml | wc -l
5552
All smart proxies updateinfo file has the same number of lines.
So it seems that from orignal repo hosted to foreman it appends a lot of stuff, and from foreman master to smart proxy it truncates a lot of stuff. 
Interesting about the smart proxies, are your smart proxies running Pulp 3 as well or Pulp 2? Iām going to guess Pulp 2 since weāre still early in Pulp 3 smart proxy development.
Got your updated errata file. I will try to reproduce it on Monday. I see youāre putting this repository in a content view, do you see the the missing erratum in the main repository you synced? There is a chance that the erratum was lost when your repository was copied to your content view version. Iād like to confirm that this erratum is also missing from the source, so to speak.
Hey, so here are the versions from the main server of services:
### Dynflow
**Version**
0.2.4
### HTTPBoot
**Version**
2.1.3
### Pulp
**Version**
2.1.0
**Pulp server version**
2.21.3
**Database connection**
**Messaging connection**
**Workers**
10
**Supported Content Types**
* deb
* puppet
### Pulpcore
**Version**
2.1.0
**Supported Content Types**
* docker
* file
* yum
### SSH
**Version**
0.3.0
### TFTP
**Version**
2.1.3
**TFTP server**
false
And here are for smart proxies:
### Dynflow
**Version**
0.2.4
### HTTPBoot
**Version**
2.1.2
### Pulp node
**Version**
2.1.0
**Pulp server version**
2.21.3
**Database connection**
**Messaging connection**
**Workers**
4
**Supported Content Types**
* deb
* docker
* file
* puppet
* yum
### SSH
**Version**
0.3.0
### TFTP
**Version**
2.1.2
**TFTP server**
false
### Templates
**Version**
2.1.2
Checking updateinfo file on main foreman server it seems to contain all errata that is should. That is why main foreman server calculates it as applicable, but when trying to apply it, server that is connected via smart proxy has no clue what to upgrade as it reads errata from smart proxy and there is a truncated file of errata.
If I were to connect servers directly to main server errata would most likely be applicable as it would get info. However due to complexity of my infrastructure, not using smart proxies would not be the most optimal solution do to network infrastructure.
Thanks, glad to know that the errata is working on the main server at least. I will try to reproduce the smart proxy issue.
1 Like
@matemikulic Iāve reproduced the issue. Maybe itās an issue with On Demand vs Immediate? Are you using the On Demand download policy for your Smart Proxies? The errata should be there regardless for sure.