Debian 11 host Warning: No installed packages and/or enabled repositories have been reported by subscription-manager

iballou · November 2, 2022, 2:48pm

Something from the Katello side that could help with debugging:

When your host is supposed to be uploading package profile information to Katello, do you see any API activity on your server? Namely something like Started PUT "/rhsm/consumers/db169785-984e-4183-be7a-18c36dce6d57/profiles ?

If the API isn’t getting hit, subscription-manager is not reaching Katello.

Another thing perhaps worth checking is the existence of the profile cache. On EL8 it’s here: /var/lib/rhsm/cache/profile.json. I’m not sure if the same can be said for Debian.

javieitez · November 2, 2022, 4:25pm

Hi @iballou,

Thank you for the clues… Where can I find the log for the API activity?

Anyway, I can asure there is communication between both ends, since the warning message is refreshed the very same second I install a package with apt.

So, the Katello server receives data from the Debian host, it just misses the package related information

Also, remote execution works, so communication works both ways.

The file exists in the same path. Also, if I delete it and reinstall a packege with apt, it is re-created.

I’m still investigating, any guidance or clues will be greatly appreciated…

Thank you all for the attention

iballou · November 2, 2022, 6:21pm

Hey there @javieitez,

You should be able to find the API hits either in /var/log/foreman/production.log or the access logs under /var/log/httpd/.

If you are, and it’s not obvious what is getting passed into the API, we can further debug by throwing some extra log statements in your code.

Do you see the right packages and repositories in /var/lib/rhsm/cache/profile.json ?

javieitez · November 3, 2022, 7:47am

OK, here’s what happens in the /var/log/foreman/production.log when I install a package on the Debian host

2022-11-03T08:31:44 [I|app|4f979351] Processing by Katello::Api::Rhsm::CandlepinProxiesController#get as JSON
2022-11-03T08:31:44 [I|app|4f979351]   Parameters: {"id"=>"176dcf1c-d750-4b6a-b840-7b2ce72ac7d5"}
2022-11-03T08:31:45 [I|app|4f979351] Completed 200 OK in 32ms (Views: 0.2ms | ActiveRecord: 3.4ms | Allocations: 3248)
2022-11-03T08:31:45 [I|app|bf4b49c4] Started GET "/rhsm/status" for 10.101.137.9 at 2022-11-03 08:31:45 +0100
2022-11-03T08:31:45 [I|app|bf4b49c4] Processing by Katello::Api::Rhsm::CandlepinProxiesController#server_status as JSON
2022-11-03T08:31:45 [I|app|bf4b49c4] Completed 200 OK in 29ms (Views: 0.4ms | ActiveRecord: 1.9ms | Allocations: 2727)
2022-11-03T08:31:45 [I|app|db48df77] Started PUT "/rhsm/consumers/3262ea70-97ca-4825-a917-169238f59367/profiles" for 10.101.137.9 at 2022-11-03 08:31:45 +0100
2022-11-03T08:31:45 [I|app|db48df77] Processing by Katello::Api::Rhsm::CandlepinDynflowProxyController#upload_profiles as JSON
2022-11-03T08:31:45 [I|app|db48df77]   Parameters: {"id"=>"3262ea70-97ca-4825-a917-169238f59367"}
2022-11-03T08:31:45 [I|app|db48df77] Completed 200 OK in 717ms (Views: 2.2ms | ActiveRecord: 130.3ms | Allocations: 141983)
2022-11-03T08:31:46 [I|app|5f6b98bf] Started PUT "/rhsm/consumers/3262ea70-97ca-4825-a917-169238f59367/tracer" for 10.101.137.9 at 2022-11-03 08:31:46 +0100
2022-11-03T08:31:46 [I|app|5f6b98bf] Processing by Katello::Api::Rhsm::CandlepinProxiesController#upload_tracer_profile as JSON
2022-11-03T08:31:46 [I|app|5f6b98bf]   Parameters: {"traces"=>{}, "id"=>"3262ea70-97ca-4825-a917-169238f59367"}
2022-11-03T08:31:46 [I|app|5f6b98bf] Completed 200 OK in 46ms (Views: 0.2ms | ActiveRecord: 17.6ms | Allocations: 8257)
2022-11-03T08:31:48 [I|app|0f9e2f8d] Started GET "/rhsm/status" for 10.101.137.9 at 2022-11-03 08:31:48 +0100
2022-11-03T08:31:48 [I|app|0f9e2f8d] Processing by Katello::Api::Rhsm::CandlepinProxiesController#server_status as JSON
2022-11-03T08:31:48 [I|app|0f9e2f8d] Completed 200 OK in 25ms (Views: 0.3ms | ActiveRecord: 3.5ms | Allocations: 2656)
2022-11-03T08:31:48 [I|app|e1e97282] Started PUT "/rhsm/consumers/3262ea70-97ca-4825-a917-169238f59367/profiles" for 10.101.137.9 at 2022-11-03 08:31:48 +0100
2022-11-03T08:31:48 [I|app|e1e97282] Processing by Katello::Api::Rhsm::CandlepinDynflowProxyController#upload_profiles as JSON
2022-11-03T08:31:48 [I|app|e1e97282]   Parameters: {"id"=>"3262ea70-97ca-4825-a917-169238f59367"}
2022-11-03T08:31:48 [I|bac|0aa6d2c9] Task {label: , execution_plan_id: 423259dd-43e2-4c2d-a93a-8f86624937e7} state changed: pending
2022-11-03T08:31:48 [I|bac|0aa6d2c9] Task {label: Actions::Katello::Applicability::Hosts::BulkGenerate, id: 92d5bbf5-bc7d-484e-ac38-ffa0adf38be6, execution_plan_id: 423259dd-43e2-4c2d-a93a-8f86624937e7} state changed: planning
2022-11-03T08:31:48 [I|bac|0aa6d2c9] Task {label: Actions::Katello::Applicability::Hosts::BulkGenerate, id: 92d5bbf5-bc7d-484e-ac38-ffa0adf38be6, execution_plan_id: 423259dd-43e2-4c2d-a93a-8f86624937e7} state changed: planned
2022-11-03T08:31:48 [I|app|0aa6d2c9] katello event handled success=true type=generate_host_applicability object_id=0 expired=false rescheduled=false duration=99.93

Repo and package information seems to be correct as well in the json file. First a list of the repos pointing to the Foreman server, then a large list of packages:

"enabled_repos": [
        {
            "repositoryid": "Default_Organization_Debian_11_main_Debian_11_main",
            "baseurl": [
                "https://******-foreman.******.local/pulp/content/Default_Organization/Prod/Debian_11_bullseye/custom/Debian_11_main/Debian_11_main"
            ]
        },
        {
            "repositoryid": "Default_Organization_Debian_11_main_Debian_11_security",
            "baseurl": [
                "https://********-foreman.********.local/pulp/content/Default_Organization/Prod/Debian_11_bullseye/custom/Debian_11_main/Debian_11_security"
            ]
        },
        {
            "repositoryid": "Default_Organization_Debian_11_main_Debian_11_updates",
            "baseurl": [
                "https://********-foreman.********.local/pulp/content/Default_Organization/Prod/Debian_11_bullseye/custom/Debian_11_main/Debian_11_updates"
            ]
        }
    ],
    "deb": [
        {
            "name": "adduser",
            "version": "3.118",
            "architecture": "all"
        },
        {
            "name": "aide",
            "version": "0.17.3-4+deb11u1",
            "architecture": "amd64"
        },
        {
            "name": "aide-common",
            "version": "0.17.3-4+deb11u1",
            "architecture": "all"
        },
        {
            "name": "apparmor",
            "version": "2.13.6-10",
            "architecture": "amd64"
        },
etc...

Thanks and kind regards

javieitez · November 3, 2022, 7:55am

Whatever it is, my guts tell me the problem is on the Debian host. On the Foreman server there are several Centos 7 and 8 hosts configured, and everything runs smoothly for them.

javieitez · November 3, 2022, 8:02am

Also, the apt transport plugin throws an error each time an apt operation is performed:

Exception ignored in: <module 'threading' from '/usr/lib/python3.9/threading.py'>
Traceback (most recent call last):
  File "/usr/lib/python3.9/threading.py", line 1388, in _shutdown
    def _shutdown():
KeyboardInterrupt:

iballou · November 3, 2022, 1:52pm

@javieitez, interesting, so the profile communication to Katello from the Debian host is seemingly working just fine.

Does your Debian host in Katello report having any installed packages?

Another thing to check:

::Host.find_by(name: 'your host Foreman name here').content_facet.bound_repositories

That will tell us if Katello is importing the repositories profile correctly. The returned data should be the proper list of repositories that your host is consuming from Katello. If it’s wrong or empty, then we know that it isn’t being imported correctly.

javieitez · November 3, 2022, 3:26pm

no, 0 packages: This host does not have any packages.

Sorry, I don’t know where should I paste this command… obviously, not in the terminal, so… where?

iballou · November 3, 2022, 3:43pm

Oops sorry, I left that bit out. That’s in the Foreman rails console. You can open it with the following:

foreman-rake console

I’m guessing it will return [] (empty) if you also have no packages reported.

javieitez · November 3, 2022, 4:01pm

Here’s the output

Loading production environment (Rails 6.0.4.7)
irb(main):001:0> ::Host.find_by(name: 'pruebakatellodebian').content_facet.bound_repositories
=> #<ActiveRecord::Associations::CollectionProxy [#<Katello::Repository id: 146, pulp_id: "1-Debian_11_bullseye-Prod-af5e459f-3145-4cbb-99f3-...", library_instance_id: 134, content_view_version_id: 13, relative_path: "Default_Organization/Prod/Debian_11_bullseye/custo...", environment_id: 2, saved_checksum_type: nil, distribution_version: nil, distribution_arch: nil, distribution_bootable: nil, distribution_family: nil, distribution_variant: nil, container_repository_name: nil, root_id: 20, remote_href: nil, publication_href: "/pulp/api/v3/publications/deb/apt/c6c0fffc-abfb-4b...", version_href: "/pulp/api/v3/repositories/deb/apt/361c3839-c936-4c...", last_contents_changed: "2022-09-28 11:52:55", last_applicability_regen: "1970-01-01 00:00:00", last_indexed: "2022-09-28 11:53:00">, #<Katello::Repository id: 147, pulp_id: "1-Debian_11_bullseye-Prod-fd00fea1-25b1-4422-bc8e-...", library_instance_id: 133, content_view_version_id: 13, relative_path: "Default_Organization/Prod/Debian_11_bullseye/custo...", environment_id: 2, saved_checksum_type: nil, distribution_version: nil, distribution_arch: nil, distribution_bootable: nil, distribution_family: nil, distribution_variant: nil, container_repository_name: nil, root_id: 19, remote_href: nil, publication_href: "/pulp/api/v3/publications/deb/apt/0ec9eb09-948d-46...", version_href: "/pulp/api/v3/repositories/deb/apt/c3cda4a0-fd83-43...", last_contents_changed: "2022-09-28 11:52:55", last_applicability_regen: "1970-01-01 00:00:00", last_indexed: "2022-09-28 11:53:03">, #<Katello::Repository id: 145, pulp_id: "1-Debian_11_bullseye-Prod-bb6b40b4-4552-4d22-8dd7-...", library_instance_id: 135, content_view_version_id: 13, relative_path: "Default_Organization/Prod/Debian_11_bullseye/custo...", environment_id: 2, saved_checksum_type: nil, distribution_version: nil, distribution_arch: nil, distribution_bootable: nil, distribution_family: nil, distribution_variant: nil, container_repository_name: nil, root_id: 21, remote_href: nil, publication_href: "/pulp/api/v3/publications/deb/apt/705d2106-4ade-47...", version_href: "/pulp/api/v3/repositories/deb/apt/a1816326-67a7-41...", last_contents_changed: "2022-09-28 11:52:55", last_applicability_regen: "1970-01-01 00:00:00", last_indexed: "2022-09-28 11:53:03">]>

iballou · November 3, 2022, 4:19pm

@javieitez I reproduced the issue and I think I have a grasp on it now.

It appears that the new host page is not ready to support Debian content. I should’ve realized that was the issue.

If you go to the top-right corner and click on the kebab button to see the dropdown actions, you should see a “Legacy content host UI button”. If you don’t (I’m forgetting when it was added), then just go to the content hosts page and use the old content hosts UI from there.

For me, on Debian 11, the old content hosts page listed the installed packages correctly. I’ll double check too if I see issues with consuming from Katello repositories per @gvde 's comments above.

iballou · November 3, 2022, 4:57pm

I also created an issue to add Debian support to the new page: Feature #35713: New host details page should show debian content - Katello - Foreman

javieitez · November 3, 2022, 5:02pm

I don’t have that button, but I can see the installed packages on the content host view

So this seems to be a bug and it will be fixed in a future update, right? We’ll just wait for it, thank you so much for your support and guidance

iballou · November 3, 2022, 5:07pm

@javieitez yeah it’s less of a bug and more of a feature gap. The new host details page is still a bit in flux since it’s so shiny and new. Regardless, we won’t deprecate the legacy page before Debian content is brought up to speed.

Bernhard_Suttner · November 10, 2022, 9:39am

@iballou in the last X years, we never had a scenario, that on a managed host, you have Debian AND RPM packages. Therefore, I guess it would make sense and it would be much more user friendly to have only a Tab “Packages” and depending on if the host uses deb or rpm, manage the corresponding packages. On the packages tab itself, you can show if the package is deb, rpm, pacman or whatever. If there would be a filter on this page to only show a certain type of pkgs, we would have all possibilities.

Your thoughts?

jeremylenz · November 10, 2022, 4:24pm

On the Module streams tab we check the host’s operating system name via regex to decide whether to show the tab. We could use something similar within the Packages tab, and either adapt the existing RPM-focused React page to also handle Deb, or display a new Deb-focused React page - whichever turns out to make more sense.

Keep in mind that the host installed packages list affects a lot, namely it’s used to calculate and store errata status (which is why the Errata tab doesn’t work for Deb) and it’s also used to store whether Tracer is installed (which is why the Traces tab doesn’t work for Debian.)

Bernhard_Suttner · November 10, 2022, 10:40pm

btw, The tracer for deb, is “under way”: Add katello-host-tools for debian/ubuntu by sbernhard · Pull Request #6958 · theforeman/foreman-packaging · GitHub
the code (katello-host-tools) for deb is done and merged already.

iballou · November 11, 2022, 3:48pm

@jeremylenz covered everything I would’ve said! I was thinking about a separate tab since that’s how the legacy content hosts page looks, but it’s better UX to load the packages accordingly to the host OS type.

cbreit · March 19, 2024, 2:37pm

Update on this situation.

I currently tested the release candidate 3 of foreman 3.10 & Katello 4.12. The problem with Debian packages not showing is resolved, as they can be viewed directly in the packages tab now.

Since there is still no solution for errata with debian packages, this results in a warning on the host global status.