Foreman automaticlly removing hosts on its own

Support
1

Problem: Foreman is removing random hosts on its own , i am trying to understand why and how to stop it as this is causing the hosts to be unsubscribed from important updates.

Expected outcome: Hosts registered should stay registered

Foreman and Proxy versions: Version 3.10.0

Foreman and Proxy plugin versions: Version 3.10.0

Distribution and version:

Other relevant data:
I am adding the hosts when they are unsubscribed however every few hours the host is getting removed from foreman on its own.
Below is audit log of one of the host, could some one help me understand the possible cause for this and what can be done to stop foreman removing hosts

image
image1635Ă—270 21.6 KB

2

I did some more digging and found that a delete call gets issued on multiple servers

Client host
Consumer profile "6f9b5eb3-14ed-499f-849d-XXXXX" has been deleted from the server. You can use command clean or unregister to remove local profile.

Foreman Server
2024-11-07 12:03:29,583 [thread=https-openssl-nio-127.0.0.1-23443-exec-180] [req=ee927e31-4f7d-4071-90cc-XXXXX, org=, csid=] INFO org.candlepin.servlet.filter.logging.LoggingFilter - Request: verb=DELETE, uri=/candlepin/consumers/6f9b5eb3-14ed-499f-849d-XXXXXX

I am not able to determine the reason why this is happening or / and if this is expected behaviour

The host is getting created again but has no subscription and i have register the host again with activation key manually

3

Katello has a clean_backend_objects rake task that runs automatically during Foreman upgrades. It removes hosts whose Candlepin consumers have no corresponding subscription facet in Foreman (determined by its Subscription UUID), or whose subscription facets have a nil UUID. Is it possible it’s this? I don’t believe that task runs regularly, unless you’ve got a cron job set up or something.

You can search the logs for phrases like

is partially missing subscription information
orphaned consumer id(s) found in candlepin
4

Appreciate your response, i dont see a log similar to what you mentioned. How can i list all jobs to check if i have similar job running in background ?

5

This would be in /var/log/foreman/production.log

6

i am afraid i dont have any entry , i see something like this which i think is not what you mentioned

 2e2ee936 | /usr/share/gems/gems/katello-4.12.0/app/models/katello/ping.rb:264:in `backend_status'
 5bd4166c | /usr/share/gems/gems/katello-4.12.0/app/models/katello/ping.rb:264:in `backend_status'
7

Does this below message from production log make any sense ?

2024-11-07T12:51:42 [E|app|9f29aeb1] RestClient::Gone: Katello::Resources::Candlepin::Consumer: 410 Gone {"displayMessage":"Unit 6f9b5eb3-14ed-499f-849d-XXXX has been deleted","requestUuid":"4501aaf5-6df6-421b-9e63-e612a53c06d9","deletedId":"6f9b5eb3-14ed-499f-849d-20cba1512d01"} (GET /candlepin/consumers/6f9b5eb3-14ed-499f-849d-XXXX)

8

That one can happen sometimes when you register hosts with the --force option. Are you doing that?

9

yes we do use the option --force

Some more logs if they can help

2024-11-07T07:45:49 [I|app|e40f3ac0] Started GET "/settings/auto_complete_search?search=unregister_delete_host" for XXXXX at 2024-11-07 07:45:49 -0500
2024-11-07T07:45:49 [I|app|e40f3ac0]   Parameters: {"search"=>"unregister_delete_host"}
2024-11-07T07:46:30 [I|app|e599d14a] Started PUT "/api/settings/unregister_delete_host" for XXXX at 2024-11-07 07:46:30 -0500
2024-11-07T07:46:30 [I|app|e599d14a]   Parameters: {"setting"=>{"value"=>"[FILTERED]"}, "apiv"=>"v2", "id"=>"unregister_delete_host"}
10

After poking around a bit I don’t think it’s that. Let’s see if anyone else has ideas…

also, do you have a reliable way I could reproduce the issue?

11

One question is, what has access to run DELETE /candlepin/consumers/:uuid? Is there a way we can see from the logs if the call is originating from the Foreman server or from the host?

12

Whatever is calling that DELETE endpoint obviously knows the host UUID, so that should be the host itself or the Foreman server.

13

Are you using GitHub - theforeman/foreman_expire_hosts: Foreman plugin for limiting host lifetime by chance?

14

I went to bed trying to figure this out, i will check the logs for the DELETE call.

I dont see any foreman_expire_hosts installed / UI option similar in settings. Let me check more on above and update you

15

Is the Anonymous Admin inbuilt user for foreman? Its the one seem to be performing all actions

16

2024-11-07T20:51:32 [I|app|8d2283b9] Started DELETE "/rhsm/consumers/60e9f021-7a8d-4fbb-8c14-XXXX" for x.x.x.x at 2024-11-07 20:51:32 -0500

This seems to be running ( found from production.log ) the IP is the proxy server IP. Not sure why its getting triggered though

17

I found something weird , the hosts are getting registered however subscription is not getting consumed. I read somewhere that hosts may be deleted if they are no longer subscribed
Could this cause issues we are seeing ? I have tried to register the hosts and they seem to be registered and getting updates however under subscriptions i see “0” as consumed

18

Found that on registering the host subscription-manager behaves differently


OLD FMN SYSTEM ( 1.20.3  )
Nov  8 03:50:47 xxxxx subscription-manager[1499869]: Registered system with identity: xxxx-1bdb-4779-xxxx-xxxxx
Nov  8 03:50:47 xxxxx subscription-manager[1499869]: Added subscription for 'rocky_linux_9' contract 'None'
Nov  8 03:50:47 xxxxx subscription-manager[1499869]: Added subscription for product 'rocky_linux_9'
Nov  8 03:50:47 xxxxx subscription-manager[1499869]: Added subscription for 'Content Access' contract 'None'
Nov  8 03:50:47 xxxxx subscription-manager[1499869]: Added subscription for product ' Content Access'

NEW FMN SYSTEM
Nov  8 01:07:52 xxxxx subscription-manager[1461154]: Added subscription for 'Content Access' contract 'None'
Nov  8 01:07:52 xxxxx subscription-manager[1461154]: Added subscription for product ' Content Access'
Nov  8 01:07:52 xxxxx subscription-manager[1461154]: Removed subscription for 'Content Access' contract 'None'
Nov  8 01:07:52 xxxxx subscription-manager[1461154]: Removed subscription for product ' Content Access'
19

Any one has any ideas on what may be wrong here ?

20

This looks normal for SCA. What doesn’t look normal is the removal.

  1. In Content > Subscriptions, do you see the subscription for your rocky_linux_9 product? It should have an end date in 2049 or something.
  2. Is your host’s system clock correct? Sometimes it can remove the subscription if it thinks it has expired or is not yet active.