Working Towards Foreman Handling 6000+ Systems

fresh-pie · December 29, 2021, 10:42pm

I apologize for the book below…

I’m in the process of using Foreman to deploy CentOS 7 to a large retail environment, which includes 6000+ systems. Currently, I’ve successfully deployed to about 800 of these systems in the past few months. It has been a very slow deployment, which is very typical in retail, but I’m planning on greatly speeding up the deployment beginning in 2022.

So thus, I’m writing this post to “check in” and ask for a little help regarding if my underlying infrastructure will be able to handle the load of thousands of systems, as well as bring up a CPU utilization spike I see on a daily basis at the same time.

I came across this awesome document which is kind of giving me an idea about what kind of infrastructure I need.

Based on the above guide, if I have 5000+ systems I would likely need the medium tuning option, with 8 cores of CPU and 32G of memory.

Currently, I have a single instance of Foreman (along with Katello) running on a system with the below specifications:

CPU:

# lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                4
On-line CPU(s) list:   0-3
Thread(s) per core:    1
Core(s) per socket:    1
Socket(s):             4
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 63
Model name:            Intel(R) Xeon(R) Platinum 8168 CPU @ 2.70GHz
Stepping:              0
CPU MHz:               2693.671
BogoMIPS:              5387.34
Hypervisor vendor:     VMware
Virtualization type:   full
L1d cache:             32K
L1i cache:             32K
L2 cache:              1024K
L3 cache:              33792K
NUMA node0 CPU(s):     0-3
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm invpcid_single ssbd ibrs ibpb stibp fsgsbase tsc_adjust bmi1 avx2 smep bmi2 invpcid xsaveopt arat md_clear spec_ctrl intel_stibp flush_l1d arch_capabilities

This already isn’t meeting what the medium tuning option would require. I’d need an additional 4 cores.

Although CPU availability has been mostly aplenty, I’m concerned this will begin to change as I add an additional thousands of systems. Looking at the output of sar, I see there is a moderate CPU spike in the early morning:

12:00:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
12:10:01 AM     all     13.24      0.00      6.60      0.04      0.00     80.12
12:20:01 AM     all      3.20      0.00      2.14      0.03      0.00     94.62
12:30:01 AM     all      2.77      0.00      1.97      0.03      0.00     95.24
12:40:01 AM     all      3.32      0.00      2.19      0.03      0.00     94.46
12:50:01 AM     all      3.37      0.00      2.22      0.03      0.00     94.39
01:00:01 AM     all      3.09      0.00      2.13      0.03      0.00     94.75
01:10:01 AM     all      8.29      0.00      3.49      0.05      0.00     88.18
01:20:01 AM     all      3.01      0.00      2.04      0.03      0.00     94.92
01:30:01 AM     all      2.90      0.00      2.00      0.03      0.00     95.07
01:40:01 AM     all      3.26      0.00      2.12      0.03      0.00     94.59
01:50:01 AM     all      3.19      0.00      2.10      0.03      0.00     94.67
02:00:02 AM     all      3.07      0.00      2.06      0.03      0.00     94.85
02:10:01 AM     all      3.69      0.00      2.33      0.04      0.00     93.94
02:20:01 AM     all      3.40      0.00      2.30      0.04      0.00     94.26
02:30:01 AM     all      3.00      0.00      2.10      0.03      0.00     94.87
02:40:01 AM     all      3.78      0.00      2.46      0.04      0.00     93.72
02:50:01 AM     all      3.48      0.00      2.29      0.03      0.00     94.20
03:00:02 AM     all      2.91      0.00      2.04      0.03      0.00     95.02
03:10:01 AM     all     16.09      0.00      5.97      0.17      0.00     77.77
03:20:01 AM     all      5.16      0.00      3.46      0.05      0.00     91.34

See the spike at 03:10 AM? I do see the below happen in the production.log around this time, which caught my eye:

2021-12-29T03:10:09 [I|app|dbc286d5] Started GET "/redhat_access/r/insights/v1/branch_info" for 127.0.0.1 at 2021-12-29 03:10:09 -0500
2021-12-29T03:10:09 [F|app|dbc286d5]
 dbc286d5 | ActionController::RoutingError (No route matches [GET] "/redhat_access/r/insights/v1/branch_info"):
 dbc286d5 |
 dbc286d5 | lib/foreman/middleware/logging_context_request.rb:11:in `call'
 dbc286d5 | katello (3.17.3) lib/katello/prevent_json_parsing.rb:12:in `call'
2021-12-29T03:10:10 [I|app|49de0ef4] Started GET "/redhat_access/r/insights/v1/branch_info" for 127.0.0.1 at 2021-12-29 03:10:10 -0500
2021-12-29T03:10:10 [F|app|49de0ef4]
 49de0ef4 | ActionController::RoutingError (No route matches [GET] "/redhat_access/r/insights/v1/branch_info"):
 49de0ef4 |
 49de0ef4 | lib/foreman/middleware/logging_context_request.rb:11:in `call'
 49de0ef4 | katello (3.17.3) lib/katello/prevent_json_parsing.rb:12:in `call'
2021-12-29T03:10:10 [I|app|fed89813] Started GET "/redhat_access/r/insights/platform/module-update-router/v1/channel?module=insights-core" for 127.0.0.1 at 2021-12-29 03:10:10 -0500
2021-12-29T03:10:10 [F|app|fed89813]
 fed89813 | ActionController::RoutingError (No route matches [GET] "/redhat_access/r/insights/platform/module-update-router/v1/channel"):
 fed89813 |
 fed89813 | lib/foreman/middleware/logging_context_request.rb:11:in `call'
 fed89813 | katello (3.17.3) lib/katello/prevent_json_parsing.rb:12:in `call'
2021-12-29T03:10:10 [I|app|7b721479] Started GET "/redhat_access/r/insights/v1/static/release/insights-core.egg" for 127.0.0.1 at 2021-12-29 03:10:10 -0500
2021-12-29T03:10:10 [F|app|7b721479]
 7b721479 | ActionController::RoutingError (No route matches [GET] "/redhat_access/r/insights/v1/static/release/insights-core.egg"):
 7b721479 |
 7b721479 | lib/foreman/middleware/logging_context_request.rb:11:in `call'
 7b721479 | katello (3.17.3) lib/katello/prevent_json_parsing.rb:12:in `call'
2021-12-29T03:10:10 [I|app|648cbae1] Started GET "/redhat_access/r/insights/v1/static/uploader.v2.json" for 127.0.0.1 at 2021-12-29 03:10:10 -0500
2021-12-29T03:10:10 [F|app|648cbae1]
 648cbae1 | ActionController::RoutingError (No route matches [GET] "/redhat_access/r/insights/v1/static/uploader.v2.json"):
 648cbae1 |
 648cbae1 | lib/foreman/middleware/logging_context_request.rb:11:in `call'
 648cbae1 | katello (3.17.3) lib/katello/prevent_json_parsing.rb:12:in `call'
2021-12-29T03:10:11 [I|app|43ba0c54] Started GET "/redhat_access/r/insights/v1/branch_info" for 127.0.0.1 at 2021-12-29 03:10:11 -0500
2021-12-29T03:10:11 [F|app|43ba0c54]
 43ba0c54 | ActionController::RoutingError (No route matches [GET] "/redhat_access/r/insights/v1/branch_info"):
 43ba0c54 |
 43ba0c54 | lib/foreman/middleware/logging_context_request.rb:11:in `call'
 43ba0c54 | katello (3.17.3) lib/katello/prevent_json_parsing.rb:12:in `call'
2021-12-29T03:10:13 [I|app|aa035ef0] Started GET "/redhat_access/r/insights/v1/systems/d36cc8d9-9b81-4944-84b3-bfc564b8e57f" for 127.0.0.1 at 2021-12-29 03:10:13 -0500
2021-12-29T03:10:13 [F|app|aa035ef0]
 aa035ef0 | ActionController::RoutingError (No route matches [GET] "/redhat_access/r/insights/v1/systems/d36cc8d9-9b81-4944-84b3-bfc564b8e57f"):
 aa035ef0 |
 aa035ef0 | lib/foreman/middleware/logging_context_request.rb:11:in `call'
 aa035ef0 | katello (3.17.3) lib/katello/prevent_json_parsing.rb:12:in `call'
2021-12-29T03:10:13 [I|app|d9887e44] Started GET "/redhat_access/r/insights/v1/branch_info" for 127.0.0.1 at 2021-12-29 03:10:13 -0500
2021-12-29T03:10:13 [F|app|d9887e44]
 d9887e44 | ActionController::RoutingError (No route matches [GET] "/redhat_access/r/insights/v1/branch_info"):
 d9887e44 |
 d9887e44 | lib/foreman/middleware/logging_context_request.rb:11:in `call'
 d9887e44 | katello (3.17.3) lib/katello/prevent_json_parsing.rb:12:in `call'
2021-12-29T03:10:16 [I|app|dd210cd6] Started GET "/redhat_access/r/insights/v1/branch_info" for 127.0.0.1 at 2021-12-29 03:10:16 -0500
2021-12-29T03:10:16 [F|app|dd210cd6]
 dd210cd6 | ActionController::RoutingError (No route matches [GET] "/redhat_access/r/insights/v1/branch_info"):
 dd210cd6 |
 dd210cd6 | lib/foreman/middleware/logging_context_request.rb:11:in `call'
 dd210cd6 | katello (3.17.3) lib/katello/prevent_json_parsing.rb:12:in `call'
2021-12-29T03:10:16 [I|app|09b7c4bc] Started GET "/redhat_access/r/insights/platform/module-update-router/v1/channel?module=insights-core" for 127.0.0.1 at 2021-12-29 03:10:16 -0500
2021-12-29T03:10:16 [F|app|09b7c4bc]
 09b7c4bc | ActionController::RoutingError (No route matches [GET] "/redhat_access/r/insights/platform/module-update-router/v1/channel"):
 09b7c4bc |
 09b7c4bc | lib/foreman/middleware/logging_context_request.rb:11:in `call'
 09b7c4bc | katello (3.17.3) lib/katello/prevent_json_parsing.rb:12:in `call'
2021-12-29T03:10:16 [I|app|634600d1] Started GET "/redhat_access/r/insights/v1/static/release/insights-core.egg" for 127.0.0.1 at 2021-12-29 03:10:16 -0500
2021-12-29T03:10:16 [F|app|634600d1]

The other thing is that all of the cash registers in the US/Eastern time zone are rebooting roughly around this time, everyday. That is hundreds of systems all around the same moment, all PXE booting iPXE and calling into the Foreman server.

I don’t really see a lot of that going on in the log around that exact time though:

2021-12-29T03:10:01 [I|app|463d201a] Started GET "/rhsm/consumers/c2025032-5307-43d1-a8df-9f07421ba741/compliance" for 127.0.0.1 at 2021-12-29 03:10:01 -0500
2021-12-29T03:10:01 [I|app|8824edb1] Started GET "/rhsm/consumers/21df8457-e950-489b-badb-1b8f41f118db/compliance" for 127.0.0.1 at 2021-12-29 03:10:01 -0500
2021-12-29T03:10:02 [I|app|aad837a2] Started GET "/rhsm/consumers/c9f593da-c3b2-4886-81ff-dfe6400ed673/compliance" for 127.0.0.1 at 2021-12-29 03:10:02 -0500
2021-12-29T03:10:03 [I|app|71028e6a] Started GET "/rhsm/consumers/75ac9653-5215-408f-b5e3-6d7b2b55637e/compliance" for 127.0.0.1 at 2021-12-29 03:10:03 -0500
2021-12-29T03:10:04 [I|app|48b05bce] Started GET "/rhsm/consumers/07493a5a-7ab7-4901-bfac-e5e6c3bca2be/compliance" for 127.0.0.1 at 2021-12-29 03:10:04 -0500
2021-12-29T03:10:04 [I|app|7035092d] Started GET "/rhsm/consumers/8aba3372-6fab-4ec9-b279-b707497e2580/compliance" for 127.0.0.1 at 2021-12-29 03:10:04 -0500
2021-12-29T03:10:04 [I|app|48f3ad7f] Started GET "/rhsm/consumers/11e4322c-01ed-4094-b7c9-21be8c50de3e/compliance" for 127.0.0.1 at 2021-12-29 03:10:04 -0500
2021-12-29T03:10:04 [I|app|302e03b6] Started GET "/rhsm/consumers/760d0910-64a4-47ef-8b00-3e87ef58085d/compliance" for 127.0.0.1 at 2021-12-29 03:10:04 -0500
2021-12-29T03:10:04 [I|app|01dd6be3] Started GET "/rhsm/consumers/2d0073e1-232c-4c7d-a618-c6b5587c2004/compliance" for 127.0.0.1 at 2021-12-29 03:10:04 -0500
2021-12-29T03:10:09 [I|app|dbc286d5] Started GET "/redhat_access/r/insights/v1/branch_info" for 127.0.0.1 at 2021-12-29 03:10:09 -0500
2021-12-29T03:10:10 [I|app|49de0ef4] Started GET "/redhat_access/r/insights/v1/branch_info" for 127.0.0.1 at 2021-12-29 03:10:10 -0500
2021-12-29T03:10:10 [I|app|fed89813] Started GET "/redhat_access/r/insights/platform/module-update-router/v1/channel?module=insights-core" for 127.0.0.1 at 2021-12-29 03:10:10 -0500
2021-12-29T03:10:10 [I|app|7b721479] Started GET "/redhat_access/r/insights/v1/static/release/insights-core.egg" for 127.0.0.1 at 2021-12-29 03:10:10 -0500
2021-12-29T03:10:10 [I|app|648cbae1] Started GET "/redhat_access/r/insights/v1/static/uploader.v2.json" for 127.0.0.1 at 2021-12-29 03:10:10 -0500
2021-12-29T03:10:11 [I|app|43ba0c54] Started GET "/redhat_access/r/insights/v1/branch_info" for 127.0.0.1 at 2021-12-29 03:10:11 -0500
2021-12-29T03:10:13 [I|app|aa035ef0] Started GET "/redhat_access/r/insights/v1/systems/d36cc8d9-9b81-4944-84b3-bfc564b8e57f" for 127.0.0.1 at 2021-12-29 03:10:13 -0500
2021-12-29T03:10:13 [I|app|d9887e44] Started GET "/redhat_access/r/insights/v1/branch_info" for 127.0.0.1 at 2021-12-29 03:10:13 -0500
2021-12-29T03:10:16 [I|app|dd210cd6] Started GET "/redhat_access/r/insights/v1/branch_info" for 127.0.0.1 at 2021-12-29 03:10:16 -0500
2021-12-29T03:10:16 [I|app|09b7c4bc] Started GET "/redhat_access/r/insights/platform/module-update-router/v1/channel?module=insights-core" for 127.0.0.1 at 2021-12-29 03:10:16 -0500
2021-12-29T03:10:16 [I|app|634600d1] Started GET "/redhat_access/r/insights/v1/static/release/insights-core.egg" for 127.0.0.1 at 2021-12-29 03:10:16 -0500
2021-12-29T03:10:16 [I|app|f8a04e0d] Started GET "/redhat_access/r/insights/v1/static/uploader.v2.json" for 127.0.0.1 at 2021-12-29 03:10:16 -0500
2021-12-29T03:10:18 [I|app|895d5a3a] Started GET "/redhat_access/r/insights/v1/branch_info" for 127.0.0.1 at 2021-12-29 03:10:18 -0500
2021-12-29T03:10:20 [I|app|005157c1] Started GET "/redhat_access/r/insights/v1/systems/c79042a9-5759-460e-a47e-9d19ceca84fb" for 127.0.0.1 at 2021-12-29 03:10:20 -0500
2021-12-29T03:10:41 [I|app|a8d53664] Started GET "/redhat_access/r/insights/v1/branch_info" for 127.0.0.1 at 2021-12-29 03:10:41 -0500
2021-12-29T03:10:44 [I|app|fd4f37db] Started GET "/redhat_access/r/insights/v1/branch_info" for 127.0.0.1 at 2021-12-29 03:10:44 -0500
2021-12-29T03:10:44 [I|app|10494985] Started GET "/redhat_access/r/insights/platform/module-update-router/v1/channel?module=insights-core" for 127.0.0.1 at 2021-12-29 03:10:44 -0500
2021-12-29T03:10:44 [I|app|3f3f9a67] Started GET "/redhat_access/r/insights/v1/static/release/insights-core.egg" for 127.0.0.1 at 2021-12-29 03:10:44 -0500
2021-12-29T03:10:44 [I|app|d050def2] Started GET "/redhat_access/r/insights/v1/static/uploader.v2.json" for 127.0.0.1 at 2021-12-29 03:10:44 -0500
2021-12-29T03:10:46 [I|app|4a3156cd] Started GET "/redhat_access/r/insights/v1/branch_info" for 127.0.0.1 at 2021-12-29 03:10:46 -0500
2021-12-29T03:10:50 [I|app|bdc94774] Started GET "/redhat_access/r/insights/v1/systems/66827462-810d-45c9-b1b1-0e693f917cf5" for 127.0.0.1 at 2021-12-29 03:10:50 -0500
2021-12-29T03:10:56 [I|app|0e25667b] Started GET "/rhsm/" for 127.0.0.1 at 2021-12-29 03:10:56 -0500
2021-12-29T03:10:56 [I|app|a444283c] Started GET "/rhsm/consumers/c2025032-5307-43d1-a8df-9f07421ba741/certificates/serials" for 127.0.0.1 at 2021-12-29 03:10:56 -0500
2021-12-29T03:10:56 [I|app|8e715fdc] Started GET "/rhsm/status" for 127.0.0.1 at 2021-12-29 03:10:56 -0500
2021-12-29T03:10:56 [I|app|a520ea7a] Started GET "/rhsm/consumers/c2025032-5307-43d1-a8df-9f07421ba741" for 127.0.0.1 at 2021-12-29 03:10:56 -0500
2021-12-29T03:10:56 [I|app|6b7a988e] Started GET "/rhsm/consumers/c2025032-5307-43d1-a8df-9f07421ba741/content_overrides" for 127.0.0.1 at 2021-12-29 03:10:56 -0500
2021-12-29T03:10:56 [I|app|33957675] Started GET "/rhsm/consumers/c2025032-5307-43d1-a8df-9f07421ba741/certificates/serials" for 127.0.0.1 at 2021-12-29 03:10:56 -0500
2021-12-29T03:10:56 [I|app|0aa93706] Started GET "/rhsm/status" for 127.0.0.1 at 2021-12-29 03:10:56 -0500
2021-12-29T03:10:56 [I|app|f87eae75] Started GET "/rhsm/consumers/c2025032-5307-43d1-a8df-9f07421ba741/content_overrides" for 127.0.0.1 at 2021-12-29 03:10:56 -0500
2021-12-29T03:10:57 [I|app|c50afd58] Started GET "/rhsm/consumers/c2025032-5307-43d1-a8df-9f07421ba741" for 127.0.0.1 at 2021-12-29 03:10:57 -0500
2021-12-29T03:10:58 [I|app|8e2a3d53] Started GET "/rhsm/" for 127.0.0.1 at 2021-12-29 03:10:58 -0500
2021-12-29T03:10:58 [I|app|69620bd4] Started GET "/rhsm/consumers/5daa0f7b-99e0-451b-8c96-24da27eb630d/certificates/serials" for 127.0.0.1 at 2021-12-29 03:10:58 -0500
2021-12-29T03:10:58 [I|app|3441760e] Started GET "/rhsm/consumers/7ee5a350-3726-49bb-9671-a4d89e7ada2c" for 127.0.0.1 at 2021-12-29 03:10:58 -0500
2021-12-29T03:10:58 [I|app|071b3e60] Started GET "/rhsm/status" for 127.0.0.1 at 2021-12-29 03:10:58 -0500
2021-12-29T03:10:59 [I|app|6999b318] Started GET "/rhsm/consumers/5daa0f7b-99e0-451b-8c96-24da27eb630d" for 127.0.0.1 at 2021-12-29 03:10:59 -0500
2021-12-29T03:10:59 [I|app|8a3ffce9] Started GET "/rhsm/consumers/5daa0f7b-99e0-451b-8c96-24da27eb630d/content_overrides" for 127.0.0.1 at 2021-12-29 03:10:59 -0500

But, I do slightly before, which could be leading to the high utilization a few minutes later:

2021-12-29T03:06:41 [I|app|e4f93768] Started GET "/unattended/iPXE?bootstrap=1&url=http%3A%2F%2Fforeman-server.example.net%3A8000" for 127.0.0.1 at 2021-12-29 03:06:41 -0500
2021-12-29T03:06:42 [I|app|a904a5e3] Started GET "/unattended/iPXE?has_raw=yes&mac=00%3A1a%3A64%3A40%3Ab7%3Ae4&url=http%3A%2F%2Fforeman-server.example.net%3A8000" for 127.0.0.1 at 2021-12-29 03:06:42 -0500
2021-12-29T03:06:42 [I|app|d008536e] Started GET "/unattended/iPXE?bootstrap=1&url=http%3A%2F%2Fforeman-server.example.net%3A8000" for 127.0.0.1 at 2021-12-29 03:06:42 -0500
2021-12-29T03:06:43 [I|app|9c7599f7] Started GET "/unattended/iPXE?bootstrap=1&url=http%3A%2F%2Fforeman-server.example.net%3A8000" for 127.0.0.1 at 2021-12-29 03:06:43 -0500
2021-12-29T03:06:43 [I|app|7cc43d36] Started GET "/unattended/iPXE?has_raw=yes&mac=00%3A1a%3A64%3A40%3Abf%3A95&url=http%3A%2F%2Fforeman-server.example.net%3A8000" for 127.0.0.1 at 2021-12-29 03:06:43 -0500
2021-12-29T03:06:43 [I|app|9c8b2347] Started GET "/unattended/iPXE?bootstrap=1&url=http%3A%2F%2Fforeman-server.example.net%3A8000" for 127.0.0.1 at 2021-12-29 03:06:43 -0500
2021-12-29T03:06:44 [I|app|a41941ec] Started GET "/rhsm/consumers/ba44926f-3ad9-4fae-8c2a-131c05561d00" for 127.0.0.1 at 2021-12-29 03:06:44 -0500
2021-12-29T03:06:44 [I|app|378efda6] Started GET "/unattended/iPXE?has_raw=yes&mac=00%3A1a%3A64%3A40%3A78%3A66&url=http%3A%2F%2Fforeman-server.example.net%3A8000" for 127.0.0.1 at 2021-12-29 03:06:44 -0500
2021-12-29T03:06:44 [I|app|f9a04ce6] Started GET "/unattended/iPXE?has_raw=yes&mac=00%3A1a%3A64%3A40%3A9a%3Aac&url=http%3A%2F%2Fforeman-server.example.net%3A8000" for 127.0.0.1 at 2021-12-29 03:06:44 -0500
2021-12-29T03:06:44 [I|app|11e24a36] Started GET "/unattended/iPXE?has_raw=yes&mac=00%3A1a%3A64%3A3c%3A57%3A29&url=http%3A%2F%2Fforeman-server.example.net%3A8000" for 127.0.0.1 at 2021-12-29 03:06:44 -0500
2021-12-29T03:06:45 [I|app|47c5e151] Started GET "/unattended/iPXE?has_raw=yes&mac=00%3A1a%3A64%3A40%3A9c%3A47&url=http%3A%2F%2Fforeman-server.example.net%3A8000" for 127.0.0.1 at 2021-12-29 03:06:45 -0500
2021-12-29T03:06:45 [I|app|3112901b] Started GET "/unattended/iPXE?has_raw=yes&mac=00%3A1a%3A64%3A40%3Ad3%3A4b&url=http%3A%2F%2Fforeman-server.example.net%3A8000" for 127.0.0.1 at 2021-12-29 03:06:45 -0500
2021-12-29T03:06:45 [I|app|e0f92c2c] Started GET "/unattended/iPXE?has_raw=yes&mac=5c%3Af3%3Afc%3Ac6%3Aff%3Ae8&url=http%3A%2F%2Fforeman-server.example.net%3A8000" for 127.0.0.1 at 2021-12-29 03:06:45 -0500
2021-12-29T03:06:45 [I|app|976ce162] Started GET "/unattended/iPXE?has_raw=yes&mac=5c%3Af3%3Afc%3Ac6%3Af8%3A08&url=http%3A%2F%2Fforeman-server.example.net%3A8000" for 127.0.0.1 at 2021-12-29 03:06:45 -0500
2021-12-29T03:06:45 [I|app|b512533a] Started GET "/unattended/iPXE?has_raw=yes&mac=00%3A1a%3A64%3A40%3Ab6%3A2f&url=http%3A%2F%2Fforeman-server.example.net%3A8000" for 127.0.0.1 at 2021-12-29 03:06:45 -0500
2021-12-29T03:06:46 [I|app|f7db1584] Started GET "/unattended/iPXE?has_raw=yes&mac=00%3A1a%3A64%3A40%3Ac3%3A5e&url=http%3A%2F%2Fforeman-server.example.net%3A8000" for 127.0.0.1 at 2021-12-29 03:06:46 -0500
2021-12-29T03:06:46 [I|app|f12a2ade] Started GET "/unattended/iPXE?bootstrap=1&url=http%3A%2F%2Fforeman-server.example.net%3A8000" for 127.0.0.1 at 2021-12-29 03:06:46 -0500
2021-12-29T03:06:46 [I|app|b2a5e9f9] Started GET "/unattended/iPXE?has_raw=yes&mac=00%3A1a%3A64%3A40%3Ab9%3A19&url=http%3A%2F%2Fforeman-server.example.net%3A8000" for 127.0.0.1 at 2021-12-29 03:06:46 -0500
2021-12-29T03:06:46 [I|app|010cb959] Started GET "/unattended/iPXE?has_raw=yes&mac=00%3A1a%3A64%3A40%3A7b%3Aed&url=http%3A%2F%2Fforeman-server.example.net%3A8000" for 127.0.0.1 at 2021-12-29 03:06:46 -0500
2021-12-29T03:06:46 [I|app|85b4a5f5] Started GET "/unattended/iPXE?has_raw=yes&mac=00%3A1a%3A64%3A40%3Aa9%3Af8&url=http%3A%2F%2Fforeman-server.example.net%3A8000" for 127.0.0.1 at 2021-12-29 03:06:46 -0500
2021-12-29T03:06:46 [I|app|efcf9ac9] Started GET "/unattended/iPXE?has_raw=yes&mac=5c%3Af3%3Afc%3Ac6%3Af4%3Adf&url=http%3A%2F%2Fforeman-server.example.net%3A8000" for 127.0.0.1 at 2021-12-29 03:06:46 -0500
2021-12-29T03:06:47 [I|app|b0d80e48] Started GET "/unattended/iPXE?bootstrap=1&url=http%3A%2F%2Fforeman-server.example.net%3A8000" for 127.0.0.1 at 2021-12-29 03:06:47 -0500
2021-12-29T03:06:47 [I|app|bca79a44] Started GET "/unattended/iPXE?has_raw=yes&mac=00%3A1a%3A64%3A40%3A7b%3Ac5&url=http%3A%2F%2Fforeman-server.example.net%3A8000" for 127.0.0.1 at 2021-12-29 03:06:47 -0500
2021-12-29T03:06:47 [I|app|951f527b] Started GET "/unattended/iPXE?has_raw=yes&mac=00%3A1a%3A64%3A40%3Ac1%3Afe&url=http%3A%2F%2Fforeman-server.example.net%3A8000" for 127.0.0.1 at 2021-12-29 03:06:47 -0500
2021-12-29T03:06:47 [I|app|96ab4dcc] Started GET "/unattended/iPXE?bootstrap=1&url=http%3A%2F%2Fforeman-server.example.net%3A8000" for 127.0.0.1 at 2021-12-29 03:06:47 -0500
2021-12-29T03:06:47 [I|app|23f41889] Started GET "/unattended/iPXE?has_raw=yes&mac=00%3A1a%3A64%3A40%3Ac2%3A10&url=http%3A%2F%2Fforeman-server.example.net%3A8000" for 127.0.0.1 at 2021-12-29 03:06:47 -0500
2021-12-29T03:06:47 [I|app|4e08f4b6] Started GET "/unattended/iPXE?has_raw=yes&mac=00%3A1a%3A64%3A43%3Aec%3A8a&url=http%3A%2F%2Fforeman-server.example.net%3A8000" for 127.0.0.1 at 2021-12-29 03:06:47 -0500
2021-12-29T03:06:47 [I|app|3cd40af1] Started GET "/unattended/iPXE?has_raw=yes&mac=00%3A1a%3A64%3A40%3A6a%3A3c&url=http%3A%2F%2Fforeman-server.example.net%3A8000" for 127.0.0.1 at 2021-12-29 03:06:47 -0500
2021-12-29T03:06:47 [I|app|15d02362] Started GET "/unattended/iPXE?has_raw=yes&mac=5c%3Af3%3Afc%3Ad5%3Ad2%3A71&url=http%3A%2F%2Fforeman-server.example.net%3A8000" for 127.0.0.1 at 2021-12-29 03:06:47 -0500
2021-12-29T03:06:47 [I|app|1de93893] Started GET "/unattended/iPXE?bootstrap=1&url=http%3A%2F%2Fforeman-server.example.net%3A8000" for 127.0.0.1 at 2021-12-29 03:06:47 -0500
2021-12-29T03:06:47 [I|app|e52c88dd] Started GET "/unattended/iPXE?has_raw=yes&mac=00%3A1a%3A64%3A40%3Aab%3A33&url=http%3A%2F%2Fforeman-server.example.net%3A8000" for 127.0.0.1 at 2021-12-29 03:06:47 -0500
2021-12-29T03:06:47 [I|app|5b694c57] Started GET "/unattended/iPXE?has_raw=yes&mac=00%3A1a%3A64%3A40%3Ac3%3A6b&url=http%3A%2F%2Fforeman-server.example.net%3A8000" for 127.0.0.1 at 2021-12-29 03:06:47 -0500
2021-12-29T03:06:47 [I|app|6a40ecb7] Started GET "/unattended/iPXE?has_raw=yes&mac=00%3A1a%3A64%3A40%3Ab6%3A9d&url=http%3A%2F%2Fforeman-server.example.net%3A8000" for 127.0.0.1 at 2021-12-29 03:06:47 -0500
2021-12-29T03:06:47 [I|app|48e2e64a] Started GET "/unattended/iPXE?has_raw=yes&mac=00%3A1a%3A64%3A40%3A70%3A3d&url=http%3A%2F%2Fforeman-server.example.net%3A8000" for 127.0.0.1 at 2021-12-29 03:06:47 -0500
2021-12-29T03:06:47 [I|app|c7ac0dab] Started GET "/unattended/iPXE?has_raw=yes&mac=00%3A1a%3A64%3A40%3Abc%3Aa1&url=http%3A%2F%2Fforeman-server.example.net%3A8000" for 127.0.0.1 at 2021-12-29 03:06:47 -0500
2021-12-29T03:06:48 [I|app|9bf726ad] Started GET "/unattended/iPXE?has_raw=yes&mac=00%3A1a%3A64%3A40%3A75%3A7c&url=http%3A%2F%2Fforeman-server.example.net%3A8000" for 127.0.0.1 at 2021-12-29 03:06:48 -0500
2021-12-29T03:06:48 [I|app|df9b2394] Started GET "/unattended/iPXE?has_raw=yes&mac=5c%3Af3%3Afc%3Ac6%3Af2%3A3c&url=http%3A%2F%2Fforeman-server.example.net%3A8000" for 127.0.0.1 at 2021-12-29 03:06:48 -0500
2021-12-29T03:06:48 [I|app|98fab29e] Started GET "/unattended/iPXE?has_raw=yes&mac=00%3A1a%3A64%3A40%3Ab6%3A89&url=http%3A%2F%2Fforeman-server.example.net%3A8000" for 127.0.0.1 at 2021-12-29 03:06:48 -0500
2021-12-29T03:06:48 [I|app|b7cf5913] Started GET "/unattended/iPXE?has_raw=yes&mac=00%3A1a%3A64%3A40%3A77%3A72&url=http%3A%2F%2Fforeman-server.example.net%3A8000" for 127.0.0.1 at 2021-12-29 03:06:48 -0500
2021-12-29T03:06:48 [I|app|1961611f] Started GET "/unattended/iPXE?bootstrap=1&url=http%3A%2F%2Fforeman-server.example.net%3A8000" for 127.0.0.1 at 2021-12-29 03:06:48 -0500
2021-12-29T03:06:48 [I|app|b024eab7] Started GET "/unattended/iPXE?has_raw=yes&mac=00%3A1a%3A64%3A40%3A78%3Ac4&url=http%3A%2F%2Fforeman-server.example.net%3A8000" for 127.0.0.1 at 2021-12-29 03:06:48 -0500
2021-12-29T03:06:48 [I|app|59547294] Started GET "/unattended/iPXE?bootstrap=1&url=http%3A%2F%2Fforeman-server.example.net%3A8000" for 127.0.0.1 at 2021-12-29 03:06:48 -0500
2021-12-29T03:06:48 [I|app|89d0aa30] Started GET "/unattended/iPXE?has_raw=yes&mac=00%3A1a%3A64%3A40%3Ab9%3A15&url=http%3A%2F%2Fforeman-server.example.net%3A8000" for 127.0.0.1 at 2021-12-29 03:06:48 -0500
2021-12-29T03:06:48 [I|app|b9c36876] Started GET "/unattended/iPXE?has_raw=yes&mac=00%3A1a%3A64%3A40%3A77%3A69&url=http%3A%2F%2Fforeman-server.example.net%3A8000" for 127.0.0.1 at 2021-12-29 03:06:48 -0500
2021-12-29T03:06:48 [I|app|9babf305] Started GET "/unattended/iPXE?has_raw=yes&mac=00%3A1a%3A64%3A40%3A9a%3A08&url=http%3A%2F%2Fforeman-server.example.net%3A8000" for 127.0.0.1 at 2021-12-29 03:06:48 -0500
2021-12-29T03:06:48 [I|app|07ba4213] Started GET "/unattended/iPXE?has_raw=yes&mac=00%3A1a%3A64%3A40%3A8b%3A02&url=http%3A%2F%2Fforeman-server.example.net%3A8000" for 127.0.0.1 at 2021-12-29 03:06:48 -0500
2021-12-29T03:06:49 [I|app|2ac5267c] Started GET "/unattended/iPXE?has_raw=yes&mac=00%3A1a%3A64%3A40%3Aa8%3A36&url=http%3A%2F%2Fforeman-server.example.net%3A8000" for 127.0.0.1 at 2021-12-29 03:06:49 -0500
2021-12-29T03:06:49 [I|app|8d79c702] Started GET "/unattended/iPXE?has_raw=yes&mac=24%3A2f%3Afa%3A01%3Aeb%3Afd&url=http%3A%2F%2Fforeman-server.example.net%3A8000" for 127.0.0.1 at 2021-12-29 03:06:49 -0500
2021-12-29T03:06:49 [I|app|e3462edd] Started GET "/unattended/iPXE?has_raw=yes&mac=00%3A1a%3A64%3A40%3A76%3A3f&url=http%3A%2F%2Fforeman-server.example.net%3A8000" for 127.0.0.1 at 2021-12-29 03:06:49 -0500
2021-12-29T03:06:49 [I|app|9f2c934b] Started GET "/unattended/iPXE?has_raw=yes&mac=00%3A1a%3A64%3A40%3A9b%3A93&url=http%3A%2F%2Fforeman-server.example.net%3A8000" for 127.0.0.1 at 2021-12-29 03:06:49 -0500
2021-12-29T03:06:49 [I|app|12ab09b5] Started GET "/unattended/iPXE?has_raw=yes&mac=00%3A1a%3A64%3A40%3Ab9%3A71&url=http%3A%2F%2Fforeman-server.example.net%3A8000" for 127.0.0.1 at 2021-12-29 03:06:49 -0500
2021-12-29T03:06:49 [I|app|f35ed153] Started GET "/unattended/iPXE?has_raw=yes&mac=00%3A1a%3A64%3A40%3Ab9%3A0b&url=http%3A%2F%2Fforeman-server.example.net%3A8000" for 127.0.0.1 at 2021-12-29 03:06:49 -0500
2021-12-29T03:06:49 [I|app|fa76850d] Started GET "/unattended/iPXE?has_raw=yes&mac=00%3A1a%3A64%3A40%3A81%3A7c&url=http%3A%2F%2Fforeman-server.example.net%3A8000" for 127.0.0.1 at 2021-12-29 03:06:49 -0500
2021-12-29T03:06:50 [I|app|e09d6a4a] Started GET "/unattended/iPXE?has_raw=yes&mac=00%3A1a%3A64%3A40%3A76%3Afc&url=http%3A%2F%2Fforeman-server.example.net%3A8000" for 127.0.0.1 at 2021-12-29 03:06:50 -0500
2021-12-29T03:06:50 [I|app|4f8d00f7] Started GET "/unattended/iPXE?has_raw=yes&mac=00%3A1a%3A64%3A40%3A6a%3A1f&url=http%3A%2F%2Fforeman-server.example.net%3A8000" for 127.0.0.1 at 2021-12-29 03:06:50 -0500
2021-12-29T03:06:51 [I|app|463a53e0] Started GET "/unattended/iPXE?has_raw=yes&mac=5c%3Af3%3Afc%3Ac6%3Afa%3Ab5&url=http%3A%2F%2Fforeman-server.example.net%3A8000" for 127.0.0.1 at 2021-12-29 03:06:51 -0500
2021-12-29T03:06:51 [I|app|6e9136e8] Started GET "/unattended/iPXE?has_raw=yes&mac=00%3A1a%3A64%3A40%3A91%3Aa3&url=http%3A%2F%2Fforeman-server.example.net%3A8000" for 127.0.0.1 at 2021-12-29 03:06:51 -0500
2021-12-29T03:06:53 [I|app|d3eeda00] Started GET "/unattended/iPXE?bootstrap=1&url=http%3A%2F%2Fforeman-server.example.net%3A8000" for 127.0.0.1 at 2021-12-29 03:06:53 -0500
2021-12-29T03:06:53 [I|app|cfbcaab8] Started GET "/unattended/iPXE?has_raw=yes&mac=00%3A1a%3A64%3A40%3Ad1%3Aca&url=http%3A%2F%2Fforeman-server.example.net%3A8000" for 127.0.0.1 at 2021-12-29 03:06:53 -0500
2021-12-29T03:06:53 [I|app|52aca90e] Started GET "/unattended/iPXE?has_raw=yes&mac=00%3A1a%3A64%3A40%3A70%3Aba&url=http%3A%2F%2Fforeman-server.example.net%3A8000" for 127.0.0.1 at 2021-12-29 03:06:53 -0500
2021-12-29T03:06:53 [I|app|35dc3f54] Started GET "/unattended/iPXE?has_raw=yes&mac=24%3A2f%3Afa%3A02%3A62%3Aec&url=http%3A%2F%2Fforeman-server.example.net%3A8000" for 127.0.0.1 at 2021-12-29 03:06:53 -0500
2021-12-29T03:06:53 [I|app|ddbb3ecf] Started GET "/unattended/iPXE?has_raw=yes&mac=24%3A2f%3Afa%3A01%3Af2%3Ae4&url=http%3A%2F%2Fforeman-server.example.net%3A8000" for 127.0.0.1 at 2021-12-29 03:06:53 -0500

I’m wondering if someone in the community would help me confirm that this is likely the cause of my CPU spike, or perhaps there is something else I should be looking for?

Memory:

# free -g
              total        used        free      shared  buff/cache   available
Mem:             62          21          28           1          12          38
Swap:             1           0           1

Looks like I should have all the memory I’d need.

Disk:

# df -h
Filesystem                         Size  Used Avail Use% Mounted on
devtmpfs                            32G     0   32G   0% /dev
tmpfs                               32G   40K   32G   1% /dev/shm
tmpfs                               32G  1.5G   30G   5% /run
tmpfs                               32G     0   32G   0% /sys/fs/cgroup
/dev/mapper/system-root            4.0G  3.2G  832M  80% /
/dev/sda1                          397M  314M   84M  79% /boot
/dev/mapper/system-var             8.5G  6.8G  1.8G  80% /var
/dev/mapper/system-opt             1.6G  1.3G  280M  83% /opt
/dev/mapper/foremanvg-mongodb      5.0G  3.8G  1.3G  76% /var/lib/mongodb
/dev/mapper/foremanvg-lib_pulp     335G  145G  191G  44% /var/lib/pulp
/dev/mapper/foremanvg-tftp         2.0G  1.4G  671M  68% /var/lib/tftpboot
/dev/mapper/system-tmp            1014M   65M  950M   7% /tmp
/dev/mapper/foremanvg-cache_pulp    10G   33M   10G   1% /var/cache/pulp

Regarding the above disk geometry and usage, it looks pretty dire. I’m working on getting some more space allocated to the logical volumes.

Anyways, I know that was a lot. I just kind of wanted to bring up what I’m setting off to accomplish and what I’ve been seeing along the way. Maybe there are some red flags or general tips/advice the community can bring to my attention.

Thank you so much.

mclausen · December 29, 2021, 11:27pm

I’m generally in favor of the K.I.S.S. principle, and beefing up a single Foreman/Katello server is not a bad idea. One thing you may want to consider is deploying Smart-Proxy systems to distribute the load to localized or regional systems (especially when these systems patch - your network folks will thank you for not flooding the WAN connections with patching data).

For my deployment, I have a primary Katello server but it services no clients. Instead the clients are registered to Smart-Proxy servers located in each of my data centers. That way the patching or provisioning traffic is handled locally instead of coming over the WAN to where my primary Katello server is located.

Keep in mind though, a single Katello server is by no means a bad deployment - but be mindful of your network connections to make sure they don’t become saturated when your 6000+ systems patch against the Katello server.

maximilian · December 30, 2021, 1:12pm

In general, I’d recommend fan out synchronizing content and patching systems as much as possible. Running multiple sync plans daily at the same time might be the reason for CPU spikes.

Also, if you manage 1000+ hosts with Foreman/Katello, scaling your host vertically should be a good idea/worthwhile investment, especially if you notice bottlenecks.

fresh-pie · January 3, 2022, 3:53pm

Thanks so much for the advice guys! I’m going to start with scaling up my Foreman instance to 8 CPUs and see how things develop from there.

We currently synchronize our CentOS repositories once a month and deploy patches in a slow rollout, I’ll continue to monitor our Foreman server during these deployments to see just how far we are pushing it.

Thanks again

Duncan_Innes · January 4, 2022, 10:26am

What version Foreman/Katello are you running? Curious to know if you are running Pulp2 or Pulp3 backend. MongoDB is a serious drag for us and the migration to Pulp3 cannot come soon enough.

lzap · January 4, 2022, 12:37pm

Keep in mind that pulp3 is more memory hungry, our installer sets N amount of workers (number of cores) and it can be actually an overkill. I think 4 should be a good enough unless you want to sync a ton of repos or promote many CVs. Check all pulp systemd units after deployment and keep an eye on memory.

fresh-pie · January 4, 2022, 1:03pm

D’oh, I should of posted that in my original post. I am running Foreman 2.2.3 with Katello 3.17.3.

I believe Pulp3 is installed:

# rpm -qa | grep python3-pulpcore
python3-pulpcore-3.7.9-1.el7.noarch

Thanks!