Hi @iballou,
I hope I have found a relevant error (ip, mac and hostname are redacted).
Those are lines from same time relevant to the machine which has problems, we have 300+ servers so some filtering is needed and I hope I have not omitted anything relevant.
Edit:
And we have a few warning Dynflows from that day but they are connected with repo syncing.
Nothing from the time of error, not even a successful task.
2021-10-19T11:57:21 [I|aud|d0e3b305] Host::Base (1379) destroy event on autoheal false
2021-10-19T11:57:21 [I|aud|d0e3b305] Host::Base (1379) destroy event on registered_at 2018-09-20 11:06:42 UTC
2021-10-19T11:57:21 [I|aud|d0e3b305] Host::Base (1379) destroy event on registered_through
2021-10-19T11:57:21 [I|aud|d0e3b305] Host::Base (1379) destroy event on user_id
2021-10-19T11:57:21 [I|aud|d0e3b305] Host::Base (1379) destroy event on hypervisor false
2021-10-19T11:57:21 [I|aud|d0e3b305] Host::Base (1379) destroy event on hypervisor_host_id
2021-10-19T11:57:21 [I|aud|d0e3b305] Host::Base (1379) destroy event on purpose_usage
2021-10-19T11:57:21 [I|aud|d0e3b305] Host::Base (1379) destroy event on purpose_role
2021-10-19T11:57:21 [I|aud|d0e3b305] Host::Base (1379) destroy event on dmi_uuid 8071C54A-D427-E611-906E-0012795D96DD
2021-10-19T11:57:21 [I|aud|d0e3b305] Host::Base (1379) destroy event on pool_ids []
2021-10-19T11:57:21 [I|aud|d0e3b305] Host::Base (725) update event on uuid 9958b46f-5d79-486e-b05a-2b6d0a2d4fe4,
2021-10-19T11:57:21 [E|app|d0e3b305] RestClient::BadRequest: Katello::Resources::Candlepin::Consumer: 400 Bad Request {"displayMessage":"Problem creating unit class ConsumerDTO {\n created: null\n updated: null\n id: null\n uuid: 9958b46f-5d79-486e-b05a-2b6d0a2d4fe4\n name: host.domain.example.com\n username: foreman_admin\n entitlementStatus: null\n serviceLevel: \n role: \n usage: \n addOns: []\n systemPurposeStatus: null\n releaseVer: null\n owner: null\n environment: class EnvironmentDTO {\n created: 2018-10-14T15:27:35.964Z\n updated: 2018-10-14T15:27:35.964Z\n id: 0bee26a340d8418acb4502bbf4425a28\n name: Library/Farm_WorkerNode_EL7\n description: null\n owner: class NestedOwnerDTO {\n id: 1367995765f6a9d70165f6aa5e7b0001\n key: FZU\n displayName: FZU\n href: /owners/FZU\n }\n environmentContent: [class EnvironmentContentDTO {\n content: class ContentDTO {\n created: 2018-09-20T12:33:13.072Z\n updated: 2021-09-24T16:40:10.855Z\n uuid: 136799577beddb47017c18adb96702e6\n id: 1537446793082\n type: yum\n label: FZU_EGI_EGI_trustanchors\n name: EGI trustanchors\n vendor: Custom\n contentUrl: /custom/EGI/EGI_trustanchors\n requiredTags: null\n releaseVer: null\n gpgUrl: ../../katello/api/v2/repositories/13/gpg_key_content\n modifiedProductIds: []\n arches: null\n metadataExpire: 1\n }\n enabled: null\n }, class EnvironmentContentDTO {\n content: class ContentDTO {\n created: 2018-09-20T12:42:49.958Z\n updated: 2021-09-24T16:48:26.354Z\n uuid: 136799577beddb47017c18b548f20bed\n id: 1537447369972\n type: yum\n label: FZU_UMD_4_EL7_UMD_4_EL7_base\n name: UMD 4 EL7 base\n vendor: Custom\n contentUrl: /custom/UMD_4_EL7/UMD_4_EL7_base\n requiredTags: null\n releaseVer: null\n gpgUrl: ../../katello/api/v2/repositories/16/gpg_key_content\n modifiedProductIds: []\n arches: null\n metadataExpire: 1\n }\n enabled: null\n }, class EnvironmentContentDTO {\n content: class ContentDTO {\n created: 2018-09-20T11:42:19.480Z\n updated: 2021-09-24T16:38:41.369Z\n uuid: 136799577beddb47017c18ac5bd9021d\n id: 1537443739484\n type: yum\n label: FZU_CentOS_7_CentOS_7_updates_x86_64\n name: CentOS 7 updates x86_64\n vendor: Custom\n contentUrl: /custom/CentOS_7/CentOS_7_updates_x86_64\n requiredTags: null\n releaseVer: null\n gpgUrl: ../../katello/api/v2/repositories/5/gpg_key_content\n modifiedProductIds: []\n arches: null\n metadataExpire: 1\n }\n enabled: null\n }, class EnvironmentContentDTO {\n content: class ContentDTO {\n created: 2018-10-14T15:20:09.316Z\n updated: 2021-09-24T16:48:40.222Z\n uuid: 136799577beddb47017c18b57f1e0c2d\n id: 1539530409325\n type: yum\n label: FZU_WLCG_EL7_WLCG_EL7\n name: WLCG EL7\n vendor: Custom\n contentUrl: /custom/WLCG_EL7/WLCG_EL7\n requiredTags: null\n releaseVer: null\n gpgUrl: ../../katello/api/v2/repositories/757/gpg_key_content\n modifiedProductIds: []\n arches: null\n metadataExpire: 1\n }\n enabled: null\n }, class EnvironmentContentDTO {\n content: class ContentDTO {\n created: 2018-10-14T15:15:21.072Z\n updated: 2021-09-24T16:39:14.755Z\n uuid: 136799577beddb47017c18acde430263\n id: 1539530121080\n type: yum\n label: FZU_CVMFS_EL7_CVMFS_EL7\n name: CVMFS EL7\n vendor: Custom\n contentUrl: /custom/CVMFS_EL7/CVMFS_EL7\n requiredTags: null\n releaseVer: null\n gpgUrl: ../../katello/api/v2/repositories/754/gpg_key_content\n modifiedProductIds: []\n arches: null\n metadataExpire: 1\n }\n enabled: null\n }, class EnvironmentContentDTO {\n content: class ContentDTO {\n created: 2018-10-19T08:18:14.419Z\n updated: 2021-09-24T16:40:53.056Z\n uuid: 136799577beddb47017c18ae5e400383\n id: 1539937094418\n type: yum\n label: FZU_EPEL_7_n_EPEL_7_n\n name: EPEL 7\n vendor: Custom\n contentUrl: /custom/EPEL_7_n/EPEL_7_n\n requiredTags: null\n releaseVer: null\n gpgUrl: ../../katello/api/v2/repositories/1286/gpg_key_content\n modifiedProductIds: []\n arches: null\n metadataExpire: 1\n }\n enabled: null\n }, class EnvironmentContentDTO {\n content: class ContentDTO {\n created: 2018-09-20T11:42:15.461Z\n updated: 2021-09-24T16:38:24.743Z\n uuid: 136799577beddb47017c18ac1ae701fa\n id: 1537443735462\n type: yum\n label: FZU_CentOS_7_CentOS_7_fasttrack_x86_64\n name: CentOS 7 fasttrack x86_64\n vendor: Custom\n contentUrl: /custom/CentOS_7/CentOS_7_fasttrack_x86_64\n requiredTags: null\n releaseVer: null\n gpgUrl: ../../katello/api/v2/repositories/3/gpg_key_content\n modifiedProductIds: []\n arches: null\n metadataExpire: 1\n }\n enabled: null\n }, class EnvironmentContentDTO {\n content: class ContentDTO {\n created: 2018-09-20T11:42:13.424Z\n updated: 2021-09-24T16:38:13.575Z\n uuid: 136799577beddb47017c18abef4701e3\n id: 1537443733425\n type: yum\n label: FZU_CentOS_7_CentOS_7_extras_x86_64\n name: CentOS 7 extras x86_64\n vendor: Custom\n contentUrl: /custom/CentOS_7/CentOS_7_extras_x86_64\n requiredTags: null\n releaseVer: null\n gpgUrl: ../../katello/api/v2/repositories/2/gpg_key_content\n modifiedProductIds: []\n arches: null\n metadataExpire: 1\n }\n enabled: null\n }, class EnvironmentContentDTO {\n content: class ContentDTO {\n created: 2019-06-18T10:49:13.320Z\n updated: 2021-09-24T16:40:21.008Z\n uuid: 136799577beddb47017c18ade1100308\n 2021-10-19T11:57:21 [I|aud|d0e3b305] Host::Base (1379) destroy event on autoheal false
2021-10-19T11:57:21 [I|aud|d0e3b305] Host::Base (1379) destroy event on registered_at 2018-09-20 11:06:42 UTC
2021-10-19T11:57:21 [I|aud|d0e3b305] Host::Base (1379) destroy event on registered_through
2021-10-19T11:57:21 [I|aud|d0e3b305] Host::Base (1379) destroy event on user_id
2021-10-19T11:57:21 [I|aud|d0e3b305] Host::Base (1379) destroy event on hypervisor false
2021-10-19T11:57:21 [I|aud|d0e3b305] Host::Base (1379) destroy event on hypervisor_host_id
2021-10-19T11:57:21 [I|aud|d0e3b305] Host::Base (1379) destroy event on purpose_usage
2021-10-19T11:57:21 [I|aud|d0e3b305] Host::Base (1379) destroy event on purpose_role
2021-10-19T11:57:21 [I|aud|d0e3b305] Host::Base (1379) destroy event on dmi_uuid 8071C54A-D427-E611-906E-0012795D96DD
2021-10-19T11:57:21 [I|aud|d0e3b305] Host::Base (1379) destroy event on pool_ids []
2021-10-19T11:57:21 [I|aud|d0e3b305] Host::Base (725) update event on uuid 9958b46f-5d79-486e-b05a-2b6d0a2d4fe4,
2021-10-19T11:57:21 [E|app|d0e3b305] RestClient::BadRequest: Katello::Resources::Candlepin::Consumer: 400 Bad Request {"displayMessage":"Problem creating unit class ConsumerDTO {\n created: null\n updated: null\n id: null\n uuid: 9958b46f-5d79-486e-b05a-2b6d0a2d4fe4\n name: host.domain.example.com\n username: foreman_admin\n entitlementStatus: null\n serviceLevel: \n role: \n usage: \n addOns: []\n systemPurposeStatus: null\n releaseVer: null\n owner: null\n environment: class EnvironmentDTO {\n created: 2018-10-14T15:27:35.964Z\n updated: 2018-10-14T15:27:35.964Z\n id: 0bee26a340d8418acb4502bbf4425a28\n name: Library/Farm_WorkerNode_EL7\n description: null\n owner: class NestedOwnerDTO {\n id: 1367995765f6a9d70165f6aa5e7b0001\n key: FZU\n displayName: FZU\n href: /owners/FZU\n }\n environmentContent: [class EnvironmentContentDTO {\n content: class ContentDTO {\n created: 2018-09-20T12:33:13.072Z\n updated: 2021-09-24T16:40:10.855Z\n uuid: 136799577beddb47017c18adb96702e6\n id: 1537446793082\n type: yum\n label: FZU_EGI_EGI_trustanchors\n name: EGI trustanchors\n vendor: Custom\n contentUrl: /custom/EGI/EGI_trustanchors\n requiredTags: null\n releaseVer: null\n gpgUrl: ../../katello/api/v2/repositories/13/gpg_key_content\n modifiedProductIds: []\n arches: null\n metadataExpire: 1\n }\n enabled: null\n }, class EnvironmentContentDTO {\n content: class ContentDTO {\n created: 2018-09-20T12:42:49.958Z\n updated: 2021-09-24T16:48:26.354Z\n uuid: 136799577beddb47017c18b548f20bed\n id: 1537447369972\n type: yum\n label: FZU_UMD_4_EL7_UMD_4_EL7_base\n name: UMD 4 EL7 base\n vendor: Custom\n contentUrl: /custom/UMD_4_EL7/UMD_4_EL7_base\n requiredTags: null\n releaseVer: null\n gpgUrl: ../../katello/api/v2/repositories/16/gpg_key_content\n modifiedProductIds: []\n arches: null\n metadataExpire: 1\n }\n enabled: null\n }, class EnvironmentContentDTO {\n content: class ContentDTO {\n created: 2018-09-20T11:42:19.480Z\n updated: 2021-09-24T16:38:41.369Z\n uuid: 136799577beddb47017c18ac5bd9021d\n id: 1537443739484\n type: yum\n label: FZU_CentOS_7_CentOS_7_updates_x86_64\n name: CentOS 7 updates x86_64\n vendor: Custom\n contentUrl: /custom/CentOS_7/CentOS_7_updates_x86_64\n requiredTags: null\n releaseVer: null\n gpgUrl: ../../katello/api/v2/repositories/5/gpg_key_content\n modifiedProductIds: []\n arches: null\n metadataExpire: 1\n }\n enabled: null\n }, class EnvironmentContentDTO {\n content: class ContentDTO {\n created: 2018-10-14T15:20:09.316Z\n updated: 2021-09-24T16:48:40.222Z\n uuid: 136799577beddb47017c18b57f1e0c2d\n id: 1539530409325\n type: yum\n label: FZU_WLCG_EL7_WLCG_EL7\n name: WLCG EL7\n vendor: Custom\n contentUrl: /custom/WLCG_EL7/WLCG_EL7\n requiredTags: null\n releaseVer: null\n gpgUrl: ../../katello/api/v2/repositories/757/gpg_key_content\n modifiedProductIds: []\n arches: null\n metadataExpire: 1\n }\n enabled: null\n }, class EnvironmentContentDTO {\n content: class ContentDTO {\n created: 2018-10-14T15:15:21.072Z\n updated: 2021-09-24T16:39:14.755Z\n uuid: 136799577beddb47017c18acde430263\n id: 1539530121080\n type: yum\n label: FZU_CVMFS_EL7_CVMFS_EL7\n name: CVMFS EL7\n vendor: Custom\n contentUrl: /custom/CVMFS_EL7/CVMFS_EL7\n requiredTags: null\n releaseVer: null\n gpgUrl: ../../katello/api/v2/repositories/754/gpg_key_content\n modifiedProductIds: []\n arches: null\n metadataExpire: 1\n }\n enabled: null\n }, class EnvironmentContentDTO {\n content: class ContentDTO {\n created: 2018-10-19T08:18:14.419Z\n updated: 2021-09-24T16:40:53.056Z\n uuid: 136799577beddb47017c18ae5e400383\n id: 1539937094418\n type: yum\n label: FZU_EPEL_7_n_EPEL_7_n\n name: EPEL 7\n vendor: Custom\n contentUrl: /custom/EPEL_7_n/EPEL_7_n\n requiredTags: null\n releaseVer: null\n gpgUrl: ../../katello/api/v2/repositories/1286/gpg_key_content\n modifiedProductIds: []\n arches: null\n metadataExpire: 1\n }\n enabled: null\n }, class EnvironmentContentDTO {\n content: class ContentDTO {\n created: 2018-09-20T11:42:15.461Z\n updated: 2021-09-24T16:38:24.743Z\n uuid: 136799577beddb47017c18ac1ae701fa\n id: 1537443735462\n type: yum\n label: FZU_CentOS_7_CentOS_7_fasttrack_x86_64\n name: CentOS 7 fasttrack x86_64\n vendor: Custom\n contentUrl: /custom/CentOS_7/CentOS_7_fasttrack_x86_64\n requiredTags: null\n releaseVer: null\n gpgUrl: ../../katello/api/v2/repositories/3/gpg_key_content\n modifiedProductIds: []\n arches: null\n metadataExpire: 1\n }\n enabled: null\n }, class EnvironmentContentDTO {\n content: class ContentDTO {\n created: 2018-09-20T11:42:13.424Z\n updated: 2021-09-24T16:38:13.575Z\n uuid: 136799577beddb47017c18abef4701e3\n id: 1537443733425\n type: yum\n label: FZU_CentOS_7_CentOS_7_extras_x86_64\n name: CentOS 7 extras x86_64\n vendor: Custom\n contentUrl: /custom/CentOS_7/CentOS_7_extras_x86_64\n requiredTags: null\n releaseVer: null\n gpgUrl: ../../katello/api/v2/repositories/2/gpg_key_content\n modifiedProductIds: []\n arches: null\n metadataExpire: 1\n }\n enabled: null\n }, class EnvironmentContentDTO {\n content: class ContentDTO {\n created: 2019-06-18T10:49:13.320Z\n updated: 2021-09-24T16:40:21.008Z\n uuid: 136799577beddb47017c18ade1100308\n id: 1560854953318\n type: yum\n label: FZU_ElasticSearch_Elastic7\n name: Elastic7\n vendor: Custom\n contentUrl: /custom/ElasticSearch/Elastic7\n requiredTags: null\n releaseVer: null\n gpgUrl: ../../katello/api/v2/repositories/17132/gpg_key_content\n modifiedProductIds: []\n arches: null\n metadataExpire: 1\n }\n enabled: null\n }, class EnvironmentContentDTO {\n content: class ContentDTO {\n created: 2018-09-20T12:34:26.654Z\n updated: 2021-09-24T16:40:04.459Z\n uuid: 136799577beddb47017c18ada06b02c3\n id: 1537446866668\n type: yum\n label: FZU_EGI_LCG_trustanchors\n name: LCG trustanchors\n vendor: Custom\n contentUrl: /custom/EGI/LCG_trustanchors\n requiredTags: null\n releaseVer: null\n gpgUrl: ../../katello/api/v2/repositories/14/gpg_key_content\n modifiedProductIds: []\n arches: null\n metadataExpire: 1\n }\n enabled: null\n }, class EnvironmentContentDTO {\n content: class ContentDTO {\n created: 2018-09-20T11:42:17.492Z\n updated: 2021-09-24T16:38:34.640Z\n uuid: 136799577beddb47017c18ac41900210\n id: 1537443737494\n type: yum\n label: FZU_CentOS_7_CentOS_7_os_x86_64\n name: CentOS 7 os x86_64\n vendor: Custom\n contentUrl: /custom/CentOS_7/CentOS_7_os_x86_64\n requiredTags: null\n releaseVer: null\n gpgUrl: ../../katello/api/v2/repositories/4/gpg_key_content\n modifiedProductIds: []\n arches: null\n metadataExpire: 1\n }\n enabled: null\n }, class EnvironmentContentDTO {\n content: class ContentDTO {\n created: 2020-05-14T14:34:38.745Z\n updated: 2021-09-24T16:46:54.451Z\n uuid: 136799577beddb47017c18b3e1f409dd\n id: 1589466878748\n type: yum\n label: FZU_Puppet_EL7_Puppet_6_EL7\n name: Puppet 6 EL7\n vendor: Custom\n contentUrl: /custom/Puppet_EL7/Puppet_6_EL7\n requiredTags: null\n releaseVer: null\n gpgUrl: ../../katello/api/v2/repositories/65525/gpg_key_content\n modifiedProductIds: []\n arches: null\n metadataExpire: 1\n }\n enabled: null\n }, class EnvironmentContentDTO {\n content: class ContentDTO {\n created: 2020-11-03T16:13:30.641Z\n updated: 2021-09-24T16:42:48.663Z\n uuid: 136799577beddb47017c18b021d7051b\n id: 1604420010622\n type: yum\n label: FZU_icinga_EL7_Icinga2_EL_7\n name: Icinga2 EL 7\n vendor: Custom\n contentUrl: /custom/icinga_EL7/Icinga2_EL_7\n requiredTags: null\n releaseVer: null\n gpgUrl: ../../katello/api/v2/repositories/87006/gpg_key_content\n modifiedProductIds: []\n arches: null\n metadataExpire: 1\n }\n enabled: null\n }, class EnvironmentContentDTO {\n content: class ContentDTO {\n created: 2021-05-13T12:16:36.039Z\n updated: 2021-09-24T16:42:09.529Z\n uuid: 136799577beddb47017c18af88f90478\n id: 1620908196038\n type: yum\n label: FZU_HTCondor_EL7_HTCondor_stable_9_0_EL7\n name: HTCondor stable 9.0 EL7\n vendor: Custom\n contentUrl: /custom/HTCondor_EL7/HTCondor_stable_9_0_EL7\n requiredTags: null\n releaseVer: null\n gpgUrl: ../../katello/api/v2/repositories/111518/gpg_key_content\n modifiedProductIds: []\n arches: null\n metadataExpire: 1\n }\n enabled: null\n }, class EnvironmentContentDTO {\n content: class ContentDTO {\n created: 2019-01-19T22:02:35.198Z\n updated: 2021-09-08T20:27:39.527Z\n uuid: 136799577bc7109a017bc7183c870059\n id: 1547935355176\n type: yum\n label: FZU_FZU_EL7_FZU_EL7\n name: FZU EL7\n vendor: Custom\n contentUrl: /custom/FZU_EL7/FZU_EL7\n requiredTags: null\n releaseVer: null\n gpgUrl: ../../katello/api/v2/repositories/5178/gpg_key_content\n modifiedProductIds: []\n arches: null\n metadataExpire: 1\n }\n enabled: null\n }, class EnvironmentContentDTO {\n content: class ContentDTO {\n created: 2018-09-20T12:43:11.077Z\n updated: 2021-09-24T16:48:31.618Z\n uuid: 136799577beddb47017c18b55d820bfa\n id: 1537447391094\n type: yum\n label: FZU_UMD_4_EL7_UMD_4_EL7_updates\n name: UMD 4 EL7 updates\n vendor: Custom\n contentUrl: /custom/UMD_4_EL7/UMD_4_EL7_updates\n requiredTags: null\n releaseVer: null\n gpgUrl: ../../katello/api/v2/repositories/17/gpg_key_content\n modifiedProductIds: []\n arches: null\n metadataExpire: 1\n }\n enabled: null\n }, class EnvironmentContentDTO {\n content: class ContentDTO {\n created: 2021-09-08T20:48:40.082Z\n updated: 2021-09-24T16:43:23.812Z\n uuid: 136799577beddb47017c18b0ab2405a0\n id: 1631134120082\n type: yum\n label: FZU_Katello_EL7_Foreman_3_0_EL7\n name: Foreman 3.0 EL7\n vendor: Custom\n contentUrl: /custom/Katello_EL7/Foreman_3_0_EL7\n requiredTags: null\n releaseVer: null\n gpgUrl: ../../katello/api/v2/repositories/138413/gpg_key_content\n modifiedProductIds: []\n arches: null\n metadataExpire: 1\n }\n enabled: null\n }]\n }\n entitlementCount: null\n facts: {dmi.memory.bank_locator=NODE 2, net.interface.lo.ipv4_netmask=8, last_boot=2021-10-19 09:48:07, uname.release=3.10.0-1160.el7.x86_64, dmi.chassis.manufacturer=..............................., proc_cpuinfo.common.wp=yes, dmi.slot.type:slottype=, dmi.bios.bios_revision=0.0, net.interface.enp6s0f0.ipv6_netmask.link_list=64, dmi.connector.external_reference_designator=JA5A1 - NIC 2, system.default_locale=Unknown, dmi.baseboard.product_name=S2600KPR, dmi.system.status=No errors detected, uname.machine=x86_64, lscpu.vendor_id=GenuineIntel, dmi.chassis.asset_tag=...................., distribution.version=7, dmi.memory.array_handle=0x002a, lscpu.l1i_cache=32K, lscpu.on-line_cpu(s)_list=0-39, distribution.id=Core, distribution.version.modifier=, dmi.memory.manufacturer=Kinston, lscpu.cpu_min_mhz=1200.0000, dmi.chassis.power_supply_state=Safe, dmi.processor.voltage=1.8 V, net.interface.lo.ipv4_netmask_list=8, dmi.bios.version=SE5C610.86B.01.01.0016.033120161139, network.ipv6_address=<redacted>, lscpu.cpu_max_mhz=3100.0000, memory.memtotal=131803148, dmi.baseboard.manufacturer=Intel Corporation, proc_cpuinfo.common.cpu_family=6, memory.swaptotal=67041276, net.interface.ib0.mac_address=<redacted>, dmi.slot.current_usage=Available, lscpu.thread(s)_per_core=2, network.fqdn=host.domain.example.com, dmi.system.product_name=S2600KPR, dmi.memory.size=16384 MB, dmi.connector.internal_reference_designator=Not Specified, network.hostname=host.domain.example.com, dmi.connector.external_connector_type=RJ-45, lscpu.numa_node0_cpu(s)=0-9,20-29, lscpu.numa_node1_cpu(s)=10-19,30-39, net.interface.enp6s0f0.mac_address=<redacted>, dmi.chassis.boot-up_state=Safe, lscpu.architecture=x86_64, net.interface.enp6s0f1.mac_address=<redacted>, dmi.processor.family=Xeon, dmi.processor.serial_number=, dmi.memory.location=System Board Or Motherboard, net.interface.enp6s0f0.ipv4_netmask=24, uname.sysname=Linux, lscpu.l2_cache=256K, dmi.system.family=Family, dmi.memory.use=System Memory, cpu.topology_source=kernel /sys cpu sibling lists, net.interface.enp6s0f0.ipv4_broadcast=147.231.25.255, dmi.system.manufacturer=Intel Corporation, net.interface.lo.ipv4_address=127.0.0.1, dmi.memory.total_width=72 bit, virt.is_guest=false, cpu.thread(s)_per_core=2, net.interface.lo.ipv4_broadcast=Unknown, net.interface.enp6s0f0.ipv4_broadcast_list=147.231.25.255, net.interface.enp6s0f0.ipv6_address.link_list=<redacted>, cpu.cpu(s)=40, dmi.bios.runtime_size=64 KB, dmi.processor.upgrade=, proc_cpuinfo.common.fpu=yes, dmi.memory.locator=DIMM_F1, dmi.system.wake-up_type=Power Switch, proc_stat.btime=1634636887, network.ipv4_address=<redacted>, dmi.chassis.lock=Not Present, lscpu.bogomips=4389.43, distribution.name=CentOS Linux, proc_cpuinfo.common.address_sizes=46 bits physical, 48 bits virtual, proc_cpuinfo.common.siblings=20, dmi.processor.l2_cache_handle=0x0019, dmi.slot.slotlength=Long, dmi.memory.speed=2133 MHz (0.5ns), dmi.connector.internal_connector_type=None, dmi.slot.type:slotbuswidth=x16, dmi.system.sku_number=SKU Number, dmi.processor.status=Populated:Enabled, dmi.bios.rom_size=16384 KB, dmi.chassis.type=Rack Mount Chassis, lscpu.byte_order=Little Endian, dmi.chassis.thermal_state=Safe, proc_cpuinfo.common.vendor_id=GenuineIntel, dmi.system.serial_number=............, proc_cpuinfo.common.flags=fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqd..., uname.nodename=host.domain.example.com, net.interface.enp6s0f0.ipv6_address.link=<redacted>, dmi.bios.relase_date=03/31/2016, dmi.processor.type=Central Processor, proc_cpuinfo.common.power_management=, lscpu.numa_node(s)=2, dmi.processor.part_number=, dmi.chassis.serial_number=.................., proc_cpuinfo.common.cpuid_level=20, lscpu.l3_cache=25600K, dmi.memory.type=, net.interface.lo.ipv6_address.host_list=::1, dmi.system.uuid=8071C54A-D427-E611-906E-0012795D96DD, proc_cpuinfo.common.stepping=1, dmi.memory.error_information_handle=No Error, dmi.processor.asset_tag=, net.interface.lo.ipv4_address_list=127.0.0.1, net.interface.lo.ipv6_netmask.host_list=128, proc_cpuinfo.common.fpu_exception=yes, dmi.memory.error_correction_type=Single-bit ECC, net.interface.enp6s0f0.ipv4_address_list=<redacted>, dmi.bios.address=0xf0000, dmi.processor.l3_cache_handle=0x001a, dmi.memory.form_factor=DIMM, proc_cpuinfo.common.cpu_cores=10, lscpu.virtualization=VT-x, proc_cpuinfo.common.cache_alignment=64, dmi.processor.socket_designation=CPU1, dmi.memory.data_width=64 bit, dmi.slot.designation=Riser 3, slot 1, net.interface.enp6s0f0.ipv4_netmask_list=24, dmi.processor.l1_cache_handle=0x0018, dmi.memory.part_number=9965600-027.A01G, dmi.system.version=...................., lscpu.core(s)_per_socket=10, net.interface.lo.ipv6_address.host=::1, lscpu.cpu_mhz=1200.170, dmi.memory.maximum_capacity= , cpu.core(s)_per_socket=10, proc_cpuinfo.common.model=79, dmi.chassis.version=.................., dmi.connector.port_type=Network Port, lscpu.cpu_op-mode(s)=32-bit, 64-bit, cpu.cpu_socket(s)=2, net.interface.enp6s0f0.ipv6_netmask.link=64, lscpu.model_name=Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz, virt.host_type=Not Applicable, dmi.baseboard.version=H13888-361, lscpu.flags=fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqd..., lscpu.model=79, uname.version=#1 SMP Mon Oct 19 16:18:59 UTC 2020, net.interface.lo.ipv6_netmask.host=128, lscpu.l1d_cache=32K, proc_cpuinfo.common.model_name=Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz, dmi.processor.version=Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz, lscpu.cpu_family=6, lscpu.socket(s)=2, dmi.memory.serial_number=<redacted>, dmi.memory.assettag=, proc_cpuinfo.common.microcode=0xb000017, proc_cpuinfo.common.cache_size=25600 KB, dmi.chassis.security_status=None, dmi.baseboard.serial_number=<redacted>, lscpu.stepping=1, dmi.bios.vendor=Intel Corporation, net.interface.enp6s0f0.ipv4_address=<redacted>, system.certificate_version=3.2, proc_cpuinfo.common.clflush_size=64, net.interface.lo.ipv4_broadcast_list=Unknown, lscpu.cpu(s)=40, dmi.meta.cpu_socket_count=2}\n lastCheckin: null\n installedProducts: []\n canActivate: null\n capabilities: null\n hypervisorId: null\n contentTags: null\n autoheal: true\n annotations: null\n contentAccessMode: null\n type: class ConsumerTypeDTO {\n created: null\n updated: null\n id: null\n label: system\n manifest: null\n }\n idCert: null\n guestIds: null\n href: null\n activationKeys: null\n serviceType: null\n}","requestUuid":"af94fb65-9c8c-4c87-84b4-aaf74d129d35"} (POST /candlepin/environments/0bee26a340d8418acb4502bbf4425a28/consumers/?activation_keys=0523f4d6-b277-4cf7-8d0a-d250e07a6261)
2021-10-19T11:57:21 [E|app|d0e3b305] /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.2.0.1/app/controllers/katello/api/rhsm/candlepin_proxies_controller.rb:234:in `consumer_activate'
2021-10-19T11:57:21 [I|app|d0e3b305] Rendering text template
2021-10-19T11:57:21 [I|app|d0e3b305] Rendered text template (Duration: 0.0ms | Allocations: 1)
2021-10-19T11:57:21 [I|app|d0e3b305] Completed 400 Bad Request in 975ms (Views: 1.0ms | ActiveRecord: 143.6ms | Allocations: 168064)
2021-10-19T11:57:21 [I|app|2b152199] Started GET "/rhsm/status" for <redacted> at 2021-10-19 11:57:21 +0200
2021-10-19T11:57:21 [I|app|2b152199] Processing by Katello::Api::Rhsm::CandlepinProxiesController#server_status as JSON
2021-10-19T11:57:21 [I|app|2b152199] Completed 200 OK in 96ms (Views: 0.5ms | ActiveRecord: 2.0ms | Allocations: 32704) id: 1560854953318\n type: yum\n label: FZU_ElasticSearch_Elastic7\n name: Elastic7\n vendor: Custom\n contentUrl: /custom/ElasticSearch/Elastic7\n requiredTags: null\n releaseVer: null\n gpgUrl: ../../katello/api/v2/repositories/17132/gpg_key_content\n modifiedProductIds: []\n arches: null\n metadataExpire: 1\n }\n enabled: null\n }, class EnvironmentContentDTO {\n content: class ContentDTO {\n created: 2018-09-20T12:34:26.654Z\n updated: 2021-09-24T16:40:04.459Z\n uuid: 136799577beddb47017c18ada06b02c3\n id: 1537446866668\n type: yum\n label: FZU_EGI_LCG_trustanchors\n name: LCG trustanchors\n vendor: Custom\n contentUrl: /custom/EGI/LCG_trustanchors\n requiredTags: null\n releaseVer: null\n gpgUrl: ../../katello/api/v2/repositories/14/gpg_key_content\n modifiedProductIds: []\n arches: null\n metadataExpire: 1\n }\n enabled: null\n }, class EnvironmentContentDTO {\n content: class ContentDTO {\n created: 2018-09-20T11:42:17.492Z\n updated: 2021-09-24T16:38:34.640Z\n uuid: 136799577beddb47017c18ac41900210\n id: 1537443737494\n type: yum\n label: FZU_CentOS_7_CentOS_7_os_x86_64\n name: CentOS 7 os x86_64\n vendor: Custom\n contentUrl: /custom/CentOS_7/CentOS_7_os_x86_64\n requiredTags: null\n releaseVer: null\n gpgUrl: ../../katello/api/v2/repositories/4/gpg_key_content\n modifiedProductIds: []\n arches: null\n metadataExpire: 1\n }\n enabled: null\n }, class EnvironmentContentDTO {\n content: class ContentDTO {\n created: 2020-05-14T14:34:38.745Z\n updated: 2021-09-24T16:46:54.451Z\n uuid: 136799577beddb47017c18b3e1f409dd\n id: 1589466878748\n type: yum\n label: FZU_Puppet_EL7_Puppet_6_EL7\n name: Puppet 6 EL7\n vendor: Custom\n contentUrl: /custom/Puppet_EL7/Puppet_6_EL7\n requiredTags: null\n releaseVer: null\n gpgUrl: ../../katello/api/v2/repositories/65525/gpg_key_content\n modifiedProductIds: []\n arches: null\n metadataExpire: 1\n }\n enabled: null\n }, class EnvironmentContentDTO {\n content: class ContentDTO {\n created: 2020-11-03T16:13:30.641Z\n updated: 2021-09-24T16:42:48.663Z\n uuid: 136799577beddb47017c18b021d7051b\n id: 1604420010622\n type: yum\n label: FZU_icinga_EL7_Icinga2_EL_7\n name: Icinga2 EL 7\n vendor: Custom\n contentUrl: /custom/icinga_EL7/Icinga2_EL_7\n requiredTags: null\n releaseVer: null\n gpgUrl: ../../katello/api/v2/repositories/87006/gpg_key_content\n modifiedProductIds: []\n arches: null\n metadataExpire: 1\n }\n enabled: null\n }, class EnvironmentContentDTO {\n content: class ContentDTO {\n created: 2021-05-13T12:16:36.039Z\n updated: 2021-09-24T16:42:09.529Z\n uuid: 136799577beddb47017c18af88f90478\n id: 1620908196038\n type: yum\n label: FZU_HTCondor_EL7_HTCondor_stable_9_0_EL7\n name: HTCondor stable 9.0 EL7\n vendor: Custom\n contentUrl: /custom/HTCondor_EL7/HTCondor_stable_9_0_EL7\n requiredTags: null\n releaseVer: null\n gpgUrl: ../../katello/api/v2/repositories/111518/gpg_key_content\n modifiedProductIds: []\n arches: null\n metadataExpire: 1\n }\n enabled: null\n }, class EnvironmentContentDTO {\n content: class ContentDTO {\n created: 2019-01-19T22:02:35.198Z\n updated: 2021-09-08T20:27:39.527Z\n uuid: 136799577bc7109a017bc7183c870059\n id: 1547935355176\n type: yum\n label: FZU_FZU_EL7_FZU_EL7\n name: FZU EL7\n vendor: Custom\n contentUrl: /custom/FZU_EL7/FZU_EL7\n requiredTags: null\n releaseVer: null\n gpgUrl: ../../katello/api/v2/repositories/5178/gpg_key_content\n modifiedProductIds: []\n arches: null\n metadataExpire: 1\n }\n enabled: null\n }, class EnvironmentContentDTO {\n content: class ContentDTO {\n created: 2018-09-20T12:43:11.077Z\n updated: 2021-09-24T16:48:31.618Z\n uuid: 136799577beddb47017c18b55d820bfa\n id: 1537447391094\n type: yum\n label: FZU_UMD_4_EL7_UMD_4_EL7_updates\n name: UMD 4 EL7 updates\n vendor: Custom\n contentUrl: /custom/UMD_4_EL7/UMD_4_EL7_updates\n requiredTags: null\n releaseVer: null\n gpgUrl: ../../katello/api/v2/repositories/17/gpg_key_content\n modifiedProductIds: []\n arches: null\n metadataExpire: 1\n }\n enabled: null\n }, class EnvironmentContentDTO {\n content: class ContentDTO {\n created: 2021-09-08T20:48:40.082Z\n updated: 2021-09-24T16:43:23.812Z\n uuid: 136799577beddb47017c18b0ab2405a0\n id: 1631134120082\n type: yum\n label: FZU_Katello_EL7_Foreman_3_0_EL7\n name: Foreman 3.0 EL7\n vendor: Custom\n contentUrl: /custom/Katello_EL7/Foreman_3_0_EL7\n requiredTags: null\n releaseVer: null\n gpgUrl: ../../katello/api/v2/repositories/138413/gpg_key_content\n modifiedProductIds: []\n arches: null\n metadataExpire: 1\n }\n enabled: null\n }]\n }\n entitlementCount: null\n facts: {dmi.memory.bank_locator=NODE 2, net.interface.lo.ipv4_netmask=8, last_boot=2021-10-19 09:48:07, uname.release=3.10.0-1160.el7.x86_64, dmi.chassis.manufacturer=..............................., proc_cpuinfo.common.wp=yes, dmi.slot.type:slottype=, dmi.bios.bios_revision=0.0, net.interface.enp6s0f0.ipv6_netmask.link_list=64, dmi.connector.external_reference_designator=JA5A1 - NIC 2, system.default_locale=Unknown, dmi.baseboard.product_name=S2600KPR, dmi.system.status=No errors detected, uname.machine=x86_64, lscpu.vendor_id=GenuineIntel, dmi.chassis.asset_tag=...................., distribution.version=7, dmi.memory.array_handle=0x002a, lscpu.l1i_cache=32K, lscpu.on-line_cpu(s)_list=0-39, distribution.id=Core, distribution.version.modifier=, dmi.memory.manufacturer=Kinston, lscpu.cpu_min_mhz=1200.0000, dmi.chassis.power_supply_state=Safe, dmi.processor.voltage=1.8 V, net.interface.lo.ipv4_netmask_list=8, dmi.bios.version=SE5C610.86B.01.01.0016.033120161139, network.ipv6_address=<redacted>, lscpu.cpu_max_mhz=3100.0000, memory.memtotal=131803148, dmi.baseboard.manufacturer=Intel Corporation, proc_cpuinfo.common.cpu_family=6, memory.swaptotal=67041276, net.interface.ib0.mac_address=<redacted>, dmi.slot.current_usage=Available, lscpu.thread(s)_per_core=2, network.fqdn=host.domain.example.com, dmi.system.product_name=S2600KPR, dmi.memory.size=16384 MB, dmi.connector.internal_reference_designator=Not Specified, network.hostname=host.domain.example.com, dmi.connector.external_connector_type=RJ-45, lscpu.numa_node0_cpu(s)=0-9,20-29, lscpu.numa_node1_cpu(s)=10-19,30-39, net.interface.enp6s0f0.mac_address=<redacted>, dmi.chassis.boot-up_state=Safe, lscpu.architecture=x86_64, net.interface.enp6s0f1.mac_address=<redacted>, dmi.processor.family=Xeon, dmi.processor.serial_number=, dmi.memory.location=System Board Or Motherboard, net.interface.enp6s0f0.ipv4_netmask=24, uname.sysname=Linux, lscpu.l2_cache=256K, dmi.system.family=Family, dmi.memory.use=System Memory, cpu.topology_source=kernel /sys cpu sibling lists, net.interface.enp6s0f0.ipv4_broadcast=147.231.25.255, dmi.system.manufacturer=Intel Corporation, net.interface.lo.ipv4_address=127.0.0.1, dmi.memory.total_width=72 bit, virt.is_guest=false, cpu.thread(s)_per_core=2, net.interface.lo.ipv4_broadcast=Unknown, net.interface.enp6s0f0.ipv4_broadcast_list=147.231.25.255, net.interface.enp6s0f0.ipv6_address.link_list=<redacted>, cpu.cpu(s)=40, dmi.bios.runtime_size=64 KB, dmi.processor.upgrade=, proc_cpuinfo.common.fpu=yes, dmi.memory.locator=DIMM_F1, dmi.system.wake-up_type=Power Switch, proc_stat.btime=1634636887, network.ipv4_address=<redacted>, dmi.chassis.lock=Not Present, lscpu.bogomips=4389.43, distribution.name=CentOS Linux, proc_cpuinfo.common.address_sizes=46 bits physical, 48 bits virtual, proc_cpuinfo.common.siblings=20, dmi.processor.l2_cache_handle=0x0019, dmi.slot.slotlength=Long, dmi.memory.speed=2133 MHz (0.5ns), dmi.connector.internal_connector_type=None, dmi.slot.type:slotbuswidth=x16, dmi.system.sku_number=SKU Number, dmi.processor.status=Populated:Enabled, dmi.bios.rom_size=16384 KB, dmi.chassis.type=Rack Mount Chassis, lscpu.byte_order=Little Endian, dmi.chassis.thermal_state=Safe, proc_cpuinfo.common.vendor_id=GenuineIntel, dmi.system.serial_number=............, proc_cpuinfo.common.flags=fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqd..., uname.nodename=host.domain.example.com, net.interface.enp6s0f0.ipv6_address.link=<redacted>, dmi.bios.relase_date=03/31/2016, dmi.processor.type=Central Processor, proc_cpuinfo.common.power_management=, lscpu.numa_node(s)=2, dmi.processor.part_number=, dmi.chassis.serial_number=.................., proc_cpuinfo.common.cpuid_level=20, lscpu.l3_cache=25600K, dmi.memory.type=, net.interface.lo.ipv6_address.host_list=::1, dmi.system.uuid=8071C54A-D427-E611-906E-0012795D96DD, proc_cpuinfo.common.stepping=1, dmi.memory.error_information_handle=No Error, dmi.processor.asset_tag=, net.interface.lo.ipv4_address_list=127.0.0.1, net.interface.lo.ipv6_netmask.host_list=128, proc_cpuinfo.common.fpu_exception=yes, dmi.memory.error_correction_type=Single-bit ECC, net.interface.enp6s0f0.ipv4_address_list=<redacted>, dmi.bios.address=0xf0000, dmi.processor.l3_cache_handle=0x001a, dmi.memory.form_factor=DIMM, proc_cpuinfo.common.cpu_cores=10, lscpu.virtualization=VT-x, proc_cpuinfo.common.cache_alignment=64, dmi.processor.socket_designation=CPU1, dmi.memory.data_width=64 bit, dmi.slot.designation=Riser 3, slot 1, net.interface.enp6s0f0.ipv4_netmask_list=24, dmi.processor.l1_cache_handle=0x0018, dmi.memory.part_number=9965600-027.A01G, dmi.system.version=...................., lscpu.core(s)_per_socket=10, net.interface.lo.ipv6_address.host=::1, lscpu.cpu_mhz=1200.170, dmi.memory.maximum_capacity= , cpu.core(s)_per_socket=10, proc_cpuinfo.common.model=79, dmi.chassis.version=.................., dmi.connector.port_type=Network Port, lscpu.cpu_op-mode(s)=32-bit, 64-bit, cpu.cpu_socket(s)=2, net.interface.enp6s0f0.ipv6_netmask.link=64, lscpu.model_name=Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz, virt.host_type=Not Applicable, dmi.baseboard.version=H13888-361, lscpu.flags=fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqd..., lscpu.model=79, uname.version=#1 SMP Mon Oct 19 16:18:59 UTC 2020, net.interface.lo.ipv6_netmask.host=128, lscpu.l1d_cache=32K, proc_cpuinfo.common.model_name=Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz, dmi.processor.version=Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz, lscpu.cpu_family=6, lscpu.socket(s)=2, dmi.memory.serial_number=<redacted>, dmi.memory.assettag=, proc_cpuinfo.common.microcode=0xb000017, proc_cpuinfo.common.cache_size=25600 KB, dmi.chassis.security_status=None, dmi.baseboard.serial_number=<redacted>, lscpu.stepping=1, dmi.bios.vendor=Intel Corporation, net.interface.enp6s0f0.ipv4_address=<redacted>, system.certificate_version=3.2, proc_cpuinfo.common.clflush_size=64, net.interface.lo.ipv4_broadcast_list=Unknown, lscpu.cpu(s)=40, dmi.meta.cpu_socket_count=2}\n lastCheckin: null\n installedProducts: []\n canActivate: null\n capabilities: null\n hypervisorId: null\n contentTags: null\n autoheal: true\n annotations: null\n contentAccessMode: null\n type: class ConsumerTypeDTO {\n created: null\n updated: null\n id: null\n label: system\n manifest: null\n }\n idCert: null\n guestIds: null\n href: null\n activationKeys: null\n serviceType: null\n}","requestUuid":"af94fb65-9c8c-4c87-84b4-aaf74d129d35"} (POST /candlepin/environments/0bee26a340d8418acb4502bbf4425a28/consumers/?activation_keys=0523f4d6-b277-4cf7-8d0a-d250e07a6261)
2021-10-19T11:57:21 [E|app|d0e3b305] /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.2.0.1/app/controllers/katello/api/rhsm/candlepin_proxies_controller.rb:234:in `consumer_activate'
2021-10-19T11:57:21 [I|app|d0e3b305] Rendering text template
2021-10-19T11:57:21 [I|app|d0e3b305] Rendered text template (Duration: 0.0ms | Allocations: 1)
2021-10-19T11:57:21 [I|app|d0e3b305] Completed 400 Bad Request in 975ms (Views: 1.0ms | ActiveRecord: 143.6ms | Allocations: 168064)
2021-10-19T11:57:21 [I|app|2b152199] Started GET "/rhsm/status" for <redacted> at 2021-10-19 11:57:21 +0200
2021-10-19T11:57:21 [I|app|2b152199] Processing by Katello::Api::Rhsm::CandlepinProxiesController#server_status as JSON
2021-10-19T11:57:21 [I|app|2b152199] Completed 200 OK in 96ms (Views: 0.5ms | ActiveRecord: 2.0ms | Allocations: 32704)
