Puppetserver keeps crashing: C [libc.so.6+0xcee8f]

Support
1

Problem:
I know this is most likely unrelated to foreman; but one of my puppetservers running on the same host as foreman keeps crashing randomly ~ once per day. I could not find anything wrong with it so far. It started after upgrading to Foreman 3.7 in August, which included upgrading to EL8 in the process.

Back in EL7 / Foreman 2.5 I never observed crashes.

Currently running foreman 3.9.1

(I manage another instance witch basically is the same: There are no such crashes)

Many thanks!

# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f7e12488e8f, pid=1338, tid=0x00007f7d781e5700
#
# JRE version: OpenJDK Runtime Environment (8.0_392-b08) (build 1.8.0_392-b08)
# Java VM: OpenJDK 64-Bit Server VM (25.392-b08 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  [libc.so.6+0xcee8f]  __memmove_avx_unaligned_erms+0x6f
[...]

# puppetserver --version
puppetserver version: 7.14.0

# rpm -aq|grep jdk
copy-jdk-configs-4.0-2.el8.noarch
java-1.8.0-openjdk-headless-1.8.0.392.b08-4.el8_8.x86_64

# /proc/cpuinfo 
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 61
model name      : Intel Core Processor (Broadwell)
stepping        : 2
microcode       : 0x1
cpu MHz         : 2097.570
cache size      : 4096 KB
physical id     : 0
siblings        : 1
core id         : 0
cpu cores       : 1
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx rdtscp lm constant_tsc rep_good nopl cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx smap xsaveopt arat
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs taa itlb_multihit srbds mmio_unknown
bogomips        : 4195.14
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:
...

# /etc/os-release 
NAME="Rocky Linux"
VERSION="8.9 (Green Obsidian)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="8.9"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Rocky Linux 8.9 (Green Obsidian)"
ANSI_COLOR="0;32"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:rocky:rocky:8:GA"
HOME_URL="https://rockylinux.org/"
BUG_REPORT_URL="https://bugs.rockylinux.org/"
SUPPORT_END="2029-05-31"
ROCKY_SUPPORT_PRODUCT="Rocky-Linux-8"
ROCKY_SUPPORT_PRODUCT_VERSION="8.9"
REDHAT_SUPPORT_PRODUCT="Rocky Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.9"

# uname -r
4.18.0-513.11.1.el8_9.x86_64
2

This sounds more like puppet issue and question for their devs, I am afraid we won’t be able to help. Foreman installs Puppet from the official open source Puppet repos.

3

I agree with Marek that you should probably ask the Puppet Community about this, but a wild idea I had:
Do you have any EL/ RPMs left on the system?

dnf list installed |grep el7

Iirc, we had some simillar errors with the Puppet client because for stupid reasons we had packages from the wrong RHEL release.

4

did you happen to open a ticket/thread on the puppet side? we have this too but only on one of the two instances we run. just did a bulk system update today hoping the puppetserver version bump might help but so far not yet. in our case it’s also not every run, it might go several hours before crashing

5

Sorry for the late reply.
I’ve since migrated my foreman-vm to a more modern CPU (AMD EPYC Genoa from Xeon Broadwell) and have not seen these crashes since. Before, I had multiple instances every month; fingers crossed:

[root@foreman ~]# ls -lth /var/log/puppetlabs/puppetserver/puppetserver_err_pid*
-rw-r-----. 1 puppet puppet 556K Mar  5 16:10 /var/log/puppetlabs/puppetserver/puppetserver_err_pid2667036.log
-rw-r-----. 1 puppet puppet 559K Feb 29 14:26 /var/log/puppetlabs/puppetserver/puppetserver_err_pid2155193.log
-rw-r-----. 1 puppet puppet 567K Feb 21 14:58 /var/log/puppetlabs/puppetserver/puppetserver_err_pid2144131.log
-rw-r-----. 1 puppet puppet 551K Feb 21 11:06 /var/log/puppetlabs/puppetserver/puppetserver_err_pid2031756.log
-rw-r-----. 1 puppet puppet 575K Feb 19 18:27 /var/log/puppetlabs/puppetserver/puppetserver_err_pid1640492.log
-rw-r-----. 1 puppet puppet 567K Feb 13 16:43 /var/log/puppetlabs/puppetserver/puppetserver_err_pid1316793.log
-rw-r-----. 1 puppet puppet 567K Feb  8 18:50 /var/log/puppetlabs/puppetserver/puppetserver_err_pid778367.log
-rw-r-----. 1 puppet puppet 561K Jan 31 14:47 /var/log/puppetlabs/puppetserver/puppetserver_err_pid707367.log
-rw-r-----. 1 puppet puppet 579K Jan 30 18:18 /var/log/puppetlabs/puppetserver/puppetserver_err_pid700131.log
-rw-r-----. 1 puppet puppet 553K Jan 30 15:48 /var/log/puppetlabs/puppetserver/puppetserver_err_pid632719.log
-rw-r-----. 1 puppet puppet 555K Jan 29 15:18 /var/log/puppetlabs/puppetserver/puppetserver_err_pid446777.log
-rw-r-----. 1 puppet puppet 571K Jan 26 16:42 /var/log/puppetlabs/puppetserver/puppetserver_err_pid438273.log
-rw-r-----. 1 puppet puppet 556K Jan 26 13:42 /var/log/puppetlabs/puppetserver/puppetserver_err_pid308863.log
-rw-r-----. 1 puppet puppet 575K Jan 23 14:25 /var/log/puppetlabs/puppetserver/puppetserver_err_pid249372.log
-rw-r-----. 1 puppet puppet 560K Jan 23 13:36 /var/log/puppetlabs/puppetserver/puppetserver_err_pid48505.log
-rw-r-----. 1 puppet puppet 576K Jan 20 04:07 /var/log/puppetlabs/puppetserver/puppetserver_err_pid34437.log
-rw-r-----. 1 puppet puppet 564K Jan 19 22:38 /var/log/puppetlabs/puppetserver/puppetserver_err_pid1338.log
-rw-r-----. 1 puppet puppet 601K Jan 16 11:52 /var/log/puppetlabs/puppetserver/puppetserver_err_pid112244.log
-rw-r-----. 1 puppet puppet 548K Jan 16 11:01 /var/log/puppetlabs/puppetserver/puppetserver_err_pid4118363.log
-rw-r-----. 1 puppet puppet 567K Jan 12 22:43 /var/log/puppetlabs/puppetserver/puppetserver_err_pid4049396.log
-rw-r-----. 1 puppet puppet 560K Jan 11 17:58 /var/log/puppetlabs/puppetserver/puppetserver_err_pid3926757.log
-rw-r-----. 1 puppet puppet 557K Jan  9 13:40 /var/log/puppetlabs/puppetserver/puppetserver_err_pid3778635.log
-rw-r-----. 1 puppet puppet 553K Jan  6 17:55 /var/log/puppetlabs/puppetserver/puppetserver_err_pid3667952.log
-rw-r-----. 1 puppet puppet 567K Jan  4 21:03 /var/log/puppetlabs/puppetserver/puppetserver_err_pid3644646.log
-rw-r-----. 1 puppet puppet 554K Jan  4 11:42 /var/log/puppetlabs/puppetserver/puppetserver_err_pid3537418.log
-rw-r-----. 1 puppet puppet 560K Jan  2 14:53 /var/log/puppetlabs/puppetserver/puppetserver_err_pid3462907.log
-rw-r-----. 1 puppet puppet 567K Jan  1 07:40 /var/log/puppetlabs/puppetserver/puppetserver_err_pid3432008.log
-rw-r-----. 1 puppet puppet 566K Dec 31 18:36 /var/log/puppetlabs/puppetserver/puppetserver_err_pid3194288.log
-rw-r-----. 1 puppet puppet 547K Dec 27 15:47 /var/log/puppetlabs/puppetserver/puppetserver_err_pid3075491.log
-rw-r-----. 1 puppet puppet 562K Dec 25 15:38 /var/log/puppetlabs/puppetserver/puppetserver_err_pid2976820.log
-rw-r-----. 1 puppet puppet 581K Dec 23 23:55 /var/log/puppetlabs/puppetserver/puppetserver_err_pid2965540.log
-rw-r-----. 1 puppet puppet 558K Dec 23 19:25 /var/log/puppetlabs/puppetserver/puppetserver_err_pid2924032.log
-rw-r-----. 1 puppet puppet 559K Dec 23 02:43 /var/log/puppetlabs/puppetserver/puppetserver_err_pid2834474.log
-rw-r-----. 1 puppet puppet 574K Dec 21 12:52 /var/log/puppetlabs/puppetserver/puppetserver_err_pid2732323.log
-rw-r-----. 1 puppet puppet 553K Dec 19 16:43 /var/log/puppetlabs/puppetserver/puppetserver_err_pid2475350.log
-rw-r-----. 1 puppet puppet 563K Dec 15 11:43 /var/log/puppetlabs/puppetserver/puppetserver_err_pid1928257.log
-rw-r-----. 1 puppet puppet 556K Dec  5 16:43 /var/log/puppetlabs/puppetserver/puppetserver_err_pid1870219.log
-rw-r-----. 1 puppet puppet 573K Dec  4 15:44 /var/log/puppetlabs/puppetserver/puppetserver_err_pid1217140.log
-rw-r-----. 1 puppet puppet 551K Nov 22 17:37 /var/log/puppetlabs/puppetserver/puppetserver_err_pid859372.log
-rw-r-----. 1 puppet puppet 566K Nov 16 09:16 /var/log/puppetlabs/puppetserver/puppetserver_err_pid751759.log
-rw-r-----. 1 puppet puppet 560K Nov 14 10:42 /var/log/puppetlabs/puppetserver/puppetserver_err_pid431577.log
-rw-r-----. 1 puppet puppet 570K Nov  8 12:54 /var/log/puppetlabs/puppetserver/puppetserver_err_pid210724.log
-rw-r-----. 1 puppet puppet 554K Nov  4 11:10 /var/log/puppetlabs/puppetserver/puppetserver_err_pid7464.log
-rw-r-----. 1 puppet puppet 583K Oct 31 18:40 /var/log/puppetlabs/puppetserver/puppetserver_err_pid1337.log
-rw-r-----. 1 puppet puppet 551K Oct 22 07:44 /var/log/puppetlabs/puppetserver/puppetserver_err_pid312669.log
-rw-r-----. 1 puppet puppet 550K Oct 20 19:39 /var/log/puppetlabs/puppetserver/puppetserver_err_pid253932.log
-rw-r-----. 1 puppet puppet 557K Oct 19 18:09 /var/log/puppetlabs/puppetserver/puppetserver_err_pid194902.log
-rw-r-----. 1 puppet puppet 567K Oct 18 15:59 /var/log/puppetlabs/puppetserver/puppetserver_err_pid143838.log
-rw-r-----. 1 puppet puppet 565K Oct 17 16:44 /var/log/puppetlabs/puppetserver/puppetserver_err_pid95805.log
-rw-r-----. 1 puppet puppet 561K Oct 16 19:36 /var/log/puppetlabs/puppetserver/puppetserver_err_pid4119788.log
-rw-r-----. 1 puppet puppet 587K Oct 13 15:19 /var/log/puppetlabs/puppetserver/puppetserver_err_pid4076619.log
-rw-r-----. 1 puppet puppet 556K Oct 12 19:19 /var/log/puppetlabs/puppetserver/puppetserver_err_pid3860225.log
-rw-r-----. 1 puppet puppet 552K Oct  8  2023 /var/log/puppetlabs/puppetserver/puppetserver_err_pid3636747.log
-rw-r-----. 1 puppet puppet 552K Oct  4  2023 /var/log/puppetlabs/puppetserver/puppetserver_err_pid3517966.log
-rw-r-----. 1 puppet puppet 555K Oct  2  2023 /var/log/puppetlabs/puppetserver/puppetserver_err_pid3464367.log
-rw-r-----. 1 puppet puppet 543K Oct  1  2023 /var/log/puppetlabs/puppetserver/puppetserver_err_pid3349378.log
-rw-r-----. 1 puppet puppet 558K Sep 29  2023 /var/log/puppetlabs/puppetserver/puppetserver_err_pid3137043.log
-rw-r-----. 1 puppet puppet 555K Sep 25  2023 /var/log/puppetlabs/puppetserver/puppetserver_err_pid2987089.log
-rw-r-----. 1 puppet puppet 575K Sep 22  2023 /var/log/puppetlabs/puppetserver/puppetserver_err_pid928201.log
-rw-r-----. 1 puppet puppet 557K Aug 14  2023 /var/log/puppetlabs/puppetserver/puppetserver_err_pid1279.log