Tomcat / Candlepin keep crashing with a task "Create organization ‘Default Organization'"

Hello everybody, this is my first post and i’m a new user,

I installed the latest Foreman 1.22.0 with Katello 3.12 to a fresh installed CentOS 7 VM (Minimal Install from ISO).

Problem:

After the installation there is a task with status error: “Create organization ‘Default Organization’” with error message:

Katello::Resources::Candlepin::Owner: Server broke connection  (POST /candlepin/owners/Default_Organization/uebercert)

The “Tomcat” service fails the only log entry i can find is:

/var/log/messages:

server: # A fatal error has been detected by the Java Runtime Environment:
server: #
server: # SIGSEGV (0xb) at pc=0x00007f8bdd8d3a4f, pid=30253, tid=0x00007f8c09b6c700
server: #
server: # JRE version: OpenJDK Runtime Environment (8.0_222-b10) (build 1.8.0_222-b10)
server: # Java VM: OpenJDK 64-Bit Server VM (25.222-b10 mixed mode linux-amd64 compressed oops)
server: # Problematic frame:
server: # C [libplds4.so+0x1a4f] PL_HashTableLookupConst+0xf
server: #
server: # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try “ulimit -c unlimited” before starting Java again
server: #
server: # An error report file with more information is saved as:
server: # /usr/share/tomcat/hs_err_pid30253.log
server: #
server: # If you would like to submit a bug report, please visit:
server: # http://bugreport.java.com/bugreport/crash.jsp
server: # The crash happened outside the Java Virtual Machine in native code.
server: # See problematic frame for where to report the bug.
server: #
systemd: tomcat.service: main process exited, code=killed, status=6/ABRT
systemd: Unit tomcat.service entered failed state.
systemd: tomcat.service failed.

I can see that Candlepin has failed with “hammer ping” and “systemctl status tomcat”.
I can start candlepin with “systemctl start tomcat” or “foreman-maintain service start”.

Expected outcome:
Resume the “Create organization ‘Default Organization’” Task

Foreman and Proxy versions:
foreman-installer-1.22.0-1.el7.noarch
foreman-1.22.0-1.el7.noarch
foreman-proxy-1.22.0-1.el7.noarch

Foreman and Proxy plugin versions:
katello-3.12.1-1.el7.noarch
candlepin-2.6.5-1.el7.noarch
pulp-server-2.19.1-1.el7.noarch
puppetserver-6.5.0-1.el7.noarch

Thanks in Advance for your help i am clueless on this issue !

Probably same issue as here (resolved) : Candlepin crash after upgrade to 1.22.1

TL;DR
Temporary fix: yum downgrade java-1.8.0-openjdk-1:1.8.0.222.b10-0.el7_6.x86_64 java-1.8.0-openjdk-headless-1:1.8.0.222.b10-0.el7_6.x86_64

thanks for your suggestion…
alas, I already tried that…
even with the downgraded version, I can’t resume the task, candlepin keep crashing…

2 Likes

Same problem here… Unable to install Foreman + Katelo… Java downgrade is not helping…

Unfortunately i have the same problem. But i can’t workaround with a java downgrade. On my CentOS 7 i have downgraded

java-1.8.0-openjdk-1:1.8.0.222.b10-1.el7_7.x86_64
java-1.8.0-openjdk-headless-1:1.8.0.222.b10-1.el7_7.x86_64

to the following version:

java-1.8.0-openjdk-headless-1.8.0.222.b03-1.el7.x86_64
java-1.8.0-openjdk-1.8.0.222.b03-1.el7.x86_64

Which version is the last working one?

Now i can answer myself… but maybe it helps someone else.

I did not pay enough attention, my Centos 7 was already updated to 7.7 but the required java packages are only available in the 7.6 repo.

So the following worked for me:

wget http://mirror.centos.org/centos-7/7.6.1810/updates/x86_64/Packages/java-1.8.0-openjdk-1.8.0.222.b10-0.el7_6.x86_64.rpm
wget http://mirror.centos.org/centos-7/7.6.1810/updates/x86_64/Packages/java-1.8.0-openjdk-headless-1.8.0.222.b10-0.el7_6.x86_64.rpm

yum downgrade java-1.8.0-openjdk-1.8.0.222.b10-0.el7_6.x86_64.rpm java-1.8.0-openjdk-headless-1.8.0.222.b10-0.el7_6.x86_64.rpm

So it is 1.8.0.222.b10-0.el7_6.x86_64

1 Like

It seems the links are no longer valid since the content has been moved to vault.centos.org
Obviously just a temporary fix until we can get back to CentOS 7.7 Java.

wget http://vault.centos.org/7.6.1810/updates/x86_64/Packages/java-1.8.0-openjdk-headless-1.8.0.222.b10-0.el7_6.x86_64.rpm
wget http://vault.centos.org/7.6.1810/updates/x86_64/Packages/java-1.8.0-openjdk-1.8.0.222.b10-0.el7_6.x86_64.rpm

Just incase anyone else was getting those 404’s

Has downgrading Java worked as a fix? I have even upgraded to 3.13-RC2 and still have the same tomcat issue.

New versions of Candlepin have been pushed to the Katello 3.10, 3.11, and 3.12 repos. A yum update candlepin followed by foreman-maintain service restart should do the trick.

1 Like

Meanwhile https://ci.theforeman.org/job/katello-3.13-rpm-pipeline/12/ is still running. After that we can kick off a nightly as well.

1 Like

Thankyou… Installation of Foreman 1.22.1 + Katello 3.12 went through without any issues…

1 Like

Sorry for the late reply, just saw your message. The Java downgrade did fix my issue of Candlepin constantly crashing. I am going to try the new Candlepin update today and update Java as well.

Thanks for the reply. I have upgraded my Katello server and all capsules to the latest patches for CentOS 7.7 and to Katello 3.13 RC-2 which still does not allow Candlepin to remain running.Candlepin-screen-shot

Screen shot of Candlepin version installed.
Candlepin-screen-shot-2

That does look like an issue, but maybe this should be a new thread?

Perhaps…but this was the same issue I was having on 3.11 after patching the OS on Monday which included the later versions of Java. In an attempt to re-mediate, I upgraded to 3.12 and then then to 3.13 RC-2. I thought it was fixed with downgrading Java yesterday, but broke again after upgrading the latest Candlepin. I have since upgraded back to the latest Java release, but the issue remains.

I appreciate all the suggestions and work going into fixing.

@Jonathon_Turel or @Justin_Sherrill could you have a look? This sounds like a blocker for 3.13 GA

@Cassius_Clay it’s possible that in your trial & error with fixing the problem the Candlepin database was not migrated (usually this happens during the installer run). In this case the service doesn’t give feedback about it even though tomcat appears to be running.

Here’s something you can try:

  • open /etc/candlepin/candlepin.conf and note the value of jpa.config.hibernate.connection.password which you’ll use as the password in a subsequent command:
  • /usr/share/candlepin/cpdb --update -u candlepin -p [password]
  • systemctl restart tomcat

Let us know if that solves the problem for you

@Jonathon_Turel so far that has seemed to start and keep the Candlepin connection alive. I’ll monitor it to make sure.

Thanks so much for you suggestion and assistance.

1 Like