/tmp does not contain “noexec” in it’s mount options.
/dev/mapper/rootvg-tmplv on /tmp type ext4 (rw,nosuid,nodev,relatime,seclabel)
/dev/mapper/rootvg-tmplv on /var/tmp type ext4 (rw,nosuid,nodev,relatime,seclabel)Just to let you know. I never installed F2.4/K4.0 on RedHat, but I did successfully on CentOS 8.3.
I used a VM (KVM) for it with 6vCPU and 8 GB. All databases are LOCAL but externally managed by PostgreSQL 12.7.
I created a iSCSI volume for /var/lib/pulp on my Synology.
This configuration works fine now for a couple of days. Not a single issue.
Even a clean install on Rocky Linux 8.3 RC1 (one of the CentOS 8 alternatives) worked fine.
As I mentioned before, the pgsql databases are local. I tried to install (using foreman-installer options) with remote PostgreSQL databases but this installation got stuck (the “Do you mean enabled?” problem).
In this forum a patch is suggested for this problem, but that not work out for me.
Because of CentOS 8 Streams, I am looking for a reliable CentOS compatible solution as well.
rgds,
-gw
So here is a fun test. Since it is virtual, and currently the build is automated. I built two identical VMS (CPU/Mem/Disk).
1 Completed successfully with the installation and 1 failed.
The failure is consistent. Its always with Candlepin 404 Not Found. Looking back through the Tomcat logs (see way above), it is the same error every failure.
It is just odd that it does succeed
It really sounds like a race condition. Perhaps there’s some async task that runs and if the service is started before it finishes, it breaks?
I was thinking the same thing, it happens in the 1400 block. I enabled DEBUG on the installer, I didn’t get much on the output.
The whole build (OS, Forman install, etc.) takes about 20-30 minutes from start to finish. I can enable anything you think would help you guys. When looking at the tomcat output, it complains that it can’t find classes. I couldn’t locate the class path, but they are indeed in the “webapps” directory for candle.
May 17 03:25:52 sec1syslog server[1178]: Exception in thread "Thread-43522 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@2d981b7b)" java.lang.NoClassDefFoundError: ch/qos/logback/classic/spi/ThrowableProxy
14-May-2021 14:46:43.271 SEVERE [main] org.apache.catalina.core.StandardContext.listenerStart Exception sending context initialized event to listener instance of class [org.candlepin.guice.CandlepinContextListener] java.lang.NoSuchFieldError: SERVER_SENT_EVENTS_TYPE
Ignore the first timestamp, but that is the same one it throws
I’m still trying to dig into this to figure out what might be the cause. With the release of RHEL 8.4, is anyone able to test an installation on it (instead of 8.3) to see if the issue persists across RHEL versions?
Looking at the df/mount outputs posted by @martux69 and @smithbr, I see you both use “non-standard” partitioning. While I don’t particularly think this should be an issue, could one of you try a vanilla (one / and that’s it) install? Or was the CentOS install that succeeded maybe like that?
If one of you is interested in an interactive debugging session (via tmate.io or similar), we can team up (I’m in central Europe)?
I can try it next weekend, but only in a virtual machine and there my try with RHEL 8.3 also worked.
But I try it and give feedback.
I can give RHEL v8.4 a go, I just need to update my build source for it.
Regarding “non standard” partitions. I am using the CIS Benchmark for my partition schema. After upgrading to RHEL v8.4, if it still fails 100% of the time, I will build a couple of test systems with a single root partition.
I would absolutely love an interactive debugging session if the above fails. As stated by @ekohl , it looks like it could be some weird race condition. I built it out like 5 times, and it worked just fine, and then 5 more times and it failed every single time.
Since my entire build, including Foreman, is automated, I can assure I didn’t fat finger something.
@evgeni : CentOS succeeded with my partitioning and I try it with one / also with RHEL 8.3 without success.
(Complete new build, not an upgrade)
cat /etc/redhat-release
Red Hat Enterprise Linux release 8.4 (Ootpa)
On the plus side, the first two side by side builds installed Foreman + Katello just fine.
I have rebuilt 2 servers 7 times each for a total of 14 builds.
Out of 14 builds, I had 14 complete 100% successfully.
I can kick off a few more to be safe, but this is significantly better than executing the installation against RHEL 8.3 (~ 50%?). It would appear that something in RHEL 8.3 with respects to Candlepin and Tomcat has been magically resolved. I only place blame there (could very much be else where) because of the webapp failing to start.
8 more rebuilds later and finally 1 failure.
2021-05-21 13:03:42 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call refresh: '/usr/sbin/foreman-rake db:seed' returned 1 instead of one of [0]
2021-05-21 13:03:42 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: '/usr/sbin/foreman-rake db:seed' returned 1 instead of one of [0]
It is Tomcat failing to start because of Candlepin
May 21 13:02:28 sec1foreman server[43025]: 21-May-2021 13:02:28.198 SEVERE [main] org.apache.catalina.core.StandardContext.startInternal One or more listeners failed to start. Full details will be found in the appropriate container log file
May 21 13:02:28 sec1foreman server[43025]: 21-May-2021 13:02:28.200 SEVERE [main] org.apache.catalina.core.StandardContext.startInternal Context [/candlepin] startup failed due to previous errors
I enabled debug during the installation, so if there is anything that might peak your curiosity that you wish to see, I am happy to share.
cat /etc/redhat-release
Red Hat Enterprise Linux release 8.4 (Ootpa)
This is the same exact build as all of the other successes.
With RHEL 8.4 my installation is running fine now.
Thanks for all the follow up testing! Even though our CI is still breaking, I’m going to pause investigating this since newer RHEL seems to fix it and my attempts to figure out what the culprit is have hit a wall.
When I initially installed foreman, I managed to get it installed on CentOS 8.3.
But I had to wipe it because somehow the whole pulp-candlepin thing ended up in a mess.
Now, I can’t seem to re-install it, it always ends with
2021-06-03 17:24:15 [NOTICE] [configure] 1300 out of 2131 done.
2021-06-03 17:24:20 [NOTICE] [configure] 1400 out of 2132 done.
2021-06-03 17:26:18 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call refresh: '/usr/sbin/foreman-rake db:seed' returned 1 instead of one of [0]
2021-06-03 17:26:18 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: '/usr/sbin/foreman-rake db:seed' returned 1 instead of one of [0]
2021-06-03 17:26:18 [NOTICE] [configure] 1500 out of 2132 done.
2021-06-03 17:24:15 [NOTICE] [configure] 1300 out of 2131 done.
2021-06-03 17:24:20 [NOTICE] [configure] 1400 out of 2132 done.
2021-06-03 17:26:18 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call refresh: '/usr/sbin/foreman-rake db:seed' returned 1 instead of one of [0]
2021-06-03 17:26:18 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: '/usr/sbin/foreman-rake db:seed' returned 1 instead of one of [0]
2021-06-03 17:26:18 [NOTICE] [configure] 1500 out of 2132 done.
2021-06-03 17:26:18 [NOTICE] [configure] 1600 out of 2132 done.
2021-06-03 17:26:18 [NOTICE] [configure] 1600 out of 2132 done.
CentOS 8.4 isn’t out yet…
Hello
Maybe this is not adding any value to the issue. I have the same problem on CentOS 8.4 and seems to be related with Candlepin. It is being stopped by Tomcat with some weird messages:
root satellite11 /var/log/tomcat #systemctl start tomcat.service
root satellite11 /var/log/tomcat #systemctl status tomcat.service
● tomcat.service - Apache Tomcat Web Application Container
Loaded: loaded (/usr/lib/systemd/system/tomcat.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2021-06-08 19:07:38 CEST; 3s ago
Main PID: 2973 (java)
Tasks: 24 (limit: 153799)
…
08-Jun-2021 19:07:43.471 WARNING [main] com.google.inject.internal.ProxyFactory. Method [public org.candlepin.model.Persisted org.candlepin.model.RulesCurator.create(org.candlepin.model.Persisted)] is
synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@496c7d29]. This could indicate a bug. The method may be intercepted twice, or may not be intercepted at all.
08-Jun-2021 19:07:43.484 WARNING [main] com.google.inject.internal.ProxyFactory. Method [public void org.candlepin.model.ContentCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being
intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@496c7d29]. This could indicate a bug. The method may be intercepted twice, or may not be intercepted at all.
08-Jun-2021 19:07:43.582 WARNING [main] com.google.inject.internal.ProxyFactory. Method [public void org.candlepin.model.EntitlementCertificateCurator.delete(org.candlepin.model.Persisted)] is syntheti
c and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@496c7d29]. This could indicate a bug. The method may be intercepted twice, or may not be intercepted at all.
08-Jun-2021 19:07:48.470 SEVERE [main] org.apache.catalina.core.StandardContext.startInternal One or more listeners failed to start. Full details will be found in the appropriate container log file
08-Jun-2021 19:07:48.472 SEVERE [main] org.apache.catalina.core.StandardContext.startInternal Context [/candlepin] startup failed due to previous errors
…
root satellite11 /var/log/tomcat #
root satellite11 /var/log/tomcat #less localhost.2021-06-08.log
08-Jun-2021 19:07:48.470 SEVERE [main] org.apache.catalina.core.StandardContext.listenerStart Exception sending context initialized event to listener instance of class [org.candlepin.guice.CandlepinContextListener]
java.lang.NoSuchFieldError: SERVER_SENT_EVENTS_TYPE
at org.jboss.resteasy.core.ResourceMethodInvoker.isSseResourceMethod(ResourceMethodInvoker.java:192)
at org.jboss.resteasy.core.ResourceMethodInvoker.(ResourceMethodInvoker.java:171)
at org.jboss.resteasy.core.ResourceMethodRegistry.processMethod(ResourceMethodRegistry.java:383)
at org.jboss.resteasy.core.ResourceMethodRegistry.register(ResourceMethodRegistry.java:310)
at org.jboss.resteasy.core.ResourceMethodRegistry.addResourceFactory(ResourceMethodRegistry.java:261)
at org.jboss.resteasy.core.ResourceMethodRegistry.addResourceFactory(ResourceMethodRegistry.java:228)
at org.jboss.resteasy.core.ResourceMethodRegistry.addResourceFactory(ResourceMethodRegistry.java:209)
at org.jboss.resteasy.core.ResourceMethodRegistry.addResourceFactory(ResourceMethodRegistry.java:193)
at org.jboss.resteasy.core.ResourceMethodRegistry.addResourceFactory(ResourceMethodRegistry.java:176)
Let me know if you are intetrsted on the full logs.
I’m trying to install Katello+Foreman 4.0 as per the instructions on CentOS 8.4
Thank you
Hello all.
This problem still appeared in CentOS 8.4.
At first I didn’t find this topic and create task in foreman redmine Bug #32865: CentOS 8 Katello::Errors::CandlepinNotRunning - Foreman
After reading this topic, I tried to update my host to CentOS-Stream (added repos and ran dnf update). But problem still exists
So, to somehow narrow down the search scope of the search, as I understand from the forum and my searches, the problem is in one of these packages:
katello
tomcatjss
java-11-openjdk
java-11-openjdk-headless
pki-servlet-4.0-api
pki-servlet-engine
I have success in instalation foreman in CentOS8 (VM). So, I had a logs with successed and failed installation. Comparing installation log not give anything, logs are same. And I start understand, that this error do not depend on package versions or installation process.