Hi all,
I've built and rebuilt Centos 6.5, Centos 7, and Redhat 6 systems using a
base install kickstart that I've been using successfully for quite some
time. I then configure all the repositories and run yum -y install
katello. All the packages install just fine.
Next, I run the katello-installer script. However, this runs for hours and
always fails with timeouts, usually on foreman-rake db:seed and
apipie:cache. The system is configured as follows:
VMware virtual machine.
1 vCPU with 4 cores. (Also tried with 2 vCPUs but didn't make a difference)
12 GB Memory
4 GB Swap
40 GB OS PV.
Large DB and Pulp PV.
After the install fails, it's only a matter of time before the VM's
resources are all used up and I can't even type without a 10 second or
longer delay.
Troubleshooting/random-button-pushing I've done:
1.) check directory permissions
2.) Check /etc/hosts for proper lookup of localhost and hostname -s
and
hostname -f
3.) Check free space…not an issue.
4.) rerun installer
5.) Add installer options like --foreman-admin-password
6.) use katello-installer --reset
7.) remove katello and try again.
8.) Rebuild system to various OS's as described above.
9.) Ensure ntp is working and only using local authentication (no ldap, etc)
10.) Log file review for something useful. It just times out!
Here's the two relevant snippets of the latest katello-installer.log:
[ INFO 2014-10-14 15:56:44 main] RESOURCE Service[httpd]
[DEBUG 2014-10-14 15:56:54 main]
/Stage[main]/Apache::Service/Service[httpd]: The container
Class[Apache::Service] will propagate my refresh event
[ INFO 2014-10-14 15:56:56 main]
/Stage[main]/Apache::Service/Service[httpd]: Unscheduling refresh on
Service[httpd]
[DEBUG 2014-10-14 15:56:56 main] Class[Apache::Service]: The container
Stage[main] will propagate my refresh event
[ INFO 2014-10-14 15:56:58 main] RESOURCE Exec[foreman-rake-db:seed]
[DEBUG 2014-10-14 15:56:58 main]
Execforeman-rake-db:seed: Executing
'/usr/sbin/foreman-rake db:seed'
[DEBUG 2014-10-14 15:56:58 main] Executing '/usr/sbin/foreman-rake db:seed'
[ERROR 2014-10-14 16:01:52 main]
/Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]:
Failed to call refresh: Command exceeded timeout
[ERROR 2014-10-14 16:01:59 main]
/Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]:
Command exceeded timeout
[ INFO 2014-10-14 16:02:00 main] /usr/lib/ruby/1.8/timeout.rb:64:in
execute' [ INFO 2014-10-14 16:02:01 main] /usr/lib/ruby/site_ruby/1.8/puppet/provider/exec.rb:63:in
run'
[ INFO 2014-10-14 16:02:02 main]
/usr/lib/ruby/site_ruby/1.8/puppet/provider/exec.rb:59:in run' [ INFO 2014-10-14 16:02:03 main] /usr/lib/ruby/site_ruby/1.8/puppet/provider/exec.rb:29:in
chdir'
[ INFO 2014-10-14 16:02:05 main]
/usr/lib/ruby/site_ruby/1.8/puppet/provider/exec.rb:29:in run' [ INFO 2014-10-14 16:02:05 main] /usr/lib/ruby/site_ruby/1.8/puppet/provider/exec/posix.rb:45:in
run'
[ INFO 2014-10-14 16:02:06 main]
/usr/lib/ruby/site_ruby/1.8/puppet/type/exec.rb:130:in `sync'
INFO 2014-10-14 16:03:14 main]
/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:384:in run' [ INFO 2014-10-14 16:03:14 main] /usr/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:146:in
run'
[ INFO 2014-10-14 16:03:14 main]
/usr/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:92:in execute' [ INFO 2014-10-14 16:03:15 main] /usr/bin/puppet:8 [ INFO 2014-10-14 16:03:15 main] RESOURCE Foreman::Rake[db:seed] [ INFO 2014-10-14 16:03:15 main] RESOURCE Foreman::Rake[apipie:cache] [ INFO 2014-10-14 16:03:15 main] Foreman::Rake[apipie:cache]: Scheduling refresh of Exec[foreman-rake-apipie:cache] [ INFO 2014-10-14 16:03:15 main] RESOURCE Exec[foreman-rake-apipie:cache] [DEBUG 2014-10-14 16:03:15 main] Exec[foreman-rake-apipie:cache](provider=posix): Executing '/usr/sbin/foreman-rake apipie:cache' [DEBUG 2014-10-14 16:03:15 main] Executing '/usr/sbin/foreman-rake apipie:cache' [ERROR 2014-10-14 16:07:13 main] /Stage[main]/Foreman::Database/Foreman::Rake[apipie:cache]/Exec[foreman-rake-apipie:cache]: Failed to call refresh: Command exceeded timeout [ERROR 2014-10-14 16:12:30 main] /Stage[main]/Foreman::Database/Foreman::Rake[apipie:cache]/Exec[foreman-rake-apipie:cache]: Command exceeded timeout [ INFO 2014-10-14 16:12:42 main] /usr/lib/ruby/1.8/timeout.rb:64:in
execute'
[ INFO 2014-10-14 16:12:44 main]
/usr/lib/ruby/site_ruby/1.8/puppet/provider/exec.rb:63:in run' [ INFO 2014-10-14 16:12:46 main] /usr/lib/ruby/site_ruby/1.8/puppet/provider/exec.rb:59:in
run'
[ INFO 2014-10-14 16:12:48 main]
/usr/lib/ruby/site_ruby/1.8/puppet/provider/exec.rb:29:in chdir' [ INFO 2014-10-14 16:12:48 main] /usr/lib/ruby/site_ruby/1.8/puppet/provider/exec.rb:29:in
run'
[ INFO 2014-10-14 16:12:49 main]
/usr/lib/ruby/site_ruby/1.8/puppet/provider/exec/posix.rb:45:in `run'
Has anyone else experienced this? Any ideas on why this becomes such a
resource hog and, most importantly, any ideas on how to fix this?
Thanks,