Katello 3.4 -> 3.5 Upgrade Step: db_seed... taking a LOOONG time.

Performed the upgrade and ran the foreman-installer upgrade step. After a
few stops and starts I got it to fully complete as bellow.

[root@fm run]# foreman-installer --scenario katello --upgrade
Upgrading, to monitor the progress on all related services, please do:
  foreman-tail | tee upgrade-$(date +%Y-%m-%d-%H%M).log
Upgrade Step: stop_services...
/usr/sbin/service-wait goferd stop
Redirecting to /bin/systemctl stop goferd.service

/usr/sbin/service-wait foreman-tasks stop
Redirecting to /bin/systemctl stop foreman-tasks.service

/usr/sbin/service-wait httpd stop
Redirecting to /bin/systemctl stop httpd.service

/usr/sbin/service-wait pulp_celerybeat stop
Redirecting to /bin/systemctl stop pulp_celerybeat.service

/usr/sbin/service-wait foreman-proxy stop
Redirecting to /bin/systemctl stop foreman-proxy.service

/usr/sbin/service-wait pulp_streamer stop
Redirecting to /bin/systemctl stop pulp_streamer.service

/usr/sbin/service-wait pulp_resource_manager stop
Redirecting to /bin/systemctl stop pulp_resource_manager.service

/usr/sbin/service-wait pulp_workers stop
Redirecting to /bin/systemctl stop pulp_workers.service

/usr/sbin/service-wait tomcat stop
Redirecting to /bin/systemctl stop tomcat.service

/usr/sbin/service-wait squid stop
Redirecting to /bin/systemctl stop squid.service

/usr/sbin/service-wait qdrouterd stop
Redirecting to /bin/systemctl stop qdrouterd.service

/usr/sbin/service-wait qpidd stop
Redirecting to /bin/systemctl stop qpidd.service

Success!
katello-service stop --exclude mongod,postgresql finished successfully!
Upgrade Step: start_databases...
/usr/sbin/service-wait mongod start
Redirecting to /bin/systemctl start mongod.service

/usr/sbin/service-wait postgresql start
Redirecting to /bin/systemctl start postgresql.service

Success!
katello-service start --only mongod,postgresql finished successfully!
Upgrade Step: update_http_conf...
grep -F -q 'Include "/etc/httpd/conf.modules.d/*.conf"'
/etc/httpd/conf/httpd.conf || echo -e '<IfVersion >=
2.4>
    Include "/etc/httpd/conf.modules.d/*.conf"
</IfVersion>' >> /etc/httpd/conf/httpd.conf finished
successfully!
Upgrade Step: migrate_pulp...
grep -qe '7.[[:digit:]]' /etc/redhat-release finished successfully!
sed -i -e 's?/var/run/mongodb/mongodb.pid?/var/run/mongodb/mongod.pid?g'
/etc/mongod.conf finished successfully!
12180
pgrep mongod finished successfully!
Attempting to connect to localhost:27017
Attempting to connect to localhost:27017
Write concern for Mongo connection: {}
Loading content types.
Loading type descriptors []
Parsing type descriptors
Validating type descriptor syntactic integrity
Validating type descriptor semantic integrity
Loading unit model: docker_blob = pulp_docker.plugins.models:Blob
Loading unit model: docker_manifest = pulp_docker.plugins.models:Manifest
Loading unit model: docker_image = pulp_docker.plugins.models:Image
Loading unit model: docker_tag = pulp_docker.plugins.models:Tag
Loading unit model: puppet_module = pulp_puppet.plugins.db.models:Module
Loading unit model: erratum = pulp_rpm.plugins.db.models:Errata
Loading unit model: distribution = pulp_rpm.plugins.db.models:Distribution
Loading unit model: srpm = pulp_rpm.plugins.db.models:SRPM
Loading unit model: package_group = pulp_rpm.plugins.db.models:PackageGroup
Loading unit model: package_category =
pulp_rpm.plugins.db.models:PackageCategory
Loading unit model: iso = pulp_rpm.plugins.db.models:ISO
Loading unit model: package_environment =
pulp_rpm.plugins.db.models:PackageEnvironment
Loading unit model: drpm = pulp_rpm.plugins.db.models:DRPM
Loading unit model: package_langpacks =
pulp_rpm.plugins.db.models:PackageLangpacks
Loading unit model: rpm = pulp_rpm.plugins.db.models:RPM
Loading unit model: yum_repo_metadata_file =
pulp_rpm.plugins.db.models:YumMetadataFile
Updating the database with types []
Found the following type definitions that were not present in the update
collection [puppet_module, drpm, package_langpacks, erratum, docker_blob,
docker_manifest, yum_repo_metadata_file, package_group, package_category,
iso, package_environment, docker_tag, srpm, rpm, distribution, docker_image]
Updating the database with types [puppet_module, drpm, package_langpacks,
erratum, docker_blob, docker_manifest, yum_repo_metadata_file,
package_group, package_category, iso, package_environment, docker_tag,
distribution, rpm, srpm, docker_image]
Content types loaded.
Ensuring the admin role and user are in place.
Admin role and user are in place.
Beginning database migrations.
Migration package pulp.server.db.migrations is up to date at version 28
Migration package pulp_docker.plugins.migrations is up to date at version 3
Migration package pulp_puppet.plugins.migrations is up to date at version 5
Migration package pulp_rpm.plugins.migrations is up to date at version 39
Loading unit model: docker_blob = pulp_docker.plugins.models:Blob
Loading unit model: docker_manifest = pulp_docker.plugins.models:Manifest
Loading unit model: docker_image = pulp_docker.plugins.models:Image
Loading unit model: docker_tag = pulp_docker.plugins.models:Tag
Loading unit model: puppet_module = pulp_puppet.plugins.db.models:Module
Loading unit model: erratum = pulp_rpm.plugins.db.models:Errata
Loading unit model: distribution = pulp_rpm.plugins.db.models:Distribution
Loading unit model: srpm = pulp_rpm.plugins.db.models:SRPM
Loading unit model: package_group = pulp_rpm.plugins.db.models:PackageGroup
Loading unit model: package_category =
pulp_rpm.plugins.db.models:PackageCategory
Loading unit model: iso = pulp_rpm.plugins.db.models:ISO
Loading unit model: package_environment =
pulp_rpm.plugins.db.models:PackageEnvironment
Loading unit model: drpm = pulp_rpm.plugins.db.models:DRPM
Loading unit model: package_langpacks =
pulp_rpm.plugins.db.models:PackageLangpacks
Loading unit model: rpm = pulp_rpm.plugins.db.models:RPM
Loading unit model: yum_repo_metadata_file =
pulp_rpm.plugins.db.models:YumMetadataFile
Database migrations complete.
su - apache -s /bin/bash -c pulp-manage-db finished successfully!
Upgrade Step: start_httpd...
/usr/sbin/service-wait httpd start
Redirecting to /bin/systemctl start httpd.service

Success!
katello-service start --only httpd finished successfully!
Upgrade Step: start_qpidd...
/usr/sbin/service-wait qpidd start
Redirecting to /bin/systemctl start qpidd.service

/usr/sbin/service-wait qdrouterd start
Redirecting to /bin/systemctl start qdrouterd.service

Success!
katello-service start --only qpidd,qdrouterd finished successfully!
Upgrade Step: start_pulp...
/usr/sbin/service-wait pulp_workers start
Redirecting to /bin/systemctl start pulp_workers.service

/usr/sbin/service-wait pulp_resource_manager start
Redirecting to /bin/systemctl start pulp_resource_manager.service

/usr/sbin/service-wait pulp_celerybeat start
Redirecting to /bin/systemctl start pulp_celerybeat.service

Success!
katello-service start --only
pulp_workers,pulp_resource_manager,pulp_celerybeat finished successfully!
Upgrade Step: migrate_candlepin...
Configuring PostgreSQL with JDBC URL: jdbc:postgresql:candlepin
Migrating candlepin database
--driver=org.postgresql.Driver
--classpath=/usr/share/java/postgresql-jdbc.jar:/var/lib/tomcat/webapps/candlepin/WEB-INF/classes/
--changeLogFile=db/changelog/changelog-update.xml
--url=jdbc:postgresql:candlepin --username=candlepin
--password=o2pvbKdcfezCyfxCNyNAjedN5HRYXMnt
Liquibase Update Successful
/usr/share/candlepin/cpdb --update --password
o2pvbKdcfezCyfxCNyNAjedN5HRYXMnt finished successfully!
Upgrade Step: start_tomcat...
/usr/sbin/service-wait tomcat start
Redirecting to /bin/systemctl start tomcat.service

Success!
katello-service start --only tomcat,tomcat6 finished successfully!
Upgrade Step: migrate_foreman...
foreman-rake -- config -k use_pulp_oauth -v true >/dev/null finished
successfully!
foreman-rake db:migrate finished successfully!
foreman-rake -- config -k use_pulp_oauth -v false >/dev/null finished
successfully!
Upgrade Step: Running installer...
Installing Done
[100%] [........................................................]
  The full log is at /var/log/foreman-installer/katello.log
Upgrade Step: restart_services...
/usr/sbin/service-wait goferd stop
Redirecting to /bin/systemctl stop goferd.service

/usr/sbin/service-wait foreman-tasks stop
Redirecting to /bin/systemctl stop foreman-tasks.service

/usr/sbin/service-wait httpd stop
Redirecting to /bin/systemctl stop httpd.service

/usr/sbin/service-wait pulp_celerybeat stop
Redirecting to /bin/systemctl stop pulp_celerybeat.service

/usr/sbin/service-wait foreman-proxy stop
Redirecting to /bin/systemctl stop foreman-proxy.service

/usr/sbin/service-wait pulp_streamer stop
Redirecting to /bin/systemctl stop pulp_streamer.service

/usr/sbin/service-wait pulp_resource_manager stop
Redirecting to /bin/systemctl stop pulp_resource_manager.service

/usr/sbin/service-wait pulp_workers stop
Redirecting to /bin/systemctl stop pulp_workers.service

/usr/sbin/service-wait tomcat stop
Redirecting to /bin/systemctl stop tomcat.service

/usr/sbin/service-wait squid stop
Redirecting to /bin/systemctl stop squid.service

/usr/sbin/service-wait qdrouterd stop
Redirecting to /bin/systemctl stop qdrouterd.service

/usr/sbin/service-wait qpidd stop
Redirecting to /bin/systemctl stop qpidd.service

/usr/sbin/service-wait postgresql stop
Redirecting to /bin/systemctl stop postgresql.service

/usr/sbin/service-wait mongod stop
Redirecting to /bin/systemctl stop mongod.service

Success!
/usr/sbin/service-wait mongod start
Redirecting to /bin/systemctl start mongod.service

/usr/sbin/service-wait postgresql start
Redirecting to /bin/systemctl start postgresql.service

/usr/sbin/service-wait qpidd start
Redirecting to /bin/systemctl start qpidd.service

/usr/sbin/service-wait qdrouterd start
Redirecting to /bin/systemctl start qdrouterd.service

/usr/sbin/service-wait squid start
Redirecting to /bin/systemctl start squid.service

/usr/sbin/service-wait tomcat start
Redirecting to /bin/systemctl start tomcat.service

/usr/sbin/service-wait pulp_workers start
Redirecting to /bin/systemctl start pulp_workers.service

/usr/sbin/service-wait pulp_resource_manager start
Redirecting to /bin/systemctl start pulp_resource_manager.service

/usr/sbin/service-wait pulp_streamer start
Redirecting to /bin/systemctl start pulp_streamer.service

/usr/sbin/service-wait foreman-proxy start
Redirecting to /bin/systemctl start foreman-proxy.service

/usr/sbin/service-wait pulp_celerybeat start
Redirecting to /bin/systemctl start pulp_celerybeat.service

/usr/sbin/service-wait httpd start
Redirecting to /bin/systemctl start httpd.service

/usr/sbin/service-wait foreman-tasks start
Redirecting to /bin/systemctl start foreman-tasks.service

/usr/sbin/service-wait goferd start
Redirecting to /bin/systemctl start goferd.service

Success!
katello-service restart finished successfully!
Upgrade Step: db_seed...

Now the system is just sitting at that db_seed step. I can access the GUI
just fine and things look ok but I have not tried to much yet. Its been
about an hour though just sitting at that one step. Should I kill it and
start over again or what?

Thanks,
Louis

Ignore this. I finally killed the update then rebooted and started the
foreman-installer again. Ran about 10 minutes but it fully completed.

Louis

···

On Tuesday, December 19, 2017 at 1:13:11 PM UTC-5, Louis Bohm wrote:

Performed the upgrade and ran the foreman-installer upgrade step. After a
few stops and starts I got it to fully complete as bellow.

[root@fm run]# foreman-installer --scenario katello --upgrade
Upgrading, to monitor the progress on all related services, please do:
  foreman-tail | tee upgrade-$(date +%Y-%m-%d-%H%M).log
Upgrade Step: stop_services...
/usr/sbin/service-wait goferd stop
Redirecting to /bin/systemctl stop goferd.service

/usr/sbin/service-wait foreman-tasks stop
Redirecting to /bin/systemctl stop foreman-tasks.service

/usr/sbin/service-wait httpd stop
Redirecting to /bin/systemctl stop httpd.service

/usr/sbin/service-wait pulp_celerybeat stop
Redirecting to /bin/systemctl stop pulp_celerybeat.service

/usr/sbin/service-wait foreman-proxy stop
Redirecting to /bin/systemctl stop foreman-proxy.service

/usr/sbin/service-wait pulp_streamer stop
Redirecting to /bin/systemctl stop pulp_streamer.service

/usr/sbin/service-wait pulp_resource_manager stop
Redirecting to /bin/systemctl stop pulp_resource_manager.service

/usr/sbin/service-wait pulp_workers stop
Redirecting to /bin/systemctl stop pulp_workers.service

/usr/sbin/service-wait tomcat stop
Redirecting to /bin/systemctl stop tomcat.service

/usr/sbin/service-wait squid stop
Redirecting to /bin/systemctl stop squid.service

/usr/sbin/service-wait qdrouterd stop
Redirecting to /bin/systemctl stop qdrouterd.service

/usr/sbin/service-wait qpidd stop
Redirecting to /bin/systemctl stop qpidd.service

Success!
katello-service stop --exclude mongod,postgresql finished successfully!
Upgrade Step: start_databases...
/usr/sbin/service-wait mongod start
Redirecting to /bin/systemctl start mongod.service

/usr/sbin/service-wait postgresql start
Redirecting to /bin/systemctl start postgresql.service

Success!
katello-service start --only mongod,postgresql finished successfully!
Upgrade Step: update_http_conf...
grep -F -q 'Include "/etc/httpd/conf.modules.d/*.conf"'
/etc/httpd/conf/httpd.conf || echo -e '<IfVersion >=
2.4>
    Include "/etc/httpd/conf.modules.d/*.conf"
</IfVersion>' >> /etc/httpd/conf/httpd.conf
finished successfully!
Upgrade Step: migrate_pulp...
grep -qe '7.[[:digit:]]' /etc/redhat-release finished successfully!
sed -i -e 's?/var/run/mongodb/mongodb.pid?/var/run/mongodb/mongod.pid?g'
/etc/mongod.conf finished successfully!
12180
pgrep mongod finished successfully!
Attempting to connect to localhost:27017
Attempting to connect to localhost:27017
Write concern for Mongo connection: {}
Loading content types.
Loading type descriptors []
Parsing type descriptors
Validating type descriptor syntactic integrity
Validating type descriptor semantic integrity
Loading unit model: docker_blob = pulp_docker.plugins.models:Blob
Loading unit model: docker_manifest = pulp_docker.plugins.models:Manifest
Loading unit model: docker_image = pulp_docker.plugins.models:Image
Loading unit model: docker_tag = pulp_docker.plugins.models:Tag
Loading unit model: puppet_module = pulp_puppet.plugins.db.models:Module
Loading unit model: erratum = pulp_rpm.plugins.db.models:Errata
Loading unit model: distribution = pulp_rpm.plugins.db.models:Distribution
Loading unit model: srpm = pulp_rpm.plugins.db.models:SRPM
Loading unit model: package_group = pulp_rpm.plugins.db.models:PackageGroup
Loading unit model: package_category =
pulp_rpm.plugins.db.models:PackageCategory
Loading unit model: iso = pulp_rpm.plugins.db.models:ISO
Loading unit model: package_environment =
pulp_rpm.plugins.db.models:PackageEnvironment
Loading unit model: drpm = pulp_rpm.plugins.db.models:DRPM
Loading unit model: package_langpacks =
pulp_rpm.plugins.db.models:PackageLangpacks
Loading unit model: rpm = pulp_rpm.plugins.db.models:RPM
Loading unit model: yum_repo_metadata_file =
pulp_rpm.plugins.db.models:YumMetadataFile
Updating the database with types []
Found the following type definitions that were not present in the update
collection [puppet_module, drpm, package_langpacks, erratum, docker_blob,
docker_manifest, yum_repo_metadata_file, package_group, package_category,
iso, package_environment, docker_tag, srpm, rpm, distribution, docker_image]
Updating the database with types [puppet_module, drpm, package_langpacks,
erratum, docker_blob, docker_manifest, yum_repo_metadata_file,
package_group, package_category, iso, package_environment, docker_tag,
distribution, rpm, srpm, docker_image]
Content types loaded.
Ensuring the admin role and user are in place.
Admin role and user are in place.
Beginning database migrations.
Migration package pulp.server.db.migrations is up to date at version 28
Migration package pulp_docker.plugins.migrations is up to date at version 3
Migration package pulp_puppet.plugins.migrations is up to date at version 5
Migration package pulp_rpm.plugins.migrations is up to date at version 39
Loading unit model: docker_blob = pulp_docker.plugins.models:Blob
Loading unit model: docker_manifest = pulp_docker.plugins.models:Manifest
Loading unit model: docker_image = pulp_docker.plugins.models:Image
Loading unit model: docker_tag = pulp_docker.plugins.models:Tag
Loading unit model: puppet_module = pulp_puppet.plugins.db.models:Module
Loading unit model: erratum = pulp_rpm.plugins.db.models:Errata
Loading unit model: distribution = pulp_rpm.plugins.db.models:Distribution
Loading unit model: srpm = pulp_rpm.plugins.db.models:SRPM
Loading unit model: package_group = pulp_rpm.plugins.db.models:PackageGroup
Loading unit model: package_category =
pulp_rpm.plugins.db.models:PackageCategory
Loading unit model: iso = pulp_rpm.plugins.db.models:ISO
Loading unit model: package_environment =
pulp_rpm.plugins.db.models:PackageEnvironment
Loading unit model: drpm = pulp_rpm.plugins.db.models:DRPM
Loading unit model: package_langpacks =
pulp_rpm.plugins.db.models:PackageLangpacks
Loading unit model: rpm = pulp_rpm.plugins.db.models:RPM
Loading unit model: yum_repo_metadata_file =
pulp_rpm.plugins.db.models:YumMetadataFile
Database migrations complete.
su - apache -s /bin/bash -c pulp-manage-db finished successfully!
Upgrade Step: start_httpd...
/usr/sbin/service-wait httpd start
Redirecting to /bin/systemctl start httpd.service

Success!
katello-service start --only httpd finished successfully!
Upgrade Step: start_qpidd...
/usr/sbin/service-wait qpidd start
Redirecting to /bin/systemctl start qpidd.service

/usr/sbin/service-wait qdrouterd start
Redirecting to /bin/systemctl start qdrouterd.service

Success!
katello-service start --only qpidd,qdrouterd finished successfully!
Upgrade Step: start_pulp...
/usr/sbin/service-wait pulp_workers start
Redirecting to /bin/systemctl start pulp_workers.service

/usr/sbin/service-wait pulp_resource_manager start
Redirecting to /bin/systemctl start pulp_resource_manager.service

/usr/sbin/service-wait pulp_celerybeat start
Redirecting to /bin/systemctl start pulp_celerybeat.service

Success!
katello-service start --only
pulp_workers,pulp_resource_manager,pulp_celerybeat finished successfully!
Upgrade Step: migrate_candlepin...
Configuring PostgreSQL with JDBC URL: jdbc:postgresql:candlepin
Migrating candlepin database
--driver=org.postgresql.Driver
--classpath=/usr/share/java/postgresql-jdbc.jar:/var/lib/tomcat/webapps/candlepin/WEB-INF/classes/
--changeLogFile=db/changelog/changelog-update.xml
--url=jdbc:postgresql:candlepin --username=candlepin
--password=o2pvbKdcfezCyfxCNyNAjedN5HRYXMnt
Liquibase Update Successful
/usr/share/candlepin/cpdb --update --password
o2pvbKdcfezCyfxCNyNAjedN5HRYXMnt finished successfully!
Upgrade Step: start_tomcat...
/usr/sbin/service-wait tomcat start
Redirecting to /bin/systemctl start tomcat.service

Success!
katello-service start --only tomcat,tomcat6 finished successfully!
Upgrade Step: migrate_foreman...
foreman-rake -- config -k use_pulp_oauth -v true >/dev/null finished
successfully!
foreman-rake db:migrate finished successfully!
foreman-rake -- config -k use_pulp_oauth -v false >/dev/null finished
successfully!
Upgrade Step: Running installer...
Installing Done
[100%] [........................................................]
  The full log is at /var/log/foreman-installer/katello.log
Upgrade Step: restart_services...
/usr/sbin/service-wait goferd stop
Redirecting to /bin/systemctl stop goferd.service

/usr/sbin/service-wait foreman-tasks stop
Redirecting to /bin/systemctl stop foreman-tasks.service

/usr/sbin/service-wait httpd stop
Redirecting to /bin/systemctl stop httpd.service

/usr/sbin/service-wait pulp_celerybeat stop
Redirecting to /bin/systemctl stop pulp_celerybeat.service

/usr/sbin/service-wait foreman-proxy stop
Redirecting to /bin/systemctl stop foreman-proxy.service

/usr/sbin/service-wait pulp_streamer stop
Redirecting to /bin/systemctl stop pulp_streamer.service

/usr/sbin/service-wait pulp_resource_manager stop
Redirecting to /bin/systemctl stop pulp_resource_manager.service

/usr/sbin/service-wait pulp_workers stop
Redirecting to /bin/systemctl stop pulp_workers.service

/usr/sbin/service-wait tomcat stop
Redirecting to /bin/systemctl stop tomcat.service

/usr/sbin/service-wait squid stop
Redirecting to /bin/systemctl stop squid.service

/usr/sbin/service-wait qdrouterd stop
Redirecting to /bin/systemctl stop qdrouterd.service

/usr/sbin/service-wait qpidd stop
Redirecting to /bin/systemctl stop qpidd.service

/usr/sbin/service-wait postgresql stop
Redirecting to /bin/systemctl stop postgresql.service

/usr/sbin/service-wait mongod stop
Redirecting to /bin/systemctl stop mongod.service

Success!
/usr/sbin/service-wait mongod start
Redirecting to /bin/systemctl start mongod.service

/usr/sbin/service-wait postgresql start
Redirecting to /bin/systemctl start postgresql.service

/usr/sbin/service-wait qpidd start
Redirecting to /bin/systemctl start qpidd.service

/usr/sbin/service-wait qdrouterd start
Redirecting to /bin/systemctl start qdrouterd.service

/usr/sbin/service-wait squid start
Redirecting to /bin/systemctl start squid.service

/usr/sbin/service-wait tomcat start
Redirecting to /bin/systemctl start tomcat.service

/usr/sbin/service-wait pulp_workers start
Redirecting to /bin/systemctl start pulp_workers.service

/usr/sbin/service-wait pulp_resource_manager start
Redirecting to /bin/systemctl start pulp_resource_manager.service

/usr/sbin/service-wait pulp_streamer start
Redirecting to /bin/systemctl start pulp_streamer.service

/usr/sbin/service-wait foreman-proxy start
Redirecting to /bin/systemctl start foreman-proxy.service

/usr/sbin/service-wait pulp_celerybeat start
Redirecting to /bin/systemctl start pulp_celerybeat.service

/usr/sbin/service-wait httpd start
Redirecting to /bin/systemctl start httpd.service

/usr/sbin/service-wait foreman-tasks start
Redirecting to /bin/systemctl start foreman-tasks.service

/usr/sbin/service-wait goferd start
Redirecting to /bin/systemctl start goferd.service

Success!
katello-service restart finished successfully!
Upgrade Step: db_seed...

Now the system is just sitting at that db_seed step. I can access the GUI
just fine and things look ok but I have not tried to much yet. Its been
about an hour though just sitting at that one step. Should I kill it and
start over again or what?

Thanks,
Louis