Katello 3.0RC install, foreman-rake db:migrate error

After correcting the problem with yum update to get Katello 2.4 up to date
(see https://groups.google.com/forum/#!topic/foreman-users/AifrfU6g9J0)
I've attempted to upgrade to 3.0RC. After correcting the URL locations in
the doc (how do I contribute to get that updated?) as below

yum update -y
https://fedorapeople.org/groups/katello/releases/yum/3.0/katello/el7/x86_64/katello-repos-latest.rpm
yum update -y
http://yum.theforeman.org/releases/1.11/el7/x86_64/foreman-release.rpm

I ran the yum clean all and yum -y update commands, both of which completed
successfully.Next I ran "foreman-installer --scenario katello --upgrade"
and it failed at the migrate_foreman step.

[root@wellkatellodev log]# foreman-installer --scenario katello --upgrade
Upgrading…
Upgrade Step: stop_services…
Upgrade Step: start_databases…
Upgrade Step: update_http_conf…
Upgrade Step: migrate_pulp…
Upgrade Step: start_httpd…
Upgrade Step: migrate_candlepin…
Upgrade Step: migrate_foreman…
Upgrade step migrate_foreman failed. Check logs for more information.
[root@wellkatellodev log]#

I have attached the katello.log. Possibly incorrectly, I then started
services and tried the db:migrate again,

Running the command manually with --trace gives the following

[root@wellkatellodev log]# foreman-rake db:migrate --trace
/opt/theforeman/tfm/root/usr/share/gems/gems/docker-api-1.17.0/lib/docker.rb:103:
warning: circular argument reference - connection
** Invoke db:migrate (first_time)
** Invoke environment (first_time)
** Execute environment
** Invoke db:load_config (first_time)
** Execute db:load_config
** Execute db:migrate
== 20150930183738 MigrateContentHosts: migrating

··· ============================== rake aborted! StandardError: An error has occurred, this and all later migrations canceled:

Could not find the inverse association for repository
(:content_facet_repositories in
MigrateContentHosts::Repository)/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/reflection.rb:294:in
check_validity_of_inverse!' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/reflection.rb:288:incheck_validity!’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/association.rb:25:in
initialize' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations.rb:155:innew’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations.rb:155:in
association' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/builder/association.rb:118:inrepository=’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/has_many_through_association.rb:87:in
build_through_record' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/has_many_through_association.rb:103:insave_through_record’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/has_many_through_association.rb:64:in
insert_record' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:522:inblock (2 levels) in concat_records’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:389:in
add_to_target' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:521:inblock in concat_records’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:519:in
each' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:519:inconcat_records’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/has_many_through_association.rb:42:in
concat_records' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:153:inblock in concat’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:168:in
block in transaction' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:199:intransaction’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/transactions.rb:208:in
transaction' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:167:intransaction’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:153:in
concat' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/has_many_through_association.rb:36:inconcat’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:507:in
replace_records' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:362:inblock in replace’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:168:in
block in transaction' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:199:intransaction’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/transactions.rb:208:in
transaction' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:167:intransaction’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:362:in
replace' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:41:inwriter’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/builder/association.rb:118:in
bound_repositories=' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0/db/migrate/20150930183738_migrate_content_hosts.rb:178:increate_content_facet’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0/db/migrate/20150930183738_migrate_content_hosts.rb:315:in
block in up' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/delegation.rb:46:ineach’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/delegation.rb:46:in
each' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0/db/migrate/20150930183738_migrate_content_hosts.rb:295:inup’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:598:in
exec_migration' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:579:inblock (2 levels) in migrate’
/opt/rh/rh-ruby22/root/usr/share/ruby/benchmark.rb:288:in measure' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:578:inblock in migrate’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/connection_pool.rb:294:in
with_connection' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:577:inmigrate’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:752:in
migrate' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:992:inblock in execute_migration_in_transaction’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:1038:in
block in ddl_transaction' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:201:inblock in transaction’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:209:in
within_new_transaction' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:201:intransaction’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/transactions.rb:208:in
transaction' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:1038:inddl_transaction’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:991:in
execute_migration_in_transaction' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:953:inblock in migrate’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:949:in
each' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:949:inmigrate’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:807:in
up' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:785:inmigrate’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/railties/databases.rake:34:in
block (2 levels) in <top (required)>' /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:240:incall’
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:240:in
block in execute' /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:235:ineach’
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:235:in
execute' /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:179:inblock in invoke_with_call_chain’
/opt/rh/rh-ruby22/root/usr/share/ruby/monitor.rb:211:in mon_synchronize' /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:172:ininvoke_with_call_chain’
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:165:in
invoke' /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:150:ininvoke_task’
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:106:in
block (2 levels) in top_level' /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:106:ineach’
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:106:in
block in top_level' /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:115:inrun_with_threads’
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:100:in
top_level' /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:78:inblock in run’
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:176:in
standard_exception_handling' /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:75:inrun’
/opt/rh/rh-ruby22/root/usr/bin/rake:33:in <main>' ActiveRecord::InverseOfAssociationNotFoundError: Could not find the inverse association for repository (:content_facet_repositories in MigrateContentHosts::Repository) /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/reflection.rb:294:incheck_validity_of_inverse!’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/reflection.rb:288:in
check_validity!' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/association.rb:25:ininitialize’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations.rb:155:in
new' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations.rb:155:inassociation’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/builder/association.rb:118:in
repository=' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/has_many_through_association.rb:87:inbuild_through_record’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/has_many_through_association.rb:103:in
save_through_record' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/has_many_through_association.rb:64:ininsert_record’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:522:in
block (2 levels) in concat_records' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:389:inadd_to_target’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:521:in
block in concat_records' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:519:ineach’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:519:in
concat_records' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/has_many_through_association.rb:42:inconcat_records’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:153:in
block in concat' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:168:inblock in transaction’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:199:in
transaction' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/transactions.rb:208:intransaction’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:167:in
transaction' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:153:inconcat’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/has_many_through_association.rb:36:in
concat' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:507:inreplace_records’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:362:in
block in replace' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:168:inblock in transaction’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:199:in
transaction' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/transactions.rb:208:intransaction’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:167:in
transaction' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:362:inreplace’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:41:in
writer' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/builder/association.rb:118:inbound_repositories=’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0/db/migrate/20150930183738_migrate_content_hosts.rb:178:in
create_content_facet' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0/db/migrate/20150930183738_migrate_content_hosts.rb:315:inblock in up’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/delegation.rb:46:in
each' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/delegation.rb:46:ineach’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0/db/migrate/20150930183738_migrate_content_hosts.rb:295:in
up' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:598:inexec_migration’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:579:in
block (2 levels) in migrate' /opt/rh/rh-ruby22/root/usr/share/ruby/benchmark.rb:288:inmeasure’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:578:in
block in migrate' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/connection_pool.rb:294:inwith_connection’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:577:in
migrate' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:752:inmigrate’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:992:in
block in execute_migration_in_transaction' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:1038:inblock in ddl_transaction’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:201:in
block in transaction' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:209:inwithin_new_transaction’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:201:in
transaction' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/transactions.rb:208:intransaction’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:1038:in
ddl_transaction' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:991:inexecute_migration_in_transaction’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:953:in
block in migrate' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:949:ineach’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:949:in
migrate' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:807:inup’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:785:in
migrate' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/railties/databases.rake:34:inblock (2 levels) in <top (required)>’
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:240:in
call' /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:240:inblock in execute’
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:235:in
each' /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:235:inexecute’
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:179:in
block in invoke_with_call_chain' /opt/rh/rh-ruby22/root/usr/share/ruby/monitor.rb:211:inmon_synchronize’
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:172:in
invoke_with_call_chain' /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:165:ininvoke’
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:150:in
`invoke_task’
/opt/rh/rh-ruby22/root

John Mitsch
Red Hat Engineering
(860)-967-7285
irc: jomitsch

> After correcting the problem with yum update to get Katello 2.4 up to date
> (see https://groups.google.com/forum/#!topic/foreman-users/AifrfU6g9J0)
> I've attempted to upgrade to 3.0RC. After correcting the URL locations in
> the doc (how do I contribute to get that updated?) as below
>
> We are working on fixing the url problem in docs currently, its a bit
complicated because they are generated dynamically and alot of changes have
to be made on our deploy script. Hopefully I can get that fixed soon :slight_smile:

>
> I've also tried using the nightly builds and get the same thing :frowning:
> Any suggestions welcome - should I file an issue?
>

Yes, please file an issue
Foreman I don't think
one exists with this exact problem. I'm spinning up a 2.4 box that I will
upgrade and see if I run into the same issue. Hopefully we can figure out a
patch.

Thanks for letting us know about this!

-John

··· On Wed, Apr 6, 2016 at 8:59 PM, Dylan Baars wrote:

bug was filed by another user who hit the same problem.
http://projects.theforeman.org/issues/14533

John Mitsch
Red Hat Engineering
(860)-967-7285
irc: jomitsch

··· On Wed, Apr 6, 2016 at 8:59 PM, Dylan Baars wrote:

After correcting the problem with yum update to get Katello 2.4 up to date
(see https://groups.google.com/forum/#!topic/foreman-users/AifrfU6g9J0)
I’ve attempted to upgrade to 3.0RC. After correcting the URL locations in
the doc (how do I contribute to get that updated?) as below

yum update -y
https://fedorapeople.org/groups/katello/releases/yum/3.0/katello/el7/x86_64/katello-repos-latest.rpm
yum update -y
http://yum.theforeman.org/releases/1.11/el7/x86_64/foreman-release.rpm

I ran the yum clean all and yum -y update commands, both of which
completed successfully.Next I ran “foreman-installer --scenario katello
–upgrade” and it failed at the migrate_foreman step.

[root@wellkatellodev log]# foreman-installer --scenario katello --upgrade
Upgrading…
Upgrade Step: stop_services…
Upgrade Step: start_databases…
Upgrade Step: update_http_conf…
Upgrade Step: migrate_pulp…
Upgrade Step: start_httpd…
Upgrade Step: migrate_candlepin…
Upgrade Step: migrate_foreman…
Upgrade step migrate_foreman failed. Check logs for more information.
[root@wellkatellodev log]#

I have attached the katello.log. Possibly incorrectly, I then started
services and tried the db:migrate again,

Running the command manually with --trace gives the following

[root@wellkatellodev log]# foreman-rake db:migrate --trace
/opt/theforeman/tfm/root/usr/share/gems/gems/docker-api-1.17.0/lib/docker.rb:103:
warning: circular argument reference - connection
** Invoke db:migrate (first_time)
** Invoke environment (first_time)
** Execute environment
** Invoke db:load_config (first_time)
** Execute db:load_config
** Execute db:migrate
== 20150930183738 MigrateContentHosts: migrating

rake aborted!
StandardError: An error has occurred, this and all later migrations
canceled:

Could not find the inverse association for repository
(:content_facet_repositories in
MigrateContentHosts::Repository)/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/reflection.rb:294:in
check_validity_of_inverse!' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/reflection.rb:288:incheck_validity!’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/association.rb:25:in
initialize' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations.rb:155:innew’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations.rb:155:in
association' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/builder/association.rb:118:inrepository=’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/has_many_through_association.rb:87:in
build_through_record' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/has_many_through_association.rb:103:insave_through_record’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/has_many_through_association.rb:64:in
insert_record' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:522:inblock (2 levels) in concat_records’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:389:in
add_to_target' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:521:inblock in concat_records’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:519:in
each' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:519:inconcat_records’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/has_many_through_association.rb:42:in
concat_records' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:153:inblock in concat’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:168:in
block in transaction' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:199:intransaction’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/transactions.rb:208:in
transaction' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:167:intransaction’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:153:in
concat' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/has_many_through_association.rb:36:inconcat’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:507:in
replace_records' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:362:inblock in replace’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:168:in
block in transaction' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:199:intransaction’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/transactions.rb:208:in
transaction' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:167:intransaction’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:362:in
replace' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:41:inwriter’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/builder/association.rb:118:in
bound_repositories=' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0/db/migrate/20150930183738_migrate_content_hosts.rb:178:increate_content_facet’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0/db/migrate/20150930183738_migrate_content_hosts.rb:315:in
block in up' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/delegation.rb:46:ineach’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/delegation.rb:46:in
each' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0/db/migrate/20150930183738_migrate_content_hosts.rb:295:inup’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:598:in
exec_migration' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:579:inblock (2 levels) in migrate’
/opt/rh/rh-ruby22/root/usr/share/ruby/benchmark.rb:288:in measure' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:578:inblock in migrate’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/connection_pool.rb:294:in
with_connection' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:577:inmigrate’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:752:in
migrate' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:992:inblock in execute_migration_in_transaction’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:1038:in
block in ddl_transaction' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:201:inblock in transaction’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:209:in
within_new_transaction' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:201:intransaction’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/transactions.rb:208:in
transaction' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:1038:inddl_transaction’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:991:in
execute_migration_in_transaction' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:953:inblock in migrate’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:949:in
each' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:949:inmigrate’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:807:in
up' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:785:inmigrate’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/railties/databases.rake:34:in
block (2 levels) in <top (required)>' /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:240:incall’
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:240:in
block in execute' /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:235:ineach’
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:235:in
execute' /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:179:inblock in invoke_with_call_chain’
/opt/rh/rh-ruby22/root/usr/share/ruby/monitor.rb:211:in mon_synchronize' /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:172:ininvoke_with_call_chain’
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:165:in
invoke' /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:150:ininvoke_task’
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:106:in
block (2 levels) in top_level' /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:106:ineach’
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:106:in
block in top_level' /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:115:inrun_with_threads’
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:100:in
top_level' /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:78:inblock in run’
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:176:in
standard_exception_handling' /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:75:inrun’
/opt/rh/rh-ruby22/root/usr/bin/rake:33:in <main>' ActiveRecord::InverseOfAssociationNotFoundError: Could not find the inverse association for repository (:content_facet_repositories in MigrateContentHosts::Repository) /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/reflection.rb:294:incheck_validity_of_inverse!’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/reflection.rb:288:in
check_validity!' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/association.rb:25:ininitialize’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations.rb:155:in
new' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations.rb:155:inassociation’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/builder/association.rb:118:in
repository=' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/has_many_through_association.rb:87:inbuild_through_record’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/has_many_through_association.rb:103:in
save_through_record' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/has_many_through_association.rb:64:ininsert_record’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:522:in
block (2 levels) in concat_records' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:389:inadd_to_target’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:521:in
block in concat_records' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:519:ineach’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:519:in
concat_records' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/has_many_through_association.rb:42:inconcat_records’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:153:in
block in concat' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:168:inblock in transaction’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:199:in
transaction' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/transactions.rb:208:intransaction’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:167:in
transaction' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:153:inconcat’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/has_many_through_association.rb:36:in
concat' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:507:inreplace_records’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:362:in
block in replace' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:168:inblock in transaction’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:199:in
transaction' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/transactions.rb:208:intransaction’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:167:in
transaction' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:362:inreplace’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:41:in
writer' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/builder/association.rb:118:inbound_repositories=’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0/db/migrate/20150930183738_migrate_content_hosts.rb:178:in
create_content_facet' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0/db/migrate/20150930183738_migrate_content_hosts.rb:315:inblock in up’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/delegation.rb:46:in
each' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/delegation.rb:46:ineach’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0/db/migrate/20150930183738_migrate_content_hosts.rb:295:in
up' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:598:inexec_migration’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:579:in
block (2 levels) in migrate' /opt/rh/rh-ruby22/root/usr/share/ruby/benchmark.rb:288:inmeasure’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:578:in
block in migrate' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/connection_pool.rb:294:inwith_connection’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:577:in
migrate' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:752:inmigrate’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:992:in
`block in execute_migration_in_transaction’
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active

Dylan,

unfortunately, I am not able to recreate this, and I don't see any recent
changes in the migration file that may be causing an issue. If you file an
issue we can look into it further, maybe other devs will have an idea of
what could be happening.

I thought you were talking about a different URL error in our docs
(unfortunately there are multiple right now :confused: ) I submitted a pull request
to fix the URL error you mentioned so it doesn't confuse others.

Thanks,

John Mitsch
Red Hat Engineering
(860)-967-7285
irc: jomitsch

··· On Thu, Apr 7, 2016 at 10:14 AM, John Mitsch wrote:

John Mitsch
Red Hat Engineering
(860)-967-7285
irc: jomitsch

On Wed, Apr 6, 2016 at 8:59 PM, Dylan Baars baarsd@gmail.com wrote:

After correcting the problem with yum update to get Katello 2.4 up to
date (see
Redirecting to Google Groups) I’ve
attempted to upgrade to 3.0RC. After correcting the URL locations in the
doc (how do I contribute to get that updated?) as below

We are working on fixing the url problem in docs currently, its a bit
complicated because they are generated dynamically and alot of changes have
to be made on our deploy script. Hopefully I can get that fixed soon :slight_smile:

I’ve also tried using the nightly builds and get the same thing :frowning:
Any suggestions welcome - should I file an issue?

Yes, please file an issue
Foreman I don’t think
one exists with this exact problem. I’m spinning up a 2.4 box that I will
upgrade and see if I run into the same issue. Hopefully we can figure out a
patch.

Thanks for letting us know about this!

-John

Ahh cool at least its not not me :slight_smile: I've watched the issue so hopefully
fixed soon!

Dylan

··· On Friday, April 8, 2016 at 7:28:42 AM UTC+12, John Mitsch wrote: > > bug was filed by another user who hit the same problem. > http://projects.theforeman.org/issues/14533 > > John Mitsch > Red Hat Engineering > (860)-967-7285 > irc: jomitsch > > On Wed, Apr 6, 2016 at 8:59 PM, Dylan Baars > wrote: > >> After correcting the problem with yum update to get Katello 2.4 up to >> date (see >> https://groups.google.com/forum/#!topic/foreman-users/AifrfU6g9J0) I've >> attempted to upgrade to 3.0RC. After correcting the URL locations in the >> doc (how do I contribute to get that updated?) as below >> >> yum update -y >> https://fedorapeople.org/groups/katello/releases/yum/3.0/katello/el7/x86_64/katello-repos-latest.rpm >> yum update -y >> http://yum.theforeman.org/releases/1.11/el7/x86_64/foreman-release.rpm >> >> I ran the yum clean all and yum -y update commands, both of which >> completed successfully.Next I ran "foreman-installer --scenario katello >> --upgrade" and it failed at the migrate_foreman step. >> >> [root@wellkatellodev log]# foreman-installer --scenario katello --upgrade >> Upgrading... >> Upgrade Step: stop_services... >> Upgrade Step: start_databases... >> Upgrade Step: update_http_conf... >> Upgrade Step: migrate_pulp... >> Upgrade Step: start_httpd... >> Upgrade Step: migrate_candlepin... >> Upgrade Step: migrate_foreman... >> Upgrade step migrate_foreman failed. Check logs for more information. >> [root@wellkatellodev log]# >> >> I have attached the katello.log. Possibly incorrectly, I then started >> services and tried the db:migrate again, >> >> Running the command manually with --trace gives the following >> >> [root@wellkatellodev log]# foreman-rake db:migrate --trace >> /opt/theforeman/tfm/root/usr/share/gems/gems/docker-api-1.17.0/lib/docker.rb:103: >> warning: circular argument reference - connection >> ** Invoke db:migrate (first_time) >> ** Invoke environment (first_time) >> ** Execute environment >> ** Invoke db:load_config (first_time) >> ** Execute db:load_config >> ** Execute db:migrate >> == 20150930183738 MigrateContentHosts: migrating >> ============================== >> rake aborted! >> StandardError: An error has occurred, this and all later migrations >> canceled: >> >> Could not find the inverse association for repository >> (:content_facet_repositories in >> MigrateContentHosts::Repository)/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/reflection.rb:294:in >> `check_validity_of_inverse!' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/reflection.rb:288:in >> `check_validity!' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/association.rb:25:in >> `initialize' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations.rb:155:in >> `new' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations.rb:155:in >> `association' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/builder/association.rb:118:in >> `repository=' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/has_many_through_association.rb:87:in >> `build_through_record' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/has_many_through_association.rb:103:in >> `save_through_record' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/has_many_through_association.rb:64:in >> `insert_record' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:522:in >> `block (2 levels) in concat_records' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:389:in >> `add_to_target' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:521:in >> `block in concat_records' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:519:in >> `each' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:519:in >> `concat_records' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/has_many_through_association.rb:42:in >> `concat_records' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:153:in >> `block in concat' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:168:in >> `block in transaction' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:199:in >> `transaction' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/transactions.rb:208:in >> `transaction' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:167:in >> `transaction' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:153:in >> `concat' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/has_many_through_association.rb:36:in >> `concat' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:507:in >> `replace_records' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:362:in >> `block in replace' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:168:in >> `block in transaction' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:199:in >> `transaction' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/transactions.rb:208:in >> `transaction' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:167:in >> `transaction' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:362:in >> `replace' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:41:in >> `writer' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/builder/association.rb:118:in >> `bound_repositories=' >> /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0/db/migrate/20150930183738_migrate_content_hosts.rb:178:in >> `create_content_facet' >> /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0/db/migrate/20150930183738_migrate_content_hosts.rb:315:in >> `block in up' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/delegation.rb:46:in >> `each' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/delegation.rb:46:in >> `each' >> /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0/db/migrate/20150930183738_migrate_content_hosts.rb:295:in >> `up' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:598:in >> `exec_migration' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:579:in >> `block (2 levels) in migrate' >> /opt/rh/rh-ruby22/root/usr/share/ruby/benchmark.rb:288:in `measure' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:578:in >> `block in migrate' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/connection_pool.rb:294:in >> `with_connection' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:577:in >> `migrate' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:752:in >> `migrate' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:992:in >> `block in execute_migration_in_transaction' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:1038:in >> `block in ddl_transaction' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:201:in >> `block in transaction' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:209:in >> `within_new_transaction' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:201:in >> `transaction' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/transactions.rb:208:in >> `transaction' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:1038:in >> `ddl_transaction' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:991:in >> `execute_migration_in_transaction' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:953:in >> `block in migrate' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:949:in >> `each' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:949:in >> `migrate' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:807:in >> `up' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:785:in >> `migrate' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/railties/databases.rake:34:in >> `block (2 levels) in ' >> /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:240:in >> `call' >> /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:240:in >> `block in execute' >> /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:235:in >> `each' >> /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:235:in >> `execute' >> /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:179:in >> `block in invoke_with_call_chain' >> /opt/rh/rh-ruby22/root/usr/share/ruby/monitor.rb:211:in `mon_synchronize' >> /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:172:in >> `invoke_with_call_chain' >> /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:165:in >> `invoke' >> /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:150:in >> `invoke_task' >> /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:106:in >> `block (2 levels) in top_level' >> /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:106:in >> `each' >> /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:106:in >> `block in top_level' >> /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:115:in >> `run_with_threads' >> /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:100:in >> `top_level' >> /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:78:in >> `block in run' >> /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:176:in >> `standard_exception_handling' >> /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:75:in >> `run' >> /opt/rh/rh-ruby22/root/usr/bin/rake:33:in `' >> ActiveRecord::InverseOfAssociationNotFoundError: Could not find the >> inverse association for repository (:content_facet_repositories in >> MigrateContentHosts::Repository) >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/reflection.rb:294:in >> `check_validity_of_inverse!' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/reflection.rb:288:in >> `check_validity!' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/association.rb:25:in >> `initialize' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations.rb:155:in >> `new' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations.rb:155:in >> `association' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/builder/association.rb:118:in >> `repository=' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/has_many_through_association.rb:87:in >> `build_through_record' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/has_many_through_association.rb:103:in >> `save_through_record' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/has_many_through_association.rb:64:in >> `insert_record' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:522:in >> `block (2 levels) in concat_records' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:389:in >> `add_to_target' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:521:in >> `block in concat_records' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:519:in >> `each' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:519:in >> `concat_records' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/has_many_through_association.rb:42:in >> `concat_records' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:153:in >> `block in concat' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:168:in >> `block in transaction' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:199:in >> `transaction' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/transactions.rb:208:in >> `transaction' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:167:in >> `transaction' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:153:in >> `concat' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/has_many_through_association.rb:36:in >> `concat' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:507:in >> `replace_records' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:362:in >> `block in replace' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:168:in >> `block in transaction' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:199:in >> `transaction' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/transactions.rb:208:in >> `transaction' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:167:in >> `transaction' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:362:in >> `replace' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/collection_association.rb:41:in >> `writer' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/builder/association.rb:118:in >> `bound_repositories=' >> /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0/db/migrate/20150930183738_migrate_content_hosts.rb:178:in >> `create_content_facet' >> /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0/db/migrate/20150930183738_migrate_content_hosts.rb:315:in >> `block in up' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/delegation.rb:46:in >> `each' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/delegation.rb:46:in >> `each' >> /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0/db/migrate/20150930183738_migrate_content_hosts.rb:295:in >> `up' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:598:in >> `exec_migration' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:579:in >> `block (2 levels) in migrate' >> /opt/rh/rh-ruby22/root/usr/share/ruby/benchmark.rb:288:in `measure' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:578:in >> `block in migrate' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/connection_pool.rb:294:in >> `with_connection' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:577:in >> `migrate' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:752:in >> `migrate' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:992:in >> `block in execute_migration_in_transaction' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:1038:in >> `block in ddl_transaction' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:201:in >> `block in transaction' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:209:in >> `within_new_transaction' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:201:in >> `transaction' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/transactions.rb:208:in >> `transaction' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:1038:in >> `ddl_transaction' >> /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:991:in >> `execute_migration_in_transaction' >>