foreman-nightly-rpm-pipeline 1485 failed

Foreman RPM nightly pipeline failed:

https://ci.theforeman.org/job/foreman-nightly-rpm-pipeline/1485/

foreman-pipeline-foreman-nightly-almalinux8-install (failed) (remote job)
foreman-pipeline-foreman-nightly-centos7-install (failed) (remote job)
foreman-pipeline-foreman-nightly-centos8-stream-install (failed) (remote job)
foreman-pipeline-foreman-nightly-centos7-upgrade (failed) (remote job)
foreman-pipeline-foreman-nightly-almalinux8-upgrade (failed) (remote job)
foreman-pipeline-foreman-nightly-centos8-stream-upgrade (failed) (remote job)

@nadjaheitmann @ezr-ondrej this seems to be on your plate:

2022-05-27 22:14:48 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: PG::UndefinedTable: ERROR:  relation "application_records" does not exist
2022-05-27 22:14:48 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: LINE 1: DELETE FROM "application_records" WHERE "application_records...
2022-05-27 22:14:48 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns:                     ^
2022-05-27 22:14:48 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/activerecord-6.1.6/lib/active_record/connection_adapters/postgresql_adapter.rb:672:in `async_exec_params'
…
2022-05-27 22:14:48 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/gems/gems/foreman_puppet-4.0.0/db/migrate_foreman/20161205142618_delete_orphaned_smart_class_parameters.foreman_puppet.rb:4:in `up'

Oh my… I did not really wanted to see this. There was a fix in foreman-tasks @aruzicka did this get released and packaged yet?

Tho this might be quite random as we still did not find a root cause.

Yes, it was: https://github.com/theforeman/foreman-packaging/pull/7896

But that was in on the 17th, but nightly only started failing with puppet 4.0.0

hmm the failing time is quite close to merge of PR#9050 might that be the root cause? This issue manifests if we load some models too early in the init process (when Setting was always first, then it was all cool) but now if the first loaded model is no longer Setting, it bombs if some (no idea what identifies some) is first, this happens … I’ll try to investigate why, but I guess the migration should not change just to resolve the issue, as it is not actually the root cause :frowning:

You don’t think this is related to the Rails upgrade then? I don’t think it is related to foreman_puppet, either. The main changes that were added in 4.0.0 are related to webpack and the host details page.

Nope, foreman-nightly-rpm-pipeline #1482 [Jenkins] was a green run with Rails 6.1.6 (once we fixed apipie-dsl and dynflow).

Yes, reverting #9050 makes the DB migrate successfully :exploding_head:

1 Like

in that case it is about load order of Rails models, the migration loads LookupValue first.

My guess (and be aware this is just working theory) is that classes named Something::Base make the Rails trip, as rails use Activerecord::Base inheritence in the algorithm to determine a table name, but without explicit namespace (so there is Base).
If there is another Base class in a scope of the class (like some Host class of ours, but also Nic, Facet), it may trip up the resolving of table_name, and if that trips the first time we’re resolving it for ApplicationRecord, then all models resolve their table name to application_records.

Possible mitigations:

  1. load ApplicationRecord explicitly early (this is bad for zetwerk, but at least it is only one specific class.
  2. switch to Zeitwerk (which should resolve the namespace confusion), but we are not ready to do so just yet (I mean core is mostly ready, but plugins are not)

Hmmm… So I edited config/application.rb and added the following the the big require block:

require File.expand_path('../app/models/concerns/host_mix', __dir__)
require File.expand_path('../app/models/concerns/has_many_common', __dir__)
require File.expand_path('../app/models/concerns/strip_whitespace', __dir__)
require File.expand_path('../app/models/concerns/parameterizable', __dir__)
require File.expand_path('../app/models/concerns/audit_associations', __dir__)
require File.expand_path('../app/models/application_record', __dir__)

and it works, but feels rather odd to me.

1 Like

Sounds like we are at one of two places to unblock nightly:

  1. Add the code suggested by @evgeni to fix the problem, and deal with Zeitwerk implications down the road
  2. Revert the change https://github.com/theforeman/foreman/pull/9050 and bring it back after we convert to Zeitwerk

Given both ways somehow, somewhere involve Zeitwerk, I’d prefer “going forward (with a workaround)” aka option 1 over “revert and re-visit” aka option 2.

PR at https://github.com/theforeman/foreman/pull/9244

enjoy :wink:

1 Like

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.