Remote Execution Foreman::PermissionMissingException on 1.16.0 -> 1.17.1 Upgrade

Support
1

Attempted to upgrade our lab Foreman environment today from 1.16.0 to 1.17.1

Below error stack presents anytime we start up a Foreman Node in the cluster. Remote Execution RPM version is: 1.4.5-2.fm1_17.el7

We are able to get everything “up” by removing this plugin, after which foreman starts without issue. However upon reinstallation (We rely on the Foreman_ansible plugin) it does not work. db_seed and db_migrate seem to kick off without any errors, but cannot get this error to clear, or foreman to start, with this plugin installed.

Any help/insight is appreciated!

ERF73-2603 [Foreman::PermissionMissingException]: some permissions were not found (Foreman::PermissionMissingException)
  /usr/share/foreman/app/models/role.rb:330:in `permission_records'
  /usr/share/foreman/app/models/role.rb:170:in `find_for_permission_removal'
  /usr/share/foreman/app/models/role.rb:211:in `remove_permissions!'
  /usr/share/foreman/app/services/foreman/plugin/role_lock.rb:47:in `block in update_plugin_role_permissions'
  /usr/share/foreman/app/models/role.rb:85:in `ignore_locking'
  /usr/share/foreman/app/services/foreman/plugin/role_lock.rb:43:in `update_plugin_role_permissions'
  /usr/share/foreman/app/services/foreman/plugin/role_lock.rb:22:in `process_role'
  /usr/share/foreman/app/services/foreman/plugin/role_lock.rb:13:in `block (2 levels) in register_role'
  /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/connection_adapters/abstract/database_statements.rb:235:in `block in transaction'
  /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/connection_adapters/abstract/transaction.rb:194:in `block in within_new_transaction'
  /opt/rh/rh-ruby24/root/usr/share/ruby/monitor.rb:214:in `mon_synchronize'
  /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/connection_adapters/abstract/transaction.rb:191:in `within_new_transaction'
  /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/connection_adapters/abstract/database_statements.rb:235:in `transaction'
  /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/transactions.rb:210:in `transaction'
  /usr/share/foreman/app/services/foreman/plugin/role_lock.rb:12:in `block in register_role'
  /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:93:in `as'
  /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:99:in `as_anonymous_admin'
  /usr/share/foreman/app/services/foreman/plugin/role_lock.rb:11:in `register_role'
  /usr/share/foreman/app/registries/foreman/plugin.rb:249:in `role'
  /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.4.5/lib/foreman_remote_execution/engine.rb:85:in `block (2 levels) in <class:Engine>'
  /usr/share/foreman/app/registries/foreman/plugin.rb:67:in `instance_eval'
  /usr/share/foreman/app/registries/foreman/plugin.rb:67:in `register'
  /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.4.5/lib/foreman_remote_execution/engine.rb:33:in `block in <class:Engine>'
  /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.4/lib/rails/initializable.rb:30:in `instance_exec'
  /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.4/lib/rails/initializable.rb:30:in `run'
  /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.4/lib/rails/initializable.rb:59:in `block in run_initializers'
  /opt/rh/rh-ruby24/root/usr/share/ruby/tsort.rb:228:in `block in tsort_each'
  /opt/rh/rh-ruby24/root/usr/share/ruby/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
  /opt/rh/rh-ruby24/root/usr/share/ruby/tsort.rb:431:in `each_strongly_connected_component_from'
  /opt/rh/rh-ruby24/root/usr/share/ruby/tsort.rb:349:in `block in each_strongly_connected_component'
  /opt/rh/rh-ruby24/root/usr/share/ruby/tsort.rb:347:in `each'
  /opt/rh/rh-ruby24/root/usr/share/ruby/tsort.rb:347:in `call'
  /opt/rh/rh-ruby24/root/usr/share/ruby/tsort.rb:347:in `each_strongly_connected_component'
  /opt/rh/rh-ruby24/root/usr/share/ruby/tsort.rb:226:in `tsort_each'
  /opt/rh/rh-ruby24/root/usr/share/ruby/tsort.rb:205:in `tsort_each'
  /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.4/lib/rails/initializable.rb:58:in `run_initializers'
  /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.4/lib/rails/application.rb:353:in `initialize!'
  /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.4/lib/rails/railtie.rb:185:in `public_send'
  /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.4/lib/rails/railtie.rb:185:in `method_missing'
  /usr/share/foreman/config/environment.rb:5:in `<top (required)>'
  /opt/rh/rh-ruby24/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require'
  /opt/rh/rh-ruby24/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require'
  config.ru:3:in `block in <main>'
  /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/rack-2.0.3/lib/rack/builder.rb:55:in `instance_eval'
  /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/rack-2.0.3/lib/rack/builder.rb:55:in `initialize'
  config.ru:1:in `new'
  config.ru:1:in `<main>'
  /usr/share/gems/gems/passenger-4.0.18/helper-scripts/rack-preloader.rb:105:in `eval'
  /usr/share/gems/gems/passenger-4.0.18/helper-scripts/rack-preloader.rb:105:in `preload_app'
  /usr/share/gems/gems/passenger-4.0.18/helper-scripts/rack-preloader.rb:150:in `<module:App>'
  /usr/share/gems/gems/passenger-4.0.18/helper-scripts/rack-preloader.rb:29:in `<module:PhusionPassenger>'
  /usr/share/gems/gems/passenger-4.0.18/helper-scripts/rack-preloader.rb:28:in `<main>'
2

As an update to this - it seems removing the RPM for the remote execution plugin does not remove it “cleanly” as we are then not able to load many UI pages. An example error below is from loading the “hosts” endpoint.
Has anyone seen this issue with remote execution before? My second thought was to “delete” all of the Default Remote Execution roles on the hopes that the rake processes could recreate them from scratch, and properly?. But, the roles are now “locked” with newer versions - so it looks like i cannot do that either…

Oops, we’re sorry but something went wrong The single-table inheritance mechanism failed to locate the subclass: ‘HostStatus::ExecutionStatus’. This error is raised because the column ‘type’ is reserved for storing the class in case of inheritance. Please rename this column if you didn’t intend it to be used for storing the inheritance class or overwrite HostStatus::Status.inheritance_column to use another column for that information.

3

Yeah, this is a problem for most of plugins. To delete these states, you need to run foreman-rake console and do something like this:

HostStatus::Status.where(:type => 'HostStatus::ExecutionStatus').destroy_all
Setting.where(:category => 'Setting::RemoteExecution').destroy_all

That should do the trick. I suggest you backup DB before running console.

4

Hey @Marek_Hulan

That got us up and running “without” remote execution and ansible plugins.

That being said - my end goal is really to “get them installed”

Looking at production.log in more detail - i see this logged as well just prior to the stacktrace that foreman logs as its final error:

2018-08-06T16:51:19 [W|app|] Could not create role ‘Remote Execution Manager’: ERF73-2603 [Foreman::PermissionMissingException]: some permissions were not found

This role “Does” already exist in my instance, and is a “locked” role.

I have a hunch if i can “wipe” the two remote execution roles, that i can re-create them with a DB:Seed rake task. Alas, as they are now “locked” plugin roles - I’m not able to delete. I’m assuming there is a console rake task to wipe Roles? I thought about wiping the rows from the table in the database (I can see them) but im assuming there are other links that this might not be the cleanest…

Thanks in advance for any further insight you may have!

5

Roles should be auto updated on Foreman restart, based on plugin definitions. That’s one of the reasons they are locked. It seems the definition may contain some missing permission. Try running foreman-rake db:seed and then restart Foreman.

6

Hey @Marek_Hulan

I’ve run a seed and reboot a few times, unfortunately no dice :frowning: .

A seed “appears” to complete successfully (no stdout logged on the console) but the same error is logged in production.log for the seed task(s):
2018-08-07T09:19:11 [W|app|] Could not create role ‘Remote Execution Manager’: ERF73-2603 [Foreman::PermissionMissingException]: some permissions were not found

Restarting Foreman (specifically the Foreman Service and httpd service) offers no change, n’or does a restart of the entire server.

My limited understanding of the error stack tells me that a specific permission is “different” on my built-in rile versus what it should be. My first thought was compare to a fresh 1.17.x or 1.18 install and adjust the permissions. But the role is locked, so that’s no go.

The console commands provided above allowed me to “forcibly remove” the plugin to an extent. But the plugin “Remote Execution” roles still remained, and caused the same error upon reinstall.

Is there a console command i can use to “remove” these plugin roles when i remove the plugin itself. Ideally, I’m hoping a completely fresh reinstall will allow the plugin to re-create the roles “as they should be”.

I’m also open to other options! To my knowledge I’ve never altered the Remote Execution roles (i never even use them TBH - I create custom/tailored roles for everything modeled off of the builtin) so im unsure how they got “broken” to begin with…

Thanks again for all the help and insight!

~Jason Lang