Fix All Mismatches - Validation failed: Taxonomy has already been taken

Problem:
Fix All Mismatches fails with Validation failed: Taxonomy has already been taken error.

Expected outcome:

Foreman and Proxy versions:
Foreman version: 2.4.1

Foreman and Proxy plugin versions:
Plugins:

  • foreman-tasks 4.0.1
  • foreman_remote_execution 4.3.0
  • katello 4.0.1.2

Distribution and version:
CentOS7
Other relevant data:

The original installation was foreman 2.0 and katello 3.15, we’ve been succefully upgrading till foreman 2.4 and katello 4.0.

We have

  • one organization - we do not use the default.
  • two locations
    – the default one where most of our servers are (and where the foreman/katello server is located)
    – another location to which the foreman/katello server does not have direct access to all the machines, so we have a smartproxy there, with which the server can communicate.

Mismatch report:

Logs:

# foreman-rake errors:fetch_log request_id=4936acf4 
2021-07-29T17:12:07 [I|app|4936acf4] Started POST "/organizations/import_mismatches" for 185.98.249.5 at 2021-07-29 17:12:07 +0100
2021-07-29T17:12:07 [I|app|4936acf4] Processing by OrganizationsController#import_mismatches as HTML
2021-07-29T17:12:07 [I|app|4936acf4]   Parameters: {"authenticity_token"=>"Refk9kdsfsanNYITLXngKpXfGcrdfVJSmUCSoxoRHQ0ARFn0oc2vPTmyAT5QPwAi9Ev8fOcJgFIdTWLFDGEuxQ=="}
2021-07-29T17:12:07 [W|app|4936acf4] Scoped order is ignored, it's forced to be batch order.
2021-07-29T17:12:07 [W|app|4936acf4] Validation failed: Taxonomy has already been taken
2021-07-29T17:12:07 [I|app|4936acf4] Backtrace for 'Validation failed: Taxonomy has already been taken' error (ActiveRecord::RecordInvalid): Validation failed: Taxonomy has already been taken
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/validations.rb:80:in `raise_validation_error'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/validations.rb:53:in `save!'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/transactions.rb:318:in `block in save!'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/transactions.rb:375:in `block in with_transaction_returning_status'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in `block in transaction'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract/transaction.rb:280:in `block in within_new_transaction'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract/transaction.rb:278:in `within_new_transaction'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in `transaction'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/transactions.rb:212:in `transaction'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/transactions.rb:366:in `with_transaction_returning_status'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/transactions.rb:318:in `save!'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/suppressor.rb:48:in `save!'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/persistence.rb:55:in `create!'
 4936acf4 | /usr/share/foreman/app/services/tax_host.rb:102:in `block in import_missing_ids'
 4936acf4 | /usr/share/foreman/app/services/tax_host.rb:95:in `each'
 4936acf4 | /usr/share/foreman/app/services/tax_host.rb:95:in `import_missing_ids'
 4936acf4 | /usr/share/foreman/app/models/taxonomy.rb:61:in `import_missing_ids'
 4936acf4 | /usr/share/foreman/app/models/taxonomy.rb:124:in `block in all_import_missing_ids'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:70:in `block (2 levels) in find_each'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:70:in `each'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:70:in `block in find_each'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:136:in `block in find_in_batches'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:238:in `block in in_batches'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:222:in `loop'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:222:in `in_batches'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:135:in `find_in_batches'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:69:in `find_each'
 4936acf4 | /usr/share/foreman/app/models/taxonomy.rb:123:in `all_import_missing_ids'
 4936acf4 | /usr/share/foreman/app/controllers/concerns/foreman/controller/taxonomies_controller.rb:129:in `import_mismatches'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/abstract_controller/base.rb:195:in `process_action'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_controller/metal/rendering.rb:30:in `process_action'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:112:in `block in run_callbacks'
 4936acf4 | /usr/share/foreman/app/controllers/concerns/foreman/controller/timezone.rb:10:in `set_timezone'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
 4936acf4 | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
 4936acf4 | /usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in `around'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in `around'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:139:in `run_callbacks'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/abstract_controller/callbacks.rb:41:in `process_action'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_controller/metal/rescue.rb:22:in `process_action'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_controller/metal/instrumentation.rb:33:in `block in process_action'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/notifications.rb:180:in `block in instrument'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/notifications.rb:180:in `instrument'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_controller/metal/params_wrapper.rb:245:in `process_action'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/railties/controller_runtime.rb:27:in `process_action'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/abstract_controller/base.rb:136:in `process'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.4/lib/action_view/rendering.rb:39:in `process'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_controller/metal.rb:190:in `dispatch'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_controller/metal.rb:254:in `dispatch'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/routing/route_set.rb:50:in `dispatch'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/routing/route_set.rb:33:in `serve'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/journey/router.rb:49:in `block in serve'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/journey/router.rb:32:in `each'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/journey/router.rb:32:in `serve'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/routing/route_set.rb:834:in `call'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.1.2/lib/katello/middleware/event_daemon.rb:10:in `call'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in `call'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in `call'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-dsl-2.3.0/lib/apipie_dsl/static_dispatcher.rb:67:in `call'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/static_dispatcher.rb:66:in `call'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/extractor/recorder.rb:137:in `call'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in `call'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in `call'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in `call'
 4936acf4 | /usr/share/foreman/lib/foreman/middleware/telemetry.rb:10:in `call'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
 4936acf4 | /usr/share/foreman/lib/foreman/middleware/catch_json_parse_errors.rb:9:in `call'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/tempfile_reaper.rb:15:in `call'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/etag.rb:27:in `call'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/conditional_get.rb:40:in `call'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/head.rb:12:in `call'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/http/content_security_policy.rb:18:in `call'
 4936acf4 | /usr/share/foreman/lib/foreman/middleware/logging_context_session.rb:22:in `call'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/session/abstract/id.rb:266:in `context'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/session/abstract/id.rb:260:in `call'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/cookies.rb:648:in `call'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:101:in `run_callbacks'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/actionable_exceptions.rb:18:in `call'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/debug_exceptions.rb:32:in `call'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.4/lib/rails/rack/logger.rb:37:in `call_app'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.4/lib/rails/rack/logger.rb:28:in `call'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in `call'
 4936acf4 | /usr/share/foreman/lib/foreman/middleware/logging_context_request.rb:11:in `call'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/request_id.rb:27:in `call'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.1.2/lib/katello/prevent_json_parsing.rb:12:in `call'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/method_override.rb:24:in `call'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/runtime.rb:22:in `call'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/executor.rb:14:in `call'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in `call'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/sendfile.rb:110:in `call'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/host_authorization.rb:76:in `call'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/secure_headers-6.3.0/lib/secure_headers/middleware.rb:11:in `call'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.4/lib/rails/engine.rb:527:in `call'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.4/lib/rails/railtie.rb:190:in `public_send'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.4/lib/rails/railtie.rb:190:in `method_missing'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:74:in `block in call'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `each'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `call'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.1.1/lib/puma/configuration.rb:246:in `call'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.1.1/lib/puma/request.rb:76:in `block in handle_request'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.1.1/lib/puma/thread_pool.rb:337:in `with_force_shutdown'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.1.1/lib/puma/request.rb:75:in `handle_request'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.1.1/lib/puma/server.rb:431:in `process_client'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.1.1/lib/puma/thread_pool.rb:145:in `block in spawn_thread'
 4936acf4 | /opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2021-07-29T17:12:07 [I|app|4936acf4]   Rendering common/500.html.erb within layouts/application
2021-07-29T17:12:07 [I|app|4936acf4]   Rendered common/500.html.erb within layouts/application (Duration: 1.5ms | Allocations: 834)
2021-07-29T17:12:07 [I|app|4936acf4]   Rendered layouts/_application_content.html.erb (Duration: 3.3ms | Allocations: 5674)
2021-07-29T17:12:07 [I|app|4936acf4]   Rendering layouts/base.html.erb
2021-07-29T17:12:07 [I|app|4936acf4]   Rendered layouts/base.html.erb (Duration: 4.2ms | Allocations: 7293)
2021-07-29T17:12:07 [I|app|4936acf4] Completed 500 Internal Server Error in 356ms (Views: 14.6ms | ActiveRecord: 92.1ms | Allocations: 196752)

Foreman version: 2.4.1
Plugins: 
 - foreman-tasks 4.0.1
 - foreman_remote_execution 4.3.0
 - katello 4.0.1.2

That’s weird, can you try to edit the object manually? E.g. edit the user or production environment and try to assign the right Location/Organization. See if the object saves fine or the form indicates some issue.

Hello Marek,

Thank you for your reply.
Trying to correct it manually using the interface I still got the same taxonomy error.

Because of your suggestion I tried using hammer to correct it:

$ hammer organization add-environment --id 3 --puppet-environment-id 1
$ hammer location add-user --id 4 --user-id 7

After correcting the organization environment and user the problem with the Default Location disappeared (autocorrected?).

Now if i run:

 $ hammer location info --id 2

I can see the smartproxy that was appearing on the mismatch report on the list of smartproxies.
Unfortunately this is a regression for us.
As I said before we have two locations. The Default Location where most our are machines are and a secondary one where we have some machines. The foreman/katello server does not have direct access to all the machines, so in order to use Remote Execution we added the smartproxy with which foreman/katello can communicate.
After adding the smartproxy all the remote execution jobs (for all the machines) started using it, so it were failing for most of our machines.
To workaround this, we removed the smartproxy completely and manually added it only on the secondary location.

We asked for help about this issue before here.

Best regards,
Armindo Silva.

A quick check, did you perform all of this under admin user or under user assigned to both locations with enough permissions?

Hello @Marek_Hulan

I was logged in in foreman/katello with the following user:

$ hammer user info --id 6
Id:                    6
Login:                 <redacted>
Name:                  <redacted>
Email:                 <redacted>
Admin:                 no
Disabled:              no
Last login:            2021/08/05 08:07:16
Authorized by:         <redacted> Freeipa
Effective admin:       no
Locale:                default
Timezone:              default
Description:           
Default organization:  
Default location:      
Roles:                 

User groups:           
 1) Usergroup: foreman-admins
    Roles:     
        Auditor
        Bookmarks manager
        Edit hosts
        Edit partition tables
        Manager
        Organization admin
        Register hosts
        Remote Execution Manager
        Remote Execution User
        Site manager
        System admin
        Tasks Manager
        Tasks Reader
        View hosts
        Viewer
 2) Usergroup: foreman-users
    Roles:     
        Tasks Reader
        View hosts
        Viewer
Inherited User groups: 

Locations:             
    <redacted>
    Default Location
Organizations:         
    Default Organization
    <redacted>
Created at:            2020/11/10 22:37:48
Updated at:            2021/05/20 11:55:12

After the first installation and configuring the Freeipa authentication I never used the admin account again, I always use my own account.
Probably that was the issue, right? My account is not and effective admin.

Well taxonomies amd permission can get tricky for non-admin users. If you encounter this again, try performing the fix by admin. If that works it gives us something to investigate.