Unable to add smart proxy Features "dynflow and salt" in this proxy are not recognized by Foreman. If these features come from a Smart Proxy plugin, make sure Foreman has the plugin installed too

Problem:
When I try to add a smart_proxy to tfm I get the error: Features “dynflow and salt” in this proxy are not recognized by Foreman. If these features come from a Smart Proxy plugin, make sure Foreman has the plugin installed too.

Expected outcome:
smart_proxy gets added

Foreman and Proxy versions:
3.4.0-1

Foreman and Proxy plugin versions:

rubygem-smart_proxy_dynflow-0.8.1-1.fm3_3.el8.noarch
rubygem-smart_proxy_salt-5.0.0-3.fm3_3.el8.noarch

Distribution and version:
rhel 8.6

Other relevant data:
When querying the smart proxy features from the foreman host, I get [“dynflow”,“salt”], so that seems to work.

No certificate errors. The smart proxy with the salt smart proxy has the foreman host as trusted host, and vice versa.

when using the hammer I get

DEBUG 2022-10-17T17:45:04 API] Using authenticator: HammerCLIForeman::Api::SessionAuthenticatorWrapper
[ERROR 2022-10-17T17:45:04 API] 422 Unprocessable Entity
[DEBUG 2022-10-17T17:45:04 API] {
“error” => {
“id” => nil,
“errors” => {
“base” => [
[0] “Features “dynflow and salt” in this proxy are not recognized by Foreman. If these features come from a Smart Proxy plugin, make sure Foreman has the plugin installed too.”
]
},
“full_messages” => [
[0] “Features “dynflow and salt” in this proxy are not recognized by Foreman. If these features come from a Smart Proxy plugin, make sure Foreman has the plugin installed too.”
]
}
}
[DEBUG 2022-10-17T17:45:04 Exception] Using exception handler HammerCLIForeman::ExceptionHandler#handle_unprocessable_entity
[ERROR 2022-10-17T17:45:04 Exception] Features “dynflow and salt” in this proxy are not recognized by Foreman. If these features come from a Smart Proxy plugin, make sure Foreman has the plugin installed too.
Could not create the proxy:
Features “dynflow and salt” in this proxy are not recognized by Foreman. If these features come from a Smart Proxy plugin, make sure Foreman has the plugin installed too.

Well, as the error says: it seems you have a smart proxy configured with those plugins but did not install the foreman server plugin to use that…

https://docs.theforeman.org/nightly/Managing_Hosts/index-foreman-el.html#salt_guide_installing_salt_plugin_managing-hosts

ok, I run the foreman installer commands and now I see the plugins in the output of

curl -H ‘accept application/json’ https://foreman.sub.domain.tld:8443/features
[“dynflow”,“logs”,“puppet”,“puppetca”,“salt”]

but now I get this error: Service unavailable ERF64-6496 [Foreman::MaintenanceException]: There are migrations pending in the system.

And if i run foreman-rake db:migrate it breaks:

== 20211108211312 AddMissingPermissions: migrating ============================
rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:

Validation failed: Name has already been taken
/usr/share/gems/gems/activerecord-6.1.6.1/lib/active_record/validations.rb:80:in raise_validation_error' /usr/share/gems/gems/activerecord-6.1.6.1/lib/active_record/validations.rb:53:in save!’
/usr/share/gems/gems/activerecord-6.1.6.1/lib/active_record/transactions.rb:302:in block in save!' /usr/share/gems/gems/activerecord-6.1.6.1/lib/active_record/transactions.rb:354:in block in with_transaction_returning_status’
/usr/share/gems/gems/activerecord-6.1.6.1/lib/active_record/connection_adapters/abstract/database_statements.rb:318:in transaction' /usr/share/gems/gems/activerecord-6.1.6.1/lib/active_record/transactions.rb:350:in with_transaction_returning_status’
/usr/share/gems/gems/activerecord-6.1.6.1/lib/active_record/transactions.rb:302:in save!' /usr/share/gems/gems/activerecord-6.1.6.1/lib/active_record/suppressor.rb:48:in save!’
/usr/share/gems/gems/activerecord-6.1.6.1/lib/active_record/persistence.rb:55:in create!' /usr/share/gems/gems/foreman_salt-15.1.0/db/migrate/20211108211312_add_missing_permissions.rb:3:in up’
/usr/share/gems/gems/activerecord-6.1.6.1/lib/active_record/migration.rb:870:in public_send' /usr/share/gems/gems/activerecord-6.1.6.1/lib/active_record/migration.rb:870:in exec_migration’
/usr/share/gems/gems/activerecord-6.1.6.1/lib/active_record/migration.rb:851:in block (2 levels) in migrate' /usr/share/gems/gems/activerecord-6.1.6.1/lib/active_record/migration.rb:850:in block in migrate’
/usr/share/gems/gems/activerecord-6.1.6.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:462:in with_connection' /usr/share/gems/gems/activerecord-6.1.6.1/lib/active_record/migration.rb:849:in migrate’
/usr/share/gems/gems/activerecord-6.1.6.1/lib/active_record/migration.rb:1037:in migrate' /usr/share/gems/gems/activerecord-6.1.6.1/lib/active_record/migration.rb:1329:in block in execute_migration_in_transaction’
/usr/share/gems/gems/activerecord-6.1.6.1/lib/active_record/migration.rb:1380:in block in ddl_transaction' /usr/share/gems/gems/activerecord-6.1.6.1/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in block in transaction’
/usr/share/gems/gems/activerecord-6.1.6.1/lib/active_record/connection_adapters/abstract/transaction.rb:319:in block in within_new_transaction' /usr/share/gems/gems/activesupport-6.1.6.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in block (2 levels) in synchronize’
/usr/share/gems/gems/activesupport-6.1.6.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in handle_interrupt' /usr/share/gems/gems/activesupport-6.1.6.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in block in synchronize’
/usr/share/gems/gems/activesupport-6.1.6.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in handle_interrupt' /usr/share/gems/gems/activesupport-6.1.6.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in synchronize’
/usr/share/gems/gems/activerecord-6.1.6.1/lib/active_record/connection_adapters/abstract/transaction.rb:317:in within_new_transaction' /usr/share/gems/gems/activerecord-6.1.6.1/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in transaction’
/usr/share/gems/gems/activerecord-6.1.6.1/lib/active_record/transactions.rb:209:in transaction' /usr/share/gems/gems/activerecord-6.1.6.1/lib/active_record/migration.rb:1380:in ddl_transaction’
/usr/share/gems/gems/activerecord-6.1.6.1/lib/active_record/migration.rb:1328:in execute_migration_in_transaction' /usr/share/gems/gems/activerecord-6.1.6.1/lib/active_record/migration.rb:1302:in each’
/usr/share/gems/gems/activerecord-6.1.6.1/lib/active_record/migration.rb:1302:in migrate_without_lock' /usr/share/gems/gems/activerecord-6.1.6.1/lib/active_record/migration.rb:1251:in block in migrate’
/usr/share/gems/gems/activerecord-6.1.6.1/lib/active_record/migration.rb:1401:in block in with_advisory_lock' /usr/share/gems/gems/activerecord-6.1.6.1/lib/active_record/migration.rb:1416:in block in with_advisory_lock_connection’
/usr/share/gems/gems/activerecord-6.1.6.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:462:in with_connection' /usr/share/gems/gems/activerecord-6.1.6.1/lib/active_record/migration.rb:1416:in with_advisory_lock_connection’
/usr/share/gems/gems/activerecord-6.1.6.1/lib/active_record/migration.rb:1397:in with_advisory_lock' /usr/share/gems/gems/activerecord-6.1.6.1/lib/active_record/migration.rb:1251:in migrate’
/usr/share/gems/gems/activerecord-6.1.6.1/lib/active_record/migration.rb:1086:in up' /usr/share/gems/gems/activerecord-6.1.6.1/lib/active_record/migration.rb:1061:in migrate’
/usr/share/gems/gems/activerecord-6.1.6.1/lib/active_record/tasks/database_tasks.rb:237:in migrate' /usr/share/gems/gems/activerecord-6.1.6.1/lib/active_record/railties/databases.rake:92:in block (3 levels) in <top (required)>’
/usr/share/gems/gems/activerecord-6.1.6.1/lib/active_record/railties/databases.rake:90:in each' /usr/share/gems/gems/activerecord-6.1.6.1/lib/active_record/railties/databases.rake:90:in block (2 levels) in <top (required)>’
/usr/share/gems/gems/rake-13.0.1/exe/rake:27:in `<top (required)>’

Caused by:
ActiveRecord::RecordInvalid: Validation failed: Name has already been taken

So it appears I have a duplicate record somewhere, but which one?

sudo -u postgres psql foreman -c ‘select id, name, organization_id, location_id from hosts;’

id | name | organization_id | location_id
----±-------------------------±----------------±------------
1 | foreman.sub.domain.tld | 1 | 2
2 | lab110-14.sub.domain.tld | 1 | 2
11 | salt.sub.domain.tld | 1 | 2
3 | lab110-15.sub.domain.tld | 1 | 2
5 | lab110-16.sub.domain.tld | 1 | 2
(5 rows)

is there a way to easily disable this validation? I am in a bit of a hurry here. I tried to dump and restore the foreman db, but this did not help.

Well, it doesn’t really improve the situation if you are not following the installation docs and use foreman-installer but instead run low-level commands (like db:migrate outside the context of foreman-installer), modify the database, etc. Very quickly, your system is broken beyond repair, because not everything is on matching versions and state. There maybe some quick fix for some issues on the way but in the long run it is only getting worse.

The migration failing is this: foreman_salt/20211108211312_add_missing_permissions.rb at master · theforeman/foreman_salt · GitHub

I guess, because you have installed the salt smart-proxy plugin without the necessary salt foreman plugin it’s got out of sync. It seems permission auth_smart_proxies_salt_autosign already exists but the migration isn’t set to have finished. So you could either remove the permission which should allow the migration to run or manually set the migration to done and try again. Either way, it’s only this single step and the next may fail again because your whole system seems to be out of order…

thanks for your comment.

If I follow the instructions, the foreman-installer replaces the certificate settings every single time. It’s easily fixeable, but annoying.

So how do I remove the auth_smart_proxy_salt_autosign permission?

I already tried
foreman=# select * from permissions where id = 182;
182 | auth_smart_proxies_salt_autosign | SmartProxy | 2022-10-17 14:30:41.008821 | 2022-10-17 14:30:41.008821

but it’s refrenced somewhere else, so delete does not work
foreman=# delete from permissions where id = 182;
ERROR: update or delete on table “permissions” violates foreign key constraint “filterings_permissions_id_fk” on table “filterings”
DETAIL: Key (id)=(182) is still referenced from table “filterings”.

or how do I get the migration manually to done?

Well, I can’t tell from that what your problem is, but usually it means you are not using the correct foreman-installer options.

And once you start installing rpms and configuring manually without using foreman-installer you’ll end up where you are…

this comment is not very constructive, the foreman installation allows for this workflow, see Foreman :: Manual

but never mind, I disable the salt proxy stuff, I can log in and get the templates I need, I will reinstall the foreman host using the foreman installer, see how it goes.

It says:

The Puppet-based Foreman installer is recommended for most environments, instead of installing only the packages as it will perform full configuration too.

So you need to know what you do if you do it manually.

Also I would recommend to use the new docs not the old: https://docs.theforeman.org/

Your comment is still not very constructive. Instead of trying to be right, and blaming the user in this case, you assume I do not know what I am doing.

In this case, there is clearly a bug in the upgrade script. It should not fail like this in such an epic manner.

On the other hand, I should have made a db backup. So that’s it. I could get the templates I needed using the hammer, I will try again. Let’s leave it at that, shall we?

I am not trying to be right. I just pointing out what happens if you start messing with the internals. You didn’t know what you were doing. Otherwise you wouldn’t have started this thread.

From what you wrote so far I gathered that you did not use the upgrade script but instead started installing individual plugin rpms. You don’t even gave a precise description on how you have made this “upgrade” and how exactly you got where you have got. You mention a lot briefly without details to understand exactly what you mean (like the certificates). You have never once mentioned until now that this might be part of an upgrade. So answers will be as superficial as your posts.

You didn’t use the recommended way with foreman-installer. It didn’t work. You made additional attempts to fix those using low-level tools like calling db:migrate manually. It made things worse. What do you expect? Maybe one of the developers has an idea how best to proceed or fix, but from user perspective you shouldn’t complain if someone wonders why you don’t use the recommended and well documented way…

This again might as well cause additional problems in the future. You seem to have some parts for the salt proxy in your database. Future database migration may not work anymore because some remnants of the salt proxy don’t go with the migration unless there is also a migration for salt (which would be there if it wasn’t disabled…).

ok, here we are again.

I reinstalled rhel 8.6, reinstalled the foreman using the foreman-installer, and after running the plugin with the new manual:

# foreman-installer \
--enable-foreman-plugin-salt \
--enable-foreman-proxy-plugin-salt

then, it breaks on exactly the same way.

022-10-18 15:02:00 [ERROR ] [configure] ‘/usr/sbin/foreman-rake db:migrate’ returned 1 instead of one of [0]
2022-10-18 15:02:00 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: change from ‘notrun’ to [‘0’] failed: ‘/usr/sbin/foreman-rake db:migrate’ returned 1 instead of one of [0]

and when looking at the foreman.log, then I come accross the same error:

2022-10-18 15:02:00 [INFO ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: == 20211108211312 AddMissingPermissions: migrating ============================
2022-10-18 15:02:00 [ERROR ] [configure] ‘/usr/sbin/foreman-rake db:migrate’ returned 1 instead of one of [0]
2022-10-18 15:02:00 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: change from ‘notrun’ to [‘0’] failed: ‘/usr/sbin/foreman-rake db:migrate’ returned 1 instead of one of [0]
2022-10-18 15:02:00 [DEBUG ] [configure] Execforeman-rake-db:migrate: Executing check ‘/usr/sbin/foreman-rake db:abort_if_pending_migrations’
2022-10-18 15:02:00 [DEBUG ] [configure] Executing with uid=foreman: ‘/usr/sbin/foreman-rake db:abort_if_pending_migrations’
2022-10-18 15:02:18 [DEBUG ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/unless: Run bin/rails db:migrate to update your database then try again.
2022-10-18 15:02:18 [DEBUG ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/unless: You have 7 pending migrations:
2022-10-18 15:02:18 [DEBUG ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/unless: 20211108211312 AddMissingPermissions
2022-10-18 15:02:18 [DEBUG ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/unless: 20211123170430 TasksSettingsToDslCategory
2022-10-18 15:02:18 [DEBUG ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/unless: 20220118160349 DropSaltHideRunSaltButtonSetting
2022-10-18 15:02:18 [DEBUG ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/unless: 20220321101835 RenameSshProviderToScript
2022-10-18 15:02:18 [DEBUG ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/unless: 20220331112719 AddSshUserToJobInvocation
2022-10-18 15:02:18 [DEBUG ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/unless: 20220406185204 FixSaltSettingCategoryToDsl
2022-10-18 15:02:18 [DEBUG ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/unless: 2021051713291621250977 AddHostProxyInvocations
2022-10-18 15:02:18 [DEBUG ] [configure] Execforeman-rake-db:migrate: Executing ‘/usr/sbin/foreman-rake db:migrate’
2022-10-18 15:02:18 [DEBUG ] [configure] Executing with uid=foreman: ‘/usr/sbin/foreman-rake db:migrate’
2022-10-18 15:02:31 [INFO ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: rake aborted!
2022-10-18 15:02:31 [INFO ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: StandardError: An error has occurred, this and all later migrations canceled:
2022-10-18 15:02:31 [INFO ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns:
2022-10-18 15:02:31 [INFO ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: Validation failed: Name has already been taken

So, unless you have something a bit more intelligent to say, I would appreciate if you stopped being so arrogant and telling me I have done stuff not right…

I’ve been using oss for about 25 years, I know how it works, thank you very much.

I just reinstalled the whole thing, foreman application and database server. So there is nothing there anymore.
And this is not an upgrade, the packages run some kind of db schema upgrade script,

So summing up:

  • brand new installation of OS
  • brand new installation of tfm with foreman-installer on a brand new db
  • after this is working, and I can login to the brand new foreman 3.4, I stop all services and run the foreman-intaller with the options to install the plugin as especified on the ‘new’ documentation, i.e.,

foreman-installer \

–enable-foreman-plugin-salt
–enable-foreman-proxy-plugin-salt

  • same error.

Looks like there is something else going on. I do appreciate the effort and time you put in your replies. The assumptions you make about what I did and that it is the case for the error irritate me.

I don’t know why you need to get offensive. I only stated the obvious. You never mentioned how you did install it before. So it’s impossible to tell, if you did it right or not. The things you wrote you have done definitively don’t help if it’s not working but rather tend to break things more. I am not saying that they did make things worse, but it’s not unlikely. What you wrote sounded very hectic and like you were trying various things to fix things which makes the whole thing even harder to follow.

You should know that if

If that’s the case then it’s obviously a bug in the salt plugin. Open an issue.

your ‘obvious’ was ‘irritating’ and ‘condescending’. If you do not see that, I’m sorry,

If you google the " Service unavailable ERF64-6496 [Foreman::MaintenanceException]: There are migrations pending in the system." all the links you get are from this forum telling people to run the foreman-rake db:migrate stuff.

So it’s not like I have not done my work. I’ve been with this issue for far longer than I would rather be.

For now, I have a working tfm 2.5.1 talking to a salt smart proxy which is what I needed.