LDAP support with load balanced Foremans?

Support
1

Problem:
LDAP auth will always fail on one of the foreman servers under our load balancer

Expected outcome:
It auths on all foreman servers

Foreman and Proxy versions:

Name Features Status Version
foreman-data-01 Logs 3.10.0
foreman-data-02 Logs 3.10.0
foreman-ui-01 Logs 3.10.0
foreman-ui-02 Logs 3.10.0
Foreman and Proxy plugin versions:

Distribution and version:
Alma9

Other relevant data:
We have a load balancer that round robins to each of our 4 foreman servers, they all point to the same redis/postgres server. If I setup a single instance of LDAP, it will auth on one of those servers. If someone tries to log in and the auth gets sent to a different server, it will fail. My current work around was to create 4 duplicate instances of the ldap server, within the UI. I had to make sure that the settings change went to each individual server. Now they can all auth, but that seems like a really poor work around.

Is this supposed to work with foreman in a load balanced way? It seems like foreman maintains this ldap server locally to each foreman server itself, but I cant find any config for it off hand.

2

I know at least one customer were this setup worked, so can you verify your rails-cache setting really points to the same central redis server? Could sometimes be a bit confusing as redis is used multiple times in the setup so perhaps this one is missing.

3

Thank you for the reply!
Rails cache testing was one of the very first things we tested and verified working.

Here is a successful login:

1724238948.915660 [0 10.222.206.152:34402] "get" "foreman:failed_login_10.222.218.3"
1724238948.918630 [0 10.222.206.152:34402] "get" "foreman:webpack_foreman_vendor_js"
1724238955.658344 [0 10.222.206.152:34402] "get" "foreman:failed_login_10.222.218.3"
1724238955.864366 [0 10.222.206.152:34402] "del" "foreman:user/5/taxonomy_and_child_ids/organizations"
1724238955.864929 [0 10.222.206.152:34402] "del" "foreman:user/5/taxonomy_and_child_ids/locations"
1724238955.865940 [0 10.222.206.152:34402] "del" "foreman:views/tabs_and_title_records-5"
1724238956.130330 [0 10.222.206.152:34402] "del" "foreman:user/5/taxonomy_and_child_ids/organizations"
1724238956.130868 [0 10.222.206.152:34402] "del" "foreman:user/5/taxonomy_and_child_ids/locations"
1724238956.132013 [0 10.222.206.152:34402] "del" "foreman:views/tabs_and_title_records-5"
1724238956.143475 [0 10.222.206.152:34402] "del" "foreman:views/tabs_and_title_records-5"
1724238956.216965 [0 10.222.206.152:34402] "del" "foreman:views/tabs_and_title_records-5"
1724238956.330156 [0 10.222.206.152:34402] "get" "foreman:webpack_foreman_vendor_js"
1724238956.332011 [0 10.222.206.152:34402] "get" "foreman:views/tabs_and_title_records-5"
1724238956.345126 [0 10.222.206.152:34402] "set" "foreman:views/tabs_and_title_records-5" "\x04\bo: ActiveSupport::Cache::Entry\n:\x0b@value\"\x02~\ax\x9c\xed[\xedo\xd46\x1c\x96\xa6\tM\xfb+\xa2N\xdb\x17\xb8\xf7\x17\xda\x1b\xb0u\xb40&\n\xa8w\xa0I\bE>\xc7\xb93ulc;w\x1cS\xff\xf7\xd9y\xe9\xc5\xb9K\x13(M\x90\xb6~h\xaa\xe4\xf9\xd9\x8f\x9d\xdf\xcbc;\xfd\xfe\x87g\a\xdf\xfd=t\x92\x9f\a>\x13(\x00\xb4%\x10\x80\xaa\x05Y\xc0\x19ET9\x14\x04\xe8\xe1\xc1\xb9\xb9{\xcc\xf9\x81\xe3\x01\x05Z\\0.\x1f\x1e\xfc\xf3\xcb\x87\x90\xa9_\t\xd8\xb0P\xc5\x7fO\x92{\x01\xa2ar\xe7mrKm8Jn\xc5\x17\x19\xce\xdd-\xee^|1\xfdY\xa83F\xb1b\xc2\x02a\xc8\xa8\x05\xf2\x81\xe3\x83\x96\x02p\xc9\x02\xa4\x90\x8d\x86KL<\x81h\x19\x1f\xacP`\x19\xa2\x8fz\xd4\tD\x89\x10%\xb7\x97* .\xe3\n3*\xd3Q_\x16\xd1?\x01r9g@xV\xcb\xa1 \x16\xaa\x13_.\xef}\x117\x1f\x10\xf9E\xe4\xfedR9S\x05T(\x91\xbc\x9e\xe0RC]iA\xebf\xfbD\xb7Q\xc2\xd2\xd7\x10w\x05H\xd8\x14\xc7\xe3\xd0\xc3e$A\x06S7\xbf\x99\x8e\x03\xafl\x12\xe3T\x10\xbdm,\x15\x86\xb2\xa32fuS\x9e^\xd1\xf8l\xda2gZ7\xf5\xbf\xd8\xbc\x84\xf4{6w1]1\b2-\x16\xd3\xf4\xf0\n{\xb9\xf4\xb6\xd3\xeb9\xe2L4\xe5`\x8f\x19\xf5\xf1\xc29\x03\x14,\x90\xce\xee\xea\xfa\xf1\xc3\b\xee\x8a\x98\xf2o\x12\x01\x01\x97\x0f\xd1J\x1b\xfa!\xb9\xfb\xf3\xe0\xe4\xae\xc9\xbc\xcd\x8c%\x9eHg\x86\x02N\x80*\xcb\x90*\x85u\xe2\xd1\xb8\xca\xb6\xbb\xe1[}\x12\xbb\xb73\x03\xf2\xa2\xa9\xe4\xb1\xed\xba,\b\x95\x81vTs\\\xcf\x11\x0c\x85\xc0t\xe1<g\x8b\xca\xb9#\xa6-R[\x97dl/\xdf\x15\x8f\xa1\x9a\x941\xe5\xd6&R d$\x12\xab\xdb\x101_^\xd6\bqv\xd9\xef\x15\tM\xe5\x1d-\\\x15\x8aHV\xe0\xd8\xa1h\xdd\x94_.tEB\xa22S\x91\x18|\x95$\xf2J\xb0\x15\x96\x9a\xa8\t\x8c)R!of\x16\x8eu\x96\xd7-C\x15\x8a\xb2\xb4\nv\xa1\xb5\xcbd-\xdf\xd7@ \xe7\x8cy\x88\x94\xf0\r2\x98\xba\x89>\xa3Z\xf1\x10\x12i\t\xe7\x0cy\x18\x94p\xddB\xea\xa6\xfa\x92#\xa1y\x1a7\xdch\xf7\x0eJf\x95\xa5p\x99E\xdf0\x1afU\x0b\xf4-\t\xdb\r\x85\x9f+.\\\xa9\x8d\x1a\xe2\xfb\n\b\x85#\xd7\x9a\x819\xa9.\x87\xb8\xca\xc0k'\x9d\xcdx\x9f\xad\xe4x\xc6\xba\xba\x9e\xbb\xb5\xc5\x84\xa3\xaa\r\xc0\xac*\xf2lo\xac^be\xaf\xb3\xb0\x05+P0k-U\xe0\xd2B6\xab`\xa2\xad\x8e\xa7\x82\x85\xbc\x82\x86Ydpu\xbf\xe6\xa7\x84\xcd\x01qt\xb0\x81h+\xab\x84.dA\xc0\xa8\xcbs\xf0\x9b\xea\x84\x90s\xa4\x9c\xd3\x17\x8f\x9b\x99\x84S\xba\xc2\x82Q\xb3~\xac\xa8\xdcy\xc4\xb8\x83v\x0ck\x17\xa2\x04\xc8\xd2\xdd\xb4\x1c\xeb\xf8\x02\xb3\x96\r\xad\xdb\xabDH\x8e|\xb2\x84o2f\xa6\x81.MN4\xf3\x95\x03\xa7\xf8\x15\xe8U\x1f\xbb\b\xb9{\x816_'\x9a\xa6\x80\xa8o=\x8e\xa4\xe6\x18\xfdr\x9b\x0f!\xb3\xe3W\x16A[\xbeZp\x87\x8d\xc9\x8b7@\xe0\nbh\xcbve\x1b\xdc\xbc,?\xa3\xbe\x00R\x890Z!Y\xd8\x9d\xda\xcc}s\xcb\x89/-\x8a\xd4\x9a\x89\x0b\xcb\xa4\xd9\"\x1d\xc7\xb1\x96l\x1fq\xe9\x8c\x1a\xa8\xcb\xb3\xd0\xfaSf\xc0C\x85\x9cs$Y(`\x19c\x18\xc3]a\xc3\x9bb\xadg\xd9\xc7\xa5\x8e\x9b\x92\xe6\x16\xba\xf6\x84\x10\xce\xb5\xb3\x96yD\x16T7\xc3\x13\x16\x00LK\x18zYP\xed\x9b\x18\xb3\xd9\xabj\xa1\xb5T\x8a7\x1bY\xe7\b\x90\xb2\x1d\x01\x91\xc1\xd4\x9f\xf5C\xa2\xf5\x06\xa3\x14\xc1\x8cE\x11\xd5\x95\x81\xbb0\x0f\xbfy\xee\x7f-Q\xa5\x83\xf10\x8fk6\xcd\x9fm\x9cc\bYXv8dh\xcb\xce\xa8\x83<|\xbb\xfa)\xbeF\x9f6h\xe9\xb8d\x9e\xd54\xbf\xda\xa9-\x1c\xd1s\xb6p^\x86\x95\x86C\xd8\xe2\xeak\x89\x9b{\xc0\xb1\x17`\x9a\xd9\x1f\xbe\xde\x0f [|Cn\xf0\xdc>\xf5,\x9a6R\xf5p\xf4\x966*\xc5\x02P\xfc\xa9\nS\xb6\x0b\xad\xff\xfb\x03\xb5\xd4\xc2\x1d\xc7S\xe6L\xab\b\x13\xa0m\xdcF5\x89\xc9c%$CYao\xe3\xf6\xd8UZ\x18\x1b\x8eM.\x83\xcfY\xa9\x9e\x13\xac1\x11\xf7\x87^Uk\xd1^v\x82<\xb7a\xf5K\x8f\x80iu|\xfa\x11\xc10\n\xa1'\bT8\xa5\x12\x91\x99\x8bR3\xd7\xb7\xccj\x17\xccHE\x87%\xd7\x93\x96\x16\xaa\xf6T5ge\x15\x13\xcc\xb3\xa5\xf2]\x023\x154\x87\x93z(\xb2\xb3D\xc0C\xc2\x1c\xda\xb3\x16D=0\x9f\xdf\x1f\x0f\xfbG#o\x0c\xe0}\xd8\x05\xde\xd1\xb0\x0f{\x83\xc3\xd1x0\x1e\x0cGp\xe0\x8d\x8e\xe6p8\xf7=\xef\xc8\xef\xa3\xd1\xc8\x1b\r\xfa\xfd\xf1!\x82\xf7\xef\xb7\xe5\xca\xae\x95\x94)\xec'y\xd5\xddaj=\x15\bb\x8ew\xf6\\\xa4b\xdc\xc5\x01\xd7\x89\x8c\xd1\x82vb\x8d\xb0\x8b\xb4\xe7\xe9J\xce\xa5\xd2\xc5|\xaf\xa0\xfbsw\x9f\xe0T\xcb\x8c\xb6\xb3\x87mm\xf0^r \x04[[}\xf8XH\xb5{\x06\x81|\xdf\x82\x11\xb0\a5\xdd\xd3\x9e^\xf9\xd8C}\xaf\x9bj'=\xff.%\xc4\x14\xb6\xf5j\xd32\x02F\xdd$V\x99\x0f0M\xa7\xd1\xa7\x19\xd4\xcd\xc9\x9c~\xb7?lu\x0f[\xfd\xde\xac;\x9e\xf4F\x93\xd1\xb8\xdd\x1b\x8c[\xdd\xd1\xa4\xdb\xb5\x9b\xde\x16=\xf7j\x8e\x86\xa9.\x8a>&\xf0\\\xa0\xf6\xb4>n\xf5G\xb3\xdep2\xeaOF\x83\xf6\xf8\xf0pO\xeb!\xf7\x8a\x1b8l\xf5\x06\xb3\xee\xa1\xa1\xd7\xed\xb5\x8f\xc6\xbd=\rp\xed\xd2k&<w\tdr\x8c2\xa1!!\xf9\xc7\xf2jk1\xfb\xd8H&\x82v\xef\x83\x95\x96\xb8\xa2\xa0M\x0f\xf9\xd1:%\xabb\xb6S\xb3\a\x98\n\xb3\xbd \xc2\xd6q(Vq6\xe3\x1c.\xa2f_,m*\x9bV\x14\x0e\xd0'F\xf7\x0c\xc8C\x12\n\xcc\xb7\x11\x92\xf4b\xb5\xeeaY\xd4r:\x8f\xbb-\xefu}\xc7\xf2\xec\xcbL8\xba\x9e`\xdcc\xebR)\xfd\xff\xc2\xee?\xb2\xb0K\xab\xd56\x8d\xeb|0\xdfd<-\xedt.\x00\xb5\x89%g\x13V\xaf\x821;\x99t\xecDl\xaf\x92\xf2\x85!}\xbc7'`b\xe2\xc3\xcd7\xf16_B\x06W\xf1\xa8\x88=MT\xaf-u7\x02\x90\xd6\x05\xb4x-\x05\xf2m\xdaW\xddt\x06-\xcbN\xeb\x11\x82`fi\x1c\xff\xa5\xf3Q\xc1\xa0\xf4\x93\xeb\xc6\xb3g9\xb6;\xa6^\xe1\x98\xa6\xd3\xf4\xf4\xb3p(V\x0f\x9d^\xcb\x98\x14\x8c\x02\x9bOs\xa8.6\xfb;r\x92\xcf;'\x8e9,2[KJKu;\x05\xa4-@F\xd2\xff\x87HZ\xf8\xa9\x1b\xfd\xd8\xbe\xac\x1d\x1e\x18\xcf\xb7\xe7\xee\xf53[\x1b\xa6\xf7\xb5\x93\xbe\x02\x8b\x94\xd8\xa8\xbbM\xb0\x9a\xc8\xe6M\xf0\x92\x9a\x13\xf4\x13=6\x95\x82\xb2\xc1\xa7g\xdc\x16\xca\xd6S\t|d\xbe\x9dJ\x1eeJ\xb9\xce\xcd\x9c\x80\xcd\x93\x0f\x1e\xd53\x90\xf9\xc4p\x0f\xe8\x05ZG\x80\x0c\xcf\xa8\x93t\xc0+\x1dg\xf9B0h\xf7\xec\xc2\xea1\xf8:\x17\xbff\xbfSN:\xe9\x89`[J\n[\x90\t\xde\x86\x84\x85^\x87`z!;\xfaI\b\xec7\xf2\xed\xaa\xad\xbcp\xb2J\x95\xdc\xeb\x00\xfa\r\xe6\x16-\xc9\xec\xa6\xb6\x8a\x81\xed\xdby\xfb\xee\xf2\xe0\xd1\x83N\xc1\x7f\r=\xfa\xf1\xce\xe4\xce\xe9\xec_r\an\x89:\r@versionI\"\x00\x06:\x06EF:\x10@created_atf\x171724238956.3441043:\x10@expires_in0:\x10@compressedT"
1724238956.917437 [0 10.222.206.152:34402] "del" "foreman:views/tabs_and_title_records-5"
1724238956.918967 [0 10.222.206.152:34402] "get" "foreman:notification-5"
1724238956.945469 [0 10.222.206.158:34962] "del" "foreman:views/tabs_and_title_records-5"
1724238956.957420 [0 10.222.206.152:34402] "del" "foreman:views/tabs_and_title_records-5"
1724238956.981609 [0 10.222.206.152:34402] "del" "foreman:views/tabs_and_title_records-5"
1724238957.040475 [0 10.222.206.152:34402] "del" "foreman:views/tabs_and_title_records-5"
1724238957.042954 [0 10.222.206.158:34962] "del" "foreman:views/tabs_and_title_records-5"
1724238957.050869 [0 10.222.206.152:39018] "del" "foreman:views/tabs_and_title_records-5"
1724238957.099073 [0 10.222.206.158:34962] "del" "foreman:views/tabs_and_title_records-5"
1724238957.112681 [0 10.222.206.152:39018] "del" "foreman:views/tabs_and_title_records-5"
1724238957.119118 [0 10.222.206.158:34962] "del" "foreman:views/tabs_and_title_records-5"
1724238957.125383 [0 10.222.206.152:34402] "del" "foreman:views/tabs_and_title_records-5"

and here is a failed login:

1724238969.936165 [0 10.222.206.152:39018] "get" "foreman:failed_login_10.222.218.3"
1724238969.944990 [0 10.222.206.152:39018] "get" "foreman:webpack_foreman_vendor_js"
1724238976.235933 [0 10.222.206.158:34962] "get" "foreman:failed_login_10.222.218.3"
1724238976.328621 [0 10.222.206.158:34962] "get" "foreman:webpack_foreman_vendor_js"

In this case, logins from .152 are working, whereas those from .158 are not.

4

More information:

Server that is failing:

irb(main):023:0> pp AuthSourceLdap.all
[#<AuthSourceLdap:0x000077d9bfa02308
  id: 4,
  type: "AuthSourceLdap",
  name: "redacted.global",
  host: "ldaps.redacted.global",
  port: 636,
  account: "redacted",
  account_password: "[FILTERED]",
  base_dn: "DC=redacted,DC=global",
  attr_login: "uid",
  attr_firstname: "givenName",
  attr_lastname: "sn",
  attr_mail: "mail",
  onthefly_register: true,
  tls: true,
  created_at: Tue, 25 Jun 2024 19:50:18.295578000 UTC +00:00,
  updated_at: Thu, 15 Aug 2024 17:47:14.014593000 UTC +00:00,
  ldap_filter: "",
  attr_photo: "jpegPhoto",
  server_type: "active_directory",
  groups_base: "OU=Salt,OU=Roles,OU=Domain Groups,DC=redacted,DC=global",
  usergroup_sync: true,
  use_netgroups: false>]
=>
[#<AuthSourceLdap:0x000077d9bfa02308
  id: 4,
  type: "AuthSourceLdap",
  name: "redacted.global",
  host: "ldaps.redacted.global",
  port: 636,
  account: "redacted",
  account_password: "[FILTERED]",
  base_dn: "DC=redacted,DC=global",
  attr_login: "uid",
  attr_firstname: "givenName",
  attr_lastname: "sn",
  attr_mail: "mail",
  onthefly_register: true,
  tls: true,
  created_at: Tue, 25 Jun 2024 19:50:18.295578000 UTC +00:00,
  updated_at: Thu, 15 Aug 2024 17:47:14.014593000 UTC +00:00,
  ldap_filter: "",
  attr_photo: "jpegPhoto",
  server_type: "active_directory",
  groups_base: "OU=Salt,OU=Roles,OU=Domain Groups,DC=redacted,DC=global",
  usergroup_sync: true,
  use_netgroups: false>]

Server that is working:

irb(main):001:0> pp AuthSourceLdap.all
[#<AuthSourceLdap:0x00007882315fe5e8
  id: 4,
  type: "AuthSourceLdap",
  name: "redacted.global",
  host: "ldaps.redacted.global",
  port: 636,
  account: "redacted",
  account_password: "[FILTERED]",
  base_dn: "DC=redacted,DC=global",
  attr_login: "uid",
  attr_firstname: "givenName",
  attr_lastname: "sn",
  attr_mail: "mail",
  onthefly_register: true,
  tls: true,
  created_at: Tue, 25 Jun 2024 19:50:18.295578000 UTC +00:00,
  updated_at: Thu, 15 Aug 2024 17:47:14.014593000 UTC +00:00,
  ldap_filter: "",
  attr_photo: "jpegPhoto",
  server_type: "active_directory",
  groups_base: "OU=Salt,OU=Roles,OU=Domain Groups,DC=redacted,DC=global",
  usergroup_sync: true,
  use_netgroups: false>]
=>
[#<AuthSourceLdap:0x00007882315fe5e8
  id: 4,
  type: "AuthSourceLdap",
  name: "redacted.global",
  host: "ldaps.redacted.global",
  port: 636,
  account: "redacted",
  account_password: "[FILTERED]",
  base_dn: "DC=redacted,DC=global",
  attr_login: "uid",
  attr_firstname: "givenName",
  attr_lastname: "sn",
  attr_mail: "mail",
  onthefly_register: true,
  tls: true,
  created_at: Tue, 25 Jun 2024 19:50:18.295578000 UTC +00:00,
  updated_at: Thu, 15 Aug 2024 17:47:14.014593000 UTC +00:00,
  ldap_filter: "",
  attr_photo: "jpegPhoto",
  server_type: "active_directory",
  groups_base: "OU=Salt,OU=Roles,OU=Domain Groups,DC=redacted,DC=global",
  usergroup_sync: true,
  use_netgroups: false>]

I dont see any difference…

However. Here is the one that works:

irb(main):002:0> source_now = AuthSourceLdap.find_by_id(4)
=>
#<AuthSourceLdap:0x00007882313d7d00
...
irb(main):003:0> conn = source_now.ldap_con
=>
#<LdapFluff:0x0000788231283878
...

And the one that fails:

irb(main):001:0> source_now = AuthSourceLdap.find_by_id(4)
=>
#<AuthSourceLdap:0x00007e86cd352668
...
irb(main):002:0> conn = source_now.ldap_con
At least one field decryption failed, check ENCRYPTION_KEY
=>
#<LdapFluff:0x00007e86cd5b9000
...
5

Ah! But if I try again randomly, it works:

irb(main):003:0> conn = source_now.ldap_con
=>
#<LdapFluff:0x00007e86cd5b9000
...

Looks like another person had a very similar or same issue:

Wondering if its the same problem with encryption.rb

6

I changed the encryption key to match that of the originally installed foreman server, and now it seems the issue is gone. Ill keep spot checking over the next couple days.

irb(main):011:0> conn = source_now.ldap_con
=>
#<LdapFluff:0x00007c937c8390c8
...
irb(main):012:0> conn = source_now.ldap_con
=>
#<LdapFluff:0x00007c937c8390c8
...
irb(main):013:0> conn = source_now.ldap_con
=>
#<LdapFluff:0x00007c937c8390c8
...
irb(main):014:0> conn = source_now.ldap_con
=>
#<LdapFluff:0x00007c937c8390c8
...
irb(main):015:0> conn = source_now.ldap_con
=>
#<LdapFluff:0x00007c937c8390c8
...
irb(main):016:0> conn = source_now.ldap_con
=>
#<LdapFluff:0x00007c937c8390c8
...
irb(main):017:0> conn = source_now.ldap_con
=>
#<LdapFluff:0x00007c937c8390c8
...
irb(main):018:0> conn = source_now.ldap_con
=>
#<LdapFluff:0x00007c937c8390c8
1 Like
7

Makes sense, in the mentioned environment systems are puppet managed and so all have the same oauth settings, so it could be the problem and the solution.

8

These are all puppet installed, but not puppet managed. I used the same (or similar) foreman answer files on install. But I must have missed that one. Alright, time to start the slow process of throwing 80,000 servers at this env!

1 Like
9

Hey @Jeff_Sparrow

Can you please provide your load balancer configuration?
What are you using to load balance foremans servers? Nginx? Haproxy?

10

messaged you.