Note: I am replacing my real FQDNs with placeholders for this.
I have four Foreman servers in production, all running 1.12.1 on RHEL6.
They all sit behind a CNAME (let's use foreman.example.com, the servers
would be foreman-0[1,4].example.com) that's loadbalanced with HA Proxy
(haproxy.example.com) in a round-robin fashion. The Foreman servers are
"clustered", extremely similar to what Chris Pisano sets up
here: Foreman :: Journey to High Availability .
If I use the standard install from running the installer, only changing the
encryption_key to match, everything works. But, it's slower than I'd like,
so I want to add memcached, which is supported through the plugin
tfm-rubygem-foreman_memcache.noarch. I also configured two memcached
servers (memcached-0[1,2].example.com). If I install and configure it on 1,
2, 3, or all 4 servers, restart httpd, I get an "Invalid Authenticity
Token" error message almost consistently when trying to log in through foreman.example.com. Logging in to each server directly (i.e. foreman-01.example.com) works and doesn't produce the error.
(Note: I verified they're all talking to my memcached servers.)
Foreman uses a few tokens to encrypt things. There are two important ones
when building a cluster of Foreman servers:
$foreman_home/config/initializers/encryption_key.rb - this is for
encrypting/decrypting passwords in the DB. This wasn't my issue.
$foreman_home/tmp/secret_token.rb - this is used to sign any cookies, and
this was my issue. Changing that to have the same key across all my
servers seems to have fixed the issue.
···
On Wednesday, August 24, 2016 at 3:24:38 PM UTC-4, Chris Baldwin wrote:
>
> Hi,
>
> Note: I am replacing my real FQDNs with placeholders for this.
>
> I have four Foreman servers in production, all running 1.12.1 on RHEL6.
> They all sit behind a CNAME (let's use foreman.example.com, the servers
> would be foreman-0[1,4].example.com) that's loadbalanced with HA Proxy (
> haproxy.example.com) in a round-robin fashion. The Foreman servers are
> "clustered", extremely similar to what Chris Pisano sets up here:
> https://theforeman.org/2015/12/journey_to_high_availability.html .
>
> If I use the standard install from running the installer, only changing
> the encryption_key to match, everything works. But, it's slower than I'd
> like, so I want to add memcached, which is supported through the plugin
> tfm-rubygem-foreman_memcache.noarch. I also configured two memcached
> servers (memcached-0[1,2].example.com). If I install and configure it on
> 1, 2, 3, or all 4 servers, restart httpd, I get an "Invalid Authenticity
> Token" error message almost consistently when trying to log in through
> foreman.example.com. Logging in to each server directly (i.e.
> foreman-01.example.com) works and doesn't produce the error.
>
> (Note: I verified they're all talking to my memcached servers.)
>
> This is very similar to
> https://groups.google.com/forum/?fromgroups#!searchin/foreman-users/authenticity|sort:relevance/foreman-users/qcPAe9n00VY/95g-GeVPBQAJ
> , but there's no solution there.
>
> Help?
>
> Error:
> *Oops, we're sorry but something went wrong *ERF42-4995
> [Foreman::Exception]: Invalid authenticity token
>
> Full trace:
> *Foreman::Exception*
> *ERF42-4995 [Foreman::Exception]: Invalid authenticity token*
> /cpos-local/usr/share/foreman/app/controllers/application_controller.rb:380:in
> `handle_unverified_request'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/action_controller/metal/request_forgery_protection.rb:204:in
> `verify_authenticity_token'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:432:in
> `block in make_lambda'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:164:in
> `call'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:164:in
> `block in halting'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:504:in
> `call'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:504:in
> `block in call'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:504:in
> `each'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:504:in
> `call'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:92:in
> `__run_callbacks__'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:778:in
> `_run_process_action_callbacks'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:81:in
> `run_callbacks'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/abstract_controller/callbacks.rb:19:in
> `process_action'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/action_controller/metal/rescue.rb:29:in
> `process_action'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/action_controller/metal/instrumentation.rb:32:in
> `block in process_action'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/notifications.rb:164:in
> `block in instrument'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/notifications/instrumenter.rb:20:in
> `instrument'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/notifications.rb:164:in
> `instrument'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/action_controller/metal/instrumentation.rb:30:in
> `process_action'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/action_controller/metal/params_wrapper.rb:250:in
> `process_action'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/railties/controller_runtime.rb:18:in
> `process_action'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/abstract_controller/base.rb:137:in
> `process'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionview-4.2.5.1/lib/action_view/rendering.rb:30:in
> `process'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/action_controller/metal.rb:196:in
> `dispatch'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/action_controller/metal/rack_delegation.rb:13:in
> `dispatch'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/action_controller/metal.rb:237:in
> `block in action'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/action_dispatch/routing/route_set.rb:74:in
> `call'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/action_dispatch/routing/route_set.rb:74:in
> `dispatch'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/action_dispatch/routing/route_set.rb:43:in
> `serve'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/action_dispatch/journey/router.rb:43:in
> `block in serve'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/action_dispatch/journey/router.rb:30:in
> `each'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/action_dispatch/journey/router.rb:30:in
> `serve'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/action_dispatch/routing/route_set.rb:815:in
> `call'
> /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.3.6/lib/apipie/static_dispatcher.rb:65:in
> `call'
> /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.3.6/lib/apipie/extractor/recorder.rb:132:in
> `call'
> /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.3.6/lib/apipie/middleware/checksum_in_headers.rb:27:in
> `call'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/rack-1.6.2/lib/rack/etag.rb:24:in
> `call'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/rack-1.6.2/lib/rack/conditionalget.rb:38:in
> `call'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/rack-1.6.2/lib/rack/head.rb:13:in
> `call'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/action_dispatch/middleware/params_parser.rb:27:in
> `call'
> /cpos-local/usr/share/foreman/lib/middleware/catch_json_parse_errors.rb:9:in
> `call'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/action_dispatch/middleware/flash.rb:260:in
> `call'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/rack-1.6.2/lib/rack/session/abstract/id.rb:225:in
> `context'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/rack-1.6.2/lib/rack/session/abstract/id.rb:220:in
> `call'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/action_dispatch/middleware/cookies.rb:560:in
> `call'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/query_cache.rb:36:in
> `call'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:653:in
> `call'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/action_dispatch/middleware/callbacks.rb:29:in
> `block in call'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:88:in
> `__run_callbacks__'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:778:in
> `_run_call_callbacks'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:81:in
> `run_callbacks'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/action_dispatch/middleware/callbacks.rb:27:in
> `call'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/action_dispatch/middleware/remote_ip.rb:78:in
> `call'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/action_dispatch/middleware/debug_exceptions.rb:17:in
> `call'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/action_dispatch/middleware/show_exceptions.rb:30:in
> `call'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/railties-4.2.5.1/lib/rails/rack/logger.rb:38:in
> `call_app'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/railties-4.2.5.1/lib/rails/rack/logger.rb:22:in
> `call'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/action_dispatch/middleware/request_id.rb:21:in
> `call'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/rack-1.6.2/lib/rack/methodoverride.rb:22:in
> `call'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/rack-1.6.2/lib/rack/runtime.rb:18:in
> `call'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/cache/strategy/local_cache_middleware.rb:28:in
> `call'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/action_dispatch/middleware/static.rb:116:in
> `call'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/rack-1.6.2/lib/rack/sendfile.rb:113:in
> `call'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/railties-4.2.5.1/lib/rails/engine.rb:518:in
> `call'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/railties-4.2.5.1/lib/rails/application.rb:165:in
> `call'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/railties-4.2.5.1/lib/rails/railtie.rb:194:in
> `public_send'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/railties-4.2.5.1/lib/rails/railtie.rb:194:in
> `method_missing'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/rack-1.6.2/lib/rack/urlmap.rb:66:in
> `block in call'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/rack-1.6.2/lib/rack/urlmap.rb:50:in
> `each'
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/rack-1.6.2/lib/rack/urlmap.rb:50:in
> `call'
> /usr/lib/ruby/gems/1.8/gems/passenger-4.0.18/lib/phusion_passenger/rack/thread_handler_extension.rb:77:in
> `process_request'
> /usr/lib/ruby/gems/1.8/gems/passenger-4.0.18/lib/phusion_passenger/request_handler/thread_handler.rb:140:in
> `accept_and_process_next_request'
> /usr/lib/ruby/gems/1.8/gems/passenger-4.0.18/lib/phusion_passenger/request_handler/thread_handler.rb:108:in
> `main_loop'
> /usr/lib/ruby/gems/1.8/gems/passenger-4.0.18/lib/phusion_passenger/request_handler.rb:441:in
> `block (3 levels) in start_threads'
> /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in
> `call'
> /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in
> `block in create_with_logging_context'
>
Oh wow, talk about a blast from the past I’m a couple years removed from
designing this, but I think I used rake security:generate_token to
regenerate the token file, added the token to an encrypted backend & the
file as a template for puppet, enforced it across my foreman cluster using
puppet + templates, and pull the value from my vault. Then restart
Foreman/httpd, and it should work.
Can anyone help to fix the below issue while generating tokens
[root@frmserver01 ~]# rake security:generate_token
rake aborted!
No Rakefile found (looking for: rakefile, Rakefile, rakefile.rb, Rakefile.rb)
/usr/local/share/gems/gems/rake-12.3.3/exe/rake:27:in `<top (required)>’
(See full trace by running task with --trace)
[root@frmserver01 ~]# foreman-rake tmp:sessions:clear
rake aborted!
Don’t know how to build task ‘tmp:sessions:clear’ (see --tasks)
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>’
(See full trace by running task with --trace)
[root@frmserver01 ~]#
Provide write permission to foreman on /usr/share/foreman/config/initializers/local_secret_token.rb file
-rw-rw---- 1 root foreman 570 Aug 5 05:46 /usr/share/foreman/config/initializers/local_secret_token.rb
and then regenerate the token :
foreman-rake security:generate_token
It started sundenlly, and I’ve already done what you’ve said (re-generated the secret token and spread it through the cluster’s nodes (3)), but no success.
I’m a couple years removed from