500 Internal Server Error when subscribing hosts

Support
1

I've been prototyping Foreman for a while now, and everything has been
working perfectly in testing. Now I've built out our production deployment
and imported content, and started trying to move hosts to the Foreman and
Katello server. When ever I run subscription-manager register, from any
host, I get 500 Internal Server Error, and the task hangs. I have to go
into the dynaflow console and skip the broken step, and resume the task to
clear locks.

I've never had this happen in any of my tests where I've built a number of
test servers to test out processes and configurations. All the test systems
were able to register clients with no issues at all. The server and client
are both fully updated RHEL 7.2 boxes, the server is
running foreman-1.12.2-1.el7.noarch and katello-3.1.0-5.el7.noarch.

The only other difference between the production and test is that I never
imported RHEL content into the test systems.

I've spent a week building out the products, repositories, content views,
etc. and would really like to recover this without having to start over,
but this is obviously a major problem!

I've googled the errors, with out finding any thing. Can anyone help me?

On the client I run:

subscription-manager register --org="myOrg"

–activationkey="rhel-72-x86_64-production"
Task 1ac48d97-a570-4a43-9a09-b5cb0be7cf29: RestClient::InternalServerError:
500 Internal Server Error

And the production.log contains (abbreviated):

2016-09-13 15:39:15 [app] [I] Started GET "/rhsm/" for XX.XX.XX.XX at
2016-09-13 15:39:15 -0500
2016-09-13 15:39:15 [app] [I] Processing by
Katello::Api::V2::RootController#rhsm_resource_list as JSON
2016-09-13 15:39:15 [app] [I] Parameters: {"root"=>{}}
2016-09-13 15:39:15 [app] [I] Rendered
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0.1/app/views/katello/api/v2/root/resource_list.json.rabl
within katello/api/v2/layouts/collection (2.4ms)
2016-09-13 15:39:15 [app] [I] Completed 200 OK in 21ms (Views: 3.8ms |
ActiveRecord: 0.0ms)
2016-09-13 15:39:15 [app] [I] Started GET "/rhsm/status" for XX.XX.XX.XX at
2016-09-13 15:39:15 -0500
2016-09-13 15:39:15 [app] [I] Processing by
Katello::Api::Rhsm::CandlepinProxiesController#server_status as JSON
2016-09-13 15:39:15 [app] [I] Completed 200 OK in 308ms (Views: 0.9ms |
ActiveRecord: 0.0ms)
2016-09-13 15:39:16 [app] [I] Started POST
"/rhsm/consumers?owner=myOrg&activation_keys=rhel-72-x86_64-production" for
XX.XX.XX.XX at 2016-09-13 15:39:16 -0500
2016-09-13 15:39:16 [app] [I] Processing by
Katello::Api::Rhsm::CandlepinProxiesController#consumer_activate as JSON
2016-09-13 15:39:16 [app] [I] Parameters: {"facts"=>"[FILTERED]",
"installedProducts"=>[{"productName"=>"Red Hat Enterprise Linux Server",
"arch"=>"x86_64", "version"=>"7.2", "productId"=>"69"}], "type"=>"system",
"name"=>"client.com", "contentTags"=>["rhel-7", "rhel-7-server"],
"owner"=>"myOrg", "activation_keys"=>"rhel-72-x86_64-production"}
2016-09-13 15:39:18 [app] [I] Import facts for 'client.example.com'
completed. Added: 0, Updated: 1, Deleted 21 facts
2016-09-13 15:39:21 [katello/pulp_rest] [E] RestClient.post
"https://foreman.example.com/pulp/api/v2/consumers/",
"{"id":"1c796228-efa5-4190-b5de-a0f7605e3074","display_name":"client.example.com"}",
"Accept"=>"/; q=0.5, application/xml", "Accept-Encoding"=>"gzip,
deflate", "Content-Length"=>"100", "accept"=>"application/json",
"content_type"=>"application/json"
> \n# => 500 InternalServerError | application/json 2551 bytes
>
2016-09-13 15:39:21 [foreman-tasks/action] [E] 500 Internal Server Error
(RestClient::InternalServerError)
>
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.8.0/lib/restclient/abstract_response.rb:74:in
return!' > /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-1.7.2/lib/runcible/base.rb:79:inblock in get_response'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.8.0/lib/restclient/request.rb:493:in
`call'

<snip>

>
/opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:77:in
block in call_job&#39; &gt; /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:333:incall'
>
/opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:333:in
run_task&#39; &gt; /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:322:inblock (3 levels) in create_worker'
>
/opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:305:in
loop&#39; &gt; /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:305:inblock (2 levels) in create_worker'
>
/opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:304:in
catch&#39; &gt; /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:304:inblock in create_worker'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in
call&#39; &gt; /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:inblock in create_with_logging_context'
2016-09-13 15:39:21 [katello/cp_proxy] [E] ForemanTasks::TaskError: Task
c837e61c-90c6-4923-81ba-d4970e1249ca: RestClient::InternalServerError: 500
Internal Server Error
>
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.18/lib/foreman_tasks.rb:40:in
block in sync_task&#39; &gt; /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.18/lib/foreman_tasks.rb:39:intap'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.18/lib/foreman_tasks.rb:39:in
sync_task&#39; &gt; /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.18/lib/foreman_tasks/triggers.rb:22:insync_task'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0.1/app/controllers/katello/api/rhsm/candlepin_proxies_controller.rb:224:in
consumer_activate&#39; &gt; /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/action_controller/metal/implicit_render.rb:4:insend_action'

<snip>

>
/usr/share/passenger/phusion_passenger/rack/thread_handler_extension.rb:74:in
process_request&#39; &gt; /usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:141:inaccept_and_process_next_request'
>
/usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:109:in
main_loop&#39; &gt; /usr/share/passenger/phusion_passenger/request_handler.rb:455:inblock
(3 levels) in start_threads'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in
call&#39; &gt; /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:inblock in create_with_logging_context'
2016-09-13 15:39:21 [app] [I] Completed 500 Internal Server Error in 5769ms
(Views: 0.4ms | ActiveRecord: 188.9ms)

2

James,

Try a couple things first:

foreman-rake katello:reindex
systemctl restart tomcat

then
'tail -f -n 0 /var/log/candlepin/* /var/log/messages/
/var/log/foreman/production.log' on your machine and retry the activation
key.
Share that output with us (if its still erroring). It will be best to pipe
this to a file and attach it as it makes it easier to parse.

Let me know if you have any questions.

Thanks,

John Mitsch
Red Hat Engineering
(860)-967-7285
irc: jomitsch

··· On Tue, Sep 13, 2016 at 4:54 PM, James Evans wrote:

I’ve been prototyping Foreman for a while now, and everything has been
working perfectly in testing. Now I’ve built out our production deployment
and imported content, and started trying to move hosts to the Foreman and
Katello server. When ever I run subscription-manager register, from any
host, I get 500 Internal Server Error, and the task hangs. I have to go
into the dynaflow console and skip the broken step, and resume the task to
clear locks.

I’ve never had this happen in any of my tests where I’ve built a number of
test servers to test out processes and configurations. All the test systems
were able to register clients with no issues at all. The server and client
are both fully updated RHEL 7.2 boxes, the server is
running foreman-1.12.2-1.el7.noarch and katello-3.1.0-5.el7.noarch.

The only other difference between the production and test is that I never
imported RHEL content into the test systems.

I’ve spent a week building out the products, repositories, content views,
etc. and would really like to recover this without having to start over,
but this is obviously a major problem!

I’ve googled the errors, with out finding any thing. Can anyone help me?

On the client I run:

subscription-manager register --org=“myOrg” --activationkey="rhel-72-x86_

64-production"
Task 1ac48d97-a570-4a43-9a09-b5cb0be7cf29: RestClient::InternalServerError:
500 Internal Server Error

And the production.log contains (abbreviated):

2016-09-13 15:39:15 [app] [I] Started GET “/rhsm/” for XX.XX.XX.XX at
2016-09-13 15:39:15 -0500
2016-09-13 15:39:15 [app] [I] Processing by Katello::Api::V2::
RootController#rhsm_resource_list as JSON
2016-09-13 15:39:15 [app] [I] Parameters: {“root”=>{}}
2016-09-13 15:39:15 [app] [I] Rendered /opt/theforeman/tfm/root/usr/
share/gems/gems/katello-3.1.0.1/app/views/katello/api/v2/root/resource_list.json.rabl
within katello/api/v2/layouts/collection (2.4ms)
2016-09-13 15:39:15 [app] [I] Completed 200 OK in 21ms (Views: 3.8ms |
ActiveRecord: 0.0ms)
2016-09-13 15:39:15 [app] [I] Started GET “/rhsm/status” for XX.XX.XX.XX
at 2016-09-13 15:39:15 -0500
2016-09-13 15:39:15 [app] [I] Processing by Katello::Api::Rhsm::
CandlepinProxiesController#server_status as JSON
2016-09-13 15:39:15 [app] [I] Completed 200 OK in 308ms (Views: 0.9ms |
ActiveRecord: 0.0ms)
2016-09-13 15:39:16 [app] [I] Started POST “/rhsm/consumers?owner=myOrg&
activation_keys=rhel-72-x86_64-production” for XX.XX.XX.XX at 2016-09-13
15:39:16 -0500
2016-09-13 15:39:16 [app] [I] Processing by Katello::Api::Rhsm::
CandlepinProxiesController#consumer_activate as JSON
2016-09-13 15:39:16 [app] [I] Parameters: {“facts”=>"[FILTERED]",
“installedProducts”=>[{“productName”=>“Red Hat Enterprise Linux Server”,
“arch”=>“x86_64”, “version”=>“7.2”, “productId”=>“69”}], “type”=>“system”,
“name”=>“client.com”, “contentTags”=>[“rhel-7”, “rhel-7-server”],
“owner”=>“myOrg”, “activation_keys”=>“rhel-72-x86_64-production”}
2016-09-13 15:39:18 [app] [I] Import facts for 'client.example.com’
completed. Added: 0, Updated: 1, Deleted 21 facts
2016-09-13 15:39:21 [katello/pulp_rest] [E] RestClient.post “
https://foreman.example.com/pulp/api/v2/consumers/”,
"{“id”:“1c796228-efa5-4190-b5de-a0f7605e3074”,“display_name”:“
client.example.com”}", “Accept”=>"/; q=0.5, application/xml",
“Accept-Encoding”=>“gzip, deflate”, “Content-Length”=>“100”,
“accept”=>“application/json”, “content_type”=>“application/json”

\n# => 500 InternalServerError | application/json 2551 bytes

2016-09-13 15:39:21 [foreman-tasks/action] [E] 500 Internal Server Error
(RestClient::InternalServerError)

/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.
8.0/lib/restclient/abstract_response.rb:74:in return!' /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-1.7.2/lib/runcible/base.rb:79:inblock in get_response’
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.
8.0/lib/restclient/request.rb:493:in `call’

/opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-
ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:77:in block in call_job' /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent- ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:333:incall’
/opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-
ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:333:in
run_task' /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent- ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:322:inblock (3 levels) in create_worker’
/opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-
ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:305:in
loop' /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent- ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:305:inblock (2 levels) in create_worker’
/opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-
ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:304:in
catch' /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent- ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:304:inblock in create_worker’
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/
lib/logging/diagnostic_context.rb:323:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/ lib/logging/diagnostic_context.rb:323:inblock in
create_with_logging_context’
2016-09-13 15:39:21 [katello/cp_proxy] [E] ForemanTasks::TaskError: Task
c837e61c-90c6-4923-81ba-d4970e1249ca: RestClient::InternalServerError:
500 Internal Server Error
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-
0.7.18/lib/foreman_tasks.rb:40:in block in sync_task' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks- 0.7.18/lib/foreman_tasks.rb:39:intap’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-
0.7.18/lib/foreman_tasks.rb:39:in sync_task' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks- 0.7.18/lib/foreman_tasks/triggers.rb:22:insync_task’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0.
1/app/controllers/katello/api/rhsm/candlepin_proxies_controller.rb:224:in
consumer_activate' /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4. 2.5.1/lib/action_controller/metal/implicit_render.rb:4:insend_action’

/usr/share/passenger/phusion_passenger/rack/thread_handler_extension.rb:74:in
process_request' /usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:141:inaccept_and_process_next_request’
/usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:109:in
main_loop' /usr/share/passenger/phusion_passenger/request_handler.rb:455:inblock (3 levels) in start_threads’
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/
lib/logging/diagnostic_context.rb:323:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/ lib/logging/diagnostic_context.rb:323:inblock in
create_with_logging_context’
2016-09-13 15:39:21 [app] [I] Completed 500 Internal Server Error in
5769ms (Views: 0.4ms | ActiveRecord: 188.9ms)

–
You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

3

Just to avoid any confusion, that tail command is going to be on your main
katello server

Thanks,

John Mitsch
Red Hat Engineering
(860)-967-7285
irc: jomitsch

··· On Tue, Sep 13, 2016 at 5:46 PM, John Mitsch wrote:

James,

Try a couple things first:

foreman-rake katello:reindex
systemctl restart tomcat

then
’tail -f -n 0 /var/log/candlepin/* /var/log/messages/
/var/log/foreman/production.log’ on your machine and retry the activation
key.
Share that output with us (if its still erroring). It will be best to pipe
this to a file and attach it as it makes it easier to parse.

Let me know if you have any questions.

Thanks,

John Mitsch
Red Hat Engineering
(860)-967-7285
irc: jomitsch

On Tue, Sep 13, 2016 at 4:54 PM, James Evans james.evans@suddenlink.com > wrote:

I’ve been prototyping Foreman for a while now, and everything has been
working perfectly in testing. Now I’ve built out our production deployment
and imported content, and started trying to move hosts to the Foreman and
Katello server. When ever I run subscription-manager register, from any
host, I get 500 Internal Server Error, and the task hangs. I have to go
into the dynaflow console and skip the broken step, and resume the task to
clear locks.

I’ve never had this happen in any of my tests where I’ve built a number
of test servers to test out processes and configurations. All the test
systems were able to register clients with no issues at all. The server and
client are both fully updated RHEL 7.2 boxes, the server is
running foreman-1.12.2-1.el7.noarch and katello-3.1.0-5.el7.noarch.

The only other difference between the production and test is that I never
imported RHEL content into the test systems.

I’ve spent a week building out the products, repositories, content views,
etc. and would really like to recover this without having to start over,
but this is obviously a major problem!

I’ve googled the errors, with out finding any thing. Can anyone help me?

On the client I run:

subscription-manager register --org=“myOrg”

–activationkey="rhel-72-x86_64-production"
Task 1ac48d97-a570-4a43-9a09-b5cb0be7cf29: RestClient::InternalServerError:
500 Internal Server Error

And the production.log contains (abbreviated):

2016-09-13 15:39:15 [app] [I] Started GET “/rhsm/” for XX.XX.XX.XX at
2016-09-13 15:39:15 -0500
2016-09-13 15:39:15 [app] [I] Processing by Katello::Api::V2::RootController#rhsm_resource_list
as JSON
2016-09-13 15:39:15 [app] [I] Parameters: {“root”=>{}}
2016-09-13 15:39:15 [app] [I] Rendered /opt/theforeman/tfm/root/usr/s
hare/gems/gems/katello-3.1.0.1/app/views/katello/api/v2/root/resource_list.json.rabl
within katello/api/v2/layouts/collection (2.4ms)
2016-09-13 15:39:15 [app] [I] Completed 200 OK in 21ms (Views: 3.8ms |
ActiveRecord: 0.0ms)
2016-09-13 15:39:15 [app] [I] Started GET “/rhsm/status” for XX.XX.XX.XX
at 2016-09-13 15:39:15 -0500
2016-09-13 15:39:15 [app] [I] Processing by Katello::Api::Rhsm::CandlepinProxiesController#server_status
as JSON
2016-09-13 15:39:15 [app] [I] Completed 200 OK in 308ms (Views: 0.9ms |
ActiveRecord: 0.0ms)
2016-09-13 15:39:16 [app] [I] Started POST “/rhsm/consumers?owner=myOrg&a
ctivation_keys=rhel-72-x86_64-production” for XX.XX.XX.XX at 2016-09-13
15:39:16 -0500
2016-09-13 15:39:16 [app] [I] Processing by Katello::Api::Rhsm::CandlepinP
roxiesController#consumer_activate as JSON
2016-09-13 15:39:16 [app] [I] Parameters: {“facts”=>"[FILTERED]",
“installedProducts”=>[{“productName”=>“Red Hat Enterprise Linux Server”,
“arch”=>“x86_64”, “version”=>“7.2”, “productId”=>“69”}], “type”=>“system”,
“name”=>“client.com”, “contentTags”=>[“rhel-7”, “rhel-7-server”],
“owner”=>“myOrg”, “activation_keys”=>“rhel-72-x86_64-production”}
2016-09-13 15:39:18 [app] [I] Import facts for 'client.example.com’
completed. Added: 0, Updated: 1, Deleted 21 facts
2016-09-13 15:39:21 [katello/pulp_rest] [E] RestClient.post “
https://foreman.example.com/pulp/api/v2/consumers/”,
"{“id”:“1c796228-efa5-4190-b5de-a0f7605e3074”,“display_name”:“
client.example.com”}", “Accept”=>"/; q=0.5, application/xml",
“Accept-Encoding”=>“gzip, deflate”, “Content-Length”=>“100”,
“accept”=>“application/json”, “content_type”=>“application/json”

\n# => 500 InternalServerError | application/json 2551 bytes

2016-09-13 15:39:21 [foreman-tasks/action] [E] 500 Internal Server Error
(RestClient::InternalServerError)

/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.8
.0/lib/restclient/abstract_response.rb:74:in return!' /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-1.7.2/lib/runcible/base.rb:79:inblock in get_response’
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.8
.0/lib/restclient/request.rb:493:in `call’

/opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-
1.0.0/lib/concurrent/executor/serialized_execution.rb:77:in block in call_job' /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby- 1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:333:incall’
/opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-
1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:333:in
run_task' /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby- 1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:322:inblock
(3 levels) in create_worker’
/opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-
1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:305:in loop' /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby- 1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:305:inblock
(2 levels) in create_worker’
/opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-
1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:304:in catch' /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby- 1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:304:inblock
in create_worker’
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/l
ib/logging/diagnostic_context.rb:323:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/l ib/logging/diagnostic_context.rb:323:inblock in
create_with_logging_context’
2016-09-13 15:39:21 [katello/cp_proxy] [E] ForemanTasks::TaskError: Task
c837e61c-90c6-4923-81ba-d4970e1249ca: RestClient::InternalServerError:
500 Internal Server Error
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0
.7.18/lib/foreman_tasks.rb:40:in block in sync_task' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0 .7.18/lib/foreman_tasks.rb:39:intap’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0
.7.18/lib/foreman_tasks.rb:39:in sync_task' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0 .7.18/lib/foreman_tasks/triggers.rb:22:insync_task’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0.1
/app/controllers/katello/api/rhsm/candlepin_proxies_controller.rb:224:in
consumer_activate' /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2. 5.1/lib/action_controller/metal/implicit_render.rb:4:insend_action’

/usr/share/passenger/phusion_passenger/rack/thread_handler_extension.rb:74:in
process_request' /usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:141:inaccept_and_process_next_request’
/usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:109:in
main_loop' /usr/share/passenger/phusion_passenger/request_handler.rb:455:inblock (3 levels) in start_threads’
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/l
ib/logging/diagnostic_context.rb:323:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/l ib/logging/diagnostic_context.rb:323:inblock in
create_with_logging_context’
2016-09-13 15:39:21 [app] [I] Completed 500 Internal Server Error in
5769ms (Views: 0.4ms | ActiveRecord: 188.9ms)

–
You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

4

Thank you for the response. I've seen several references to
katello:reindex, but I don't seem to have that task on my production
machine:

foreman-rake katello:reindex

rake aborted!
Don't know how to build task 'katello:reindex'

(See full trace by running task with --trace)

Interestingly enough, I do seem to have the task on my test server. This is
pretty confusing as I scripted out the install on both machines, so they
should be identical.
A quick look at rpm -qVa doesn't so anything obvious, what package provides
this task?

james

··· On Tuesday, September 13, 2016 at 4:46:24 PM UTC-5, John Mitsch wrote: > > James, > > Try a couple things first: > > foreman-rake katello:reindex > systemctl restart tomcat > > then > 'tail -f -n 0 /var/log/candlepin/* /var/log/messages/ > /var/log/foreman/production.log' on your machine and retry the activation > key. > Share that output with us (if its still erroring). It will be best to pipe > this to a file and attach it as it makes it easier to parse. > > Let me know if you have any questions. > > Thanks, > > John Mitsch > Red Hat Engineering > (860)-967-7285 > irc: jomitsch > > On Tue, Sep 13, 2016 at 4:54 PM, James Evans > wrote: > >> I've been prototyping Foreman for a while now, and everything has been >> working perfectly in testing. Now I've built out our production deployment >> and imported content, and started trying to move hosts to the Foreman and >> Katello server. When ever I run subscription-manager register, from any >> host, I get 500 Internal Server Error, and the task hangs. I have to go >> into the dynaflow console and skip the broken step, and resume the task to >> clear locks. >> >> I've never had this happen in any of my tests where I've built a number >> of test servers to test out processes and configurations. All the test >> systems were able to register clients with no issues at all. The server and >> client are both fully updated RHEL 7.2 boxes, the server is >> running foreman-1.12.2-1.el7.noarch and katello-3.1.0-5.el7.noarch. >> >> The only other difference between the production and test is that I never >> imported RHEL content into the test systems. >> >> I've spent a week building out the products, repositories, content views, >> etc. and would really like to recover this without having to start over, >> but this is obviously a major problem! >> >> I've googled the errors, with out finding any thing. Can anyone help me? >> >> On the client I run: >> >> # subscription-manager register --org="myOrg" >> --activationkey="rhel-72-x86_64-production" >> Task 1ac48d97-a570-4a43-9a09-b5cb0be7cf29: >> RestClient::InternalServerError: 500 Internal Server Error >> >> And the production.log contains (abbreviated): >> >> 2016-09-13 15:39:15 [app] [I] Started GET "/rhsm/" for XX.XX.XX.XX at >> 2016-09-13 15:39:15 -0500 >> 2016-09-13 15:39:15 [app] [I] Processing by >> Katello::Api::V2::RootController#rhsm_resource_list as JSON >> 2016-09-13 15:39:15 [app] [I] Parameters: {"root"=>{}} >> 2016-09-13 15:39:15 [app] [I] Rendered >> /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0.1/app/views/katello/api/v2/root/resource_list.json.rabl >> within katello/api/v2/layouts/collection (2.4ms) >> 2016-09-13 15:39:15 [app] [I] Completed 200 OK in 21ms (Views: 3.8ms | >> ActiveRecord: 0.0ms) >> 2016-09-13 15:39:15 [app] [I] Started GET "/rhsm/status" for XX.XX.XX.XX >> at 2016-09-13 15:39:15 -0500 >> 2016-09-13 15:39:15 [app] [I] Processing by >> Katello::Api::Rhsm::CandlepinProxiesController#server_status as JSON >> 2016-09-13 15:39:15 [app] [I] Completed 200 OK in 308ms (Views: 0.9ms | >> ActiveRecord: 0.0ms) >> 2016-09-13 15:39:16 [app] [I] Started POST >> "/rhsm/consumers?owner=myOrg&activation_keys=rhel-72-x86_64-production" for >> XX.XX.XX.XX at 2016-09-13 15:39:16 -0500 >> 2016-09-13 15:39:16 [app] [I] Processing by >> Katello::Api::Rhsm::CandlepinProxiesController#consumer_activate as JSON >> 2016-09-13 15:39:16 [app] [I] Parameters: {"facts"=>"[FILTERED]", >> "installedProducts"=>[{"productName"=>"Red Hat Enterprise Linux Server", >> "arch"=>"x86_64", "version"=>"7.2", "productId"=>"69"}], "type"=>"system", >> "name"=>"client.com", "contentTags"=>["rhel-7", "rhel-7-server"], >> "owner"=>"myOrg", "activation_keys"=>"rhel-72-x86_64-production"} >> 2016-09-13 15:39:18 [app] [I] Import facts for 'client.example.com' >> completed. Added: 0, Updated: 1, Deleted 21 facts >> 2016-09-13 15:39:21 [katello/pulp_rest] [E] RestClient.post " >> https://foreman.example.com/pulp/api/v2/consumers/", >> "{\"id\":\"1c796228-efa5-4190-b5de-a0f7605e3074\",\"display_name\":\" >> client.example.com\"}", "Accept"=>"*/*; q=0.5, application/xml", >> "Accept-Encoding"=>"gzip, deflate", "Content-Length"=>"100", >> "accept"=>"application/json", "content_type"=>"application/json" >> > \n# => 500 InternalServerError | application/json 2551 bytes >> > >> 2016-09-13 15:39:21 [foreman-tasks/action] [E] 500 Internal Server Error >> (RestClient::InternalServerError) >> > >> /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.8.0/lib/restclient/abstract_response.rb:74:in >> `return!' >> > >> /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-1.7.2/lib/runcible/base.rb:79:in >> `block in get_response' >> > >> /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.8.0/lib/restclient/request.rb:493:in >> `call' >> >> >> >> > >> /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:77:in >> `block in call_job' >> > >> /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:333:in >> `call' >> > >> /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:333:in >> `run_task' >> > >> /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:322:in >> `block (3 levels) in create_worker' >> > >> /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:305:in >> `loop' >> > >> /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:305:in >> `block (2 levels) in create_worker' >> > >> /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:304:in >> `catch' >> > >> /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:304:in >> `block in create_worker' >> > >> /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in >> `call' >> > >> /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in >> `block in create_with_logging_context' >> 2016-09-13 15:39:21 [katello/cp_proxy] [E] ForemanTasks::TaskError: Task >> c837e61c-90c6-4923-81ba-d4970e1249ca: RestClient::InternalServerError: 500 >> Internal Server Error >> > >> /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.18/lib/foreman_tasks.rb:40:in >> `block in sync_task' >> > >> /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.18/lib/foreman_tasks.rb:39:in >> `tap' >> > >> /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.18/lib/foreman_tasks.rb:39:in >> `sync_task' >> > >> /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.18/lib/foreman_tasks/triggers.rb:22:in >> `sync_task' >> > >> /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0.1/app/controllers/katello/api/rhsm/candlepin_proxies_controller.rb:224:in >> `consumer_activate' >> > >> /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/action_controller/metal/implicit_render.rb:4:in >> `send_action' >> >> >> >> > >> /usr/share/passenger/phusion_passenger/rack/thread_handler_extension.rb:74:in >> `process_request' >> > >> /usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:141:in >> `accept_and_process_next_request' >> > >> /usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:109:in >> `main_loop' >> > /usr/share/passenger/phusion_passenger/request_handler.rb:455:in >> `block (3 levels) in start_threads' >> > >> /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in >> `call' >> > >> /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in >> `block in create_with_logging_context' >> 2016-09-13 15:39:21 [app] [I] Completed 500 Internal Server Error in >> 5769ms (Views: 0.4ms | ActiveRecord: 188.9ms) >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Foreman users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to foreman-user...@googlegroups.com . >> To post to this group, send email to forema...@googlegroups.com >> . >> Visit this group at https://groups.google.com/group/foreman-users. >> For more options, visit https://groups.google.com/d/optout. >> > >
5

I ran foreman-rake katello:reimport as that seems to be the replacement for
the functionality of katello:reindex, restarted all katello services, and
tried again. It's still failing. I've attached the requested log file.

Thanks,

james

··· On Tuesday, September 13, 2016 at 4:46:24 PM UTC-5, John Mitsch wrote: > > James, > > Try a couple things first: > > foreman-rake katello:reindex > systemctl restart tomcat > > then > 'tail -f -n 0 /var/log/candlepin/* /var/log/messages/ > /var/log/foreman/production.log' on your machine and retry the activation > key. > Share that output with us (if its still erroring). It will be best to pipe > this to a file and attach it as it makes it easier to parse. > > Let me know if you have any questions. > > Thanks, > > John Mitsch > Red Hat Engineering > (860)-967-7285 > irc: jomitsch > > On Tue, Sep 13, 2016 at 4:54 PM, James Evans > wrote: > >> I've been prototyping Foreman for a while now, and everything has been >> working perfectly in testing. Now I've built out our production deployment >> and imported content, and started trying to move hosts to the Foreman and >> Katello server. When ever I run subscription-manager register, from any >> host, I get 500 Internal Server Error, and the task hangs. I have to go >> into the dynaflow console and skip the broken step, and resume the task to >> clear locks. >> >> I've never had this happen in any of my tests where I've built a number >> of test servers to test out processes and configurations. All the test >> systems were able to register clients with no issues at all. The server and >> client are both fully updated RHEL 7.2 boxes, the server is >> running foreman-1.12.2-1.el7.noarch and katello-3.1.0-5.el7.noarch. >> >> The only other difference between the production and test is that I never >> imported RHEL content into the test systems. >> >> I've spent a week building out the products, repositories, content views, >> etc. and would really like to recover this without having to start over, >> but this is obviously a major problem! >> >> I've googled the errors, with out finding any thing. Can anyone help me? >> >> On the client I run: >> >> # subscription-manager register --org="myOrg" >> --activationkey="rhel-72-x86_64-production" >> Task 1ac48d97-a570-4a43-9a09-b5cb0be7cf29: >> RestClient::InternalServerError: 500 Internal Server Error >> >> And the production.log contains (abbreviated): >> >> 2016-09-13 15:39:15 [app] [I] Started GET "/rhsm/" for XX.XX.XX.XX at >> 2016-09-13 15:39:15 -0500 >> 2016-09-13 15:39:15 [app] [I] Processing by >> Katello::Api::V2::RootController#rhsm_resource_list as JSON >> 2016-09-13 15:39:15 [app] [I] Parameters: {"root"=>{}} >> 2016-09-13 15:39:15 [app] [I] Rendered >> /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0.1/app/views/katello/api/v2/root/resource_list.json.rabl >> within katello/api/v2/layouts/collection (2.4ms) >> 2016-09-13 15:39:15 [app] [I] Completed 200 OK in 21ms (Views: 3.8ms | >> ActiveRecord: 0.0ms) >> 2016-09-13 15:39:15 [app] [I] Started GET "/rhsm/status" for XX.XX.XX.XX >> at 2016-09-13 15:39:15 -0500 >> 2016-09-13 15:39:15 [app] [I] Processing by >> Katello::Api::Rhsm::CandlepinProxiesController#server_status as JSON >> 2016-09-13 15:39:15 [app] [I] Completed 200 OK in 308ms (Views: 0.9ms | >> ActiveRecord: 0.0ms) >> 2016-09-13 15:39:16 [app] [I] Started POST >> "/rhsm/consumers?owner=myOrg&activation_keys=rhel-72-x86_64-production" for >> XX.XX.XX.XX at 2016-09-13 15:39:16 -0500 >> 2016-09-13 15:39:16 [app] [I] Processing by >> Katello::Api::Rhsm::CandlepinProxiesController#consumer_activate as JSON >> 2016-09-13 15:39:16 [app] [I] Parameters: {"facts"=>"[FILTERED]", >> "installedProducts"=>[{"productName"=>"Red Hat Enterprise Linux Server", >> "arch"=>"x86_64", "version"=>"7.2", "productId"=>"69"}], "type"=>"system", >> "name"=>"client.com", "contentTags"=>["rhel-7", "rhel-7-server"], >> "owner"=>"myOrg", "activation_keys"=>"rhel-72-x86_64-production"} >> 2016-09-13 15:39:18 [app] [I] Import facts for 'client.example.com' >> completed. Added: 0, Updated: 1, Deleted 21 facts >> 2016-09-13 15:39:21 [katello/pulp_rest] [E] RestClient.post " >> https://foreman.example.com/pulp/api/v2/consumers/", >> "{\"id\":\"1c796228-efa5-4190-b5de-a0f7605e3074\",\"display_name\":\" >> client.example.com\"}", "Accept"=>"*/*; q=0.5, application/xml", >> "Accept-Encoding"=>"gzip, deflate", "Content-Length"=>"100", >> "accept"=>"application/json", "content_type"=>"application/json" >> > \n# => 500 InternalServerError | application/json 2551 bytes >> > >> 2016-09-13 15:39:21 [foreman-tasks/action] [E] 500 Internal Server Error >> (RestClient::InternalServerError) >> > >> /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.8.0/lib/restclient/abstract_response.rb:74:in >> `return!' >> > >> /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-1.7.2/lib/runcible/base.rb:79:in >> `block in get_response' >> > >> /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.8.0/lib/restclient/request.rb:493:in >> `call' >> >> >> >> > >> /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:77:in >> `block in call_job' >> > >> /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:333:in >> `call' >> > >> /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:333:in >> `run_task' >> > >> /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:322:in >> `block (3 levels) in create_worker' >> > >> /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:305:in >> `loop' >> > >> /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:305:in >> `block (2 levels) in create_worker' >> > >> /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:304:in >> `catch' >> > >> /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:304:in >> `block in create_worker' >> > >> /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in >> `call' >> > >> /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in >> `block in create_with_logging_context' >> 2016-09-13 15:39:21 [katello/cp_proxy] [E] ForemanTasks::TaskError: Task >> c837e61c-90c6-4923-81ba-d4970e1249ca: RestClient::InternalServerError: 500 >> Internal Server Error >> > >> /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.18/lib/foreman_tasks.rb:40:in >> `block in sync_task' >> > >> /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.18/lib/foreman_tasks.rb:39:in >> `tap' >> > >> /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.18/lib/foreman_tasks.rb:39:in >> `sync_task' >> > >> /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.18/lib/foreman_tasks/triggers.rb:22:in >> `sync_task' >> > >> /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0.1/app/controllers/katello/api/rhsm/candlepin_proxies_controller.rb:224:in >> `consumer_activate' >> > >> /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/action_controller/metal/implicit_render.rb:4:in >> `send_action' >> >> >> >> > >> /usr/share/passenger/phusion_passenger/rack/thread_handler_extension.rb:74:in >> `process_request' >> > >> /usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:141:in >> `accept_and_process_next_request' >> > >> /usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:109:in >> `main_loop' >> > /usr/share/passenger/phusion_passenger/request_handler.rb:455:in >> `block (3 levels) in start_threads' >> > >> /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in >> `call' >> > >> /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in >> `block in create_with_logging_context' >> 2016-09-13 15:39:21 [app] [I] Completed 500 Internal Server Error in >> 5769ms (Views: 0.4ms | ActiveRecord: 188.9ms) >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Foreman users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to foreman-user...@googlegroups.com . >> To post to this group, send email to forema...@googlegroups.com >> . >> Visit this group at https://groups.google.com/group/foreman-users. >> For more options, visit https://groups.google.com/d/optout. >> > >
6

One difference between the test and production instances I forgot to
mention. The test is running katello-3.1.0-4.rc2.el7.noarch while the
production is running katello-3.1.0-5.el7.noarch.
I noticed that the test instance has a katello:reindex while the production
has katello:reimport

james

7

> I ran foreman-rake katello:reimport as that seems to be the
> replacement for the functionality of katello:reindex, restarted all
> katello services, and tried again. It's still failing. I've attached
> the requested log file.

I recognize an error:

Error opening CA Certificate /etc/pki/pulp/ca.crt
140466849802144:error:02001002:system library:fopen:No such file or
directory:bss_file.c:398:fopen('/etc/pki/pulp/ca.crt','r')
140466849802144:error:20074002:BIO routines:FILE_CTRL:system
lib:bss_file.c:400:
unable to load certificate
unable to write 'random state'

can you try running:

pulp-gen-ca-certificate
katello-service restart

-Justin

··· On 09/14/2016 10:17 AM, James Evans wrote:

Thanks,

james

On Tuesday, September 13, 2016 at 4:46:24 PM UTC-5, John Mitsch wrote:

James,

Try a couple things first:

foreman-rake katello:reindex
systemctl restart tomcat

then
 'tail -f -n 0 /var/log/candlepin/* /var/log/messages/
/var/log/foreman/production.log' on your machine and retry the
activation key.
Share that output with us (if its still erroring). It will be best
to pipe this to a file and attach it as it makes it easier to parse.

Let me know if you have any questions.

Thanks,

John Mitsch
Red Hat Engineering
(860)-967-7285
irc: jomitsch

On Tue, Sep 13, 2016 at 4:54 PM, James Evans >     <james...@suddenlink.com <javascript:>> wrote:

    I've been prototyping Foreman for a while now, and everything
    has been working perfectly in testing. Now I've built out our
    production deployment and imported content, and started trying
    to move hosts to the Foreman and Katello server. When ever I
    run subscription-manager register, from any host, I get 500
    Internal Server Error, and the task hangs. I have to go into
    the dynaflow console and skip the broken step, and resume the
    task to clear locks.

    I've never had this happen in any of my tests where I've built
    a number of test servers to test out processes and
    configurations. All the test systems were able to register
    clients with no issues at all. The server and client are both
    fully updated RHEL 7.2 boxes, the server is
    running foreman-1.12.2-1.el7.noarch
    and katello-3.1.0-5.el7.noarch.

    The only other difference between the production and test is
    that I never imported RHEL content into the test systems.

    I've spent a week building out the products, repositories,
    content views, etc. and would really like to recover this
    without having to start over, but this is obviously a major
    problem!

    I've googled the errors, with out finding any thing. Can
    anyone help me?

    On the client I run:

    # subscription-manager register --org="myOrg"
    --activationkey="rhel-72-x86_64-production"
    Task 1ac48d97-a570-4a43-9a09-b5cb0be7cf29:
    RestClient::InternalServerError: 500 Internal Server Error

    And the production.log contains (abbreviated):

    2016-09-13 15:39:15 [app] [I] Started GET "/rhsm/" for
    XX.XX.XX.XX at 2016-09-13 15:39:15 -0500
    2016-09-13 15:39:15 [app] [I] Processing by
    Katello::Api::V2::RootController#rhsm_resource_list as JSON
    2016-09-13 15:39:15 [app] [I]   Parameters: {"root"=>{}}
    2016-09-13 15:39:15 [app] [I]   Rendered
    /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0.1/app/views/katello/api/v2/root/resource_list.json.rabl
    within katello/api/v2/layouts/collection (2.4ms)
    2016-09-13 15:39:15 [app] [I] Completed 200 OK in 21ms (Views:
    3.8ms | ActiveRecord: 0.0ms)
    2016-09-13 15:39:15 [app] [I] Started GET "/rhsm/status" for
    XX.XX.XX.XX at 2016-09-13 15:39:15 -0500
    2016-09-13 15:39:15 [app] [I] Processing by
    Katello::Api::Rhsm::CandlepinProxiesController#server_status
    as JSON
    2016-09-13 15:39:15 [app] [I] Completed 200 OK in 308ms
    (Views: 0.9ms | ActiveRecord: 0.0ms)
    2016-09-13 15:39:16 [app] [I] Started POST
    "/rhsm/consumers?owner=myOrg&activation_keys=rhel-72-x86_64-production"
    for XX.XX.XX.XX at 2016-09-13 15:39:16 -0500
    2016-09-13 15:39:16 [app] [I] Processing by
    Katello::Api::Rhsm::CandlepinProxiesController#consumer_activate
    as JSON
    2016-09-13 15:39:16 [app] [I]   Parameters:
    {"facts"=>"[FILTERED]",
    "installedProducts"=>[{"productName"=>"Red Hat Enterprise
    Linux Server", "arch"=>"x86_64", "version"=>"7.2",
    "productId"=>"69"}], "type"=>"system", "name"=>"client.com
    <http://client.com>", "contentTags"=>["rhel-7",
    "rhel-7-server"], "owner"=>"myOrg",
    "activation_keys"=>"rhel-72-x86_64-production"}
    2016-09-13 15:39:18 [app] [I] Import facts for
    'client.example.com <http://client.example.com>' completed.
    Added: 0, Updated: 1, Deleted 21 facts
    2016-09-13 15:39:21 [katello/pulp_rest] [E] RestClient.post
    "https://foreman.example.com/pulp/api/v2/consumers/
    <https://foreman.example.com/pulp/api/v2/consumers/>",
    "{\"id\":\"1c796228-efa5-4190-b5de-a0f7605e3074\",\"display_name\":\"client.example.com
    <http://client.example.com>\"}", "Accept"=>"*/*; q=0.5,
    application/xml", "Accept-Encoding"=>"gzip, deflate",
    "Content-Length"=>"100", "accept"=>"application/json",
    "content_type"=>"application/json"
     > \n# => 500 InternalServerError | application/json 2551 bytes
     >
    2016-09-13 15:39:21 [foreman-tasks/action] [E] 500 Internal
    Server Error (RestClient::InternalServerError)
     >
    /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.8.0/lib/restclient/abstract_response.rb:74:in
    `return!'
     >
    /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-1.7.2/lib/runcible/base.rb:79:in
    `block in get_response'
     >
    /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.8.0/lib/restclient/request.rb:493:in
    `call'

    <snip>

     >
    /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:77:in
    `block in call_job'
     >
    /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:333:in
    `call'
     >
    /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:333:in
    `run_task'
     >
    /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:322:in
    `block (3 levels) in create_worker'
     >
    /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:305:in
    `loop'
     >
    /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:305:in
    `block (2 levels) in create_worker'
     >
    /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:304:in
    `catch'
     >
    /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:304:in
    `block in create_worker'
     >
    /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in
    `call'
     >
    /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in
    `block in create_with_logging_context'
    2016-09-13 15:39:21 [katello/cp_proxy] [E]
    ForemanTasks::TaskError: Task
    c837e61c-90c6-4923-81ba-d4970e1249ca:
    RestClient::InternalServerError: 500 Internal Server Error
     >
    /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.18/lib/foreman_tasks.rb:40:in
    `block in sync_task'
     >
    /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.18/lib/foreman_tasks.rb:39:in
    `tap'
     >
    /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.18/lib/foreman_tasks.rb:39:in
    `sync_task'
     >
    /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.18/lib/foreman_tasks/triggers.rb:22:in
    `sync_task'
     >
    /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0.1/app/controllers/katello/api/rhsm/candlepin_proxies_controller.rb:224:in
    `consumer_activate'
     >
    /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/action_controller/metal/implicit_render.rb:4:in
    `send_action'

    <snip>

     >
    /usr/share/passenger/phusion_passenger/rack/thread_handler_extension.rb:74:in
    `process_request'
     >
    /usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:141:in
    `accept_and_process_next_request'
     >
    /usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:109:in
    `main_loop'
     >
    /usr/share/passenger/phusion_passenger/request_handler.rb:455:in
    `block (3 levels) in start_threads'
     >
    /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in
    `call'
     >
    /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in
    `block in create_with_logging_context'
    2016-09-13 15:39:21 [app] [I] Completed 500 Internal Server
    Error in 5769ms (Views: 0.4ms | ActiveRecord: 188.9ms)

    -- 
    You received this message because you are subscribed to the
    Google Groups "Foreman users" group.
    To unsubscribe from this group and stop receiving emails from
    it, send an email to foreman-user...@googlegroups.com
    <javascript:>.
    To post to this group, send email to
    forema...@googlegroups.com <javascript:>.
    Visit this group at
    https://groups.google.com/group/foreman-users
    <https://groups.google.com/group/foreman-users>.
    For more options, visit https://groups.google.com/d/optout
    <https://groups.google.com/d/optout>.

–
You received this message because you are subscribed to the Google
Groups “Foreman users” group.
To unsubscribe from this group and stop receiving emails from it, send
an email to foreman-users+unsubscribe@googlegroups.com
mailto:foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com
mailto:foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

8

That seems to have done it! I can register systems now.

Thank you for your help,

james

··· On Wednesday, September 14, 2016 at 11:16:24 AM UTC-5, jsherril wrote: > > On 09/14/2016 10:17 AM, James Evans wrote: > > I ran foreman-rake katello:reimport as that seems to be the replacement > for the functionality of katello:reindex, restarted all katello services, > and tried again. It's still failing. I've attached the requested log file. > > > I recognize an error: > > > Error opening CA Certificate /etc/pki/pulp/ca.crt > 140466849802144:error:02001002:system library:fopen:No such file or > directory:bss_file.c:398:fopen('/etc/pki/pulp/ca.crt','r') > 140466849802144:error:20074002:BIO routines:FILE_CTRL:system > lib:bss_file.c:400: > unable to load certificate > unable to write 'random state' > > can you try running: > > pulp-gen-ca-certificate > katello-service restart > > > -Justin > > > Thanks, > > james > > On Tuesday, September 13, 2016 at 4:46:24 PM UTC-5, John Mitsch wrote: >> >> James, >> >> Try a couple things first: >> >> foreman-rake katello:reindex >> systemctl restart tomcat >> >> then >> 'tail -f -n 0 /var/log/candlepin/* /var/log/messages/ >> /var/log/foreman/production.log' on your machine and retry the activation >> key. >> Share that output with us (if its still erroring). It will be best to >> pipe this to a file and attach it as it makes it easier to parse. >> >> Let me know if you have any questions. >> >> Thanks, >> >> John Mitsch >> Red Hat Engineering >> (860)-967-7285 >> irc: jomitsch >> >> On Tue, Sep 13, 2016 at 4:54 PM, James Evans >> wrote: >> >>> I've been prototyping Foreman for a while now, and everything has been >>> working perfectly in testing. Now I've built out our production deployment >>> and imported content, and started trying to move hosts to the Foreman and >>> Katello server. When ever I run subscription-manager register, from any >>> host, I get 500 Internal Server Error, and the task hangs. I have to go >>> into the dynaflow console and skip the broken step, and resume the task to >>> clear locks. >>> >>> I've never had this happen in any of my tests where I've built a number >>> of test servers to test out processes and configurations. All the test >>> systems were able to register clients with no issues at all. The server and >>> client are both fully updated RHEL 7.2 boxes, the server is >>> running foreman-1.12.2-1.el7.noarch and katello-3.1.0-5.el7.noarch. >>> >>> The only other difference between the production and test is that I >>> never imported RHEL content into the test systems. >>> >>> I've spent a week building out the products, repositories, content >>> views, etc. and would really like to recover this without having to start >>> over, but this is obviously a major problem! >>> >>> I've googled the errors, with out finding any thing. Can anyone help me? >>> >>> On the client I run: >>> >>> # subscription-manager register --org="myOrg" >>> --activationkey="rhel-72-x86_64-production" >>> Task 1ac48d97-a570-4a43-9a09-b5cb0be7cf29: >>> RestClient::InternalServerError: 500 Internal Server Error >>> >>> And the production.log contains (abbreviated): >>> >>> 2016-09-13 15:39:15 [app] [I] Started GET "/rhsm/" for XX.XX.XX.XX at >>> 2016-09-13 15:39:15 -0500 >>> 2016-09-13 15:39:15 [app] [I] Processing by >>> Katello::Api::V2::RootController#rhsm_resource_list as JSON >>> 2016-09-13 15:39:15 [app] [I] Parameters: {"root"=>{}} >>> 2016-09-13 15:39:15 [app] [I] Rendered >>> /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0.1/app/views/katello/api/v2/root/resource_list.json.rabl >>> within katello/api/v2/layouts/collection (2.4ms) >>> 2016-09-13 15:39:15 [app] [I] Completed 200 OK in 21ms (Views: 3.8ms | >>> ActiveRecord: 0.0ms) >>> 2016-09-13 15:39:15 [app] [I] Started GET "/rhsm/status" for XX.XX.XX.XX >>> at 2016-09-13 15:39:15 -0500 >>> 2016-09-13 15:39:15 [app] [I] Processing by >>> Katello::Api::Rhsm::CandlepinProxiesController#server_status as JSON >>> 2016-09-13 15:39:15 [app] [I] Completed 200 OK in 308ms (Views: 0.9ms | >>> ActiveRecord: 0.0ms) >>> 2016-09-13 15:39:16 [app] [I] Started POST >>> "/rhsm/consumers?owner=myOrg&activation_keys=rhel-72-x86_64-production" for >>> XX.XX.XX.XX at 2016-09-13 15:39:16 -0500 >>> 2016-09-13 15:39:16 [app] [I] Processing by >>> Katello::Api::Rhsm::CandlepinProxiesController#consumer_activate as JSON >>> 2016-09-13 15:39:16 [app] [I] Parameters: {"facts"=>"[FILTERED]", >>> "installedProducts"=>[{"productName"=>"Red Hat Enterprise Linux Server", >>> "arch"=>"x86_64", "version"=>"7.2", "productId"=>"69"}], "type"=>"system", >>> "name"=>"client.com", "contentTags"=>["rhel-7", "rhel-7-server"], >>> "owner"=>"myOrg", "activation_keys"=>"rhel-72-x86_64-production"} >>> 2016-09-13 15:39:18 [app] [I] Import facts for 'client.example.com' >>> completed. Added: 0, Updated: 1, Deleted 21 facts >>> 2016-09-13 15:39:21 [katello/pulp_rest] [E] RestClient.post " >>> https://foreman.example.com/pulp/api/v2/consumers/", >>> "{\"id\":\"1c796228-efa5-4190-b5de-a0f7605e3074\",\"display_name\":\" >>> client.example.com\"}", "Accept"=>"*/*; q=0.5, application/xml", >>> "Accept-Encoding"=>"gzip, deflate", "Content-Length"=>"100", >>> "accept"=>"application/json", "content_type"=>"application/json" >>> > \n# => 500 InternalServerError | application/json 2551 bytes >>> > >>> 2016-09-13 15:39:21 [foreman-tasks/action] [E] 500 Internal Server Error >>> (RestClient::InternalServerError) >>> > >>> /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.8.0/lib/restclient/abstract_response.rb:74:in >>> `return!' >>> > >>> /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-1.7.2/lib/runcible/base.rb:79:in >>> `block in get_response' >>> > >>> /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.8.0/lib/restclient/request.rb:493:in >>> `call' >>> >>> >>> >>> > >>> /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:77:in >>> `block in call_job' >>> > >>> /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:333:in >>> `call' >>> > >>> /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:333:in >>> `run_task' >>> > >>> /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:322:in >>> `block (3 levels) in create_worker' >>> > >>> /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:305:in >>> `loop' >>> > >>> /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:305:in >>> `block (2 levels) in create_worker' >>> > >>> /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:304:in >>> `catch' >>> > >>> /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:304:in >>> `block in create_worker' >>> > >>> /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in >>> `call' >>> > >>> /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in >>> `block in create_with_logging_context' >>> 2016-09-13 15:39:21 [katello/cp_proxy] [E] ForemanTasks::TaskError: Task >>> c837e61c-90c6-4923-81ba-d4970e1249ca: RestClient::InternalServerError: 500 >>> Internal Server Error >>> > >>> /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.18/lib/foreman_tasks.rb:40:in >>> `block in sync_task' >>> > >>> /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.18/lib/foreman_tasks.rb:39:in >>> `tap' >>> > >>> /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.18/lib/foreman_tasks.rb:39:in >>> `sync_task' >>> > >>> /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.18/lib/foreman_tasks/triggers.rb:22:in >>> `sync_task' >>> > >>> /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0.1/app/controllers/katello/api/rhsm/candlepin_proxies_controller.rb:224:in >>> `consumer_activate' >>> > >>> /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/action_controller/metal/implicit_render.rb:4:in >>> `send_action' >>> >>> >>> >>> > >>> /usr/share/passenger/phusion_passenger/rack/thread_handler_extension.rb:74:in >>> `process_request' >>> > >>> /usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:141:in >>> `accept_and_process_next_request' >>> > >>> /usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:109:in >>> `main_loop' >>> > /usr/share/passenger/phusion_passenger/request_handler.rb:455:in >>> `block (3 levels) in start_threads' >>> > >>> /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in >>> `call' >>> > >>> /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in >>> `block in create_with_logging_context' >>> 2016-09-13 15:39:21 [app] [I] Completed 500 Internal Server Error in >>> 5769ms (Views: 0.4ms | ActiveRecord: 188.9ms) >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Foreman users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to foreman-user...@googlegroups.com. >>> To post to this group, send email to forema...@googlegroups.com. >>> Visit this group at https://groups.google.com/group/foreman-users. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- > You received this message because you are subscribed to the Google Groups > "Foreman users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to foreman-user...@googlegroups.com . > To post to this group, send email to forema...@googlegroups.com > . > Visit this group at https://groups.google.com/group/foreman-users. > For more options, visit https://groups.google.com/d/optout. > > >