Cannot re-register a host after failed register

katello

#1

Problem:
After multiple failed register (vi the bootstrap.py script) rhel system, I cannot register again and endup with a “Completed 500 Internal Server Error in 784ms”
The host is visible in katello, interface but when i try to un-register i have the same error code
I dunno how to get rid of it? Database delete maybe? but where?

Expected outcome:
the host to be correctly registered to the katello server

Foreman and Proxy versions:
katello_repositories_version: 3.8
katello_repositories_environment: release
foreman_repositories_environment: release
foreman_repositories_version: 1.19
katello_repositories_pulp_version: 2.16
Foreman and Proxy plugin versions:

Other relevant data:

2019-01-10T17:33:57 [I|app|6372c] Processing by Katello::Api::Rhsm::CandlepinProxiesController#consumer_activate as JSON
2019-01-10T17:33:57 [I|app|6372c]   Parameters: {"facts"=>"[FILTERED]", "installedProducts"=>[{"productName"=>"Red Hat Enterprise Linux Server", "arch"=>"x86_64", "version"=>"7.6", "productId"=>"69"}], "type"=>"system", "name"=>"app.example.com", "contentTags"=>["rhel-7", "rhel-7-server"], "owner"=>"IFS", "activation_keys"=>"ak-UAT-biz-redhat7"}
2019-01-10T17:33:57 [I|app|6a9bd] Processing by Katello::Api::V2::RootController#rhsm_resource_list as JSON
2019-01-10T17:33:57 [I|app|6a9bd]   Parameters: {"root"=>{}}
2019-01-10T17:33:57 [I|app|6a9bd]   Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.8.1/app/views/katello/api/v2/root/resource_list.json.rabl within katello/api/v2/layouts/collection
2019-01-10T17:33:57 [I|app|6a9bd]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.8.1/app/views/katello/api/v2/root/resource_list.json.rabl within katello/api/v2/layouts/collection (1.4ms)
2019-01-10T17:33:57 [I|app|6a9bd] Completed 200 OK in 13ms (Views: 2.3ms | ActiveRecord: 1.4ms)
2019-01-10T17:33:57 [I|kat|6372c] GET: https://katello-a.example.com/pulp/api/v2/users/: {"content_type"=>"application/json", "accept"=>"application/json"}
Response: 200: [{"_id": {"$oid": "5bfc196968de2a30b1cc7b3d"}, "name": "admin", "roles": ["super-users"], "_ns": "users", "login": "admin", "_href": "/pulp/api/v2/users/admin/"}]
2019-01-10T17:33:57 [I|app|984dc] Processing by Katello::Api::Rhsm::CandlepinProxiesController#server_status as JSON
2019-01-10T17:33:57 [I|app|984dc] Completed 200 OK in 16ms (Views: 0.3ms | ActiveRecord: 1.5ms)
2019-01-10T17:33:57 [W|app|6372c] Candlepin consumer c8cb8058-04ce-41d0-bc6b-8994a4b3983f has already been removed
2019-01-10T17:33:57 [I|app|1e8ba] Processing by Katello::Api::V2::RootController#rhsm_resource_list as JSON
2019-01-10T17:33:57 [I|app|1e8ba]   Parameters: {"root"=>{}}
2019-01-10T17:33:57 [I|app|1e8ba]   Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.8.1/app/views/katello/api/v2/root/resource_list.json.rabl within katello/api/v2/layouts/collection
2019-01-10T17:33:57 [I|app|1e8ba]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.8.1/app/views/katello/api/v2/root/resource_list.json.rabl within katello/api/v2/layouts/collection (1.5ms)
2019-01-10T17:33:57 [I|app|1e8ba] Completed 200 OK in 13ms (Views: 2.2ms | ActiveRecord: 1.5ms)
2019-01-10T17:33:57 [E|kat|6372c] DELETE: https://katello-a.example.com/pulp/api/v2/consumers/c8cb8058-04ce-41d0-bc6b-8994a4b3983f/: {"content_type"=>"application/json", "accept"=>"application/json"}
500 Internal Server Error: {"http_request_method": "DELETE", "exception": ["OperationalError: connection aborted\n"], "error_message": "connection aborted", "_href": "/pulp/api/v2/consumers/c8cb8058-04ce-41d0-bc6b-8994a4b3983f/", "http_status": 500, "traceback": ["  File \"/usr/lib/python2.7/site-packages/django/core/handlers/base.py\", line 185, in _get_response\n    response = wrapped_callback(request, *callback_args, **callback_kwargs)\n", "  File \"/usr/lib/python2.7/site-packages/django/views/generic/base.py\", line 68, in view\n    return self.dispatch(request, *args, **kwargs)\n", "  File \"/usr/lib/python2.7/site-packages/django/views/generic/base.py\", line 88, in dispatch\n    return handler(request, *args, **kwargs)\n", "  File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py\", line 241, in _auth_decorator\n    return _verify_auth(self, operation, super_user_only, method, *args, **kwargs)\n", "  File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py\", line 195, in _verify_auth\n    value = method(self, *args, **kwargs)\n", "  File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/views/consumers.py\", line 258, in delete\n    response = manager.unregister(consumer_id)\n", "  File \"/usr/lib/python2.7/site-packages/pulp/server/managers/consumer/cud.py\", line 115, in unregister\n    agent_consumer.unregister(consumer_id)\n", "  File \"/usr/lib/python2.7/site-packages/pulp/server/managers/consumer/agent.py\", line 63, in unregister\n    tags=task_tags)\n", "  File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 471, in apply_async\n    async_result = super(Task, self).apply_async(*args, **kwargs)\n", "  File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 99, in apply_async\n    return super(PulpTask, self).apply_async(*args, **kwargs)\n", "  File \"/usr/lib/python2.7/site-packages/celery/app/task.py\", line 535, in apply_async\n    **options\n", "  File \"/usr/lib/python2.7/site-packages/celery/app/base.py\", line 737, in send_task\n    amqp.send_task_message(P, name, message, **options)\n", "  File \"/usr/lib/python2.7/site-packages/celery/app/amqp.py\", line 558, in send_task_message\n    **properties\n", "  File \"/usr/lib/python2.7/site-packages/kombu/messaging.py\", line 181, in publish\n    exchange_name, declare,\n", "  File \"/usr/lib/python2.7/site-packages/kombu/connection.py\", line 527, in _ensured\n    errback and errback(exc, 0)\n", "  File \"/usr/lib64/python2.7/contextlib.py\", line 35, in __exit__\n    self.gen.throw(type, value, traceback)\n", "  File \"/usr/lib/python2.7/site-packages/kombu/connection.py\", line 419, in _reraise_as_library_errors\n    sys.exc_info()[2])\n", "  File \"/usr/lib/python2.7/site-packages/kombu/connection.py\", line 414, in _reraise_as_library_errors\n    yield\n", "  File \"/usr/lib/python2.7/site-packages/kombu/connection.py\", line 494, in _ensured\n    return fun(*args, **kwargs)\n", "  File \"/usr/lib/python2.7/site-packages/kombu/messaging.py\", line 203, in _publish\n    mandatory=mandatory, immediate=immediate,\n", "  File \"/usr/lib/python2.7/site-packages/kombu/transport/qpid.py\", line 1138, in basic_publish\n    self._put(routing_key, message, exchange, **kwargs)\n", "  File \"/usr/lib/python2.7/site-packages/kombu/transport/qpid.py\", line 484, in _put\n    sender.close()\n", "  File \"<string>\", line 6, in close\n", "  File \"/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py\", line 1031, in close\n    self.sync(timeout=timeout)\n", "  File \"<string>\", line 6, in sync\n", "  File \"/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py\", line 1015, in sync\n    if not self._ewait(lambda: self.acked >= mno, timeout=timeout):\n", "  File \"/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py\", line 928, in _ewait\n    result = self.session._ewait(lambda: self.error or predicate(), timeout)\n", "  File \"/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py\", line 637, in _ewait\n    result = self.connection._ewait(lambda: self.error or predicate(), timeout)\n", "  File \"/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py\", line 274, in _ewait\n    self.check_error()\n", "  File \"/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py\", line 267, in check_error\n    raise e\n"]}
2019-01-10T17:33:57 [E|app|6372c] RestClient::InternalServerError: 500 Internal Server Error
Body: {"http_request_method": "DELETE", "exception": ["OperationalError: connection aborted\n"], "error_message": "connection aborted", "_href": "/pulp/api/v2/consumers/c8cb8058-04ce-41d0-bc6b-8994a4b3983f/", "http_status": 500, "traceback": ["  File \"/usr/lib/python2.7/site-packages/django/core/handlers/base.py\", line 185, in _get_response\n    response = wrapped_callback(request, *callback_args, **callback_kwargs)\n", "  File \"/usr/lib/python2.7/site-packages/django/views/generic/base.py\", line 68, in view\n    return self.dispatch(request, *args, **kwargs)\n", "  File \"/usr/lib/python2.7/site-packages/django/views/generic/base.py\", line 88, in dispatch\n    return handler(request, *args, **kwargs)\n", "  File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py\", line 241, in _auth_decorator\n    return _verify_auth(self, operation, super_user_only, method, *args, **kwargs)\n", "  File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py\", line 195, in _verify_auth\n    value = method(self, *args, **kwargs)\n", "  File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/views/consumers.py\", line 258, in delete\n    response = manager.unregister(consumer_id)\n", "  File \"/usr/lib/python2.7/site-packages/pulp/server/managers/consumer/cud.py\", line 115, in unregister\n    agent_consumer.unregister(consumer_id)\n", "  File \"/usr/lib/python2.7/site-packages/pulp/server/managers/consumer/agent.py\", line 63, in unregister\n    tags=task_tags)\n", "  File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 471, in apply_async\n    async_result = super(Task, self).apply_async(*args, **kwargs)\n", "  File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 99, in apply_async\n    return super(PulpTask, self).apply_async(*args, **kwargs)\n", "  File \"/usr/lib/python2.7/site-packages/celery/app/task.py\", line 535, in apply_async\n    **options\n", "  File \"/usr/lib/python2.7/site-packages/celery/app/base.py\", line 737, in send_task\n    amqp.send_task_message(P, name, message, **options)\n", "  File \"/usr/lib/python2.7/site-packages/celery/app/amqp.py\", line 558, in send_task_message\n    **properties\n", "  File \"/usr/lib/python2.7/site-packages/kombu/messaging.py\", line 181, in publish\n    exchange_name, declare,\n", "  File \"/usr/lib/python2.7/site-packages/kombu/connection.py\", line 527, in _ensured\n    errback and errback(exc, 0)\n", "  File \"/usr/lib64/python2.7/contextlib.py\", line 35, in __exit__\n    self.gen.throw(type, value, traceback)\n", "  File \"/usr/lib/python2.7/site-packages/kombu/connection.py\", line 419, in _reraise_as_library_errors\n    sys.exc_info()[2])\n", "  File \"/usr/lib/python2.7/site-packages/kombu/connection.py\", line 414, in _reraise_as_library_errors\n    yield\n", "  File \"/usr/lib/python2.7/site-packages/kombu/connection.py\", line 494, in _ensured\n    return fun(*args, **kwargs)\n", "  File \"/usr/lib/python2.7/site-packages/kombu/messaging.py\", line 203, in _publish\n    mandatory=mandatory, immediate=immediate,\n", "  File \"/usr/lib/python2.7/site-packages/kombu/transport/qpid.py\", line 1138, in basic_publish\n    self._put(routing_key, message, exchange, **kwargs)\n", "  File \"/usr/lib/python2.7/site-packages/kombu/transport/qpid.py\", line 484, in _put\n    sender.close()\n", "  File \"<string>\", line 6, in close\n", "  File \"/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py\", line 1031, in close\n    self.sync(timeout=timeout)\n", "  File \"<string>\", line 6, in sync\n", "  File \"/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py\", line 1015, in sync\n    if not self._ewait(lambda: self.acked >= mno, timeout=timeout):\n", "  File \"/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py\", line 928, in _ewait\n    result = self.session._ewait(lambda: self.error or predicate(), timeout)\n", "  File \"/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py\", line 637, in _ewait\n    result = self.connection._ewait(lambda: self.error or predicate(), timeout)\n", "  File \"/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py\", line 274, in _ewait\n    self.check_error()\n", "  File \"/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py\", line 267, in check_error\n    raise e\n"]}
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/abstract_response.rb:223:in `exception_with_response'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/abstract_response.rb:103:in `return!'
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.8.1/lib/runcible/base.rb:93:in `block in get_response'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:807:in `process_result'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:725:in `block in transmit'
/opt/rh/rh-ruby24/root/usr/share/ruby/net/http.rb:877:in `start'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:715:in `transmit'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:145:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:52:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/resource.rb:94:in `delete'
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.8.1/lib/runcible/base.rb:92:in `get_response'
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.8.1/lib/runcible/base.rb:72:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.8.1/lib/runcible/resources/consumer.rb:52:in `delete'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.8.1/app/services/katello/registration_manager.rb:161:in `pulp_consumer_destory'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.8.1/app/services/katello/registration_manager.rb:19:in `unregister_host'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.8.1/app/services/katello/registration_manager.rb:39:in `register_host'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.8.1/app/controllers/katello/api/rhsm/candlepin_proxies_controller.rb:241:in `consumer_activate'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.6/lib/action_controller/metal/basic_implicit_render.rb:4:in `send_action'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.6/lib/abstract_controller/base.rb:186:in `process_action'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.6/lib/action_controller/metal/rendering.rb:30:in `process_action'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.6/lib/abstract_controller/callbacks.rb:20:in `block in process_action'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.6/lib/active_support/callbacks.rb:108:in `block in run_callbacks'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.8.1/app/controllers/katello/api/rhsm/candlepin_proxies_controller.rb:35:in `repackage_message'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.6/lib/active_support/callbacks.rb:117:in `block in run_callbacks'
/usr/share/foreman/app/controllers/api/v2/base_controller.rb:163:in `disable_json_root'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.6/lib/active_support/callbacks.rb:117:in `block in run_callbacks'
/usr/share/foreman/app/controllers/concerns/application_shared.rb:15:in `set_timezone'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.6/lib/active_support/callbacks.rb:117:in `block in run_callbacks'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.6/lib/active_support/callbacks.rb:117:in `block in run_callbacks'
/usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.6/lib/active_support/callbacks.rb:117:in `block in run_callbacks'
/opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.7.1/lib/audited/sweeper.rb:14:in `around'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.6/lib/active_support/callbacks.rb:117:in `block in run_callbacks'
/opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.7.1/lib/audited/sweeper.rb:14:in `around'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.6/lib/active_support/callbacks.rb:117:in `block in run_callbacks'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.6/lib/active_support/callbacks.rb:135:in `run_callbacks'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.6/lib/abstract_controller/callbacks.rb:19:in `process_action'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.6/lib/action_controller/metal/rescue.rb:20:in `process_action'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.6/lib/action_controller/metal/instrumentation.rb:32:in `block in process_action'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.6/lib/active_support/notifications.rb:166:in `block in instrument'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.6/lib/active_support/notifications/instrumenter.rb:21:in `instrument'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.6/lib/active_support/notifications.rb:166:in `instrument'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.6/lib/action_controller/metal/instrumentation.rb:30:in `process_action'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.6/lib/action_controller/metal/params_wrapper.rb:252:in `process_action'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/railties/controller_runtime.rb:22:in `process_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.8.1/app/controllers/katello/concerns/api/api_controller.rb:39:in `process_action'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.6/lib/abstract_controller/base.rb:124:in `process'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionview-5.1.6/lib/action_view/rendering.rb:30:in `process'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.6/lib/action_controller/metal.rb:189:in `dispatch'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.6/lib/action_controller/metal.rb:253:in `dispatch'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.6/lib/action_dispatch/routing/route_set.rb:49:in `dispatch'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.6/lib/action_dispatch/routing/route_set.rb:31:in `serve'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.6/lib/action_dispatch/routing/mapper.rb:16:in `block in <class:Constraints>'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.6/lib/action_dispatch/routing/mapper.rb:46:in `serve'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.6/lib/action_dispatch/journey/router.rb:50:in `block in serve'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.6/lib/action_dispatch/journey/router.rb:33:in `each'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.6/lib/action_dispatch/journey/router.rb:33:in `serve'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.6/lib/action_dispatch/routing/route_set.rb:844:in `call'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.6/lib/rails/engine.rb:522:in `call'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.6/lib/rails/railtie.rb:185:in `public_send'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.6/lib/rails/railtie.rb:185:in `method_missing'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.6/lib/action_dispatch/routing/mapper.rb:17:in `block in <class:Constraints>'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.6/lib/action_dispatch/routing/mapper.rb:46:in `serve'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.6/lib/action_dispatch/journey/router.rb:50:in `block in serve'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.6/lib/action_dispatch/journey/router.rb:33:in `each'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.6/lib/action_dispatch/journey/router.rb:33:in `serve'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.6/lib/action_dispatch/routing/route_set.rb:844:in `call'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.6/lib/action_dispatch/middleware/static.rb:125:in `call'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.6/lib/action_dispatch/middleware/static.rb:125:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.9/lib/apipie/static_dispatcher.rb:65:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.9/lib/apipie/extractor/recorder.rb:136:in `call'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.6/lib/action_dispatch/middleware/static.rb:125:in `call'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.6/lib/action_dispatch/middleware/static.rb:125:in `call'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.6/lib/action_dispatch/middleware/static.rb:125:in `call'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.6/lib/action_dispatch/middleware/static.rb:125:in `call'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.6/lib/action_dispatch/middleware/static.rb:125:in `call'
/usr/share/foreman/lib/middleware/telemetry.rb:9:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.9/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
/usr/share/foreman/lib/middleware/catch_json_parse_errors.rb:8:in `call'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/rack-2.0.3/lib/rack/etag.rb:25:in `call'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/rack-2.0.3/lib/rack/conditional_get.rb:38:in `call'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/rack-2.0.3/lib/rack/head.rb:12:in `call'
/usr/share/foreman/lib/middleware/logging_context.rb:23:in `call'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/rack-2.0.3/lib/rack/session/abstract/id.rb:232:in `context'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/rack-2.0.3/lib/rack/session/abstract/id.rb:226:in `call'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.6/lib/action_dispatch/middleware/cookies.rb:613:in `call'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.6/lib/action_dispatch/middleware/callbacks.rb:26:in `block in call'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.6/lib/active_support/callbacks.rb:97:in `run_callbacks'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.6/lib/action_dispatch/middleware/callbacks.rb:24:in `call'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.6/lib/action_dispatch/middleware/debug_exceptions.rb:59:in `call'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.6/lib/action_dispatch/middleware/show_exceptions.rb:31:in `call'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.6/lib/rails/rack/logger.rb:36:in `call_app'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.6/lib/rails/rack/logger.rb:26:in `call'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in `call'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.6/lib/action_dispatch/middleware/remote_ip.rb:79:in `call'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.6/lib/action_dispatch/middleware/request_id.rb:25:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.8.1/lib/katello/prevent_json_parsing.rb:12:in `call'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/rack-2.0.3/lib/rack/method_override.rb:22:in `call'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/rack-2.0.3/lib/rack/runtime.rb:22:in `call'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.6/lib/active_support/cache/strategy/local_cache_middleware.rb:27:in `call'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.6/lib/action_dispatch/middleware/executor.rb:12:in `call'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.6/lib/action_dispatch/middleware/static.rb:125:in `call'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/rack-2.0.3/lib/rack/sendfile.rb:111:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/secure_headers-5.0.5/lib/secure_headers/middleware.rb:13:in `call'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.6/lib/rails/engine.rb:522:in `call'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.6/lib/rails/railtie.rb:185:in `public_send'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.6/lib/rails/railtie.rb:185:in `method_missing'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/rack-2.0.3/lib/rack/urlmap.rb:68:in `block in call'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/rack-2.0.3/lib/rack/urlmap.rb:53:in `each'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/rack-2.0.3/lib/rack/urlmap.rb:53:in `call'
/usr/share/passenger/phusion_passenger/rack/thread_handler_extension.rb:74:in `process_request'
/usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:141:in `accept_and_process_next_request'
/usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:109:in `main_loop'
/usr/share/passenger/phusion_passenger/request_handler.rb:455:in `block (3 levels) in start_threads'
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2019-01-10T17:33:57 [I|app|6372c]   Rendering text template
2019-01-10T17:33:57 [I|app|6372c]   Rendered text template (0.0ms)
2019-01-10T17:33:57 [I|app|6372c] Completed 500 Internal Server Error in 784ms (Views: 1.1ms | ActiveRecord: 29.8ms)

Thank in advance
Regards
Mike


#2

FYI i found this:
https://bugzilla.redhat.com/show_bug.cgi?id=1500059

[root@katello-a ~]# qpid-stat -q --ssl-certificate=/etc/pki/katello/qpid_client_striped.crt   -b amqps://localhost:5671 |grep c8cb8058-04ce-41d0-bc6b-8994a4b3983f
  pulp.agent.c8cb8058-04ce-41d0-bc6b-8994a4b3983f              Y                     19    19      0    8.80k  8.80k       0         0     1

i wonder if the del queue could help me?

i was lead here by the following logs:

2019-01-10 18:52:42.440 katello-a qpidd:2019-01-10 18:52:42 [Protocol] error Connection qpid.127.0.0.1:5671-127.0.0.1:55690 closed by error: Queue celery: MessageStoreImpl::store() failed: jexception 0x0803 wmgr::enqueue() threw JERR_WMGR_ENQDISCONT: Enqueued new dtok when previous enqueue returned partly completed (state ENQ_PART). (This data_tok: id=26717 state=NONE) (/builddir/build/BUILD/qpid-cpp-1.38.0/src/qpid/linearstore/MessageStoreImpl.cpp:1226)(501)
2019-01-10 18:52:42.441 katello-a qpidd:2019-01-10 18:52:42 [Protocol] error Connection qpid.127.0.0.1:5671-127.0.0.1:55690 closed by error: illegal-argument: Value for replyText is too large(320)
2019-01-10 18:52:42.441 katello-a qpidd[15130]:2019-01-10 18:52:42 [Protocol] error Connection qpid.127.0.0.1:5671-127.0.0.1:55690 closed by error: Queue celery: MessageStoreImpl::store() failed: jexception 0x0803 wmgr::enqueue() threw JERR_WMGR_ENQDISCONT: Enqueued new dtok when previous enqueue returned partly completed (state ENQ_PART). (This data_tok: id=26717 state=NONE) (/builddir/build/BUILD/qpid-cpp-1.38.0/src/qpid/linearstore/MessageStoreImpl.cpp:1226)(501)
2019-01-10 18:52:42.441 katello-a qpidd[15130]:2019-01-10 18:52:42 [Protocol] error Connection qpid.127.0.0.1:5671-127.0.0.1:55690 closed by error: illegal-argument: Value for replyText is too large(320)
2019-01-10 18:52:42.441 katello-a pulp:pulp.server.webservices.middleware.exception:ERROR: Unhandled Exception
2019-01-10 18:52:42.442 katello-a pulp:pulp.server.webservices.middleware.exception:ERROR: (15530-85440) illegal-argument: Value for replyText is too large(320)
2019-01-10 18:52:42.442 katello-a pulp:pulp.server.webservices.middleware.exception:ERROR: (15530-85440) Traceback (most recent call last):
2019-01-10 18:52:42.443 katello-a pulp:pulp.server.webservices.middleware.exception:ERROR: (15530-85440)   File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 185, in _get_response
2019-01-10 18:52:42.443 katello-a pulp:pulp.server.webservices.middleware.exception:ERROR: (15530-85440)     response = wrapped_callback(request, *callback_args, **callback_kwargs)
2019-01-10 18:52:42.443 katello-a pulp:pulp.server.webservices.middleware.exception:ERROR: (15530-85440)   File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 68, in view
2019-01-10 18:52:42.443 katello-a pulp:pulp.server.webservices.middleware.exception:ERROR: (15530-85440)     return self.dispatch(request, *args, **kwargs)
2019-01-10 18:52:42.443 katello-a pulp:pulp.server.webservices.middleware.exception:ERROR: (15530-85440)   File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 88, in dispatch
2019-01-10 18:52:42.443 katello-a pulp:pulp.server.webservices.middleware.exception:ERROR: (15530-85440)     return handler(request, *args, **kwargs)
2019-01-10 18:52:42.444 katello-a pulp:pulp.server.webservices.middleware.exception:ERROR: (15530-85440)   File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py", line 241, in _auth_decorator
2019-01-10 18:52:42.444 katello-a pulp:pulp.server.webservices.middleware.exception:ERROR: (15530-85440)     return _verify_auth(self, operation, super_user_only, method, *args, **kwargs)
2019-01-10 18:52:42.444 katello-a pulp:pulp.server.webservices.middleware.exception:ERROR: (15530-85440)   File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py", line 195, in _verify_auth
2019-01-10 18:52:42.444 katello-a pulp:pulp.server.webservices.middleware.exception:ERROR: (15530-85440)     value = method(self, *args, **kwargs)
2019-01-10 18:52:42.444 katello-a pulp:pulp.server.webservices.middleware.exception:ERROR: (15530-85440)   File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/consumers.py", line 258, in delete
2019-01-10 18:52:42.445 katello-a pulp:pulp.server.webservices.middleware.exception:ERROR: (15530-85440)     response = manager.unregister(consumer_id)
2019-01-10 18:52:42.445 katello-a pulp:pulp.server.webservices.middleware.exception:ERROR: (15530-85440)   File "/usr/lib/python2.7/site-packages/pulp/server/managers/consumer/cud.py", line 115, in unregister
2019-01-10 18:52:42.445 katello-a pulp:pulp.server.webservices.middleware.exception:ERROR: (15530-85440)     agent_consumer.unregister(consumer_id)
2019-01-10 18:52:42.445 katello-a pulp:pulp.server.webservices.middleware.exception:ERROR: (15530-85440)   File "/usr/lib/python2.7/site-packages/pulp/server/managers/consumer/agent.py", line 63, in unregister
2019-01-10 18:52:42.445 katello-a pulp:pulp.server.webservices.middleware.exception:ERROR: (15530-85440)     tags=task_tags)
2019-01-10 18:52:42.445 katello-a pulp:pulp.server.webservices.middleware.exception:ERROR: (15530-85440)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 471, in apply_async
2019-01-10 18:52:42.446 katello-a pulp:pulp.server.webservices.middleware.exception:ERROR: (15530-85440)     async_result = super(Task, self).apply_async(*args, **kwargs)
2019-01-10 18:52:42.446 katello-a pulp:pulp.server.webservices.middleware.exception:ERROR: (15530-85440)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 99, in apply_async
2019-01-10 18:52:42.446 katello-a pulp:pulp.server.webservices.middleware.exception:ERROR: (15530-85440)     return super(PulpTask, self).apply_async(*args, **kwargs)
2019-01-10 18:52:42.446 katello-a pulp:pulp.server.webservices.middleware.exception:ERROR: (15530-85440)   File "/usr/lib/python2.7/site-packages/celery/app/task.py", line 535, in apply_async
2019-01-10 18:52:42.446 katello-a pulp:pulp.server.webservices.middleware.exception:ERROR: (15530-85440)     **options
2019-01-10 18:52:42.446 katello-a pulp:pulp.server.webservices.middleware.exception:ERROR: (15530-85440)   File "/usr/lib/python2.7/site-packages/celery/app/base.py", line 737, in send_task
2019-01-10 18:52:42.447 katello-a pulp:pulp.server.webservices.middleware.exception:ERROR: (15530-85440)     amqp.send_task_message(P, name, message, **options)
2019-01-10 18:52:42.447 katello-a pulp:pulp.server.webservices.middleware.exception:ERROR: (15530-85440)   File "/usr/lib/python2.7/site-packages/celery/app/amqp.py", line 558, in send_task_message
2019-01-10 18:52:42.447 katello-a pulp:pulp.server.webservices.middleware.exception:ERROR: (15530-85440)     **properties
2019-01-10 18:52:42.447 katello-a pulp:pulp.server.webservices.middleware.exception:ERROR: (15530-85440)   File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 181, in publish
2019-01-10 18:52:42.447 katello-a pulp:pulp.server.webservices.middleware.exception:ERROR: (15530-85440)     exchange_name, declare,
2019-01-10 18:52:42.447 katello-a pulp:pulp.server.webservices.middleware.exception:ERROR: (15530-85440)   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 527, in _ensured
2019-01-10 18:52:42.448 katello-a pulp:pulp.server.webservices.middleware.exception:ERROR: (15530-85440)     errback and errback(exc, 0)
2019-01-10 18:52:42.448 katello-a pulp:pulp.server.webservices.middleware.exception:ERROR: (15530-85440)   File "/usr/lib64/python2.7/contextlib.py", line 35, in __exit__
2019-01-10 18:52:42.448 katello-a pulp:pulp.server.webservices.middleware.exception:ERROR: (15530-85440)     self.gen.throw(type, value, traceback)
2019-01-10 18:52:42.448 katello-a pulp:pulp.server.webservices.middleware.exception:ERROR: (15530-85440)   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 419, in _reraise_as_library_errors
2019-01-10 18:52:42.448 katello-a pulp:pulp.server.webservices.middleware.exception:ERROR: (15530-85440)     sys.exc_info()[2])
2019-01-10 18:52:42.448 katello-a pulp:pulp.server.webservices.middleware.exception:ERROR: (15530-85440)   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 414, in _reraise_as_library_errors
2019-01-10 18:52:42.449 katello-a pulp:pulp.server.webservices.middleware.exception:ERROR: (15530-85440)     yield
2019-01-10 18:52:42.449 katello-a pulp:pulp.server.webservices.middleware.exception:ERROR: (15530-85440)   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 494, in _ensured
2019-01-10 18:52:42.449 katello-a pulp:pulp.server.webservices.middleware.exception:ERROR: (15530-85440)     return fun(*args, **kwargs)
2019-01-10 18:52:42.449 katello-a pulp:pulp.server.webservices.middleware.exception:ERROR: (15530-85440)   File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 203, in _publish
2019-01-10 18:52:42.449 katello-a pulp:pulp.server.webservices.middleware.exception:ERROR: (15530-85440)     mandatory=mandatory, immediate=immediate,
2019-01-10 18:52:42.450 katello-a pulp:pulp.server.webservices.middleware.exception:ERROR: (15530-85440)   File "/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line 1138, in basic_publish
2019-01-10 18:52:42.450 katello-a pulp:pulp.server.webservices.middleware.exception:ERROR: (15530-85440)     self._put(routing_key, message, exchange, **kwargs)
2019-01-10 18:52:42.450 katello-a pulp:pulp.server.webservices.middleware.exception:ERROR: (15530-85440)   File "/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line 484, in _put
2019-01-10 18:52:42.450 katello-a pulp:pulp.server.webservices.middleware.exception:ERROR: (15530-85440)     sender.close()
2019-01-10 18:52:42.450 katello-a pulp:pulp.server.webservices.middleware.exception:ERROR: (15530-85440)   File "<string>", line 6, in close
2019-01-10 18:52:42.450 katello-a pulp:pulp.server.webservices.middleware.exception:ERROR: (15530-85440)   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 1031, in close
2019-01-10 18:52:42.451 katello-a pulp:pulp.server.webservices.middleware.exception:ERROR: (15530-85440)     self.sync(timeout=timeout)
2019-01-10 18:52:42.451 katello-a pulp:pulp.server.webservices.middleware.exception:ERROR: (15530-85440)   File "<string>", line 6, in sync
2019-01-10 18:52:42.451 katello-a pulp:pulp.server.webservices.middleware.exception:ERROR: (15530-85440)   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 1015, in sync
2019-01-10 18:52:42.451 katello-a pulp:pulp.server.webservices.middleware.exception:ERROR: (15530-85440)     if not self._ewait(lambda: self.acked >= mno, timeout=timeout):
2019-01-10 18:52:42.451 katello-a pulp:pulp.server.webservices.middleware.exception:ERROR: (15530-85440)   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 928, in _ewait
2019-01-10 18:52:42.451 katello-a pulp:pulp.server.webservices.middleware.exception:ERROR: (15530-85440)     result = self.session._ewait(lambda: self.error or predicate(), timeout)
2019-01-10 18:52:42.452 katello-a pulp:pulp.server.webservices.middleware.exception:ERROR: (15530-85440)   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 637, in _ewait
2019-01-10 18:52:42.452 katello-a pulp:pulp.server.webservices.middleware.exception:ERROR: (15530-85440)     result = self.connection._ewait(lambda: self.error or predicate(), timeout)
2019-01-10 18:52:42.452 katello-a pulp:pulp.server.webservices.middleware.exception:ERROR: (15530-85440)   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 274, in _ewait
2019-01-10 18:52:42.452 katello-a pulp:pulp.server.webservices.middleware.exception:ERROR: (15530-85440)     self.check_error()
2019-01-10 18:52:42.452 katello-a pulp:pulp.server.webservices.middleware.exception:ERROR: (15530-85440)   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 267, in check_error
2019-01-10 18:52:42.452 katello-a pulp:pulp.server.webservices.middleware.exception:ERROR: (15530-85440)     raise e
2019-01-10 18:52:42.453 katello-a pulp:pulp.server.webservices.middleware.exception:ERROR: (15530-85440) OperationalError: illegal-argument: Value for replyText is too large(320)

#3

Ok you can close this, a restart of the katello server solve the issue \o/


#4

Glad you were able to work through it. Nice work!