Problem:
i’ve inherited a foreman+katello installation and it have some issues - candlepin and pulp related.
I upgraded it from version 1.23.x to 1.24 and during the upgrade I had this error:
Upgrade Step 3/3: katello:clean_backend_objects. This may take a long while.
Failed upgrade task: katello:clean_backend_objects, see logs for more information.
foreman-rake upgrade:run finished successfully!
Upgrade completed!
Ummm ok, I ran a foreman-rake katello:clean_backend_objects --trace and the result is:
** Invoke katello:clean_backend_objects (first_time)
** Invoke environment (first_time)
** Execute environment
** Invoke katello:check_ping (first_time)
** Invoke environment
** Execute katello:check_ping
** Execute katello:clean_backend_objects
The following changes will not actually be performed. Rerun with COMMIT=true to apply the changes
rake aborted!
RestClient::NotFound: Katello::Resources::Candlepin::Consumer: 404 Not Found {"displayMessage":"owner with key: MA was not found.","requestUuid":"7e5cc8e0-c5eb-437f-ae1a-eb2198408b10"} (GET /candlepin/consumers/?owner=MA&include=uuid&per_page=1500&page=1)
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/abstract_response.rb:223:in `exception_with_response'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/abstract_response.rb:103:in `return!'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:809:in `process_result'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:725:in `block in transmit'
/opt/rh/rh-ruby25/root/usr/share/ruby/net/http.rb:910:in `start'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:715:in `transmit'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:145:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:52:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/resource.rb:51:in `get'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/katello/http_resource.rb:88:in `get'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/katello/resources/candlepin/consumer.rb:24:in `block in get'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/katello/resources/candlepin.rb:65:in `block in fetch_paged'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/katello/resources/candlepin.rb:63:in `loop'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/katello/resources/candlepin.rb:63:in `fetch_paged'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/katello/resources/candlepin/consumer.rb:23:in `get'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/katello/resources/candlepin/consumer.rb:12:in `block in all_uuids'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/relation/delegation.rb:41:in `each'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/relation/delegation.rb:41:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/katello/resources/candlepin/consumer.rb:11:in `map'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/katello/resources/candlepin/consumer.rb:11:in `all_uuids'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/lib/katello/tasks/clean_backend_objects.rake:13:in `populate!'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/lib/katello/tasks/clean_backend_objects.rake:106:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in `block in execute'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in `each'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in `execute'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:195:in `block in invoke_with_call_chain'
/opt/rh/rh-ruby25/root/usr/share/ruby/monitor.rb:226:in `mon_synchronize'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:188:in `invoke_with_call_chain'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:181:in `invoke'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:160:in `invoke_task'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in `each'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in `block in top_level'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:125:in `run_with_threads'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:110:in `top_level'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:83:in `block in run'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:186:in `standard_exception_handling'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:80:in `run'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
/opt/rh/rh-ruby25/root/usr/bin/rake:23:in `load'
/opt/rh/rh-ruby25/root/usr/bin/rake:23:in `<main>'
Tasks: TOP => katello:clean_backend_objects
as I know the user/admin MA doesn’t exists anymore…(or it was never present).
After that I tried to run a sync, it was not synced since ages and after 5/10 minutes I had a connection refused - ServerSelectionTimeoutError: localhost:27017: [Errno 111] Connection refused, the backend had this error
Backend System Status
Component Status Message
pulp FAIL Pulp database connection issue at https://foreman.mydomain.lan/pulp/api/v2.
pulp_auth FAIL Skipped pulp_auth check after failed pulp check
hammer ping returned the same (of course)
All the sync went down/incomplete and I had to restart katello/foreman.
The previous admin told me the error was still present in 1.23…so now i’m (still) out of the blue…
The logs are below.
Thanks,
M
Expected outcome:
sync - pulp+candlepin are working
Foreman and Proxy versions:
Server Centos 7
candlepin-2.9.14-1.el7.noarch
candlepin-selinux-2.9.14-1.el7.noarch
foreman-1.24.2-1.el7.noarch
katello-3.14.1-1.el7.noarch
pulp-client-1.0-1.noarch
pulp-deb-plugins-1.10.1-1.el7.noarch
pulp-docker-plugins-3.2.6-1.el7.noarch
pulp-katello-1.0.3-1.el7.noarch
pulp-puppet-plugins-2.21.1-1.el7.noarch
pulp-puppet-tools-2.21.1-1.el7.noarch
pulp-rpm-plugins-2.21.1-1.el7.noarch
pulp-selinux-2.21.1-1.el7.noarch
pulp-server-2.21.1-1.el7.noarch
qpid-cpp-client-1.39.0-1.el7.x86_64
qpid-cpp-client-devel-1.39.0-1.el7.x86_64
qpid-cpp-server-1.39.0-1.el7.x86_64
qpid-cpp-server-linearstore-1.39.0-1.el7.x86_64
qpid-dispatch-router-1.11.0-1.el7.x86_64
qpid-proton-c-0.30.0-2.el7.x86_64
qpid-qmf-1.39.0-1.el7.x86_64
qpid-tools-1.39.0-1.el7.noarch
Foreman and Proxy plugin versions:
Distribution and version:
Centos 7
Other relevant data:
During the sync error foreman-tail printed:
foreman-tail
==> /var/log/messages <==
May 14 11:25:41 foreman pulp: kombu.async.hub:ERROR: (41168-04480) Error in timer: ServerSelectionTimeoutError('localhost:27017: [Errno 111] Connection refused',)
May 14 11:25:41 foreman pulp: kombu.async.hub:ERROR: (41168-04480) Traceback (most recent call last):
May 14 11:25:41 foreman pulp: kombu.async.hub:ERROR: (41168-04480) File "/usr/lib/python2.7/site-packages/kombu/async/hub.py", line 137, in fire_timers
May 14 11:25:41 foreman pulp: kombu.async.hub:ERROR: (41168-04480) entry()
May 14 11:25:41 foreman pulp: kombu.async.hub:ERROR: (41168-04480) File "/usr/lib/python2.7/site-packages/kombu/async/timer.py", line 68, in __call__
May 14 11:25:41 foreman pulp: kombu.async.hub:ERROR: (41171-75936) Error in timer: ServerSelectionTimeoutError('localhost:27017: [Errno 111] Connection refused',)
May 14 11:25:41 foreman pulp: kombu.async.hub:ERROR: (41168-04480) return self.fun(*self.args, **self.kwargs)
May 14 11:25:41 foreman pulp: kombu.async.hub:ERROR: (41168-04480) File "/usr/lib/python2.7/site-packages/kombu/async/timer.py", line 127, in _reschedules
May 14 11:25:41 foreman pulp: kombu.async.hub:ERROR: (41171-75936) Traceback (most recent call last):
May 14 11:25:41 foreman pulp: kombu.async.hub:ERROR: (41171-75936) File "/usr/lib/python2.7/site-packages/kombu/async/hub.py", line 137, in fire_timers
May 14 11:25:41 foreman pulp: kombu.async.hub:ERROR: (41168-04480) return fun(*args, **kwargs)
May 14 11:25:41 foreman pulp: kombu.async.hub:ERROR: (41171-75936) entry()
May 14 11:25:41 foreman pulp: kombu.async.hub:ERROR: (41168-04480) File "/usr/lib/python2.7/site-packages/pulp/server/async/app.py", line 119, in _record_heartbeat
May 14 11:25:41 foreman pulp: kombu.async.hub:ERROR: (41171-75936) File "/usr/lib/python2.7/site-packages/kombu/async/timer.py", line 68, in __call__
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41168-04480) worker_watcher.handle_worker_heartbeat(name)
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41171-75936) return self.fun(*self.args, **self.kwargs)
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41168-04480) File "/usr/lib/python2.7/site-packages/pulp/server/async/worker_watcher.py", line 39, in handle_worker_heartbeat
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41168-04480) existing_worker = Worker.objects(name=worker_name).first()
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41171-75936) File "/usr/lib/python2.7/site-packages/kombu/async/timer.py", line 127, in _reschedules
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41168-04480) File "/usr/lib/python2.7/site-packages/mongoengine/queryset/base.py", line 264, in first
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41171-75936) return fun(*args, **kwargs)
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41168-04480) result = queryset[0]
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41171-75936) File "/usr/lib/python2.7/site-packages/pulp/server/async/app.py", line 119, in _record_heartbeat
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41168-04480) File "/usr/lib/python2.7/site-packages/mongoengine/queryset/base.py", line 161, in __getitem__
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41171-75936) worker_watcher.handle_worker_heartbeat(name)
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41168-04480) return queryset._document._from_son(queryset._cursor[key],
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41171-75936) File "/usr/lib/python2.7/site-packages/pulp/server/async/worker_watcher.py", line 39, in handle_worker_heartbeat
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41168-04480) File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 565, in __getitem__
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41171-75936) existing_worker = Worker.objects(name=worker_name).first()
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41168-04480) for doc in clone:
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41171-75936) File "/usr/lib/python2.7/site-packages/mongoengine/queryset/base.py", line 264, in first
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41168-04480) File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 1097, in next
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41171-75936) result = queryset[0]
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41168-04480) if len(self.__data) or self._refresh():
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41171-75936) File "/usr/lib/python2.7/site-packages/mongoengine/queryset/base.py", line 161, in __getitem__
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41168-04480) File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 1019, in _refresh
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41171-75936) return queryset._document._from_son(queryset._cursor[key],
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41168-04480) self.__read_concern))
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41168-04480) File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 850, in __send_message
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41171-75936) File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 565, in __getitem__
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41168-04480) **kwargs)
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41171-75936) for doc in clone:
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41168-04480) File "/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py", line 777, in _send_message_with_response
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41171-75936) File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 1097, in next
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41168-04480) server = topology.select_server(selector)
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41171-75936) if len(self.__data) or self._refresh():
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41168-04480) File "/usr/lib64/python2.7/site-packages/pymongo/topology.py", line 141, in select_server
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41171-75936) File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 1019, in _refresh
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41168-04480) address))
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41171-75936) self.__read_concern))
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41168-04480) File "/usr/lib64/python2.7/site-packages/pymongo/topology.py", line 117, in select_servers
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41171-75936) File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 850, in __send_message
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41168-04480) self._error_message(selector))
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41171-75936) **kwargs)
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41168-04480) ServerSelectionTimeoutError: localhost:27017: [Errno 111] Connection refused
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41171-75936) File "/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py", line 777, in _send_message_with_response
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41171-75936) server = topology.select_server(selector)
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41171-75936) File "/usr/lib64/python2.7/site-packages/pymongo/topology.py", line 141, in select_server
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41171-75936) address))
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41171-75936) File "/usr/lib64/python2.7/site-packages/pymongo/topology.py", line 117, in select_servers
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41171-75936) self._error_message(selector))
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41171-75936) ServerSelectionTimeoutError: localhost:27017: [Errno 111] Connection refused
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41157-20736) Error in timer: ServerSelectionTimeoutError('localhost:27017: [Errno 111] Connection refused',)
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41157-20736) Traceback (most recent call last):
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41157-20736) File "/usr/lib/python2.7/site-packages/kombu/async/hub.py", line 137, in fire_timers
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41157-20736) entry()
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41157-20736) File "/usr/lib/python2.7/site-packages/kombu/async/timer.py", line 68, in __call__
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41157-20736) return self.fun(*self.args, **self.kwargs)
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41157-20736) File "/usr/lib/python2.7/site-packages/kombu/async/timer.py", line 127, in _reschedules
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41157-20736) return fun(*args, **kwargs)
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41157-20736) File "/usr/lib/python2.7/site-packages/pulp/server/async/app.py", line 119, in _record_heartbeat
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41157-20736) worker_watcher.handle_worker_heartbeat(name)
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41157-20736) File "/usr/lib/python2.7/site-packages/pulp/server/async/worker_watcher.py", line 39, in handle_worker_heartbeat
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41157-20736) existing_worker = Worker.objects(name=worker_name).first()
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41157-20736) File "/usr/lib/python2.7/site-packages/mongoengine/queryset/base.py", line 264, in first
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41157-20736) result = queryset[0]
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41157-20736) File "/usr/lib/python2.7/site-packages/mongoengine/queryset/base.py", line 161, in __getitem__
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41157-20736) return queryset._document._from_son(queryset._cursor[key],
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41157-20736) File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 565, in __getitem__
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41157-20736) for doc in clone:
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41157-20736) File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 1097, in next
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41157-20736) if len(self.__data) or self._refresh():
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41157-20736) File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 1019, in _refresh
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41157-20736) self.__read_concern))
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41157-20736) File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 850, in __send_message
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41157-20736) **kwargs)
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41157-20736) File "/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py", line 777, in _send_message_with_response
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41157-20736) server = topology.select_server(selector)
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41157-20736) File "/usr/lib64/python2.7/site-packages/pymongo/topology.py", line 141, in select_server
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41157-20736) address))
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41157-20736) File "/usr/lib64/python2.7/site-packages/pymongo/topology.py", line 117, in select_servers
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41157-20736) self._error_message(selector))
May 14 11:25:42 foreman pulp: kombu.async.hub:ERROR: (41157-20736) ServerSelectionTimeoutError: localhost:27017: [Errno 111] Connection refused
May 14 11:26:00 foreman pulp: pulp.server.async.tasks:INFO: Task failed : [8beba124-0f69-41af-a361-eeca88d9c137]
May 14 11:26:00 foreman pulp: celery.app.trace:ERROR: (41396-04480) Task pulp.server.managers.consumer.applicability.batch_regenerate_applicability[8beba124-0f69-41af-a361-eeca88d9c137] raised unexpected: ServerSelectionTimeoutError('localhost:27017: [Errno 111] Connection refused',)
May 14 11:26:00 foreman pulp: celery.app.trace:ERROR: (41396-04480) Traceback (most recent call last):
May 14 11:26:00 foreman pulp: celery.app.trace:ERROR: (41396-04480) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 381, in trace_task
May 14 11:26:00 foreman pulp: celery.app.trace:ERROR: (41396-04480) I, R, state, retval = on_error(task_request, exc, uuid)
May 14 11:26:00 foreman pulp: celery.app.trace:ERROR: (41396-04480) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 323, in on_error
May 14 11:26:00 foreman pulp: celery.app.trace:ERROR: (41396-04480) task, request, eager=eager, call_errbacks=call_errbacks,
May 14 11:26:00 foreman pulp: celery.app.trace:ERROR: (41396-04480) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 157, in handle_error_state
May 14 11:26:00 foreman pulp: celery.app.trace:ERROR: (41396-04480) call_errbacks=call_errbacks)
May 14 11:26:00 foreman pulp: celery.app.trace:ERROR: (41396-04480) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 205, in handle_failure
May 14 11:26:00 foreman pulp: celery.app.trace:ERROR: (41396-04480) task.on_failure(exc, req.id, req.args, req.kwargs, einfo)
May 14 11:26:00 foreman pulp: celery.app.trace:ERROR: (41396-04480) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 762, in on_failure
May 14 11:26:00 foreman pulp: celery.app.trace:ERROR: (41396-04480) task_status = TaskStatus.objects.get(task_id=task_id)
May 14 11:26:00 foreman pulp: celery.app.trace:ERROR: (41396-04480) File "/usr/lib/python2.7/site-packages/mongoengine/queryset/base.py", line 238, in get
May 14 11:26:00 foreman pulp: celery.app.trace:ERROR: (41396-04480) result = queryset.next()
May 14 11:26:00 foreman pulp: celery.app.trace:ERROR: (41396-04480) File "/usr/lib/python2.7/site-packages/mongoengine/queryset/base.py", line 1407, in next
May 14 11:26:00 foreman pulp: celery.app.trace:ERROR: (41396-04480) raw_doc = self._cursor.next()
May 14 11:26:00 foreman pulp: celery.app.trace:ERROR: (41396-04480) File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 1097, in next
May 14 11:26:00 foreman pulp: celery.app.trace:ERROR: (41396-04480) if len(self.__data) or self._refresh():
May 14 11:26:00 foreman pulp: celery.app.trace:ERROR: (41396-04480) File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 1019, in _refresh
May 14 11:26:00 foreman pulp: celery.app.trace:ERROR: (41396-04480) self.__read_concern))
May 14 11:26:00 foreman pulp: celery.app.trace:ERROR: (41396-04480) File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 850, in __send_message
May 14 11:26:00 foreman pulp: celery.app.trace:ERROR: (41396-04480) **kwargs)
May 14 11:26:00 foreman pulp: celery.app.trace:ERROR: (41396-04480) File "/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py", line 777, in _send_message_with_response
May 14 11:26:00 foreman pulp: celery.app.trace:ERROR: (41396-04480) server = topology.select_server(selector)
May 14 11:26:00 foreman pulp: celery.app.trace:ERROR: (41396-04480) File "/usr/lib64/python2.7/site-packages/pymongo/topology.py", line 141, in select_server
May 14 11:26:00 foreman pulp: celery.app.trace:ERROR: (41396-04480) address))
May 14 11:26:00 foreman pulp: celery.app.trace:ERROR: (41396-04480) File "/usr/lib64/python2.7/site-packages/pymongo/topology.py", line 117, in select_servers
May 14 11:26:00 foreman pulp: celery.app.trace:ERROR: (41396-04480) self._error_message(selector))
May 14 11:26:00 foreman pulp: celery.app.trace:ERROR: (41396-04480) ServerSelectionTimeoutError: localhost:27017: [Errno 111] Connection refused
May 14 11:26:06 foreman pulp: pulp.server.webservices.middleware.exception:ERROR: Unhandled Exception
May 14 11:26:06 foreman pulp: pulp.server.webservices.middleware.exception:ERROR: (41461-83552) localhost:27017: [Errno 111] Connection refused
May 14 11:26:06 foreman pulp: pulp.server.webservices.middleware.exception:ERROR: (41461-83552) Traceback (most recent call last):
May 14 11:26:06 foreman pulp: pulp.server.webservices.middleware.exception:ERROR: (41461-83552) File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 185, in _get_response
May 14 11:26:06 foreman pulp: pulp.server.webservices.middleware.exception:ERROR: (41461-83552) response = wrapped_callback(request, *callback_args, **callback_kwargs)
May 14 11:26:06 foreman pulp: pulp.server.webservices.middleware.exception:ERROR: (41461-83552) File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 68, in view
May 14 11:26:06 foreman pulp: pulp.server.webservices.middleware.exception:ERROR: (41461-83552) return self.dispatch(request, *args, **kwargs)
May 14 11:26:06 foreman pulp: pulp.server.webservices.middleware.exception:ERROR: (41461-83552) File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 88, in dispatch
May 14 11:26:06 foreman pulp: pulp.server.webservices.middleware.exception:ERROR: (41461-83552) return handler(request, *args, **kwargs)
May 14 11:26:06 foreman pulp: pulp.server.webservices.middleware.exception:ERROR: (41461-83552) File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py", line 241, in _auth_decorator
May 14 11:26:06 foreman pulp: pulp.server.webservices.middleware.exception:ERROR: (41461-83552) return _verify_auth(self, operation, super_user_only, method, *args, **kwargs)
May 14 11:26:06 foreman pulp: pulp.server.webservices.middleware.exception:ERROR: (41461-83552) File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py", line 154, in _verify_auth
May 14 11:26:06 foreman pulp: pulp.server.webservices.middleware.exception:ERROR: (41461-83552) login = authenticate_user()
May 14 11:26:06 foreman pulp: pulp.server.webservices.middleware.exception:ERROR: (41461-83552) File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py", line 30, in check_preauthenticated
May 14 11:26:06 foreman pulp: pulp.server.webservices.middleware.exception:ERROR: (41461-83552) userid = factory.authentication_manager().check_username_password(username)
May 14 11:26:06 foreman pulp: pulp.server.webservices.middleware.exception:ERROR: (41461-83552) File "/usr/lib/python2.7/site-packages/pulp/server/managers/auth/authentication.py", line 95, in check_username_password
May 14 11:26:06 foreman pulp: pulp.server.webservices.middleware.exception:ERROR: (41461-83552) user = self._check_username_password_local(username, password)
May 14 11:26:06 foreman pulp: pulp.server.webservices.middleware.exception:ERROR: (41461-83552) File "/usr/lib/python2.7/site-packages/pulp/server/managers/auth/authentication.py", line 35, in _check_username_password_local
May 14 11:26:06 foreman pulp: pulp.server.webservices.middleware.exception:ERROR: (41461-83552) user = model.User.objects(login=username).first()
May 14 11:26:06 foreman pulp: pulp.server.webservices.middleware.exception:ERROR: (41461-83552) File "/usr/lib/python2.7/site-packages/mongoengine/queryset/base.py", line 264, in first
May 14 11:26:06 foreman pulp: pulp.server.webservices.middleware.exception:ERROR: (41461-83552) result = queryset[0]
May 14 11:26:06 foreman pulp: pulp.server.webservices.middleware.exception:ERROR: (41461-83552) File "/usr/lib/python2.7/site-packages/mongoengine/queryset/base.py", line 161, in __getitem__
May 14 11:26:06 foreman pulp: pulp.server.webservices.middleware.exception:ERROR: (41461-83552) return queryset._document._from_son(queryset._cursor[key],
May 14 11:26:06 foreman pulp: pulp.server.webservices.middleware.exception:ERROR: (41461-83552) File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 565, in __getitem__
May 14 11:26:06 foreman pulp: pulp.server.webservices.middleware.exception:ERROR: (41461-83552) for doc in clone:
May 14 11:26:06 foreman pulp: pulp.server.webservices.middleware.exception:ERROR: (41461-83552) File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 1097, in next
May 14 11:26:06 foreman pulp: pulp.server.webservices.middleware.exception:ERROR: (41461-83552) if len(self.__data) or self._refresh():
May 14 11:26:06 foreman pulp: pulp.server.webservices.middleware.exception:ERROR: (41461-83552) File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 1019, in _refresh
May 14 11:26:06 foreman pulp: pulp.server.webservices.middleware.exception:ERROR: (41461-83552) self.__read_concern))
May 14 11:26:06 foreman pulp: pulp.server.webservices.middleware.exception:ERROR: (41461-83552) File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 850, in __send_message
May 14 11:26:06 foreman pulp: pulp.server.webservices.middleware.exception:ERROR: (41461-83552) **kwargs)
May 14 11:26:06 foreman pulp: pulp.server.webservices.middleware.exception:ERROR: (41461-83552) File "/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py", line 777, in _send_message_with_response
May 14 11:26:06 foreman pulp: pulp.server.webservices.middleware.exception:ERROR: (41461-83552) server = topology.select_server(selector)
May 14 11:26:06 foreman pulp: pulp.server.webservices.middleware.exception:ERROR: (41461-83552) File "/usr/lib64/python2.7/site-packages/pymongo/topology.py", line 141, in select_server
May 14 11:26:06 foreman pulp: pulp.server.webservices.middleware.exception:ERROR: (41461-83552) address))
May 14 11:26:06 foreman pulp: pulp.server.webservices.middleware.exception:ERROR: (41461-83552) File "/usr/lib64/python2.7/site-packages/pymongo/topology.py", line 117, in select_servers
May 14 11:26:06 foreman pulp: pulp.server.webservices.middleware.exception:ERROR: (41461-83552) self._error_message(selector))
May 14 11:26:06 foreman pulp: pulp.server.webservices.middleware.exception:ERROR: (41461-83552) ServerSelectionTimeoutError: localhost:27017: [Errno 111] Connection refused
==> /var/log/httpd/foreman-ssl_access_ssl.log <==
10.101.2.151 - admin [14/May/2020:11:25:36 -0400] "POST /pulp/api/v2/consumers/content/applicability/ HTTP/1.1" 500 2917 "-" "rest-client/2.0.2 (linux-gnu x86_64) ruby/2.5.5p157"
==> /var/log/foreman/production.log <==
2020-05-14T11:26:06 [E|kat|ea308bec] POST: https://foreman.myserver.lan/pulp/api/v2/consumers/content/applicability/: {"criteria":{"filters":{"id":{"$in":["26766efb-c742-447e-961c-65e198dedc01"]}}},"content_types":["modulemd"]}: {"content_type"=>"application/json", "accept"=>"application/json"}
500 Internal Server Error: {"http_request_method": "POST", "exception": ["ServerSelectionTimeoutError: localhost:27017: [Errno 111] Connection refused\n"], "error_message": "localhost:27017: [Errno 111] Connection refused", "_href": "/pulp/api/v2/consumers/content/applicability/", "http_status": 500, "traceback": [" File \"/usr/lib/python2.7/site-packages/django/core/handlers/base.py\", line 185, in _get_response\n response = wrapped_callback(request, *callback_args, **callback_kwargs)\n", " File \"/usr/lib/python2.7/site-packages/django/views/generic/base.py\", line 68, in view\n return self.dispatch(request, *args, **kwargs)\n", " File \"/usr/lib/python2.7/site-packages/django/views/generic/base.py\", line 88, in dispatch\n return handler(request, *args, **kwargs)\n", " File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py\", line 241, in _auth_decorator\n return _verify_auth(self, operation, super_user_only, method, *args, **kwargs)\n", " File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py\", line 154, in _verify_auth\n login = authenticate_user()\n", " File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py\", line 30, in check_preauthenticated\n userid = factory.authentication_manager().check_username_password(username)\n", " File \"/usr/lib/python2.7/site-packages/pulp/server/managers/auth/authentication.py\", line 95, in check_username_password\n user = self._check_username_password_local(username, password)\n", " File \"/usr/lib/python2.7/site-packages/pulp/server/managers/auth/authentication.py\", line 35, in _check_username_password_local\n user = model.User.objects(login=username).first()\n", " File \"/usr/lib/python2.7/site-packages/mongoengine/queryset/base.py\", line 264, in first\n result = queryset[0]\n", " File \"/usr/lib/python2.7/site-packages/mongoengine/queryset/base.py\", line 161, in __getitem__\n return queryset._document._from_son(queryset._cursor[key],\n", " File \"/usr/lib64/python2.7/site-packages/pymongo/cursor.py\", line 565, in __getitem__\n for doc in clone:\n", " File \"/usr/lib64/python2.7/site-packages/pymongo/cursor.py\", line 1097, in next\n if len(self.__data) or self._refresh():\n", " File \"/usr/lib64/python2.7/site-packages/pymongo/cursor.py\", line 1019, in _refresh\n self.__read_concern))\n", " File \"/usr/lib64/python2.7/site-packages/pymongo/cursor.py\", line 850, in __send_message\n **kwargs)\n", " File \"/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py\", line 777, in _send_message_with_response\n server = topology.select_server(selector)\n", " File \"/usr/lib64/python2.7/site-packages/pymongo/topology.py\", line 141, in select_server\n address))\n", " File \"/usr/lib64/python2.7/site-packages/pymongo/topology.py\", line 117, in select_servers\n self._error_message(selector))\n"]}
2020-05-14T11:26:06 [E|kat|ea308bec] event_queue_error: type=import_host_applicability, object_id=76
2020-05-14T11:26:06 [E|kat|ea308bec] 500 Internal Server Error
2020-05-14T11:26:06 [E|kat|ea308bec] /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/abstract_response.rb:223:in `exception_with_response'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/abstract_response.rb:103:in `return!'
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.12.1/lib/runcible/base.rb:95:in `block in get_response'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:807:in `process_result'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:725:in `block in transmit'
/opt/rh/rh-ruby25/root/usr/share/ruby/net/http.rb:910:in `start'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:715:in `transmit'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:145:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:52:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/resource.rb:67:in `post'
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.12.1/lib/runcible/base.rb:94:in `get_response'
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.12.1/lib/runcible/base.rb:74:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.12.1/lib/runcible/resources/consumer.rb:171:in `applicability'
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.12.1/lib/runcible/extensions/consumer.rb:174:in `applicable_for_type'
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.12.1/lib/runcible/extensions/consumer.rb:162:in `applicable_module_streams'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/services/katello/pulp/consumer.rb:36:in `applicable_ids'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/services/katello/applicable_content_helper.rb:85:in `applicable_differences'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/services/katello/applicable_content_helper.rb:11:in `import'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/models/katello/host/content_facet.rb:137:in `import_module_stream_applicability'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/models/katello/host/content_facet.rb:108:in `import_applicability'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/object/try.rb:19:in `public_send'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/object/try.rb:19:in `try!'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/object/try.rb:8:in `try'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/models/katello/events/import_host_applicability.rb:12:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/services/katello/event_monitor/poller_thread.rb:67:in `block in run_event'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:108:in `as'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:114:in `as_anonymous_admin'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/services/katello/event_monitor/poller_thread.rb:65:in `run_event'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/services/katello/event_monitor/poller_thread.rb:96:in `block (4 levels) in poll_for_events'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/katello/util/support.rb:76:in `with_db_connection'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/services/katello/event_monitor/poller_thread.rb:94:in `block (3 levels) in poll_for_events'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/execution_wrapper.rb:87:in `wrap'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/services/katello/event_monitor/poller_thread.rb:93:in `block (2 levels) in poll_for_events'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/services/katello/event_monitor/poller_thread.rb:92:in `loop'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/services/katello/event_monitor/poller_thread.rb:92:in `block in poll_for_events'
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
production.log
2020-05-14T11:26:53 [E|kat|61b70c0d] POST: https://foreman.myserver.lan/pulp/api/v2/consumers/content/applicability/: {"criteria":{"filters":{"id":{"$in":["4d665385-8e26-4081-a2fe-12198ea1bfca"]}}},"content_types":["modulemd"]}: {"content_type"=>"application/json", "accept"=>"application/json"}
500 Internal Server Error: {"http_request_method": "POST", "exception": ["ServerSelectionTimeoutError: localhost:27017: [Errno 111] Connection refused\n"], "error_message": "localhost:27017: [Errno 111] Connection refused", "_href": "/pulp/api/v2/consumers/content/applicability/", "http_status": 500, "traceback": [" File \"/usr/lib/python2.7/site-packages/django/core/handlers/base.py\", line 185, in _get_response\n response = wrapped_callback(request, *callback_args, **callback_kwargs)\n", " File \"/usr/lib/python2.7/site-packages/django/views/generic/base.py\", line 68, in view\n return self.dispatch(request, *args, **kwargs)\n", " File \"/usr/lib/python2.7/site-packages/django/views/generic/base.py\", line 88, in dispatch\n return handler(request, *args, **kwargs)\n", " File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py\", line 241, in _auth_decorator\n return _verify_auth(self, operation, super_user_only, method, *args, **kwargs)\n", " File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py\", line 154, in _verify_auth\n login = authenticate_user()\n", " File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py\", line 30, in check_preauthenticated\n userid = factory.authentication_manager().check_username_password(username)\n", " File \"/usr/lib/python2.7/site-packages/pulp/server/managers/auth/authentication.py\", line 95, in check_username_password\n user = self._check_username_password_local(username, password)\n", " File \"/usr/lib/python2.7/site-packages/pulp/server/managers/auth/authentication.py\", line 35, in _check_username_password_local\n user = model.User.objects(login=username).first()\n", " File \"/usr/lib/python2.7/site-packages/mongoengine/queryset/base.py\", line 264, in first\n result = queryset[0]\n", " File \"/usr/lib/python2.7/site-packages/mongoengine/queryset/base.py\", line 161, in __getitem__\n return queryset._document._from_son(queryset._cursor[key],\n", " File \"/usr/lib64/python2.7/site-packages/pymongo/cursor.py\", line 565, in __getitem__\n for doc in clone:\n", " File \"/usr/lib64/python2.7/site-packages/pymongo/cursor.py\", line 1097, in next\n if len(self.__data) or self._refresh():\n", " File \"/usr/lib64/python2.7/site-packages/pymongo/cursor.py\", line 1019, in _refresh\n self.__read_concern))\n", " File \"/usr/lib64/python2.7/site-packages/pymongo/cursor.py\", line 850, in __send_message\n **kwargs)\n", " File \"/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py\", line 777, in _send_message_with_response\n server = topology.select_server(selector)\n", " File \"/usr/lib64/python2.7/site-packages/pymongo/topology.py\", line 141, in select_server\n address))\n", " File \"/usr/lib64/python2.7/site-packages/pymongo/topology.py\", line 117, in select_servers\n self._error_message(selector))\n"]}
2020-05-14T11:26:53 [E|kat|61b70c0d] event_queue_error: type=import_host_applicability, object_id=47
2020-05-14T11:26:53 [E|kat|61b70c0d] 500 Internal Server Error
2020-05-14T11:26:53 [E|kat|61b70c0d] /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/abstract_response.rb:223:in `exception_with_response'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/abstract_response.rb:103:in `return!'
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.12.1/lib/runcible/base.rb:95:in `block in get_response'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:807:in `process_result'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:725:in `block in transmit'
/opt/rh/rh-ruby25/root/usr/share/ruby/net/http.rb:910:in `start'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:715:in `transmit'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:145:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:52:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/resource.rb:67:in `post'
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.12.1/lib/runcible/base.rb:94:in `get_response'
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.12.1/lib/runcible/base.rb:74:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.12.1/lib/runcible/resources/consumer.rb:171:in `applicability'
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.12.1/lib/runcible/extensions/consumer.rb:174:in `applicable_for_type'
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.12.1/lib/runcible/extensions/consumer.rb:162:in `applicable_module_streams'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/services/katello/pulp/consumer.rb:36:in `applicable_ids'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/services/katello/applicable_content_helper.rb:85:in `applicable_differences'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/services/katello/applicable_content_helper.rb:11:in `import'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/models/katello/host/content_facet.rb:137:in `import_module_stream_applicability'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/models/katello/host/content_facet.rb:108:in `import_applicability'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/object/try.rb:19:in `public_send'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/object/try.rb:19:in `try!'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/object/try.rb:8:in `try'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/models/katello/events/import_host_applicability.rb:12:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/services/katello/event_monitor/poller_thread.rb:67:in `block in run_event'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:108:in `as'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:114:in `as_anonymous_admin'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/services/katello/event_monitor/poller_thread.rb:65:in `run_event'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/services/katello/event_monitor/poller_thread.rb:96:in `block (4 levels) in poll_for_events'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/katello/util/support.rb:76:in `with_db_connection'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/services/katello/event_monitor/poller_thread.rb:94:in `block (3 levels) in poll_for_events'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/execution_wrapper.rb:87:in `wrap'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/services/katello/event_monitor/poller_thread.rb:93:in `block (2 levels) in poll_for_events'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/services/katello/event_monitor/poller_thread.rb:92:in `loop'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/services/katello/event_monitor/poller_thread.rb:92:in `block in poll_for_events'
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2020-05-14T11:27:20 [I|app|972ec6dd] Started GET "/notification_recipients" for 10.100.0.58 at 2020-05-14 11:27:20 -0400
2020-05-14T11:27:20 [I|app|972ec6dd] Processing by NotificationRecipientsController#index as JSON
2020-05-14T11:27:20 [I|app|972ec6dd] Completed 200 OK in 16ms (Views: 0.3ms | ActiveRecord: 2.4ms)
2020-05-14T11:27:23 [E|kat|61b70c0d] POST: https://foreman.mydomain.lan/pulp/api/v2/consumers/content/applicability/: {"criteria":{"filters":{"id":{"$in":["6e199b7e-a2ba-49b9-85d8-bd97bdf9b0ef"]}}},"content_types":["modulemd"]}: {"content_type"=>"application/json", "accept"=>"application/json"}
500 Internal Server Error: {"http_request_method": "POST", "exception": ["ServerSelectionTimeoutError: localhost:27017: [Errno 111] Connection refused\n"], "error_message": "localhost:27017: [Errno 111] Connection refused", "_href": "/pulp/api/v2/consumers/content/applicability/", "http_status": 500, "traceback": [" File \"/usr/lib/python2.7/site-packages/django/core/handlers/base.py\", line 185, in _get_response\n response = wrapped_callback(request, *callback_args, **callback_kwargs)\n", " File \"/usr/lib/python2.7/site-packages/django/views/generic/base.py\", line 68, in view\n return self.dispatch(request, *args, **kwargs)\n", " File \"/usr/lib/python2.7/site-packages/django/views/generic/base.py\", line 88, in dispatch\n return handler(request, *args, **kwargs)\n", " File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py\", line 241, in _auth_decorator\n return _verify_auth(self, operation, super_user_only, method, *args, **kwargs)\n", " File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py\", line 154, in _verify_auth\n login = authenticate_user()\n", " File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py\", line 30, in check_preauthenticated\n userid = factory.authentication_manager().check_username_password(username)\n", " File \"/usr/lib/python2.7/site-packages/pulp/server/managers/auth/authentication.py\", line 95, in check_username_password\n user = self._check_username_password_local(username, password)\n", " File \"/usr/lib/python2.7/site-packages/pulp/server/managers/auth/authentication.py\", line 35, in _check_username_password_local\n user = model.User.objects(login=username).first()\n", " File \"/usr/lib/python2.7/site-packages/mongoengine/queryset/base.py\", line 264, in first\n result = queryset[0]\n", " File \"/usr/lib/python2.7/site-packages/mongoengine/queryset/base.py\", line 161, in __getitem__\n return queryset._document._from_son(queryset._cursor[key],\n", " File \"/usr/lib64/python2.7/site-packages/pymongo/cursor.py\", line 565, in __getitem__\n for doc in clone:\n", " File \"/usr/lib64/python2.7/site-packages/pymongo/cursor.py\", line 1097, in next\n if len(self.__data) or self._refresh():\n", " File \"/usr/lib64/python2.7/site-packages/pymongo/cursor.py\", line 1019, in _refresh\n self.__read_concern))\n", " File \"/usr/lib64/python2.7/site-packages/pymongo/cursor.py\", line 850, in __send_message\n **kwargs)\n", " File \"/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py\", line 777, in _send_message_with_response\n server = topology.select_server(selector)\n", " File \"/usr/lib64/python2.7/site-packages/pymongo/topology.py\", line 141, in select_server\n address))\n", " File \"/usr/lib64/python2.7/site-packages/pymongo/topology.py\", line 117, in select_servers\n self._error_message(selector))\n"]}
2020-05-14T11:27:23 [E|kat|61b70c0d] event_queue_error: type=import_host_applicability, object_id=30
2020-05-14T11:27:23 [E|kat|61b70c0d] 500 Internal Server Error
2020-05-14T11:27:23 [E|kat|61b70c0d] /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/abstract_response.rb:223:in `exception_with_response'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/abstract_response.rb:103:in `return!'
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.12.1/lib/runcible/base.rb:95:in `block in get_response'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:807:in `process_result'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:725:in `block in transmit'
/opt/rh/rh-ruby25/root/usr/share/ruby/net/http.rb:910:in `start'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:715:in `transmit'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:145:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:52:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/resource.rb:67:in `post'
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.12.1/lib/runcible/base.rb:94:in `get_response'
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.12.1/lib/runcible/base.rb:74:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.12.1/lib/runcible/resources/consumer.rb:171:in `applicability'
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.12.1/lib/runcible/extensions/consumer.rb:174:in `applicable_for_type'
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.12.1/lib/runcible/extensions/consumer.rb:162:in `applicable_module_streams'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/services/katello/pulp/consumer.rb:36:in `applicable_ids'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/services/katello/applicable_content_helper.rb:85:in `applicable_differences'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/services/katello/applicable_content_helper.rb:11:in `import'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/models/katello/host/content_facet.rb:137:in `import_module_stream_applicability'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/models/katello/host/content_facet.rb:108:in `import_applicability'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/object/try.rb:19:in `public_send'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/object/try.rb:19:in `try!'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/object/try.rb:8:in `try'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/models/katello/events/import_host_applicability.rb:12:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/services/katello/event_monitor/poller_thread.rb:67:in `block in run_event'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:108:in `as'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:114:in `as_anonymous_admin'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/services/katello/event_monitor/poller_thread.rb:65:in `run_event'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/services/katello/event_monitor/poller_thread.rb:96:in `block (4 levels) in poll_for_events'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/katello/util/support.rb:76:in `with_db_connection'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/services/katello/event_monitor/poller_thread.rb:94:in `block (3 levels) in poll_for_events'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/execution_wrapper.rb:87:in `wrap'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/services/katello/event_monitor/poller_thread.rb:93:in `block (2 levels) in poll_for_events'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/services/katello/event_monitor/poller_thread.rb:92:in `loop'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/services/katello/event_monitor/poller_thread.rb:92:in `block in poll_for_events'
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2020-05-14T11:27:28 [I|app|13b3a722] Started POST "/katello/sync_management/sync" for 10.100.0.58 at 2020-05-14 11:27:28 -0400
2020-05-14T11:27:28 [I|app|13b3a722] Processing by Katello::SyncManagementController#sync as JS
2020-05-14T11:27:28 [I|app|13b3a722] Parameters: {"utf8"=>"✓", "sync_status_url"=>"", "repoids"=>["50"], "commit"=>"Synchronize Now"}
2020-05-14T11:27:28 [I|bac|13b3a722] Task {label: Actions::Katello::Repository::Sync, id: 53fce27e-33e6-459f-a61b-6e042f46ca35, execution_plan_id: 9bd7d0b7-81fb-4fdf-b6fa-1a4bf4180d83} state changed: planning