Content Hosts page - nothing past 40 hosts displayed, not able to search for host. pulp_rest ERROR?

Eric_du_Toit · June 10, 2015, 2:13pm

Running Katello 2.2.1 on a 4 vCPU 16GB RAM VMWare machine. Checking
CPU/Load performance data on the machine during these steps - it is not
ever above 25% CPU and always below 1.0 load avg. Memory utilization is
under 5GB.

The behavior I'm experiencing seems to be similar to these:
https://bugzilla.redhat.com/show_bug.cgi?id=1191711
http://projects.theforeman.org/issues/10152
http://projects.theforeman.org/issues/9264

Content Hosts page worked well when there were 3 RHEL 7 hosts registered.
I added 46 VMWare hosts using VirtWho

I can scroll down through the first 20 hosts where the page grays for
several seconds and then displays 20 additional hosts (40 of 49).
Scrolling down to the end causes the page to gray indefinitely and never
display the last 9 hosts.

The production.log shows the first and second 20 hosts page "Rendered", but
the last 9 hang for several minutes until a 500 error is logged but never
displayed in the browser/page.

ERROR 2015-06-10 08:55:01 pulp_rest #17541] RestClient.get
"https://KATELLOHOST.myorg.org/pulp/api/v2/consumers/15930fd2-ff35-4242-af15-b25c4ab4d2e9/profiles/rpm//",
"Accept"=>"/; q=0.5, application/xml", "Accept-Encoding"=>"gzip,
deflate", "Authorization"=>"OAuth oauth_consumer_key="<trim>",
oauth_nonce="<trim> ", oauth_signature="<trim>",
oauth_signature_method="HMAC-SHA1", oauth_timestamp="<trim>",
oauth_version="1.0"", "accept"=>"application/json",
"content_type"=>"application/json", "pulp-user"=>"myuser-4e572dbb"
>
2015-06-10 08:55:01 [I] Rendered
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.1/app/views/katello/api/v2/systems/index.json.rabl
within katello/api/v2/layouts/collection (121242.7ms)
2015-06-10 08:55:01 [E] exception when talking to a remote client: Request
Timeout RestClient::RequestTimeout: Request Timeout
Body:

/opt/rh/ruby193/root/usr/share/gems/gems/rbovirt-0.0.35/lib/restclient_ext/request.rb:56:in
rescue in transmit' /opt/rh/ruby193/root/usr/share/gems/gems/rbovirt-0.0.35/lib/restclient_ext/request.rb:11:intransmit'
/opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:64:in
execute' /opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:33:inexecute'
/opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/resource.rb:51:in
get' /opt/rh/ruby193/root/usr/share/gems/gems/runcible-1.3.1/lib/runcible/base.rb:91:inget_response'
/opt/rh/ruby193/root/usr/share/gems/gems/runcible-1.3.1/lib/runcible/base.rb:82:in
`call'

<snip>

2015-06-10 08:55:01 [I] Completed 400 Bad Request in 121300ms (Views: 0.4ms
> ActiveRecord: 21.2ms)

The Content Hosts page stays gray.

Searching for a specific host in Content Hosts displays "Loading"
indefinitely even after the error / 400 :

2015-06-10 08:59:19 [I] Processing by
Katello::Api::V2::SystemsController#index as JSON
2015-06-10 08:59:19 [I] Parameters: {"organization_id"=>"1", "page"=>"1",
"search"=>"PARITALHOSTNAME*", "sort_by"=>"name", "sort_order"=>"ASC",
"api_version"=>"v2"}
2015-06-10 08:59:21 [I] Connecting to database specified by database.yml
2015-06-10 08:59:30 [W] Creating scope :completer_scope. Overwriting
existing method Location.completer_scope.
2015-06-10 08:59:30 [W] Creating scope :completer_scope. Overwriting
existing method Organization.completer_scope.
2015-06-10 08:59:58 [I] Client connected.
2015-06-10 08:59:58 [I] Connected to server.
2015-06-10 09:00:08 [I] Connecting to database specified by database.yml
2015-06-10 09:00:09 [I] Connecting to database specified by database.yml
2015-06-10 09:00:18 [W] Creating scope :completer_scope. Overwriting
existing method Location.completer_scope.
2015-06-10 09:00:18 [W] Creating scope :completer_scope. Overwriting
existing method Location.completer_scope.
2015-06-10 09:00:18 [W] Creating scope :completer_scope. Overwriting
existing method Organization.completer_scope.
2015-06-10 09:00:18 [W] Creating scope :completer_scope. Overwriting
existing method Organization.completer_scope.
2015-06-10 09:00:24 [I] Client connected.
2015-06-10 09:00:24 [I] Connected to server.
2015-06-10 09:00:25 [I] Connected to server.
2015-06-10 09:00:26 [I] Client connected.
2015-06-10 09:00:28 [I] Client disconnected.
2015-06-10 09:00:30 [I] Client disconnected.

[ERROR 2015-06-10 09:01:19 pulp_rest #17541] RestClient.get
"https://KATELLOHOST.myorg.org/pulp/api/v2/consumers/15930fd2-ff35-4242-af15-b25c4ab4d2e9/profiles/rpm//",
"Accept"=>"/; q=0.5, application/xml", "Accept-Encoding"=>"gzip,
deflate", "Authorization"=>"OAuth oauth_consumer_key="no",
oauth_nonce="no", oauth_signature="no",
oauth_signature_method="HMAC-SHA1", oauth_timestamp="no",
oauth_version="1.0"", "accept"=>"application/json",
"content_type"=>"application/json", "pulp-user"=>"myusername-4e572dbb"
>
2015-06-10 09:01:19 [I] Rendered
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.1/app/views/katello/api/v2/systems/index.json.rabl
within katello/api/v2/layouts/collection (120190.2ms)
2015-06-10 09:01:19 [E] exception when talking to a remote client: Request
Timeout RestClient::RequestTimeout: Request Timeout
Body:
/opt/rh/ruby193/root/usr/share/gems/gems/rbovirt-0.0.35/lib/restclient_ext/request.rb:56:in
rescue in transmit' /opt/rh/ruby193/root/usr/share/gems/gems/rbovirt-0.0.35/lib/restclient_ext/request.rb:11:intransmit'
/opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:64:in
`execute'

<snip>

2015-06-10 09:01:19 [I] Completed 400 Bad Request in 120243ms (Views: 0.3ms
> ActiveRecord: 0.0ms)

It looks like the error is with pulp, but I don't know where to start to
correct it.

Eric_du_Toit · June 10, 2015, 7:40pm

After restarting katello using katello-service restart, I figured, why not
reboot. That appears to have solved the issue - just not sure why.

···

On Wednesday, June 10, 2015 at 9:13:33 AM UTC-5, Eric du Toit wrote: > > > It looks like the error is with pulp, but I don't know where to start to > correct it. >

Eric_du_Toit · June 10, 2015, 7:54pm

And managed to create another issue :

Dashboard :

Oops, we're sorry but something went wrong

Warning!No route matches {:controller=>"reports", :host_id=>nil}

If you feel this is an error with Foreman itself, please open a new issue
with Foreman ticketing system, You would probably need to attach the Full
trace and relevant log entries.

Back

ActionController::RoutingError
No route matches {:controller=>"reports", :host_id=>nil}
app/views/dashboard/_reports_widget.html.erb:15:in block in _app_views_dashboard__reports_widget_html_erb___2135940476048849752_70256718343900' app/views/dashboard/_reports_widget.html.erb:13:in_app_views_dashboard__reports_widget_html_erb___2135940476048849752_70256718343900'
app/views/dashboard/index.html.erb:8:in block in _app_views_dashboard_index_html_erb___4123694470954109684_70256721218040' app/views/dashboard/index.html.erb:7:inmap'
app/views/dashboard/index.html.erb:7:in
_app_views_dashboard_index_html_erb___4123694470954109684_70256721218040' app/controllers/dashboard_controller.rb:6:inindex'
app/controllers/concerns/application_shared.rb:13:in set_timezone' app/models/concerns/foreman/thread_session.rb:32:inclear_thread'
lib/middleware/catch_json_parse_errors.rb:9:in `call'

On the positive side, if I visit a specific URL like /products or
/content_hosts, things work - even the timeout I was experiencing with
content_hosts & pulp

···

On Wednesday, June 10, 2015 at 2:40:10 PM UTC-5, Eric du Toit wrote: > > > > After restarting katello using katello-service restart, I figured, why not > reboot. That appears to have solved the issue - just not sure why. >

Eric_du_Toit · June 10, 2015, 9:00pm

>
> No route matches {:controller=>"reports", :host_id=>nil}
>
>
>
This matches
http://projects.theforeman.org/issues/10632
http://projects.theforeman.org/issues/10627

I renamed a VMWare host in Content Hosts to match the actual hostname and
not the UUID which resulted in a 404 error. Here are the relevant logs;

2015-06-10 14:45:22 [I] Processing by
Katello::Api::V2::SystemsController#update as JSON
2015-06-10 14:45:22 [I] Parameters: {"content_overrides"=>[],
"service_level"=>"", "type"=>"Hypervisor",
"compliance"=>{"date"=>"2015-06-10T19:45:04.078+0000", "compliantUntil"=>
nil, "nonCompliantProducts"=>[], "compliantProducts"=>{},
"partiallyCompliantProducts"=>{}, "partialStacks"=>{}, "reasons"=>[],
"compliant"=>true, "status"=>"valid"}, "facts"=>"[FI
LTERED]", "katello_id"=>72, "uuid"=>"63b8dbbb-8f0c-4b8a-8fb8-2c8ca8be1863",
"name"=>"abc5vmwarehost04.myorg.org", "description"=>"Initial Registration
Params", "location"=>"None",
"content_view"=>{"id"=>5, "name"=>"RedHat 7", "label"=>"RedHat_7",
"description"=>"RedHat Enterprise Linux 7 base packages",
"organization_id"=>1, "default"=>false, "created_at"=>"
2015-04-28T00:43:30-05:00", "updated_at"=>"2015-04-28T00:43:30-05:00",
"composite"=>nil, "next_version"=>7, "organization"=>"My Org Friendly
Name", "environments"=>["Test", "Dev", "Library", "
Prod"], "versions"=>["4.0", "5.0", "6.0"],
"versions_details"=>[{"version"=>"4.0", "published"=>"2015-06-02 16:26:55
-0500", "environments"=>[]}, {"version"=>"5.0", "published"=>"2
015-06-03 18:52:33 -0500", "environments"=>[]}, {"version"=>"6.0",
"published"=>"2015-06-03 21:45:43 -0500", "environments"=>["Library",
"Dev", "Test", "Prod"]}]}, "content_view_id
"=>5, "distribution"=>" ", "katello_agent_installed"=>false,
"registered_by"=>"15930fd2-ff35-4242-af15-b25c4ab4d2e9",
"entitlementStatus"=>"valid", "autoheal"=>true, "release"=>nil
, "checkin_time"=>"2015-06-09T11:06:21-05:00",
"created"=>"2015-06-09T16:05:45.904+0000", "installedProducts"=>[],
"environment"=>{"library"=>false, "id"=>3, "name"=>"Test", "label
"=>"Test", "description"=>"Testing", "organization"=>{"name"=>"My Org
Friendly Name", "label"=>"My__Organization", "id"=>1},
"created_at"=>"2015-04-27T23:54:35-05:00", "updated_at"=>"2015-04-27T23:54:
35-05:00", "prior"=>{"name"=>"Dev", "id"=>2}, "successor"=>{"name"=>"Prod",
"id"=>4}, "counts"=>{"content_hosts"=>47, "content_views"=>4},
"permissions"=>{"view_lifecycle_environme
nts"=>true, "edit_lifecycle_environments"=>true,
"destroy_lifecycle_environments"=>false,
"promote_or_remove_content_views_to_environments"=>true},
"selected"=>true}, "activation_k
eys"=>[], "id"=>"63b8dbbb-8f0c-4b8a-8fb8-2c8ca8be1863",
"errata_counts"=>{"security"=>0, "bugfix"=>0, "enhancement"=>0,
"total"=>0}, "products"=>[], "host"=>nil, "hostCollections"=
>[], "customInfo"=>[], "release_ver"=>nil, "virtual_guests"=>[{"id"=>30,
"uuid"=>"1092fe6f-80f9-4586-9311-a2c20b4ff503",
"name"=>"abc3vlptime02.myorg.org", "description"=>"Initial
Registration Params", "location"=>"None", "environment_id"=>4,
"created_at"=>"2015-06-08T11:25:34-05:00",
"updated_at"=>"2015-06-08T11:25:35-05:00", "type"=>"Katello::System", "co
ntent_view_id"=>5, "host_id"=>13}], "permissions"=>{"editable"=>true},
"selected"=>false, "api_version"=>"v2",
"system"=>{"id"=>"63b8dbbb-8f0c-4b8a-8fb8-2c8ca8be1863", "uuid"=>"63b
8dbbb-8f0c-4b8a-8fb8-2c8ca8be1863", "name"=>"abc5vmwarehost04.myorg.org",
"description"=>"Initial Registration Params", "location"=>"None",
"type"=>"Hypervisor", "content_view_id"=
>5, "autoheal"=>true, "facts"=>"[FILTERED]", "content_view"=>{"id"=>5,
"name"=>"RedHat 7", "label"=>"RedHat_7", "description"=>"RedHat Enterprise
Linux 7 base packages", "organizat
ion_id"=>1, "default"=>false, "created_at"=>"2015-04-28T00:43:30-05:00",
"updated_at"=>"2015-04-28T00:43:30-05:00", "composite"=>nil,
"next_version"=>7, "organization"=>"My Org Friendly Name",
"environments"=>["Test", "Dev", "Library", "Prod"], "versions"=>["4.0",
"5.0", "6.0"], "versions_details"=>[{"version"=>"4.0",
"published"=>"2015-06-02 16:26:55 -0500", "environme
nts"=>[]}, {"version"=>"5.0", "published"=>"2015-06-03 18:52:33 -0500",
"environments"=>[]}, {"version"=>"6.0", "published"=>"2015-06-03 21:45:43
-0500", "environments"=>["Library", "Dev", "Test", "Prod"]}]},
"environment"=>{"library"=>false, "id"=>3, "name"=>"Test", "label"=>"Test",
"description"=>"Testing", "organization"=>{"name"=>"My Org Friendly Name",
"label"=>"My__Organization", "id"=>1},
"created_at"=>"2015-04-27T23:54:35-05:00",
"updated_at"=>"2015-04-27T23:54:35-05:00", "prior"=>{"name"=>"Dev",
"id"=>2}, "successor"=>{"name"=>"Prod", "id"=>4},
"counts"=>{"content_hosts"=>47, "content_views"=>4},
"permissions"=>{"view_lifecycle_environments"=>true,
"edit_lifecycle_environments"=>true,
"destroy_lifecycle_environments"=>false,
"promote_or_remove_content_views_to_environments"=>true},
"selected"=>true}}}
[ERROR 2015-06-10 14:45:23 pulp_rest #2918] RestClient.post
"https://myKatelloHost.myorg.org/pulp/api/v2/repositories/My__Organization-CentOS-CentOS_7_x86_64/search/units/",
"{"criteria":{"type_ids":["distribution"]}}", "Accept"=>"/; q=0.5,
application/xml", "Accept-Encoding"=>"gzip, deflate",
"Authorization"=>"OAuth oauth_body_hash="no",
oauth_consumer_key="katello", oauth_nonce="no", oauth_signature="no",
oauth_signature_method="HMAC-SHA1", oauth_timestamp="1433965523",
oauth_version="1.0"", "Content-Length"=>"42",
"accept"=>"application/json", "content_type"=>"application/json",
"pulp-user"=>"admin"
2015-06-10 14:45:23 [E] 404 Resource Not Found
(RestClient::ResourceNotFound)
2015-06-10 14:45:25 [E] ForemanTasks::TaskError: Task
f2502e38-9af2-413e-879e-c77bfb2e4730: RestClient::ResourceNotFound: 404
Resource Not Found
2015-06-10 14:45:25 [I] Completed 500 Internal Server Error in 3122ms
(Views: 0.3ms | ActiveRecord: 57.9ms)

The VMWare host does have the correct name now in Content Hosts view, but I
believe this is what may have caused the Dashboard to error.

···

On Wednesday, June 10, 2015 at 2:54:38 PM UTC-5, Eric du Toit wrote: