Katello 2.1 cannot show content hosts any more

Hi there,

I was trying to upgrade to katello 2.2, but for some reason, I failed, the
reason I want to upgrade is because I have some issue running on 2.1, one
of the issue was I can not list content hosts in "Hosts"–> "Content Hosts"
page, it only showed the circular loading icon spinning continuously,
After analyzing via firebug, I found one GET method got 400 return code,
the get url is:
https://katello.anim.odw.com.cn/katello/api/v2/systems?organization_id=3&page=1&search=&sort_by=name&sort_order=ASC,
looks normal, but the response I got was:

{"displayMessage":"Katello::Resources::Candlepin::Consumer: 410 Gone {"displayMessage":"Unit 455ae32a-8faa-428c-ad92-18a816627f96 has been deleted","requestUuid":"695a9943-de23-4ce0-852b-7998a9bc1708","deletedId":"455ae32a-8faa-428c-ad92-18a816627f96"}
(GET /candlepin/consumers/455ae32a-8faa-428c-ad92-18a816627f96)","errors":["Katello::Resources::Candlepin::Consumer: 410 Gone
{"displayMessage":"Unit 455ae32a-8faa-428c-ad92-18a816627f96 has been deleted","requestUuid":"695a9943-de23-4ce0-852b-7998a9bc1708","deletedId":"455ae32a-8faa-428c-ad92-18a816627f96"}
(GET /candlepin/consumers/455ae32a-8faa-428c-ad92-18a816627f96)"]}

it seems something was deleted, I am not quite clear what was that, and how we can fix it.

Any one has any ideas please?

Thanks
Sinux

I alsl get the error from the katello log:

Processing by Katello::Api::V2::SystemsController#index as JSON
Parameters: {"organization_id"=>"3", "page"=>"1", "search"=>"",
"sort_by"=>"name", "sort_order"=>"ASC", "api_version"=>"v2"}
Failed to retrieve all Katello::System search results (20/20 found.)
Rendered
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.1.0/app/views/katello/api/v2/systems/index.json.rabl
within katello/api/v2/layouts/collection (5821.9ms)
exception when talking to a remote client:
Katello::Resources::Candlepin::Consumer: 410 Gone {"displayMessage":"Unit
455ae32a-8faa-428c-ad92-18a816627f96 has been
deleted","requestUuid":"a748e055-ac6f-46a7-a957-d60373944e62","deletedId":"455ae32a-8faa-428c-ad92-18a816627f96"}
(GET /candlepin/consumers/455ae32a-8faa-428c-ad92-18a816627f96)
RestClient::Gone: Katello::Resources::Candlepin::Consumer: 410 Gone
{"displayMessage":"Unit 455ae32a-8faa-428c-ad92-18a816627f96 has been
deleted","requestUuid":"a748e055-ac6f-46a7-a957-d60373944e62","deletedId":"455ae32a-8faa-428c-ad92-18a816627f96"}
(GET /candlepin/consumers/455ae32a-8faa-428c-ad92-18a816627f96)
Body: {"displayMessage":"Unit 455ae32a-8faa-428c-ad92-18a816627f96 has been
deleted","requestUuid":"a748e055-ac6f-46a7-a957-d60373944e62","deletedId":"455ae32a-8faa-428c-ad92-18a816627f96"}

··· On Tuesday, May 19, 2015 at 11:29:38 AM UTC+8, sinux shen wrote: > > Hi there, > > I was trying to upgrade to katello 2.2, but for some reason, I failed, the > reason I want to upgrade is because I have some issue running on 2.1, one > of the issue was I can not list content hosts in "Hosts"--> "Content Hosts" > page, it only showed the circular loading icon spinning continuously, > After analyzing via firebug, I found one GET method got 400 return code, > the get url is: > https://katello.anim.odw.com.cn/katello/api/v2/systems?organization_id=3&page=1&search=&sort_by=name&sort_order=ASC, > looks normal, but the response I got was: > > {"displayMessage":"Katello::Resources::Candlepin::Consumer: 410 Gone {\"displayMessage\":\"Unit 455ae32a-8faa-428c-ad92-18a816627f96 has been deleted\",\"requestUuid\":\"695a9943-de23-4ce0-852b-7998a9bc1708\",\"deletedId\":\"455ae32a-8faa-428c-ad92-18a816627f96\"} > (GET /candlepin/consumers/455ae32a-8faa-428c-ad92-18a816627f96)","errors":["Katello::Resources::Candlepin::Consumer: 410 Gone > {\"displayMessage\":\"Unit 455ae32a-8faa-428c-ad92-18a816627f96 has been deleted\",\"requestUuid\":\"695a9943-de23-4ce0-852b-7998a9bc1708\",\"deletedId\":\"455ae32a-8faa-428c-ad92-18a816627f96\"} > (GET /candlepin/consumers/455ae32a-8faa-428c-ad92-18a816627f96)"]} > > it seems something was deleted, I am not quite clear what was that, and how we can fix it. > > Any one has any ideas please? > > Thanks > Sinux > > >

I ran clean_backend_objects command again, this time, the error was a
little different,

before I ran this, I checked the message log, and I found tons of error
like this:
May 25 10:46:11 katello qpidd[5978]: 2015-05-25 10:46:11 [System] error
Could not accept socket: Too many open files
(/builddir/build/BUILD/qpid-0.26/cpp/src/qpid/sys/ssl/SslSocket.cpp:195)
May 25 10:46:11 katello qpidd[5978]: 2015-05-25 10:46:11 [System] error
Could not accept socket: Too many open files
(/builddir/build/BUILD/qpid-0.26/cpp/src/qpid/sys/ssl/SslSocket.cpp:195)
May 25 10:46:11 katello qpidd[5978]: 2015-05-25 10:46:11 [System] error
Could not accept socket: Too many open files
(/builddir/build/BUILD/qpid-0.26/cpp/src/qpid/sys/ssl/SslSocket.cpp:195)

And then I restarted qpidd, and I ran clean_backend_objects, it gave me the
following errors:
katello ODW [/home/sxin] 1001# foreman-rake katello:clean_backend_objects
System 110 dogtag.anim.odw.com.cn 455ae32a-8faa-428c-ad92-18a816627f96 is
partially missing. Cleaning.
System 111 dogtag.anim.odw.com.cn 2d734c21-8c11-4565-987f-d338840d0ab2 is
partially missing. Cleaning.
rake aborted!
PGError: 错误: 在 "katello_systems" 上的更新或删除操作违反了在
"katello_system_repositories" 上的外键约束
"katello_system_repositories_system_id_fk"
DETAIL: 键值对(id)=(111)仍然是从表"katello_system_repositories"引用的.
: DELETE FROM "katello_systems" WHERE "katello_systems"."id" = 111

Tasks: TOP => katello:clean_backend_objects
(See full trace by running task with --trace)
And in message, it has one lines of log shows:
May 25 10:50:39 katello pulp:
pulp.server.webservices.middleware.exception:INFO: Missing resource(s):
consumer=455ae32a-8faa-428c-ad92-18a816627f96

Sorry, it contains some Chinese, I remember that when I tried to upgrade
katello to 2.2, it has similar errors.

··· On Tuesday, May 19, 2015 at 11:29:38 AM UTC+8, sinux shen wrote: > > Hi there, > > I was trying to upgrade to katello 2.2, but for some reason, I failed, the > reason I want to upgrade is because I have some issue running on 2.1, one > of the issue was I can not list content hosts in "Hosts"--> "Content Hosts" > page, it only showed the circular loading icon spinning continuously, > After analyzing via firebug, I found one GET method got 400 return code, > the get url is: > https://katello.anim.odw.com.cn/katello/api/v2/systems?organization_id=3&page=1&search=&sort_by=name&sort_order=ASC, > looks normal, but the response I got was: > > {"displayMessage":"Katello::Resources::Candlepin::Consumer: 410 Gone {\"displayMessage\":\"Unit 455ae32a-8faa-428c-ad92-18a816627f96 has been deleted\",\"requestUuid\":\"695a9943-de23-4ce0-852b-7998a9bc1708\",\"deletedId\":\"455ae32a-8faa-428c-ad92-18a816627f96\"} > (GET /candlepin/consumers/455ae32a-8faa-428c-ad92-18a816627f96)","errors":["Katello::Resources::Candlepin::Consumer: 410 Gone > {\"displayMessage\":\"Unit 455ae32a-8faa-428c-ad92-18a816627f96 has been deleted\",\"requestUuid\":\"695a9943-de23-4ce0-852b-7998a9bc1708\",\"deletedId\":\"455ae32a-8faa-428c-ad92-18a816627f96\"} > (GET /candlepin/consumers/455ae32a-8faa-428c-ad92-18a816627f96)"]} > > it seems something was deleted, I am not quite clear what was that, and how we can fix it. > > Any one has any ideas please? > > Thanks > Sinux > > >

> I alsl get the error from the katello log:
>
> Processing by Katello::Api::V2::SystemsController#index as JSON
> Parameters: {"organization_id"=>"3", "page"=>"1", "search"=>"",
> "sort_by"=>"name", "sort_order"=>"ASC", "api_version"=>"v2"}
> Failed to retrieve all Katello::System search results (20/20 found.)
> Rendered
> /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.1.0/app/views/katello/api/v2/systems/index.json.rabl
> within katello/api/v2/layouts/collection (5821.9ms)
> exception when talking to a remote client:
> Katello::Resources::Candlepin::Consumer: 410 Gone {"displayMessage":"Unit
> 455ae32a-8faa-428c-ad92-18a816627f96 has been
> deleted","requestUuid":"a748e055-ac6f-46a7-a957-d60373944e62","deletedId":"455ae32a-8faa-428c-ad92-18a816627f96"}
> (GET /candlepin/consumers/455ae32a-8faa-428c-ad92-18a816627f96)
> RestClient::Gone: Katello::Resources::Candlepin::Consumer: 410 Gone
> {"displayMessage":"Unit 455ae32a-8faa-428c-ad92-18a816627f96 has been
> deleted","requestUuid":"a748e055-ac6f-46a7-a957-d60373944e62","deletedId":"455ae32a-8faa-428c-ad92-18a816627f96"}
> (GET /candlepin/consumers/455ae32a-8faa-428c-ad92-18a816627f96)
> Body: {"displayMessage":"Unit 455ae32a-8faa-428c-ad92-18a816627f96 has been
> deleted","requestUuid":"a748e055-ac6f-46a7-a957-d60373944e62","deletedId":"455ae32a-8faa-428c-ad92-18a816627f96"}
>

Thanks Sinux, logs are helpful.

Could you try to run foreman-rake katello:clean_backend_objects as root and see
if you still get the same error?

>
> >
> > Hi there,
> >
> > I was trying to upgrade to katello 2.2, but for some reason, I failed, the
> > reason I want to upgrade is because I have some issue running on 2.1, one
> > of the issue was I can not list content hosts in "Hosts"–> "Content Hosts"
> > page, it only showed the circular loading icon spinning continuously,
> > After analyzing via firebug, I found one GET method got 400 return code,
> > the get url is:
> > https://katello.anim.odw.com.cn/katello/api/v2/systems?organization_id=3&page=1&search=&sort_by=name&sort_order=ASC,
> > looks normal, but the response I got was:
> >
> > {"displayMessage":"Katello::Resources::Candlepin::Consumer: 410 Gone {"displayMessage":"Unit 455ae32a-8faa-428c-ad92-18a816627f96 has been deleted","requestUuid":"695a9943-de23-4ce0-852b-7998a9bc1708","deletedId":"455ae32a-8faa-428c-ad92-18a816627f96"}
> > (GET /candlepin/consumers/455ae32a-8faa-428c-ad92-18a816627f96)","errors":["Katello::Resources::Candlepin::Consumer: 410 Gone
> > {"displayMessage":"Unit 455ae32a-8faa-428c-ad92-18a816627f96 has been deleted","requestUuid":"695a9943-de23-4ce0-852b-7998a9bc1708","deletedId":"455ae32a-8faa-428c-ad92-18a816627f96"}
> > (GET /candlepin/consumers/455ae32a-8faa-428c-ad92-18a816627f96)"]}
> >
> > it seems something was deleted, I am not quite clear what was that, and how we can fix it.
> >
> > Any one has any ideas please?
> >
> > Thanks
> > Sinux
> >
> >
> >
>
> –
> You received this message because you are subscribed to the Google Groups "Foreman users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to foreman-users+unsubscribe@googlegroups.com.
> To post to this group, send email to foreman-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/foreman-users.
> For more options, visit https://groups.google.com/d/optout.

Best,

··· On 05/18, sinux shen wrote: > On Tuesday, May 19, 2015 at 11:29:38 AM UTC+8, sinux shen wrote:


Daniel Lobato Garcia

@eLobatoss
blog.daniellobato.me
daniellobato.me

GPG: http://keys.gnupg.net/pks/lookup?op=get&search=0x7A92D6DD38D6DE30
Keybase: https://keybase.io/elobato

Hi Eric & Daniel,

Can anyone of you help with this or please let me know if you need more
infomations, I was stuck here and don't know what to do to fix it.

Thanks
Sinux

··· On Monday, May 25, 2015 at 10:58:15 AM UTC+8, sinux shen wrote: > > I ran clean_backend_objects command again, this time, the error was a > little different, > > before I ran this, I checked the message log, and I found tons of error > like this: > May 25 10:46:11 katello qpidd[5978]: 2015-05-25 10:46:11 [System] error > Could not accept socket: Too many open files > (/builddir/build/BUILD/qpid-0.26/cpp/src/qpid/sys/ssl/SslSocket.cpp:195) > May 25 10:46:11 katello qpidd[5978]: 2015-05-25 10:46:11 [System] error > Could not accept socket: Too many open files > (/builddir/build/BUILD/qpid-0.26/cpp/src/qpid/sys/ssl/SslSocket.cpp:195) > May 25 10:46:11 katello qpidd[5978]: 2015-05-25 10:46:11 [System] error > Could not accept socket: Too many open files > (/builddir/build/BUILD/qpid-0.26/cpp/src/qpid/sys/ssl/SslSocket.cpp:195) > > And then I restarted qpidd, and I ran clean_backend_objects, it gave me > the following errors: > katello ODW [/home/sxin] 1001# foreman-rake katello:clean_backend_objects > System 110 dogtag.anim.odw.com.cn 455ae32a-8faa-428c-ad92-18a816627f96 is > partially missing. Cleaning. > System 111 dogtag.anim.odw.com.cn 2d734c21-8c11-4565-987f-d338840d0ab2 is > partially missing. Cleaning. > rake aborted! > PGError: 错误: 在 "katello_systems" 上的更新或删除操作违反了在 > "katello_system_repositories" 上的外键约束 > "katello_system_repositories_system_id_fk" > DETAIL: 键值对(id)=(111)仍然是从表"katello_system_repositories"引用的. > : DELETE FROM "katello_systems" WHERE "katello_systems"."id" = 111 > > Tasks: TOP => katello:clean_backend_objects > (See full trace by running task with --trace) > And in message, it has one lines of log shows: > May 25 10:50:39 katello pulp: > pulp.server.webservices.middleware.exception:INFO: Missing resource(s): > consumer=455ae32a-8faa-428c-ad92-18a816627f96 > > > Sorry, it contains some Chinese, I remember that when I tried to upgrade > katello to 2.2, it has similar errors. > > > On Tuesday, May 19, 2015 at 11:29:38 AM UTC+8, sinux shen wrote: >> >> Hi there, >> >> I was trying to upgrade to katello 2.2, but for some reason, I failed, >> the reason I want to upgrade is because I have some issue running on 2.1, >> one of the issue was I can not list content hosts in "Hosts"--> "Content >> Hosts" page, it only showed the circular loading icon spinning >> continuously, >> After analyzing via firebug, I found one GET method got 400 return code, >> the get url is: >> https://katello.anim.odw.com.cn/katello/api/v2/systems?organization_id=3&page=1&search=&sort_by=name&sort_order=ASC, >> looks normal, but the response I got was: >> >> {"displayMessage":"Katello::Resources::Candlepin::Consumer: 410 Gone {\"displayMessage\":\"Unit 455ae32a-8faa-428c-ad92-18a816627f96 has been deleted\",\"requestUuid\":\"695a9943-de23-4ce0-852b-7998a9bc1708\",\"deletedId\":\"455ae32a-8faa-428c-ad92-18a816627f96\"} >> (GET /candlepin/consumers/455ae32a-8faa-428c-ad92-18a816627f96)","errors":["Katello::Resources::Candlepin::Consumer: 410 Gone >> {\"displayMessage\":\"Unit 455ae32a-8faa-428c-ad92-18a816627f96 has been deleted\",\"requestUuid\":\"695a9943-de23-4ce0-852b-7998a9bc1708\",\"deletedId\":\"455ae32a-8faa-428c-ad92-18a816627f96\"} >> (GET /candlepin/consumers/455ae32a-8faa-428c-ad92-18a816627f96)"]} >> >> it seems something was deleted, I am not quite clear what was that, and how we can fix it. >> >> Any one has any ideas please? >> >> Thanks >> Sinux >> >> >>

Hi Daniel,

Thanks for the quick response, I ran the command, it shows some errors:

katello ODW [/home/sxin] 1000# foreman-rake katello:clean_backend_objects
System 110 dogtag.anim.odw.com.cn 455ae32a-8faa-428c-ad92-18a816627f96 is
partially missing. Cleaning.
rake aborted!
500 Internal Server Error

Tasks: TOP => katello:clean_backend_objects
(See full trace by running task with --trace)

and then I re-run this command by adding --trace
katello ODW [/home/sxin] 1001# foreman-rake katello:clean_backend_objects
–trace
** Invoke katello:clean_backend_objects (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute katello:clean_backend_objects
System 110 dogtag.anim.odw.com.cn 455ae32a-8faa-428c-ad92-18a816627f96 is
partially missing. Cleaning.
rake aborted!
500 Internal Server Error
/opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/abstract_response.rb:48:in
return!' /opt/rh/ruby193/root/usr/share/gems/gems/runcible-1.3.0/lib/runcible/base.rb:92:inblock in get_response'
/opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:228:in
call' /opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:228:inprocess_result'
/opt/rh/ruby193/root/usr/share/gems/gems/rbovirt-0.0.29/lib/restclient_ext/request.rb:50:in
block in transmit' /opt/rh/ruby193/root/usr/share/ruby/net/http.rb:746:instart'
/opt/rh/ruby193/root/usr/share/gems/gems/rbovirt-0.0.29/lib/restclient_ext/request.rb:44:in
transmit' /opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:64:inexecute'
/opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:33:in
execute' /opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/resource.rb:94:indelete'
/opt/rh/ruby193/root/usr/share/gems/gems/runcible-1.3.0/lib/runcible/base.rb:91:in
get_response' /opt/rh/ruby193/root/usr/share/gems/gems/runcible-1.3.0/lib/runcible/base.rb:82:incall'
/opt/rh/ruby193/root/usr/share/gems/gems/runcible-1.3.0/lib/runcible/resources/consumer.rb:68:in
delete' /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.1.0/app/models/katello/glue/pulp/consumer.rb:127:indel_pulp_consumer'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.1.0/lib/katello/tasks/clean_backend_objects.rake:31:in
block in cleanup_systems' /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/relation/batches.rb:21:inblock (2 levels) in find_each'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/relation/batches.rb:21:in
each' /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/relation/batches.rb:21:inblock in find_each'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/relation/batches.rb:72:in
find_in_batches' /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/relation/batches.rb:20:infind_each'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/querying.rb:8:in
find_each' /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.1.0/lib/katello/tasks/clean_backend_objects.rake:5:incleanup_systems'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.1.0/lib/katello/tasks/clean_backend_objects.rake:55:in
block (2 levels) in <top (required)>' /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:205:incall'
/opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:205:in block in execute' /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:200:ineach'
/opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:200:in execute' /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:158:inblock in
invoke_with_call_chain'
/opt/rh/ruby193/root/usr/share/ruby/monitor.rb:211:in mon_synchronize' /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:151:ininvoke_with_call_chain'
/opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:144:in invoke' /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:116:ininvoke_task'
/opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:94:in block (2 levels) in top_level' /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:94:ineach'
/opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:94:in block in top_level' /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:133:instandard_exception_handling'
/opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:88:in top_level' /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:66:inblock in run'
/opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:133:in
standard_exception_handling' /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:63:inrun'
/opt/rh/ruby193/root/usr/bin/rake:32:in `<main>'
Tasks: TOP => katello:clean_backend_objects

Then I went back the the foreman GUI, I click Hosts–> Content Hosts, but
it was still keep showing the circular loading icon, firebug show the same
400 GET error, like what I mentioned previously.

Thank you so much for you help.

··· On Tuesday, May 19, 2015 at 4:17:05 PM UTC+8, Daniel Lobato wrote: > > On 05/18, sinux shen wrote: > > I alsl get the error from the katello log: > > > > Processing by Katello::Api::V2::SystemsController#index as JSON > > Parameters: {"organization_id"=>"3", "page"=>"1", "search"=>"", > > "sort_by"=>"name", "sort_order"=>"ASC", "api_version"=>"v2"} > > Failed to retrieve all Katello::System search results (20/20 found.) > > Rendered > > > /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.1.0/app/views/katello/api/v2/systems/index.json.rabl > > > within katello/api/v2/layouts/collection (5821.9ms) > > exception when talking to a remote client: > > Katello::Resources::Candlepin::Consumer: 410 Gone > {"displayMessage":"Unit > > 455ae32a-8faa-428c-ad92-18a816627f96 has been > > > deleted","requestUuid":"a748e055-ac6f-46a7-a957-d60373944e62","deletedId":"455ae32a-8faa-428c-ad92-18a816627f96"} > > > (GET /candlepin/consumers/455ae32a-8faa-428c-ad92-18a816627f96) > > RestClient::Gone: Katello::Resources::Candlepin::Consumer: 410 Gone > > {"displayMessage":"Unit 455ae32a-8faa-428c-ad92-18a816627f96 has been > > > deleted","requestUuid":"a748e055-ac6f-46a7-a957-d60373944e62","deletedId":"455ae32a-8faa-428c-ad92-18a816627f96"} > > > (GET /candlepin/consumers/455ae32a-8faa-428c-ad92-18a816627f96) > > Body: {"displayMessage":"Unit 455ae32a-8faa-428c-ad92-18a816627f96 has > been > > > deleted","requestUuid":"a748e055-ac6f-46a7-a957-d60373944e62","deletedId":"455ae32a-8faa-428c-ad92-18a816627f96"} > > > > > Thanks Sinux, logs are helpful. > > Could you try to run foreman-rake katello:clean_backend_objects as root > and see > if you still get the same error? > > > > > On Tuesday, May 19, 2015 at 11:29:38 AM UTC+8, sinux shen wrote: > > > > > > Hi there, > > > > > > I was trying to upgrade to katello 2.2, but for some reason, I failed, > the > > > reason I want to upgrade is because I have some issue running on 2.1, > one > > > of the issue was I can not list content hosts in "Hosts"--> "Content > Hosts" > > > page, it only showed the circular loading icon spinning continuously, > > > After analyzing via firebug, I found one GET method got 400 return > code, > > > the get url is: > > > > https://katello.anim.odw.com.cn/katello/api/v2/systems?organization_id=3&page=1&search=&sort_by=name&sort_order=ASC, > > > > looks normal, but the response I got was: > > > > > > {"displayMessage":"Katello::Resources::Candlepin::Consumer: 410 Gone > {\"displayMessage\":\"Unit 455ae32a-8faa-428c-ad92-18a816627f96 has been > deleted\",\"requestUuid\":\"695a9943-de23-4ce0-852b-7998a9bc1708\",\"deletedId\":\"455ae32a-8faa-428c-ad92-18a816627f96\"} > > > > (GET > /candlepin/consumers/455ae32a-8faa-428c-ad92-18a816627f96)","errors":["Katello::Resources::Candlepin::Consumer: > 410 Gone > > > {\"displayMessage\":\"Unit 455ae32a-8faa-428c-ad92-18a816627f96 has > been > deleted\",\"requestUuid\":\"695a9943-de23-4ce0-852b-7998a9bc1708\",\"deletedId\":\"455ae32a-8faa-428c-ad92-18a816627f96\"} > > > > (GET /candlepin/consumers/455ae32a-8faa-428c-ad92-18a816627f96)"]} > > > > > > it seems something was deleted, I am not quite clear what was that, > and how we can fix it. > > > > > > Any one has any ideas please? > > > > > > Thanks > > > Sinux > > > > > > > > > > > > > -- > > You received this message because you are subscribed to the Google > Groups "Foreman users" group. > > To unsubscribe from this group and stop receiving emails from it, send > an email to foreman-user...@googlegroups.com . > > To post to this group, send email to forema...@googlegroups.com > . > > Visit this group at http://groups.google.com/group/foreman-users. > > For more options, visit https://groups.google.com/d/optout. > > Best, > > -- > Daniel Lobato Garcia > > @eLobatoss > blog.daniellobato.me > daniellobato.me > > GPG: http://keys.gnupg.net/pks/lookup?op=get&search=0x7A92D6DD38D6DE30 > Keybase: https://keybase.io/elobato >

> Hi Eric & Daniel,
>
> Can anyone of you help with this or please let me know if you need
> more infomations, I was stuck here and don't know what to do to fix it.

The 500 Internal Server Error from candlepin is concerning and likely
the cause of your content host issue. Would you be able to run
foreman-debug and upload the results (according to the instructions
given at the end of the command)? Then simply reply with the filename
it created.

Thanks,

Justin

··· On 05/26/2015 01:42 AM, sinux shen wrote:

Thanks
Sinux

On Monday, May 25, 2015 at 10:58:15 AM UTC+8, sinux shen wrote:

I ran clean_backend_objects command again, this time, the error
was a little different,

before I ran this, I checked the message log, and I found tons of
error like this:
May 25 10:46:11 katello qpidd[5978]: 2015-05-25 10:46:11 [System]
error Could not accept socket: Too many open files
(/builddir/build/BUILD/qpid-0.26/cpp/src/qpid/sys/ssl/SslSocket.cpp:195)
May 25 10:46:11 katello qpidd[5978]: 2015-05-25 10:46:11 [System]
error Could not accept socket: Too many open files
(/builddir/build/BUILD/qpid-0.26/cpp/src/qpid/sys/ssl/SslSocket.cpp:195)
May 25 10:46:11 katello qpidd[5978]: 2015-05-25 10:46:11 [System]
error Could not accept socket: Too many open files
(/builddir/build/BUILD/qpid-0.26/cpp/src/qpid/sys/ssl/SslSocket.cpp:195)

And then I restarted qpidd, and I ran clean_backend_objects, it
gave me the following errors:
katello ODW [/home/sxin] 1001# foreman-rake
katello:clean_backend_objects
System 110 dogtag.anim.odw.com.cn <http://dogtag.anim.odw.com.cn>
455ae32a-8faa-428c-ad92-18a816627f96 is partially missing.  Cleaning.
System 111 dogtag.anim.odw.com.cn <http://dogtag.anim.odw.com.cn>
2d734c21-8c11-4565-987f-d338840d0ab2 is partially missing.  Cleaning.
rake aborted!
PGError: 错误:  在 "katello_systems" 上的更新或删除操作违反了在
"katello_system_repositories" 上的外键约束
"katello_system_repositories_system_id_fk"
DETAIL:  键值对(id)=(111)仍然是从表"katello_system_repositories"
引用的.
: DELETE FROM "katello_systems" WHERE "katello_systems"."id" = 111

Tasks: TOP => katello:clean_backend_objects
(See full trace by running task with --trace)
And in message, it has one lines of log shows:
May 25 10:50:39 katello pulp:
pulp.server.webservices.middleware.exception:INFO: Missing
resource(s): consumer=455ae32a-8faa-428c-ad92-18a816627f96


Sorry, it contains some Chinese, I remember that when I tried to
upgrade katello to 2.2, it has similar errors.


On Tuesday, May 19, 2015 at 11:29:38 AM UTC+8, sinux shen wrote:

    Hi there,

    I was trying to upgrade to katello 2.2, but for some reason, I
    failed, the reason I want to upgrade is because I have some
    issue running on 2.1, one of the issue was I can not list
    content hosts in "Hosts"--> "Content Hosts" page, it only
    showed the circular loading icon spinning continuously,
    After analyzing via firebug, I found one GET method got 400
    return code, the get url is:
    https://katello.anim.odw.com.cn/katello/api/v2/systems?organization_id=3&page=1&search=&sort_by=name&sort_order=ASC
    <https://katello.anim.odw.com.cn/katello/api/v2/systems?organization_id=3&page=1&search=&sort_by=name&sort_order=ASC>,
    looks normal, but the response I got was:

    {"displayMessage":"Katello::Resources::Candlepin::Consumer: 410 Gone {\"displayMessage\":\"Unit 455ae32a-8faa-428c-ad92-18a816627f96 has been deleted\",\"requestUuid\":\"695a9943-de23-4ce0-852b-7998a9bc1708\",\"deletedId\":\"455ae32a-8faa-428c-ad92-18a816627f96\"}
    (GET /candlepin/consumers/455ae32a-8faa-428c-ad92-18a816627f96)","errors":["Katello::Resources::Candlepin::Consumer: 410 Gone
    {\"displayMessage\":\"Unit 455ae32a-8faa-428c-ad92-18a816627f96 has been deleted\",\"requestUuid\":\"695a9943-de23-4ce0-852b-7998a9bc1708\",\"deletedId\":\"455ae32a-8faa-428c-ad92-18a816627f96\"}
    (GET /candlepin/consumers/455ae32a-8faa-428c-ad92-18a816627f96)"]}

    it seems something was deleted, I am not quite clear what was that, and how we can fix it.

    Any one has any ideas please?

    Thanks
    Sinux


You received this message because you are subscribed to the Google
Groups “Foreman users” group.
To unsubscribe from this group and stop receiving emails from it, send
an email to foreman-users+unsubscribe@googlegroups.com
mailto:foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com
mailto:foreman-users@googlegroups.com.
Visit this group at http://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

Hi, Daniel,

Is there any more info I need to supply? I was still stuck here, I can't
get any content hosts listed in both WebGUI or hammer shell, but the
response server gave me was all the same, something like:

> content-host list --organization odw
Katello::Resources::Candlepin::Consumer: 410 Gone {"displayMessage":"Unit
455ae32a-8faa-428c-ad92-18a816627f96 has been
deleted","requestUuid":"8ba7b052-f7cf-4de0-b1d3-fa37805ef9ab","deletedId":"455ae32a-8faa-428c-ad92-18a816627f96"}
(GET /candlepin/consumers/455ae32a-8faa-428c-ad92-18a816627f96)

··· On Wednesday, May 20, 2015 at 11:37:09 AM UTC+8, sinux shen wrote: > > Hi Daniel, > > Thanks for the quick response, I ran the command, it shows some errors: > > katello ODW [/home/sxin] 1000# foreman-rake katello:clean_backend_objects > System 110 dogtag.anim.odw.com.cn 455ae32a-8faa-428c-ad92-18a816627f96 is > partially missing. Cleaning. > rake aborted! > 500 Internal Server Error > > Tasks: TOP => katello:clean_backend_objects > (See full trace by running task with --trace) > > and then I re-run this command by adding --trace > katello ODW [/home/sxin] 1001# foreman-rake katello:clean_backend_objects > --trace > ** Invoke katello:clean_backend_objects (first_time) > ** Invoke environment (first_time) > ** Execute environment > ** Execute katello:clean_backend_objects > System 110 dogtag.anim.odw.com.cn 455ae32a-8faa-428c-ad92-18a816627f96 is > partially missing. Cleaning. > rake aborted! > 500 Internal Server Error > /opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/abstract_response.rb:48:in > `return!' > /opt/rh/ruby193/root/usr/share/gems/gems/runcible-1.3.0/lib/runcible/base.rb:92:in > `block in get_response' > /opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:228:in > `call' > /opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:228:in > `process_result' > /opt/rh/ruby193/root/usr/share/gems/gems/rbovirt-0.0.29/lib/restclient_ext/request.rb:50:in > `block in transmit' > /opt/rh/ruby193/root/usr/share/ruby/net/http.rb:746:in `start' > /opt/rh/ruby193/root/usr/share/gems/gems/rbovirt-0.0.29/lib/restclient_ext/request.rb:44:in > `transmit' > /opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:64:in > `execute' > /opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:33:in > `execute' > /opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/resource.rb:94:in > `delete' > /opt/rh/ruby193/root/usr/share/gems/gems/runcible-1.3.0/lib/runcible/base.rb:91:in > `get_response' > /opt/rh/ruby193/root/usr/share/gems/gems/runcible-1.3.0/lib/runcible/base.rb:82:in > `call' > /opt/rh/ruby193/root/usr/share/gems/gems/runcible-1.3.0/lib/runcible/resources/consumer.rb:68:in > `delete' > /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.1.0/app/models/katello/glue/pulp/consumer.rb:127:in > `del_pulp_consumer' > /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.1.0/lib/katello/tasks/clean_backend_objects.rake:31:in > `block in cleanup_systems' > /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/relation/batches.rb:21:in > `block (2 levels) in find_each' > /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/relation/batches.rb:21:in > `each' > /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/relation/batches.rb:21:in > `block in find_each' > /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/relation/batches.rb:72:in > `find_in_batches' > /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/relation/batches.rb:20:in > `find_each' > /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/querying.rb:8:in > `find_each' > /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.1.0/lib/katello/tasks/clean_backend_objects.rake:5:in > `cleanup_systems' > /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.1.0/lib/katello/tasks/clean_backend_objects.rake:55:in > `block (2 levels) in ' > /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:205:in `call' > /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:205:in `block in execute' > /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:200:in `each' > /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:200:in `execute' > /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:158:in `block in > invoke_with_call_chain' > /opt/rh/ruby193/root/usr/share/ruby/monitor.rb:211:in `mon_synchronize' > /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:151:in > `invoke_with_call_chain' > /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:144:in `invoke' > /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:116:in > `invoke_task' > /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:94:in `block (2 > levels) in top_level' > /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:94:in `each' > /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:94:in `block in > top_level' > /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:133:in > `standard_exception_handling' > /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:88:in `top_level' > /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:66:in `block in > run' > /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:133:in > `standard_exception_handling' > /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:63:in `run' > /opt/rh/ruby193/root/usr/bin/rake:32:in `' > Tasks: TOP => katello:clean_backend_objects > > Then I went back the the foreman GUI, I click Hosts--> Content Hosts, but > it was still keep showing the circular loading icon, firebug show the same > 400 GET error, like what I mentioned previously. > > Thank you so much for you help. > > > On Tuesday, May 19, 2015 at 4:17:05 PM UTC+8, Daniel Lobato wrote: >> >> On 05/18, sinux shen wrote: >> > I alsl get the error from the katello log: >> > >> > Processing by Katello::Api::V2::SystemsController#index as JSON >> > Parameters: {"organization_id"=>"3", "page"=>"1", "search"=>"", >> > "sort_by"=>"name", "sort_order"=>"ASC", "api_version"=>"v2"} >> > Failed to retrieve all Katello::System search results (20/20 found.) >> > Rendered >> > >> /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.1.0/app/views/katello/api/v2/systems/index.json.rabl >> >> > within katello/api/v2/layouts/collection (5821.9ms) >> > exception when talking to a remote client: >> > Katello::Resources::Candlepin::Consumer: 410 Gone >> {"displayMessage":"Unit >> > 455ae32a-8faa-428c-ad92-18a816627f96 has been >> > >> deleted","requestUuid":"a748e055-ac6f-46a7-a957-d60373944e62","deletedId":"455ae32a-8faa-428c-ad92-18a816627f96"} >> >> > (GET /candlepin/consumers/455ae32a-8faa-428c-ad92-18a816627f96) >> > RestClient::Gone: Katello::Resources::Candlepin::Consumer: 410 Gone >> > {"displayMessage":"Unit 455ae32a-8faa-428c-ad92-18a816627f96 has been >> > >> deleted","requestUuid":"a748e055-ac6f-46a7-a957-d60373944e62","deletedId":"455ae32a-8faa-428c-ad92-18a816627f96"} >> >> > (GET /candlepin/consumers/455ae32a-8faa-428c-ad92-18a816627f96) >> > Body: {"displayMessage":"Unit 455ae32a-8faa-428c-ad92-18a816627f96 has >> been >> > >> deleted","requestUuid":"a748e055-ac6f-46a7-a957-d60373944e62","deletedId":"455ae32a-8faa-428c-ad92-18a816627f96"} >> >> > >> >> Thanks Sinux, logs are helpful. >> >> Could you try to run foreman-rake katello:clean_backend_objects as root >> and see >> if you still get the same error? >> >> > >> > On Tuesday, May 19, 2015 at 11:29:38 AM UTC+8, sinux shen wrote: >> > > >> > > Hi there, >> > > >> > > I was trying to upgrade to katello 2.2, but for some reason, I >> failed, the >> > > reason I want to upgrade is because I have some issue running on 2.1, >> one >> > > of the issue was I can not list content hosts in "Hosts"--> "Content >> Hosts" >> > > page, it only showed the circular loading icon spinning continuously, >> > > After analyzing via firebug, I found one GET method got 400 return >> code, >> > > the get url is: >> > > >> https://katello.anim.odw.com.cn/katello/api/v2/systems?organization_id=3&page=1&search=&sort_by=name&sort_order=ASC, >> >> > > looks normal, but the response I got was: >> > > >> > > {"displayMessage":"Katello::Resources::Candlepin::Consumer: 410 Gone >> {\"displayMessage\":\"Unit 455ae32a-8faa-428c-ad92-18a816627f96 has been >> deleted\",\"requestUuid\":\"695a9943-de23-4ce0-852b-7998a9bc1708\",\"deletedId\":\"455ae32a-8faa-428c-ad92-18a816627f96\"} >> >> > > (GET >> /candlepin/consumers/455ae32a-8faa-428c-ad92-18a816627f96)","errors":["Katello::Resources::Candlepin::Consumer: >> 410 Gone >> > > {\"displayMessage\":\"Unit 455ae32a-8faa-428c-ad92-18a816627f96 has >> been >> deleted\",\"requestUuid\":\"695a9943-de23-4ce0-852b-7998a9bc1708\",\"deletedId\":\"455ae32a-8faa-428c-ad92-18a816627f96\"} >> >> > > (GET /candlepin/consumers/455ae32a-8faa-428c-ad92-18a816627f96)"]} >> > > >> > > it seems something was deleted, I am not quite clear what was that, >> and how we can fix it. >> > > >> > > Any one has any ideas please? >> > > >> > > Thanks >> > > Sinux >> > > >> > > >> > > >> > >> > -- >> > You received this message because you are subscribed to the Google >> Groups "Foreman users" group. >> > To unsubscribe from this group and stop receiving emails from it, send >> an email to foreman-user...@googlegroups.com. >> > To post to this group, send email to forema...@googlegroups.com. >> > Visit this group at http://groups.google.com/group/foreman-users. >> > For more options, visit https://groups.google.com/d/optout. >> >> Best, >> >> -- >> Daniel Lobato Garcia >> >> @eLobatoss >> blog.daniellobato.me >> daniellobato.me >> >> GPG: http://keys.gnupg.net/pks/lookup?op=get&search=0x7A92D6DD38D6DE30 >> Keybase: https://keybase.io/elobato >> >

Was this before or after upgrading? Can you re-run the
clean_backend_objects command and then check messages/journalctl to see if
there are pulp errors listed?

··· On Tue, May 19, 2015 at 11:37 PM, sinux shen wrote:

Hi Daniel,

Thanks for the quick response, I ran the command, it shows some errors:

katello ODW [/home/sxin] 1000# foreman-rake katello:clean_backend_objects
System 110 dogtag.anim.odw.com.cn 455ae32a-8faa-428c-ad92-18a816627f96 is
partially missing. Cleaning.
rake aborted!
500 Internal Server Error

Tasks: TOP => katello:clean_backend_objects
(See full trace by running task with --trace)

and then I re-run this command by adding --trace
katello ODW [/home/sxin] 1001# foreman-rake katello:clean_backend_objects
–trace
** Invoke katello:clean_backend_objects (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute katello:clean_backend_objects
System 110 dogtag.anim.odw.com.cn 455ae32a-8faa-428c-ad92-18a816627f96 is
partially missing. Cleaning.
rake aborted!
500 Internal Server Error
/opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/abstract_response.rb:48:in
return!' /opt/rh/ruby193/root/usr/share/gems/gems/runcible-1.3.0/lib/runcible/base.rb:92:inblock in get_response’
/opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:228:in
call' /opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:228:inprocess_result’
/opt/rh/ruby193/root/usr/share/gems/gems/rbovirt-0.0.29/lib/restclient_ext/request.rb:50:in
block in transmit' /opt/rh/ruby193/root/usr/share/ruby/net/http.rb:746:instart’
/opt/rh/ruby193/root/usr/share/gems/gems/rbovirt-0.0.29/lib/restclient_ext/request.rb:44:in
transmit' /opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:64:inexecute’
/opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:33:in
execute' /opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/resource.rb:94:indelete’
/opt/rh/ruby193/root/usr/share/gems/gems/runcible-1.3.0/lib/runcible/base.rb:91:in
get_response' /opt/rh/ruby193/root/usr/share/gems/gems/runcible-1.3.0/lib/runcible/base.rb:82:incall’
/opt/rh/ruby193/root/usr/share/gems/gems/runcible-1.3.0/lib/runcible/resources/consumer.rb:68:in
delete' /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.1.0/app/models/katello/glue/pulp/consumer.rb:127:indel_pulp_consumer’
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.1.0/lib/katello/tasks/clean_backend_objects.rake:31:in
block in cleanup_systems' /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/relation/batches.rb:21:inblock (2 levels) in find_each’
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/relation/batches.rb:21:in
each' /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/relation/batches.rb:21:inblock in find_each’
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/relation/batches.rb:72:in
find_in_batches' /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/relation/batches.rb:20:infind_each’
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/querying.rb:8:in
find_each' /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.1.0/lib/katello/tasks/clean_backend_objects.rake:5:incleanup_systems’
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.1.0/lib/katello/tasks/clean_backend_objects.rake:55:in
block (2 levels) in <top (required)>' /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:205:incall’
/opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:205:in block in execute' /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:200:ineach’
/opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:200:in execute' /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:158:inblock in
invoke_with_call_chain’
/opt/rh/ruby193/root/usr/share/ruby/monitor.rb:211:in mon_synchronize' /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:151:ininvoke_with_call_chain’
/opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:144:in invoke' /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:116:ininvoke_task’
/opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:94:in block (2 levels) in top_level' /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:94:ineach’
/opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:94:in block in top_level' /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:133:instandard_exception_handling’
/opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:88:in top_level' /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:66:inblock in
run’
/opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:133:in
standard_exception_handling' /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:63:inrun’
/opt/rh/ruby193/root/usr/bin/rake:32:in `'
Tasks: TOP => katello:clean_backend_objects

Then I went back the the foreman GUI, I click Hosts–> Content Hosts, but
it was still keep showing the circular loading icon, firebug show the same
400 GET error, like what I mentioned previously.

Thank you so much for you help.

On Tuesday, May 19, 2015 at 4:17:05 PM UTC+8, Daniel Lobato wrote:

On 05/18, sinux shen wrote:

I alsl get the error from the katello log:

Processing by Katello::Api::V2::SystemsController#index as JSON
Parameters: {“organization_id”=>“3”, “page”=>“1”, “search”=>"",
“sort_by”=>“name”, “sort_order”=>“ASC”, “api_version”=>“v2”}
Failed to retrieve all Katello::System search results (20/20 found.)
Rendered

/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.1.0/app/views/katello/api/v2/systems/index.json.rabl

within katello/api/v2/layouts/collection (5821.9ms)
exception when talking to a remote client:
Katello::Resources::Candlepin::Consumer: 410 Gone
{“displayMessage”:"Unit
455ae32a-8faa-428c-ad92-18a816627f96 has been

deleted",“requestUuid”:“a748e055-ac6f-46a7-a957-d60373944e62”,“deletedId”:“455ae32a-8faa-428c-ad92-18a816627f96”}

(GET /candlepin/consumers/455ae32a-8faa-428c-ad92-18a816627f96)
RestClient::Gone: Katello::Resources::Candlepin::Consumer: 410 Gone
{“displayMessage”:"Unit 455ae32a-8faa-428c-ad92-18a816627f96 has been

deleted",“requestUuid”:“a748e055-ac6f-46a7-a957-d60373944e62”,“deletedId”:“455ae32a-8faa-428c-ad92-18a816627f96”}

(GET /candlepin/consumers/455ae32a-8faa-428c-ad92-18a816627f96)
Body: {“displayMessage”:"Unit 455ae32a-8faa-428c-ad92-18a816627f96 has
been

deleted",“requestUuid”:“a748e055-ac6f-46a7-a957-d60373944e62”,“deletedId”:“455ae32a-8faa-428c-ad92-18a816627f96”}

Thanks Sinux, logs are helpful.

Could you try to run foreman-rake katello:clean_backend_objects as root
and see
if you still get the same error?

On Tuesday, May 19, 2015 at 11:29:38 AM UTC+8, sinux shen wrote:

Hi there,

I was trying to upgrade to katello 2.2, but for some reason, I
failed, the

reason I want to upgrade is because I have some issue running on 2.1,
one

of the issue was I can not list content hosts in “Hosts”–> “Content
Hosts”

page, it only showed the circular loading icon spinning continuously,
After analyzing via firebug, I found one GET method got 400 return
code,

the get url is:

https://katello.anim.odw.com.cn/katello/api/v2/systems?organization_id=3&page=1&search=&sort_by=name&sort_order=ASC,

looks normal, but the response I got was:

{“displayMessage”:"Katello::Resources::Candlepin::Consumer: 410 Gone
{“displayMessage”:“Unit 455ae32a-8faa-428c-ad92-18a816627f96 has been
deleted”,“requestUuid”:“695a9943-de23-4ce0-852b-7998a9bc1708”,“deletedId”:“455ae32a-8faa-428c-ad92-18a816627f96”}

(GET
/candlepin/consumers/455ae32a-8faa-428c-ad92-18a816627f96)",“errors”:["Katello::Resources::Candlepin::Consumer:
410 Gone

{“displayMessage”:“Unit 455ae32a-8faa-428c-ad92-18a816627f96 has
been
deleted”,“requestUuid”:“695a9943-de23-4ce0-852b-7998a9bc1708”,“deletedId”:“455ae32a-8faa-428c-ad92-18a816627f96”}

(GET /candlepin/consumers/455ae32a-8faa-428c-ad92-18a816627f96)"]}

it seems something was deleted, I am not quite clear what was that,
and how we can fix it.

Any one has any ideas please?

Thanks
Sinux


You received this message because you are subscribed to the Google
Groups “Foreman users” group.
To unsubscribe from this group and stop receiving emails from it, send
an email to foreman-user...@googlegroups.com.
To post to this group, send email to forema...@googlegroups.com.
Visit this group at http://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

Best,


Daniel Lobato Garcia

@eLobatoss
blog.daniellobato.me
daniellobato.me

GPG: http://keys.gnupg.net/pks/lookup?op=get&search=0x7A92D6DD38D6DE30
Keybase: https://keybase.io/elobato


You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at http://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

Thanks Justin,

Yes, indeed, it's something wrong with the content host, from the error
here:
katello ODW [/home/sxin] 1001# foreman-rake katello:clean_backend_objects
System 110 dogtag.anim.odw.com.cn 455ae32a-8faa-428c-ad92-18a816627f96 is
partially missing. Cleaning.
System 111 dogtag.anim.odw.com.cn 2d734c21-8c11-4565-987f-
d338840d0ab2 is partially missing. Cleaning.
rake aborted!
PGError: 错误: 在 "katello_systems" 上的更新或删除操作违反了在
"katello_system_repositories" 上的外键约束
"katello_system_repositories_system_id_fk"
DETAIL: 键值对(id)=(111)仍然是从表"katello_system_repositories"引用的.
: DELETE FROM "katello_systems" WHERE "katello_systems"."id" = 111

but I can't find this content hosts any more.

Anyway, I will run foreman-debug and let you know what happened, thanks
again.

Thanks
Sinux

··· On Wednesday, May 27, 2015 at 10:33:32 PM UTC+8, jsherril wrote: > > On 05/26/2015 01:42 AM, sinux shen wrote: > > Hi Eric & Daniel, > > Can anyone of you help with this or please let me know if you need more > infomations, I was stuck here and don't know what to do to fix it. > > > The 500 Internal Server Error from candlepin is concerning and likely the > cause of your content host issue. Would you be able to run foreman-debug > and upload the results (according to the instructions given at the end of > the command)? Then simply reply with the filename it created. > > Thanks, > > Justin > > > Thanks > Sinux > > On Monday, May 25, 2015 at 10:58:15 AM UTC+8, sinux shen wrote: >> >> I ran clean_backend_objects command again, this time, the error was a >> little different, >> >> before I ran this, I checked the message log, and I found tons of error >> like this: >> May 25 10:46:11 katello qpidd[5978]: 2015-05-25 10:46:11 [System] error >> Could not accept socket: Too many open files >> (/builddir/build/BUILD/qpid-0.26/cpp/src/qpid/sys/ssl/SslSocket.cpp:195) >> May 25 10:46:11 katello qpidd[5978]: 2015-05-25 10:46:11 [System] error >> Could not accept socket: Too many open files >> (/builddir/build/BUILD/qpid-0.26/cpp/src/qpid/sys/ssl/SslSocket.cpp:195) >> May 25 10:46:11 katello qpidd[5978]: 2015-05-25 10:46:11 [System] error >> Could not accept socket: Too many open files >> (/builddir/build/BUILD/qpid-0.26/cpp/src/qpid/sys/ssl/SslSocket.cpp:195) >> >> And then I restarted qpidd, and I ran clean_backend_objects, it gave me >> the following errors: >> katello ODW [/home/sxin] 1001# foreman-rake katello:clean_backend_objects >> System 110 dogtag.anim.odw.com.cn 455ae32a-8faa-428c-ad92-18a816627f96 >> is partially missing. Cleaning. >> System 111 dogtag.anim.odw.com.cn 2d734c21-8c11-4565-987f-d338840d0ab2 >> is partially missing. Cleaning. >> rake aborted! >> PGError: 错误: 在 "katello_systems" 上的更新或删除操作违反了在 >> "katello_system_repositories" 上的外键约束 >> "katello_system_repositories_system_id_fk" >> DETAIL: 键值对(id)=(111)仍然是从表"katello_system_repositories" 引用的. >> : DELETE FROM "katello_systems" WHERE "katello_systems"."id" = 111 >> >> Tasks: TOP => katello:clean_backend_objects >> (See full trace by running task with --trace) >> And in message, it has one lines of log shows: >> May 25 10:50:39 katello pulp: >> pulp.server.webservices.middleware.exception:INFO: Missing resource(s): >> consumer=455ae32a-8faa-428c-ad92-18a816627f96 >> >> >> Sorry, it contains some Chinese, I remember that when I tried to upgrade >> katello to 2.2, it has similar errors. >> >> >> On Tuesday, May 19, 2015 at 11:29:38 AM UTC+8, sinux shen wrote: >>> >>> Hi there, >>> >>> I was trying to upgrade to katello 2.2, but for some reason, I failed, >>> the reason I want to upgrade is because I have some issue running on 2.1, >>> one of the issue was I can not list content hosts in "Hosts"--> "Content >>> Hosts" page, it only showed the circular loading icon spinning >>> continuously, >>> After analyzing via firebug, I found one GET method got 400 return code, >>> the get url is: >>> https://katello.anim.odw.com.cn/katello/api/v2/systems?organization_id=3&page=1&search=&sort_by=name&sort_order=ASC, >>> looks normal, but the response I got was: >>> >>> {"displayMessage":"Katello::Resources::Candlepin::Consumer: 410 Gone {\"displayMessage\":\"Unit 455ae32a-8faa-428c-ad92-18a816627f96 has been deleted\",\"requestUuid\":\"695a9943-de23-4ce0-852b-7998a9bc1708\",\"deletedId\":\"455ae32a-8faa-428c-ad92-18a816627f96\"} >>> (GET /candlepin/consumers/455ae32a-8faa-428c-ad92-18a816627f96)","errors":["Katello::Resources::Candlepin::Consumer: 410 Gone >>> {\"displayMessage\":\"Unit 455ae32a-8faa-428c-ad92-18a816627f96 has been deleted\",\"requestUuid\":\"695a9943-de23-4ce0-852b-7998a9bc1708\",\"deletedId\":\"455ae32a-8faa-428c-ad92-18a816627f96\"} >>> (GET /candlepin/consumers/455ae32a-8faa-428c-ad92-18a816627f96)"]} >>> >>> it seems something was deleted, I am not quite clear what was that, and how we can fix it. >>> >>> Any one has any ideas please? >>> >>> Thanks >>> Sinux >>> >>> >>> -- > You received this message because you are subscribed to the Google Groups > "Foreman users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to foreman-user...@googlegroups.com . > To post to this group, send email to forema...@googlegroups.com > . > Visit this group at http://groups.google.com/group/foreman-users. > For more options, visit https://groups.google.com/d/optout. > > >

it was before the upgrading, aslo, I tried to upgrade, but it failed, I
have another email thread ask about that.

··· On Thursday, May 21, 2015 at 8:47:31 PM UTC+8, Eric Helms wrote: > > Was this before or after upgrading? Can you re-run the > clean_backend_objects command and then check messages/journalctl to see if > there are pulp errors listed? > > On Tue, May 19, 2015 at 11:37 PM, sinux shen > wrote: > >> Hi Daniel, >> >> Thanks for the quick response, I ran the command, it shows some errors: >> >> katello ODW [/home/sxin] 1000# foreman-rake katello:clean_backend_objects >> System 110 dogtag.anim.odw.com.cn 455ae32a-8faa-428c-ad92-18a816627f96 >> is partially missing. Cleaning. >> rake aborted! >> 500 Internal Server Error >> >> Tasks: TOP => katello:clean_backend_objects >> (See full trace by running task with --trace) >> >> and then I re-run this command by adding --trace >> katello ODW [/home/sxin] 1001# foreman-rake katello:clean_backend_objects >> --trace >> ** Invoke katello:clean_backend_objects (first_time) >> ** Invoke environment (first_time) >> ** Execute environment >> ** Execute katello:clean_backend_objects >> System 110 dogtag.anim.odw.com.cn 455ae32a-8faa-428c-ad92-18a816627f96 >> is partially missing. Cleaning. >> rake aborted! >> 500 Internal Server Error >> /opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/abstract_response.rb:48:in >> `return!' >> /opt/rh/ruby193/root/usr/share/gems/gems/runcible-1.3.0/lib/runcible/base.rb:92:in >> `block in get_response' >> /opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:228:in >> `call' >> /opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:228:in >> `process_result' >> /opt/rh/ruby193/root/usr/share/gems/gems/rbovirt-0.0.29/lib/restclient_ext/request.rb:50:in >> `block in transmit' >> /opt/rh/ruby193/root/usr/share/ruby/net/http.rb:746:in `start' >> /opt/rh/ruby193/root/usr/share/gems/gems/rbovirt-0.0.29/lib/restclient_ext/request.rb:44:in >> `transmit' >> /opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:64:in >> `execute' >> /opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:33:in >> `execute' >> /opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/resource.rb:94:in >> `delete' >> /opt/rh/ruby193/root/usr/share/gems/gems/runcible-1.3.0/lib/runcible/base.rb:91:in >> `get_response' >> /opt/rh/ruby193/root/usr/share/gems/gems/runcible-1.3.0/lib/runcible/base.rb:82:in >> `call' >> /opt/rh/ruby193/root/usr/share/gems/gems/runcible-1.3.0/lib/runcible/resources/consumer.rb:68:in >> `delete' >> /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.1.0/app/models/katello/glue/pulp/consumer.rb:127:in >> `del_pulp_consumer' >> /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.1.0/lib/katello/tasks/clean_backend_objects.rake:31:in >> `block in cleanup_systems' >> /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/relation/batches.rb:21:in >> `block (2 levels) in find_each' >> /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/relation/batches.rb:21:in >> `each' >> /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/relation/batches.rb:21:in >> `block in find_each' >> /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/relation/batches.rb:72:in >> `find_in_batches' >> /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/relation/batches.rb:20:in >> `find_each' >> /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/querying.rb:8:in >> `find_each' >> /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.1.0/lib/katello/tasks/clean_backend_objects.rake:5:in >> `cleanup_systems' >> /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.1.0/lib/katello/tasks/clean_backend_objects.rake:55:in >> `block (2 levels) in ' >> /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:205:in `call' >> /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:205:in `block in execute' >> /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:200:in `each' >> /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:200:in `execute' >> /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:158:in `block in >> invoke_with_call_chain' >> /opt/rh/ruby193/root/usr/share/ruby/monitor.rb:211:in `mon_synchronize' >> /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:151:in >> `invoke_with_call_chain' >> /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:144:in `invoke' >> /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:116:in >> `invoke_task' >> /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:94:in `block (2 >> levels) in top_level' >> /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:94:in `each' >> /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:94:in `block in >> top_level' >> /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:133:in >> `standard_exception_handling' >> /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:88:in `top_level' >> /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:66:in `block in >> run' >> /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:133:in >> `standard_exception_handling' >> /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:63:in `run' >> /opt/rh/ruby193/root/usr/bin/rake:32:in `' >> Tasks: TOP => katello:clean_backend_objects >> >> Then I went back the the foreman GUI, I click Hosts--> Content Hosts, but >> it was still keep showing the circular loading icon, firebug show the same >> 400 GET error, like what I mentioned previously. >> >> Thank you so much for you help. >> >> >> >> On Tuesday, May 19, 2015 at 4:17:05 PM UTC+8, Daniel Lobato wrote: >> >>> On 05/18, sinux shen wrote: >>> > I alsl get the error from the katello log: >>> > >>> > Processing by Katello::Api::V2::SystemsController#index as JSON >>> > Parameters: {"organization_id"=>"3", "page"=>"1", "search"=>"", >>> > "sort_by"=>"name", "sort_order"=>"ASC", "api_version"=>"v2"} >>> > Failed to retrieve all Katello::System search results (20/20 found.) >>> > Rendered >>> > >>> /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.1.0/app/views/katello/api/v2/systems/index.json.rabl >>> >>> > within katello/api/v2/layouts/collection (5821.9ms) >>> > exception when talking to a remote client: >>> > Katello::Resources::Candlepin::Consumer: 410 Gone >>> {"displayMessage":"Unit >>> > 455ae32a-8faa-428c-ad92-18a816627f96 has been >>> > >>> deleted","requestUuid":"a748e055-ac6f-46a7-a957-d60373944e62","deletedId":"455ae32a-8faa-428c-ad92-18a816627f96"} >>> >>> > (GET /candlepin/consumers/455ae32a-8faa-428c-ad92-18a816627f96) >>> > RestClient::Gone: Katello::Resources::Candlepin::Consumer: 410 Gone >>> > {"displayMessage":"Unit 455ae32a-8faa-428c-ad92-18a816627f96 has been >>> > >>> deleted","requestUuid":"a748e055-ac6f-46a7-a957-d60373944e62","deletedId":"455ae32a-8faa-428c-ad92-18a816627f96"} >>> >>> > (GET /candlepin/consumers/455ae32a-8faa-428c-ad92-18a816627f96) >>> > Body: {"displayMessage":"Unit 455ae32a-8faa-428c-ad92-18a816627f96 has >>> been >>> > >>> deleted","requestUuid":"a748e055-ac6f-46a7-a957-d60373944e62","deletedId":"455ae32a-8faa-428c-ad92-18a816627f96"} >>> >>> > >>> >>> Thanks Sinux, logs are helpful. >>> >>> Could you try to run foreman-rake katello:clean_backend_objects as root >>> and see >>> if you still get the same error? >>> >>> > >>> > On Tuesday, May 19, 2015 at 11:29:38 AM UTC+8, sinux shen wrote: >>> > > >>> > > Hi there, >>> > > >>> > > I was trying to upgrade to katello 2.2, but for some reason, I >>> failed, the >>> > > reason I want to upgrade is because I have some issue running on >>> 2.1, one >>> > > of the issue was I can not list content hosts in "Hosts"--> "Content >>> Hosts" >>> > > page, it only showed the circular loading icon spinning >>> continuously, >>> > > After analyzing via firebug, I found one GET method got 400 return >>> code, >>> > > the get url is: >>> > > >>> https://katello.anim.odw.com.cn/katello/api/v2/systems?organization_id=3&page=1&search=&sort_by=name&sort_order=ASC, >>> >>> > > looks normal, but the response I got was: >>> > > >>> > > {"displayMessage":"Katello::Resources::Candlepin::Consumer: 410 Gone >>> {\"displayMessage\":\"Unit 455ae32a-8faa-428c-ad92-18a816627f96 has been >>> deleted\",\"requestUuid\":\"695a9943-de23-4ce0-852b-7998a9bc1708\",\"deletedId\":\"455ae32a-8faa-428c-ad92-18a816627f96\"} >>> >>> > > (GET >>> /candlepin/consumers/455ae32a-8faa-428c-ad92-18a816627f96)","errors":["Katello::Resources::Candlepin::Consumer: >>> 410 Gone >>> > > {\"displayMessage\":\"Unit 455ae32a-8faa-428c-ad92-18a816627f96 has >>> been >>> deleted\",\"requestUuid\":\"695a9943-de23-4ce0-852b-7998a9bc1708\",\"deletedId\":\"455ae32a-8faa-428c-ad92-18a816627f96\"} >>> >>> > > (GET /candlepin/consumers/455ae32a-8faa-428c-ad92-18a816627f96)"]} >>> > > >>> > > it seems something was deleted, I am not quite clear what was that, >>> and how we can fix it. >>> > > >>> > > Any one has any ideas please? >>> > > >>> > > Thanks >>> > > Sinux >>> > > >>> > > >>> > > >>> > >>> > -- >>> > You received this message because you are subscribed to the Google >>> Groups "Foreman users" group. >>> > To unsubscribe from this group and stop receiving emails from it, send >>> an email to foreman-user...@googlegroups.com. >>> > To post to this group, send email to forema...@googlegroups.com. >>> > Visit this group at http://groups.google.com/group/foreman-users. >>> > For more options, visit https://groups.google.com/d/optout. >>> >>> Best, >>> >>> -- >>> Daniel Lobato Garcia >>> >>> @eLobatoss >>> blog.daniellobato.me >>> daniellobato.me >>> >>> GPG: http://keys.gnupg.net/pks/lookup?op=get&search=0x7A92D6DD38D6DE30 >>> Keybase: https://keybase.io/elobato >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Foreman users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to foreman-user...@googlegroups.com . >> To post to this group, send email to forema...@googlegroups.com >> . >> Visit this group at http://groups.google.com/group/foreman-users. >> For more options, visit https://groups.google.com/d/optout. >> > >

Hi Justin,

Can you please check if this has been uploaded? File name
foreman-debug-aHtUS.tar.xz
the out put was a little confused though:
Uploading…
rsync: failed to connect to theforeman.org: Connection timed out (110)
rsync error: error in socket IO (code 10) at clientserver.c(124)
[sender=3.0.6]
The tarball has been uploaded, please contact us on mailing list or IRC.

··· On Thursday, May 28, 2015 at 9:53:39 AM UTC+8, sinux shen wrote: > > Thanks Justin, > > Yes, indeed, it's something wrong with the content host, from the error > here: > katello ODW [/home/sxin] 1001# foreman-rake katello:clean_backend_objects > System 110 dogtag.anim.odw.com.cn 455ae32a-8faa-428c-ad92-18a816627f96 is > partially missing. Cleaning. > System 111 dogtag.anim.odw.com.cn 2d734c21-8c11-4565-987f- > d338840d0ab2 is partially missing. Cleaning. > rake aborted! > PGError: 错误: 在 "katello_systems" 上的更新或删除操作违反了在 > "katello_system_repositories" 上的外键约束 > "katello_system_repositories_system_id_fk" > DETAIL: 键值对(id)=(111)仍然是从表"katello_system_repositories"引用的. > : DELETE FROM "katello_systems" WHERE "katello_systems"."id" = 111 > > but I can't find this content hosts any more. > > Anyway, I will run foreman-debug and let you know what happened, thanks > again. > > Thanks > Sinux > > > On Wednesday, May 27, 2015 at 10:33:32 PM UTC+8, jsherril wrote: >> >> On 05/26/2015 01:42 AM, sinux shen wrote: >> >> Hi Eric & Daniel, >> >> Can anyone of you help with this or please let me know if you need more >> infomations, I was stuck here and don't know what to do to fix it. >> >> >> The 500 Internal Server Error from candlepin is concerning and likely the >> cause of your content host issue. Would you be able to run foreman-debug >> and upload the results (according to the instructions given at the end of >> the command)? Then simply reply with the filename it created. >> >> Thanks, >> >> Justin >> >> >> Thanks >> Sinux >> >> On Monday, May 25, 2015 at 10:58:15 AM UTC+8, sinux shen wrote: >>> >>> I ran clean_backend_objects command again, this time, the error was a >>> little different, >>> >>> before I ran this, I checked the message log, and I found tons of error >>> like this: >>> May 25 10:46:11 katello qpidd[5978]: 2015-05-25 10:46:11 [System] error >>> Could not accept socket: Too many open files >>> (/builddir/build/BUILD/qpid-0.26/cpp/src/qpid/sys/ssl/SslSocket.cpp:195) >>> May 25 10:46:11 katello qpidd[5978]: 2015-05-25 10:46:11 [System] error >>> Could not accept socket: Too many open files >>> (/builddir/build/BUILD/qpid-0.26/cpp/src/qpid/sys/ssl/SslSocket.cpp:195) >>> May 25 10:46:11 katello qpidd[5978]: 2015-05-25 10:46:11 [System] error >>> Could not accept socket: Too many open files >>> (/builddir/build/BUILD/qpid-0.26/cpp/src/qpid/sys/ssl/SslSocket.cpp:195) >>> >>> And then I restarted qpidd, and I ran clean_backend_objects, it gave me >>> the following errors: >>> katello ODW [/home/sxin] 1001# foreman-rake katello:clean_backend_objects >>> System 110 dogtag.anim.odw.com.cn 455ae32a-8faa-428c-ad92-18a816627f96 >>> is partially missing. Cleaning. >>> System 111 dogtag.anim.odw.com.cn 2d734c21-8c11-4565-987f-d338840d0ab2 >>> is partially missing. Cleaning. >>> rake aborted! >>> PGError: 错误: 在 "katello_systems" 上的更新或删除操作违反了在 >>> "katello_system_repositories" 上的外键约束 >>> "katello_system_repositories_system_id_fk" >>> DETAIL: 键值对(id)=(111)仍然是从表"katello_system_repositories" 引用的. >>> : DELETE FROM "katello_systems" WHERE "katello_systems"."id" = 111 >>> >>> Tasks: TOP => katello:clean_backend_objects >>> (See full trace by running task with --trace) >>> And in message, it has one lines of log shows: >>> May 25 10:50:39 katello pulp: >>> pulp.server.webservices.middleware.exception:INFO: Missing resource(s): >>> consumer=455ae32a-8faa-428c-ad92-18a816627f96 >>> >>> >>> Sorry, it contains some Chinese, I remember that when I tried to upgrade >>> katello to 2.2, it has similar errors. >>> >>> >>> On Tuesday, May 19, 2015 at 11:29:38 AM UTC+8, sinux shen wrote: >>>> >>>> Hi there, >>>> >>>> I was trying to upgrade to katello 2.2, but for some reason, I failed, >>>> the reason I want to upgrade is because I have some issue running on 2.1, >>>> one of the issue was I can not list content hosts in "Hosts"--> "Content >>>> Hosts" page, it only showed the circular loading icon spinning >>>> continuously, >>>> After analyzing via firebug, I found one GET method got 400 return >>>> code, the get url is: >>>> https://katello.anim.odw.com.cn/katello/api/v2/systems?organization_id=3&page=1&search=&sort_by=name&sort_order=ASC, >>>> looks normal, but the response I got was: >>>> >>>> {"displayMessage":"Katello::Resources::Candlepin::Consumer: 410 Gone {\"displayMessage\":\"Unit 455ae32a-8faa-428c-ad92-18a816627f96 has been deleted\",\"requestUuid\":\"695a9943-de23-4ce0-852b-7998a9bc1708\",\"deletedId\":\"455ae32a-8faa-428c-ad92-18a816627f96\"} >>>> (GET /candlepin/consumers/455ae32a-8faa-428c-ad92-18a816627f96)","errors":["Katello::Resources::Candlepin::Consumer: 410 Gone >>>> {\"displayMessage\":\"Unit 455ae32a-8faa-428c-ad92-18a816627f96 has been deleted\",\"requestUuid\":\"695a9943-de23-4ce0-852b-7998a9bc1708\",\"deletedId\":\"455ae32a-8faa-428c-ad92-18a816627f96\"} >>>> (GET /candlepin/consumers/455ae32a-8faa-428c-ad92-18a816627f96)"]} >>>> >>>> it seems something was deleted, I am not quite clear what was that, and how we can fix it. >>>> >>>> Any one has any ideas please? >>>> >>>> Thanks >>>> Sinux >>>> >>>> >>>> -- >> You received this message because you are subscribed to the Google Groups >> "Foreman users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to foreman-user...@googlegroups.com. >> To post to this group, send email to forema...@googlegroups.com. >> Visit this group at http://groups.google.com/group/foreman-users. >> For more options, visit https://groups.google.com/d/optout. >> >> >>

so I did an yum update, couple of foreman/katello/pulp packages were
updated, here is the version related to all of that:
katello ODW [/home/sxin] 1003# rpm -qa | egrep "katello|foreman|pulp"
python-isodate-0.5.0-4.pulp.el6.noarch
katello-installer-2.1.1-4.el6.noarch
pulp-server-2.5.1-1.el6.noarch
ruby193-rubygem-foreman-tasks-0.6.10-1.el6.noarch
katello.anim.odw.com.cn-puppet-client-1.0-1.noarch
python-pulp-puppet-common-2.5.1-2.katello.el6.noarch
pulp-nodes-common-2.5.1-1.el6.noarch
katello.anim.odw.com.cn-foreman-proxy-1.0-1.noarch
katello-installer-base-2.1.1-4.el6.noarch
rubygem-smart_proxy_pulp-1.0.1-1.el6.noarch
rubygem-hammer_cli_foreman-0.1.3-1.el6.noarch
foreman-libvirt-1.7.5-1.el6.noarch
katello.anim.odw.com.cn-foreman-client-1.0-1.noarch
python-pulp-rpm-common-2.5.1-1.el6.noarch
katello.anim.odw.com.cn-qpid-client-cert-1.0-1.noarch
pulp-nodes-parent-2.5.1-1.el6.noarch
rubygem-hammer_cli_katello-0.0.7-1.el6.noarch
foreman-postgresql-1.7.5-1.el6.noarch
katello-certs-tools-2.0.1-1.el6.noarch
mod_wsgi-3.4-2.pulp.el6.x86_64
python-kombu-3.0.15-12.pulp.el6.noarch
katello-repos-2.1.3-1.el6.noarch
foreman-selinux-1.7.5-1.el6.noarch
m2crypto-0.21.1.pulp-8.el6.x86_64
foreman-vmware-1.7.5-1.el6.noarch
katello-default-ca-1.0-1.noarch
katello.anim.odw.com.cn-qpid-broker-1.0-1.noarch
pulp-puppet-tools-2.5.1-2.katello.el6.noarch
katello-2.1.0-2.el6.noarch
foreman-compute-1.7.5-1.el6.noarch
ruby193-rubygem-foreman_docker-1.3.1-1.fm1_7.el6.noarch
rubygem-hammer_cli_foreman_tasks-0.0.3-2.201409091410gitc96619d.git.0.37f3704.el6.noarch
foreman-gce-1.7.5-1.el6.noarch
foreman-release-1.7.5-1.el6.noarch
ruby193-rubygem-foreman_hooks-0.3.7-2.el6.noarch
katello.anim.odw.com.cn-apache-1.0-1.noarch
katello.anim.odw.com.cn-parent-cert-1.0-1.noarch
pulp-puppet-plugins-2.5.1-2.katello.el6.noarch
rubygem-hammer_cli_foreman_bootdisk-0.1.2-1.el6.noarch
ruby193-rubygem-foreman_discovery-2.0.2-1.el6.noarch
pulp-katello-0.3-3.el6.noarch
foreman-1.7.5-1.el6.noarch
ruby193-rubygem-katello-2.1.2-3.el6.noarch
foreman-ovirt-1.7.5-1.el6.noarch
katello-server-ca-1.0-1.noarch
pulp-selinux-2.5.1-1.el6.noarch
python-pulp-common-2.5.1-1.el6.noarch
python-pulp-bindings-2.5.1-1.el6.noarch
pulp-docker-plugins-0.2.1-0.2.beta.el6.noarch
ruby193-rubygem-foreman_bootdisk-4.0.2-1.el6.noarch
foreman-proxy-1.7.5-1.el6.noarch
python-pulp-docker-common-0.2.1-0.2.beta.el6.noarch
pulp-rpm-plugins-2.5.1-1.el6.noarch

And I re-ran clean_backend_objects again, it seems to have the same error,
from the error, dogtag used to be one of the content host, how I can get
around this? here is the trace:
katello ODW [/home/sxin] 1002# foreman-rake katello:clean_backend_objects
–trace** Invoke katello:clean_backend_objects (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute katello:clean_backend_objects
System 110 dogtag.anim.odw.com.cn 455ae32a-8faa-428c-ad92-18a816627f96 is
partially missing. Cleaning.
rake aborted!
500 Internal Server Error
/opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/abstract_response.rb:48:in
return!&#39; /opt/rh/ruby193/root/usr/share/gems/gems/runcible-1.3.0/lib/runcible/base.rb:92:inblock in get_response'
/opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:228:in
call&#39; /opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:228:inprocess_result'
/opt/rh/ruby193/root/usr/share/gems/gems/rbovirt-0.0.29/lib/restclient_ext/request.rb:50:in
block in transmit&#39; /opt/rh/ruby193/root/usr/share/ruby/net/http.rb:746:instart'
/opt/rh/ruby193/root/usr/share/gems/gems/rbovirt-0.0.29/lib/restclient_ext/request.rb:44:in
transmit&#39; /opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:64:inexecute'
/opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:33:in
execute&#39; /opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/resource.rb:94:indelete'
/opt/rh/ruby193/root/usr/share/gems/gems/runcible-1.3.0/lib/runcible/base.rb:91:in
get_response&#39; /opt/rh/ruby193/root/usr/share/gems/gems/runcible-1.3.0/lib/runcible/base.rb:82:incall'
/opt/rh/ruby193/root/usr/share/gems/gems/runcible-1.3.0/lib/runcible/resources/consumer.rb:68:in
delete&#39; /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.1.2/app/models/katello/glue/pulp/consumer.rb:127:indel_pulp_consumer'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.1.2/lib/katello/tasks/clean_backend_objects.rake:31:in
block in cleanup_systems&#39; /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/relation/batches.rb:21:inblock (2 levels) in find_each'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/relation/batches.rb:21:in
each&#39; /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/relation/batches.rb:21:inblock in find_each'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/relation/batches.rb:72:in
find_in_batches&#39; /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/relation/batches.rb:20:infind_each'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/querying.rb:8:in
find_each&#39; /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.1.2/lib/katello/tasks/clean_backend_objects.rake:5:incleanup_systems'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.1.2/lib/katello/tasks/clean_backend_objects.rake:57:in
block (2 levels) in &lt;top (required)&gt;&#39; /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:205:incall'
/opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:205:in block in execute&#39; /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:200:ineach'
/opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:200:in execute&#39; /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:158:inblock in
invoke_with_call_chain'
/opt/rh/ruby193/root/usr/share/ruby/monitor.rb:211:in mon_synchronize&#39; /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:151:ininvoke_with_call_chain'
/opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:144:in invoke&#39; /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:116:ininvoke_task'
/opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:94:in block (2 levels) in top_level&#39; /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:94:ineach'
/opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:94:in block in top_level&#39; /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:133:instandard_exception_handling'
/opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:88:in top_level&#39; /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:66:inblock in run'
/opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:133:in
standard_exception_handling&#39; /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:63:inrun'
/opt/rh/ruby193/root/usr/bin/rake:32:in `<main>'
Tasks: TOP => katello:clean_backend_objects

··· On Friday, May 22, 2015 at 11:31:33 AM UTC+8, sinux shen wrote: > > it was before the upgrading, aslo, I tried to upgrade, but it failed, I > have another email thread ask about that. > > On Thursday, May 21, 2015 at 8:47:31 PM UTC+8, Eric Helms wrote: >> >> Was this before or after upgrading? Can you re-run the >> clean_backend_objects command and then check messages/journalctl to see if >> there are pulp errors listed? >> >> On Tue, May 19, 2015 at 11:37 PM, sinux shen wrote: >> >>> Hi Daniel, >>> >>> Thanks for the quick response, I ran the command, it shows some errors: >>> >>> katello ODW [/home/sxin] 1000# foreman-rake katello:clean_backend_objects >>> System 110 dogtag.anim.odw.com.cn 455ae32a-8faa-428c-ad92-18a816627f96 >>> is partially missing. Cleaning. >>> rake aborted! >>> 500 Internal Server Error >>> >>> Tasks: TOP => katello:clean_backend_objects >>> (See full trace by running task with --trace) >>> >>> and then I re-run this command by adding --trace >>> katello ODW [/home/sxin] 1001# foreman-rake >>> katello:clean_backend_objects --trace >>> ** Invoke katello:clean_backend_objects (first_time) >>> ** Invoke environment (first_time) >>> ** Execute environment >>> ** Execute katello:clean_backend_objects >>> System 110 dogtag.anim.odw.com.cn 455ae32a-8faa-428c-ad92-18a816627f96 >>> is partially missing. Cleaning. >>> rake aborted! >>> 500 Internal Server Error >>> /opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/abstract_response.rb:48:in >>> `return!' >>> /opt/rh/ruby193/root/usr/share/gems/gems/runcible-1.3.0/lib/runcible/base.rb:92:in >>> `block in get_response' >>> /opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:228:in >>> `call' >>> /opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:228:in >>> `process_result' >>> /opt/rh/ruby193/root/usr/share/gems/gems/rbovirt-0.0.29/lib/restclient_ext/request.rb:50:in >>> `block in transmit' >>> /opt/rh/ruby193/root/usr/share/ruby/net/http.rb:746:in `start' >>> /opt/rh/ruby193/root/usr/share/gems/gems/rbovirt-0.0.29/lib/restclient_ext/request.rb:44:in >>> `transmit' >>> /opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:64:in >>> `execute' >>> /opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:33:in >>> `execute' >>> /opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/resource.rb:94:in >>> `delete' >>> /opt/rh/ruby193/root/usr/share/gems/gems/runcible-1.3.0/lib/runcible/base.rb:91:in >>> `get_response' >>> /opt/rh/ruby193/root/usr/share/gems/gems/runcible-1.3.0/lib/runcible/base.rb:82:in >>> `call' >>> /opt/rh/ruby193/root/usr/share/gems/gems/runcible-1.3.0/lib/runcible/resources/consumer.rb:68:in >>> `delete' >>> /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.1.0/app/models/katello/glue/pulp/consumer.rb:127:in >>> `del_pulp_consumer' >>> /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.1.0/lib/katello/tasks/clean_backend_objects.rake:31:in >>> `block in cleanup_systems' >>> /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/relation/batches.rb:21:in >>> `block (2 levels) in find_each' >>> /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/relation/batches.rb:21:in >>> `each' >>> /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/relation/batches.rb:21:in >>> `block in find_each' >>> /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/relation/batches.rb:72:in >>> `find_in_batches' >>> /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/relation/batches.rb:20:in >>> `find_each' >>> /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/querying.rb:8:in >>> `find_each' >>> /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.1.0/lib/katello/tasks/clean_backend_objects.rake:5:in >>> `cleanup_systems' >>> /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.1.0/lib/katello/tasks/clean_backend_objects.rake:55:in >>> `block (2 levels) in ' >>> /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:205:in `call' >>> /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:205:in `block in >>> execute' >>> /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:200:in `each' >>> /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:200:in `execute' >>> /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:158:in `block in >>> invoke_with_call_chain' >>> /opt/rh/ruby193/root/usr/share/ruby/monitor.rb:211:in `mon_synchronize' >>> /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:151:in >>> `invoke_with_call_chain' >>> /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:144:in `invoke' >>> /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:116:in >>> `invoke_task' >>> /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:94:in `block (2 >>> levels) in top_level' >>> /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:94:in `each' >>> /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:94:in `block in >>> top_level' >>> /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:133:in >>> `standard_exception_handling' >>> /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:88:in `top_level' >>> /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:66:in `block in >>> run' >>> /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:133:in >>> `standard_exception_handling' >>> /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:63:in `run' >>> /opt/rh/ruby193/root/usr/bin/rake:32:in `' >>> Tasks: TOP => katello:clean_backend_objects >>> >>> Then I went back the the foreman GUI, I click Hosts--> Content Hosts, >>> but it was still keep showing the circular loading icon, firebug show the >>> same 400 GET error, like what I mentioned previously. >>> >>> Thank you so much for you help. >>> >>> >>> >>> On Tuesday, May 19, 2015 at 4:17:05 PM UTC+8, Daniel Lobato wrote: >>> >>>> On 05/18, sinux shen wrote: >>>> > I alsl get the error from the katello log: >>>> > >>>> > Processing by Katello::Api::V2::SystemsController#index as JSON >>>> > Parameters: {"organization_id"=>"3", "page"=>"1", "search"=>"", >>>> > "sort_by"=>"name", "sort_order"=>"ASC", "api_version"=>"v2"} >>>> > Failed to retrieve all Katello::System search results (20/20 found.) >>>> > Rendered >>>> > >>>> /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.1.0/app/views/katello/api/v2/systems/index.json.rabl >>>> >>>> > within katello/api/v2/layouts/collection (5821.9ms) >>>> > exception when talking to a remote client: >>>> > Katello::Resources::Candlepin::Consumer: 410 Gone >>>> {"displayMessage":"Unit >>>> > 455ae32a-8faa-428c-ad92-18a816627f96 has been >>>> > >>>> deleted","requestUuid":"a748e055-ac6f-46a7-a957-d60373944e62","deletedId":"455ae32a-8faa-428c-ad92-18a816627f96"} >>>> >>>> > (GET /candlepin/consumers/455ae32a-8faa-428c-ad92-18a816627f96) >>>> > RestClient::Gone: Katello::Resources::Candlepin::Consumer: 410 Gone >>>> > {"displayMessage":"Unit 455ae32a-8faa-428c-ad92-18a816627f96 has been >>>> > >>>> deleted","requestUuid":"a748e055-ac6f-46a7-a957-d60373944e62","deletedId":"455ae32a-8faa-428c-ad92-18a816627f96"} >>>> >>>> > (GET /candlepin/consumers/455ae32a-8faa-428c-ad92-18a816627f96) >>>> > Body: {"displayMessage":"Unit 455ae32a-8faa-428c-ad92-18a816627f96 >>>> has been >>>> > >>>> deleted","requestUuid":"a748e055-ac6f-46a7-a957-d60373944e62","deletedId":"455ae32a-8faa-428c-ad92-18a816627f96"} >>>> >>>> > >>>> >>>> Thanks Sinux, logs are helpful. >>>> >>>> Could you try to run foreman-rake katello:clean_backend_objects as root >>>> and see >>>> if you still get the same error? >>>> >>>> > >>>> > On Tuesday, May 19, 2015 at 11:29:38 AM UTC+8, sinux shen wrote: >>>> > > >>>> > > Hi there, >>>> > > >>>> > > I was trying to upgrade to katello 2.2, but for some reason, I >>>> failed, the >>>> > > reason I want to upgrade is because I have some issue running on >>>> 2.1, one >>>> > > of the issue was I can not list content hosts in "Hosts"--> >>>> "Content Hosts" >>>> > > page, it only showed the circular loading icon spinning >>>> continuously, >>>> > > After analyzing via firebug, I found one GET method got 400 return >>>> code, >>>> > > the get url is: >>>> > > >>>> https://katello.anim.odw.com.cn/katello/api/v2/systems?organization_id=3&page=1&search=&sort_by=name&sort_order=ASC, >>>> >>>> > > looks normal, but the response I got was: >>>> > > >>>> > > {"displayMessage":"Katello::Resources::Candlepin::Consumer: 410 >>>> Gone {\"displayMessage\":\"Unit 455ae32a-8faa-428c-ad92-18a816627f96 has >>>> been >>>> deleted\",\"requestUuid\":\"695a9943-de23-4ce0-852b-7998a9bc1708\",\"deletedId\":\"455ae32a-8faa-428c-ad92-18a816627f96\"} >>>> >>>> > > (GET >>>> /candlepin/consumers/455ae32a-8faa-428c-ad92-18a816627f96)","errors":["Katello::Resources::Candlepin::Consumer: >>>> 410 Gone >>>> > > {\"displayMessage\":\"Unit 455ae32a-8faa-428c-ad92-18a816627f96 has >>>> been >>>> deleted\",\"requestUuid\":\"695a9943-de23-4ce0-852b-7998a9bc1708\",\"deletedId\":\"455ae32a-8faa-428c-ad92-18a816627f96\"} >>>> >>>> > > (GET /candlepin/consumers/455ae32a-8faa-428c-ad92-18a816627f96)"]} >>>> > > >>>> > > it seems something was deleted, I am not quite clear what was that, >>>> and how we can fix it. >>>> > > >>>> > > Any one has any ideas please? >>>> > > >>>> > > Thanks >>>> > > Sinux >>>> > > >>>> > > >>>> > > >>>> > >>>> > -- >>>> > You received this message because you are subscribed to the Google >>>> Groups "Foreman users" group. >>>> > To unsubscribe from this group and stop receiving emails from it, >>>> send an email to foreman-user...@googlegroups.com. >>>> > To post to this group, send email to forema...@googlegroups.com. >>>> > Visit this group at http://groups.google.com/group/foreman-users. >>>> > For more options, visit https://groups.google.com/d/optout. >>>> >>>> Best, >>>> >>>> -- >>>> Daniel Lobato Garcia >>>> >>>> @eLobatoss >>>> blog.daniellobato.me >>>> daniellobato.me >>>> >>>> GPG: http://keys.gnupg.net/pks/lookup?op=get&search=0x7A92D6DD38D6DE30 >>>> Keybase: https://keybase.io/elobato >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Foreman users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to foreman-user...@googlegroups.com. >>> To post to this group, send email to forema...@googlegroups.com. >>> Visit this group at http://groups.google.com/group/foreman-users. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >>

Sorry, it was blocked by our firewall, I moved the debug file to another
host which has the access and uploaded again, it should be their now.

··· On Thursday, May 28, 2015 at 10:01:53 AM UTC+8, sinux shen wrote: > > Hi Justin, > > Can you please check if this has been uploaded? File name > foreman-debug-aHtUS.tar.xz > the out put was a little confused though: > Uploading... > rsync: failed to connect to theforeman.org: Connection timed out (110) > rsync error: error in socket IO (code 10) at clientserver.c(124) > [sender=3.0.6] > The tarball has been uploaded, please contact us on mailing list or IRC. > > > On Thursday, May 28, 2015 at 9:53:39 AM UTC+8, sinux shen wrote: >> >> Thanks Justin, >> >> Yes, indeed, it's something wrong with the content host, from the error >> here: >> katello ODW [/home/sxin] 1001# foreman-rake katello:clean_backend_objects >> System 110 dogtag.anim.odw.com.cn 455ae32a-8faa-428c-ad92-18a816627f96 >> is partially missing. Cleaning. >> System 111 dogtag.anim.odw.com.cn 2d734c21-8c11-4565-987f- >> d338840d0ab2 is partially missing. Cleaning. >> rake aborted! >> PGError: 错误: 在 "katello_systems" 上的更新或删除操作违反了在 >> "katello_system_repositories" 上的外键约束 >> "katello_system_repositories_system_id_fk" >> DETAIL: 键值对(id)=(111)仍然是从表"katello_system_repositories"引用的. >> : DELETE FROM "katello_systems" WHERE "katello_systems"."id" = 111 >> >> but I can't find this content hosts any more. >> >> Anyway, I will run foreman-debug and let you know what happened, thanks >> again. >> >> Thanks >> Sinux >> >> >> On Wednesday, May 27, 2015 at 10:33:32 PM UTC+8, jsherril wrote: >>> >>> On 05/26/2015 01:42 AM, sinux shen wrote: >>> >>> Hi Eric & Daniel, >>> >>> Can anyone of you help with this or please let me know if you need more >>> infomations, I was stuck here and don't know what to do to fix it. >>> >>> >>> The 500 Internal Server Error from candlepin is concerning and likely >>> the cause of your content host issue. Would you be able to run >>> foreman-debug and upload the results (according to the instructions given >>> at the end of the command)? Then simply reply with the filename it created. >>> >>> Thanks, >>> >>> Justin >>> >>> >>> Thanks >>> Sinux >>> >>> On Monday, May 25, 2015 at 10:58:15 AM UTC+8, sinux shen wrote: >>>> >>>> I ran clean_backend_objects command again, this time, the error was a >>>> little different, >>>> >>>> before I ran this, I checked the message log, and I found tons of error >>>> like this: >>>> May 25 10:46:11 katello qpidd[5978]: 2015-05-25 10:46:11 [System] error >>>> Could not accept socket: Too many open files >>>> (/builddir/build/BUILD/qpid-0.26/cpp/src/qpid/sys/ssl/SslSocket.cpp:195) >>>> May 25 10:46:11 katello qpidd[5978]: 2015-05-25 10:46:11 [System] error >>>> Could not accept socket: Too many open files >>>> (/builddir/build/BUILD/qpid-0.26/cpp/src/qpid/sys/ssl/SslSocket.cpp:195) >>>> May 25 10:46:11 katello qpidd[5978]: 2015-05-25 10:46:11 [System] error >>>> Could not accept socket: Too many open files >>>> (/builddir/build/BUILD/qpid-0.26/cpp/src/qpid/sys/ssl/SslSocket.cpp:195) >>>> >>>> And then I restarted qpidd, and I ran clean_backend_objects, it gave me >>>> the following errors: >>>> katello ODW [/home/sxin] 1001# foreman-rake >>>> katello:clean_backend_objects >>>> System 110 dogtag.anim.odw.com.cn 455ae32a-8faa-428c-ad92-18a816627f96 >>>> is partially missing. Cleaning. >>>> System 111 dogtag.anim.odw.com.cn 2d734c21-8c11-4565-987f-d338840d0ab2 >>>> is partially missing. Cleaning. >>>> rake aborted! >>>> PGError: 错误: 在 "katello_systems" 上的更新或删除操作违反了在 >>>> "katello_system_repositories" 上的外键约束 >>>> "katello_system_repositories_system_id_fk" >>>> DETAIL: 键值对(id)=(111)仍然是从表"katello_system_repositories" 引用的. >>>> : DELETE FROM "katello_systems" WHERE "katello_systems"."id" = 111 >>>> >>>> Tasks: TOP => katello:clean_backend_objects >>>> (See full trace by running task with --trace) >>>> And in message, it has one lines of log shows: >>>> May 25 10:50:39 katello pulp: >>>> pulp.server.webservices.middleware.exception:INFO: Missing resource(s): >>>> consumer=455ae32a-8faa-428c-ad92-18a816627f96 >>>> >>>> >>>> Sorry, it contains some Chinese, I remember that when I tried to >>>> upgrade katello to 2.2, it has similar errors. >>>> >>>> >>>> On Tuesday, May 19, 2015 at 11:29:38 AM UTC+8, sinux shen wrote: >>>>> >>>>> Hi there, >>>>> >>>>> I was trying to upgrade to katello 2.2, but for some reason, I failed, >>>>> the reason I want to upgrade is because I have some issue running on 2.1, >>>>> one of the issue was I can not list content hosts in "Hosts"--> "Content >>>>> Hosts" page, it only showed the circular loading icon spinning >>>>> continuously, >>>>> After analyzing via firebug, I found one GET method got 400 return >>>>> code, the get url is: >>>>> https://katello.anim.odw.com.cn/katello/api/v2/systems?organization_id=3&page=1&search=&sort_by=name&sort_order=ASC, >>>>> looks normal, but the response I got was: >>>>> >>>>> {"displayMessage":"Katello::Resources::Candlepin::Consumer: 410 Gone {\"displayMessage\":\"Unit 455ae32a-8faa-428c-ad92-18a816627f96 has been deleted\",\"requestUuid\":\"695a9943-de23-4ce0-852b-7998a9bc1708\",\"deletedId\":\"455ae32a-8faa-428c-ad92-18a816627f96\"} >>>>> (GET /candlepin/consumers/455ae32a-8faa-428c-ad92-18a816627f96)","errors":["Katello::Resources::Candlepin::Consumer: 410 Gone >>>>> {\"displayMessage\":\"Unit 455ae32a-8faa-428c-ad92-18a816627f96 has been deleted\",\"requestUuid\":\"695a9943-de23-4ce0-852b-7998a9bc1708\",\"deletedId\":\"455ae32a-8faa-428c-ad92-18a816627f96\"} >>>>> (GET /candlepin/consumers/455ae32a-8faa-428c-ad92-18a816627f96)"]} >>>>> >>>>> it seems something was deleted, I am not quite clear what was that, and how we can fix it. >>>>> >>>>> Any one has any ideas please? >>>>> >>>>> Thanks >>>>> Sinux >>>>> >>>>> >>>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Foreman users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to foreman-user...@googlegroups.com. >>> To post to this group, send email to forema...@googlegroups.com. >>> Visit this group at http://groups.google.com/group/foreman-users. >>> For more options, visit https://groups.google.com/d/optout. >>> >>> >>>

Any error messages in /var/log/messages when you run the clean backend
script?

··· On Fri, May 22, 2015 at 2:30 AM, sinux shen wrote:

so I did an yum update, couple of foreman/katello/pulp packages were
updated, here is the version related to all of that:
katello ODW [/home/sxin] 1003# rpm -qa | egrep "katello|foreman|pulp"
python-isodate-0.5.0-4.pulp.el6.noarch
katello-installer-2.1.1-4.el6.noarch
pulp-server-2.5.1-1.el6.noarch
ruby193-rubygem-foreman-tasks-0.6.10-1.el6.noarch
katello.anim.odw.com.cn-puppet-client-1.0-1.noarch
python-pulp-puppet-common-2.5.1-2.katello.el6.noarch
pulp-nodes-common-2.5.1-1.el6.noarch
katello.anim.odw.com.cn-foreman-proxy-1.0-1.noarch
katello-installer-base-2.1.1-4.el6.noarch
rubygem-smart_proxy_pulp-1.0.1-1.el6.noarch
rubygem-hammer_cli_foreman-0.1.3-1.el6.noarch
foreman-libvirt-1.7.5-1.el6.noarch
katello.anim.odw.com.cn-foreman-client-1.0-1.noarch
python-pulp-rpm-common-2.5.1-1.el6.noarch
katello.anim.odw.com.cn-qpid-client-cert-1.0-1.noarch
pulp-nodes-parent-2.5.1-1.el6.noarch
rubygem-hammer_cli_katello-0.0.7-1.el6.noarch
foreman-postgresql-1.7.5-1.el6.noarch
katello-certs-tools-2.0.1-1.el6.noarch
mod_wsgi-3.4-2.pulp.el6.x86_64
python-kombu-3.0.15-12.pulp.el6.noarch
katello-repos-2.1.3-1.el6.noarch
foreman-selinux-1.7.5-1.el6.noarch
m2crypto-0.21.1.pulp-8.el6.x86_64
foreman-vmware-1.7.5-1.el6.noarch
katello-default-ca-1.0-1.noarch
katello.anim.odw.com.cn-qpid-broker-1.0-1.noarch
pulp-puppet-tools-2.5.1-2.katello.el6.noarch
katello-2.1.0-2.el6.noarch
foreman-compute-1.7.5-1.el6.noarch
ruby193-rubygem-foreman_docker-1.3.1-1.fm1_7.el6.noarch

rubygem-hammer_cli_foreman_tasks-0.0.3-2.201409091410gitc96619d.git.0.37f3704.el6.noarch
foreman-gce-1.7.5-1.el6.noarch
foreman-release-1.7.5-1.el6.noarch
ruby193-rubygem-foreman_hooks-0.3.7-2.el6.noarch
katello.anim.odw.com.cn-apache-1.0-1.noarch
katello.anim.odw.com.cn-parent-cert-1.0-1.noarch
pulp-puppet-plugins-2.5.1-2.katello.el6.noarch
rubygem-hammer_cli_foreman_bootdisk-0.1.2-1.el6.noarch
ruby193-rubygem-foreman_discovery-2.0.2-1.el6.noarch
pulp-katello-0.3-3.el6.noarch
foreman-1.7.5-1.el6.noarch
ruby193-rubygem-katello-2.1.2-3.el6.noarch
foreman-ovirt-1.7.5-1.el6.noarch
katello-server-ca-1.0-1.noarch
pulp-selinux-2.5.1-1.el6.noarch
python-pulp-common-2.5.1-1.el6.noarch
python-pulp-bindings-2.5.1-1.el6.noarch
pulp-docker-plugins-0.2.1-0.2.beta.el6.noarch
ruby193-rubygem-foreman_bootdisk-4.0.2-1.el6.noarch
foreman-proxy-1.7.5-1.el6.noarch
python-pulp-docker-common-0.2.1-0.2.beta.el6.noarch
pulp-rpm-plugins-2.5.1-1.el6.noarch

And I re-ran clean_backend_objects again, it seems to have the same error,
from the error, dogtag used to be one of the content host, how I can get
around this? here is the trace:
katello ODW [/home/sxin] 1002# foreman-rake katello:clean_backend_objects
–trace** Invoke katello:clean_backend_objects (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute katello:clean_backend_objects
System 110 dogtag.anim.odw.com.cn 455ae32a-8faa-428c-ad92-18a816627f96 is
partially missing. Cleaning.
rake aborted!
500 Internal Server Error
/opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/abstract_response.rb:48:in
return!' /opt/rh/ruby193/root/usr/share/gems/gems/runcible-1.3.0/lib/runcible/base.rb:92:inblock in get_response’
/opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:228:in
call' /opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:228:inprocess_result’
/opt/rh/ruby193/root/usr/share/gems/gems/rbovirt-0.0.29/lib/restclient_ext/request.rb:50:in
block in transmit' /opt/rh/ruby193/root/usr/share/ruby/net/http.rb:746:instart’
/opt/rh/ruby193/root/usr/share/gems/gems/rbovirt-0.0.29/lib/restclient_ext/request.rb:44:in
transmit' /opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:64:inexecute’
/opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:33:in
execute' /opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/resource.rb:94:indelete’
/opt/rh/ruby193/root/usr/share/gems/gems/runcible-1.3.0/lib/runcible/base.rb:91:in
get_response' /opt/rh/ruby193/root/usr/share/gems/gems/runcible-1.3.0/lib/runcible/base.rb:82:incall’
/opt/rh/ruby193/root/usr/share/gems/gems/runcible-1.3.0/lib/runcible/resources/consumer.rb:68:in
delete' /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.1.2/app/models/katello/glue/pulp/consumer.rb:127:indel_pulp_consumer’
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.1.2/lib/katello/tasks/clean_backend_objects.rake:31:in
block in cleanup_systems' /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/relation/batches.rb:21:inblock (2 levels) in find_each’
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/relation/batches.rb:21:in
each' /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/relation/batches.rb:21:inblock in find_each’
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/relation/batches.rb:72:in
find_in_batches' /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/relation/batches.rb:20:infind_each’
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/querying.rb:8:in
find_each' /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.1.2/lib/katello/tasks/clean_backend_objects.rake:5:incleanup_systems’
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.1.2/lib/katello/tasks/clean_backend_objects.rake:57:in
block (2 levels) in <top (required)>' /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:205:incall’
/opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:205:in block in execute' /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:200:ineach’
/opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:200:in execute' /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:158:inblock in
invoke_with_call_chain’
/opt/rh/ruby193/root/usr/share/ruby/monitor.rb:211:in mon_synchronize' /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:151:ininvoke_with_call_chain’
/opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:144:in invoke' /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:116:ininvoke_task’
/opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:94:in block (2 levels) in top_level' /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:94:ineach’
/opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:94:in block in top_level' /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:133:instandard_exception_handling’
/opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:88:in top_level' /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:66:inblock in
run’
/opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:133:in
standard_exception_handling' /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:63:inrun’
/opt/rh/ruby193/root/usr/bin/rake:32:in `'
Tasks: TOP => katello:clean_backend_objects

On Friday, May 22, 2015 at 11:31:33 AM UTC+8, sinux shen wrote:

it was before the upgrading, aslo, I tried to upgrade, but it failed, I
have another email thread ask about that.

On Thursday, May 21, 2015 at 8:47:31 PM UTC+8, Eric Helms wrote:

Was this before or after upgrading? Can you re-run the
clean_backend_objects command and then check messages/journalctl to see if
there are pulp errors listed?

On Tue, May 19, 2015 at 11:37 PM, sinux shen sinux...@gmail.com wrote:

Hi Daniel,

Thanks for the quick response, I ran the command, it shows some errors:

katello ODW [/home/sxin] 1000# foreman-rake
katello:clean_backend_objects
System 110 dogtag.anim.odw.com.cn 455ae32a-8faa-428c-ad92-18a816627f96
is partially missing. Cleaning.
rake aborted!
500 Internal Server Error

Tasks: TOP => katello:clean_backend_objects
(See full trace by running task with --trace)

and then I re-run this command by adding --trace
katello ODW [/home/sxin] 1001# foreman-rake
katello:clean_backend_objects --trace
** Invoke katello:clean_backend_objects (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute katello:clean_backend_objects
System 110 dogtag.anim.odw.com.cn 455ae32a-8faa-428c-ad92-18a816627f96
is partially missing. Cleaning.
rake aborted!
500 Internal Server Error
/opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/abstract_response.rb:48:in
return!' /opt/rh/ruby193/root/usr/share/gems/gems/runcible-1.3.0/lib/runcible/base.rb:92:inblock in get_response’
/opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:228:in
call' /opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:228:inprocess_result’
/opt/rh/ruby193/root/usr/share/gems/gems/rbovirt-0.0.29/lib/restclient_ext/request.rb:50:in
block in transmit' /opt/rh/ruby193/root/usr/share/ruby/net/http.rb:746:instart’
/opt/rh/ruby193/root/usr/share/gems/gems/rbovirt-0.0.29/lib/restclient_ext/request.rb:44:in
transmit' /opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:64:inexecute’
/opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:33:in
execute' /opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/resource.rb:94:indelete’
/opt/rh/ruby193/root/usr/share/gems/gems/runcible-1.3.0/lib/runcible/base.rb:91:in
get_response' /opt/rh/ruby193/root/usr/share/gems/gems/runcible-1.3.0/lib/runcible/base.rb:82:incall’
/opt/rh/ruby193/root/usr/share/gems/gems/runcible-1.3.0/lib/runcible/resources/consumer.rb:68:in
delete' /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.1.0/app/models/katello/glue/pulp/consumer.rb:127:indel_pulp_consumer’
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.1.0/lib/katello/tasks/clean_backend_objects.rake:31:in
block in cleanup_systems' /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/relation/batches.rb:21:inblock (2 levels) in find_each’
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/relation/batches.rb:21:in
each' /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/relation/batches.rb:21:inblock in find_each’
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/relation/batches.rb:72:in
find_in_batches' /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/relation/batches.rb:20:infind_each’
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/querying.rb:8:in
find_each' /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.1.0/lib/katello/tasks/clean_backend_objects.rake:5:incleanup_systems’
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.1.0/lib/katello/tasks/clean_backend_objects.rake:55:in
block (2 levels) in <top (required)>' /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:205:incall’
/opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:205:in block in execute' /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:200:ineach’
/opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:200:in execute' /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:158:inblock in
invoke_with_call_chain’
/opt/rh/ruby193/root/usr/share/ruby/monitor.rb:211:in mon_synchronize' /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:151:ininvoke_with_call_chain’
/opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:144:in invoke' /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:116:ininvoke_task’
/opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:94:in block (2 levels) in top_level' /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:94:ineach’
/opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:94:in block in top_level' /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:133:instandard_exception_handling’
/opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:88:in
top_level' /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:66:inblock in
run’
/opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:133:in
standard_exception_handling' /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:63:inrun’
/opt/rh/ruby193/root/usr/bin/rake:32:in `'
Tasks: TOP => katello:clean_backend_objects

Then I went back the the foreman GUI, I click Hosts–> Content Hosts,
but it was still keep showing the circular loading icon, firebug show the
same 400 GET error, like what I mentioned previously.

Thank you so much for you help.

On Tuesday, May 19, 2015 at 4:17:05 PM UTC+8, Daniel Lobato wrote:

On 05/18, sinux shen wrote:

I alsl get the error from the katello log:

Processing by Katello::Api::V2::SystemsController#index as JSON
Parameters: {“organization_id”=>“3”, “page”=>“1”, “search”=>"",
“sort_by”=>“name”, “sort_order”=>“ASC”, “api_version”=>“v2”}
Failed to retrieve all Katello::System search results (20/20 found.)
Rendered

/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.1.0/app/views/katello/api/v2/systems/index.json.rabl

within katello/api/v2/layouts/collection (5821.9ms)
exception when talking to a remote client:
Katello::Resources::Candlepin::Consumer: 410 Gone
{“displayMessage”:"Unit
455ae32a-8faa-428c-ad92-18a816627f96 has been

deleted",“requestUuid”:“a748e055-ac6f-46a7-a957-d60373944e62”,“deletedId”:“455ae32a-8faa-428c-ad92-18a816627f96”}

(GET /candlepin/consumers/455ae32a-8faa-428c-ad92-18a816627f96)
RestClient::Gone: Katello::Resources::Candlepin::Consumer: 410 Gone
{“displayMessage”:"Unit 455ae32a-8faa-428c-ad92-18a816627f96 has
been

deleted",“requestUuid”:“a748e055-ac6f-46a7-a957-d60373944e62”,“deletedId”:“455ae32a-8faa-428c-ad92-18a816627f96”}

(GET /candlepin/consumers/455ae32a-8faa-428c-ad92-18a816627f96)
Body: {“displayMessage”:"Unit 455ae32a-8faa-428c-ad92-18a816627f96
has been

deleted",“requestUuid”:“a748e055-ac6f-46a7-a957-d60373944e62”,“deletedId”:“455ae32a-8faa-428c-ad92-18a816627f96”}

Thanks Sinux, logs are helpful.

Could you try to run foreman-rake katello:clean_backend_objects as
root and see
if you still get the same error?

On Tuesday, May 19, 2015 at 11:29:38 AM UTC+8, sinux shen wrote:

Hi there,

I was trying to upgrade to katello 2.2, but for some reason, I
failed, the

reason I want to upgrade is because I have some issue running on
2.1, one

of the issue was I can not list content hosts in “Hosts”–>
“Content Hosts”

page, it only showed the circular loading icon spinning
continuously,

After analyzing via firebug, I found one GET method got 400 return
code,

the get url is:

https://katello.anim.odw.com.cn/katello/api/v2/systems?organization_id=3&page=1&search=&sort_by=name&sort_order=ASC,

looks normal, but the response I got was:

{“displayMessage”:"Katello::Resources::Candlepin::Consumer: 410
Gone {“displayMessage”:“Unit 455ae32a-8faa-428c-ad92-18a816627f96 has
been
deleted”,“requestUuid”:“695a9943-de23-4ce0-852b-7998a9bc1708”,“deletedId”:“455ae32a-8faa-428c-ad92-18a816627f96”}

(GET
/candlepin/consumers/455ae32a-8faa-428c-ad92-18a816627f96)",“errors”:["Katello::Resources::Candlepin::Consumer:
410 Gone

{“displayMessage”:“Unit 455ae32a-8faa-428c-ad92-18a816627f96
has been
deleted”,“requestUuid”:“695a9943-de23-4ce0-852b-7998a9bc1708”,“deletedId”:“455ae32a-8faa-428c-ad92-18a816627f96”}

(GET /candlepin/consumers/455ae32a-8faa-428c-ad92-18a816627f96)"]}

it seems something was deleted, I am not quite clear what was
that, and how we can fix it.