I'm running Katello 2.1 on RH6, and seeing huge numbers of open files for
qpidd. So many that qpidd is hitting ulimit, which seems to fatally mess up
Katello to the point that a re-install is required.
Watching lsof while registering and de-registering hosts with
subscription-manager shows qpidd opening 7 new file handles every time we
register/de-register a host, and those files are always held open even
after restart. The number of open files only ever goes up. Is this normal
for qpid? Is there a setting somewhere to limit this, or a way to clear
down the queues for de-registered hosts?
> I'm running Katello 2.1 on RH6, and seeing huge numbers of open files for
> qpidd. So many that qpidd is hitting ulimit, which seems to fatally mess up
> Katello to the point that a re-install is required
re-install or restart? I guess simply restarting qpidd should help.
> Watching lsof while registering and de-registering hosts with
> subscription-manager shows qpidd opening 7 new file handles every time we
> register/de-register a host, and those files are always held open even
> after restart. The number of open files only ever goes up. Is this normal
> for qpid? Is there a setting somewhere to limit this, or a way to clear
> down the queues for de-registered hosts?
There is a bug in qpid damon that was fixed in version 0.14. What is
your version?
> > I'm running Katello 2.1 on RH6, and seeing huge numbers of open files
> for
> > qpidd. So many that qpidd is hitting ulimit, which seems to fatally mess
> up
> > Katello to the point that a re-install is required
>
> re-install or restart? I guess simply restarting qpidd should help.
>
> Once qpidd hits ulimit, pulp either sees or throws (not eniterly sure
which) '500 internal server error' for every unregister operation
(subscription-manager, katello, or hammer). Restarting the whole stack
doesn't fix it, neither does manually removing hosts from pulp/mongo. The
only thing that I've been able to do has been a re-install (or an upgrade
from 2.0 to 2.1 in this case).
Registering clients and other functionality works fine it's just
unregistering that breaks.
> > Watching lsof while registering and de-registering hosts with
> > subscription-manager shows qpidd opening 7 new file handles every time
> we
> > register/de-register a host, and those files are always held open even
> > after restart. The number of open files only ever goes up. Is this
> normal
> > for qpid? Is there a setting somewhere to limit this, or a way to clear
> > down the queues for de-registered hosts?
>
> There is a bug in qpid damon that was fixed in version 0.14. What is
> your version?
>
> https://access.redhat.com/solutions/107713
> https://issues.apache.org/jira/browse/QPID-3447
>
> I see v0.30 in RHEL 6.6 so that should be fine. But this is likely an
> issue in qpidd itself.
>
>
> Hello,
>
>
> Hi.
>
> > I'm running Katello 2.1 on RH6, and seeing huge numbers of open
> files for
> > qpidd. So many that qpidd is hitting ulimit, which seems to
> fatally mess up
> > Katello to the point that a re-install is required
>
> re-install or restart? I guess simply restarting qpidd should help.
>
> Once qpidd hits ulimit, pulp either sees or throws (not eniterly sure
> which) '500 internal server error' for every unregister operation
> (subscription-manager, katello, or hammer). Restarting the whole stack
> doesn't fix it, neither does manually removing hosts from pulp/mongo.
> The only thing that I've been able to do has been a re-install (or an
> upgrade from 2.0 to 2.1 in this case).
> Registering clients and other functionality works fine it's just
> unregistering that breaks.
It has been fixed for pulp 2.6 and katello 2.2 should ship with it.
In the meantime bumping the ulimit and deleting queues manually is the
only work around: You would have to be sure to only delete queues for
content host uuids that are no longer in use. An email to the pulp
mailing list may provide more information on how to do this.
-Justin
···
On 02/12/2015 09:25 AM, David Stark wrote:
> On Thursday, February 12, 2015 at 1:21:23 PM UTC, Lukas Zapletal wrote:
> Watching lsof while registering and de-registering hosts with
> subscription-manager shows qpidd opening 7 new file handles
every time we
> register/de-register a host, and those files are always held
open even
> after restart. The number of open files only ever goes up. Is
this normal
> for qpid? Is there a setting somewhere to limit this, or a way
to clear
> down the queues for de-registered hosts?
There is a bug in qpid damon that was fixed in version 0.14. What is
your version?
https://access.redhat.com/solutions/107713
<https://access.redhat.com/solutions/107713>
https://issues.apache.org/jira/browse/QPID-3447
<https://issues.apache.org/jira/browse/QPID-3447>
I see v0.30 in RHEL 6.6 so that should be fine. But this is likely an
issue in qpidd itself.
>
>
>
> Once qpidd hits ulimit, pulp either sees or throws (not eniterly sure
>> which) '500 internal server error' for every unregister operation
>> (subscription-manager, katello, or hammer). Restarting the whole stack
>> doesn't fix it, neither does manually removing hosts from pulp/mongo. The
>> only thing that I've been able to do has been a re-install (or an upgrade
>> from 2.0 to 2.1 in this case).
>> Registering clients and other functionality works fine it's just
>> unregistering that breaks.
>>
>
> Sadly this is a known issue in pulp:
> https://bugzilla.redhat.com/show_bug.cgi?id=1159303
>
> It has been fixed for pulp 2.6 and katello 2.2 should ship with it.
>
> In the meantime bumping the ulimit and deleting queues manually is the
> only work around: You would have to be sure to only delete queues for
> content host uuids that are no longer in use. An email to the pulp mailing
> list may provide more information on how to do this.
>
> -Justin
>
>
Ah. Cool cool. I'll keep an eye out for a release.
Cheers!
···
> On Thursday, February 12, 2015 at 7:47:06 PM UTC, jsherril wrote:
