401 Unauthorized for db:seed and in candlepin (katello 2.4)

I updated my katello 2.3 system -> 2.4 last night and ran into issues. When
doing foreman-rake db:seed I got the following error.

foreman-rake db:seed --trace

** Invoke db:seed (first_time)
** Execute db:seed
** Invoke db:abort_if_pending_migrations (first_time)
** Invoke environment (first_time)
** Execute environment
** Invoke db:load_config (first_time)
** Execute db:load_config
** Execute db:abort_if_pending_migrations
Seeding /usr/share/foreman/db/seeds.d/03-auth_sources.rb
Seeding /usr/share/foreman/db/seeds.d/03-permissions.rb
Seeding /usr/share/foreman/db/seeds.d/03-roles.rb
Seeding /usr/share/foreman/db/seeds.d/04-admin.rb
Seeding /usr/share/foreman/db/seeds.d/05-taxonomies.rb
Seeding /usr/share/foreman/db/seeds.d/06-architectures.rb
Seeding /usr/share/foreman/db/seeds.d/07-provisioning_templates.rb
Seeding /usr/share/foreman/db/seeds.d/08-partition_tables.rb
Seeding /usr/share/foreman/db/seeds.d/10-installation_media.rb
Seeding /usr/share/foreman/db/seeds.d/11-smart_proxy_features.rb
Seeding /usr/share/foreman/db/seeds.d/13-compute_profiles.rb
Seeding /usr/share/foreman/db/seeds.d/15-bookmarks.rb
Seeding /usr/share/foreman/db/seeds.d/16-mail_notifications.rb
Seeding
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.10/db/seeds.d/20-foreman_tasks_permissions.rb
Seeding
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_bootdisk-6.0.0/db/seeds.d/50-bootdisk_templates.rb
Seeding
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.10/db/seeds.d/60-dynflow_proxy_feature.rb
Seeding
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.10/db/seeds.d/61-foreman_tasks_bookmarks.rb
All seed files executed
rake aborted!
401 Unauthorized
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/abstract_response.rb:48:in
return!' /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-1.3.5/lib/runcible/base.rb:92:inblock in get_response'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:228:in
call' /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:228:inprocess_result'
/opt/theforeman/tfm/root/usr/share/gems/gems/rbovirt-0.0.35/lib/restclient_ext/request.rb:50:in
block in transmit' /opt/rh/ruby193/root/usr/share/ruby/net/http.rb:746:instart'
/opt/theforeman/tfm/root/usr/share/gems/gems/rbovirt-0.0.35/lib/restclient_ext/request.rb:44:in
transmit' /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:64:inexecute'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:33:in
execute' /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/resource.rb:51:inget'
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-1.3.5/lib/runcible/base.rb:91:in
get_response' /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-1.3.5/lib/runcible/base.rb:79:incall'
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-1.3.5/lib/runcible/resources/event_notifier.rb:71:in
list' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-2.4.0/app/models/katello/glue/pulp/repo.rb:33:inensure_sync_notification'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-2.4.0/db/seeds.rb:31:in
<top (required)>' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/dependencies.rb:245:inload'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/dependencies.rb:245:in
block in load' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/dependencies.rb:236:inload_dependency'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/dependencies.rb:245:in
load' /opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/engine.rb:520:inload_seed'
/opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/railtie/configurable.rb:30:in
method_missing&#39; /opt/theforeman/tfm/root/usr/share/gems/gems/katello-2.4.0/lib/katello/engine.rb:192:inblock (2 levels) in <class:Engine>'
/opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:205:in call&#39; /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:205:inblock in execute'
/opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:200:in each&#39; /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:200:inexecute'
/opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:158:in block in invoke_with_call_chain&#39; /opt/rh/ruby193/root/usr/share/ruby/monitor.rb:211:inmon_synchronize'
/opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:151:in
invoke_with_call_chain&#39; /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:144:ininvoke'
/opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:116:in invoke_task&#39; /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:94:inblock (2
levels) in top_level'
/opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:94:in each&#39; /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:94:inblock in
top_level'
/opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:133:in
standard_exception_handling&#39; /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:88:intop_level'
/opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:66:in block in run&#39; /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:133:instandard_exception_handling'
/opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:63:in run&#39; /opt/rh/ruby193/root/usr/bin/rake:32:in<main>'
Tasks: TOP => db:seed

When trying to manually sync a repo I see the following errors for
Actions::Katello::Repository::Sync RuntimeError: There was an issue with
the backend service pulp: Pulp message bus connection issue.

My product pages don't load at all and when I try to get info from hammer product info --id 29 I get Katello::Resources::Candlepin::Product: 401
Unauthorized {"displayMessage":"Invalid oauth unit
or secret","requestUuid":"64aba749-0384-409a-a32c-098a6f7064a1"} (GET
/candlepin/products/71)

Not really sure how I got in this state but could use some help. I'm in
#theforeman regularly if it's easier to help there. @jgarr

Thanks

Hello Justin,

Did you get any help with this Pulp message bus connection issue ?
I have the same problem, and cannot use pulp-related features
(products/repos/sync) in foreman/katello.
katello-service restart may help for a few minutes, but the problem comes
back very quickly.:

[app] [W] Pulp message bus connection issue. [app] [E] RuntimeError: There
was an issue with the backend service pulp: Pulp message bus connection
issue.

There seems to be a relevant Satellite bug that was fixed (
https://bugzilla.redhat.com/show_bug.cgi?id=1236096) in Satellite version
6.1.1, but we still see this error with the latest katello 2.4 / foreman
1.10.

··· > When trying to manually sync a repo I see the following errors for > Actions::Katello::Repository::Sync RuntimeError: There was an issue with > the backend service pulp: Pulp message bus connection issue. > > My product pages don't load at all and when I try to get info from `hammer > product info --id 29` I get Katello::Resources::Candlepin::Product: 401 > Unauthorized {"displayMessage":"Invalid oauth unit > or secret","requestUuid":"64aba749-0384-409a-a32c-098a6f7064a1"} (GET > /candlepin/products/71) > > Not really sure how I got in this state but could use some help. I'm in > #theforeman regularly if it's easier to help there. @jgarr > > Thanks > >

I don't actually remember with this specific issue. I don't have any
outstanding issues on my Satellite server and my Katello system got
reinstalled.

Sorry

– Justin

··· On Wed, Mar 30, 2016 at 11:51 AM, dcp wrote:

Hello Justin,

Did you get any help with this Pulp message bus connection issue ?
I have the same problem, and cannot use pulp-related features
(products/repos/sync) in foreman/katello.
katello-service restart may help for a few minutes, but the problem comes
back very quickly.:

[app] [W] Pulp message bus connection issue. [app] [E] RuntimeError: There
was an issue with the backend service pulp: Pulp message bus connection
issue.

There seems to be a relevant Satellite bug that was fixed (
https://bugzilla.redhat.com/show_bug.cgi?id=1236096) in Satellite version
6.1.1, but we still see this error with the latest katello 2.4 / foreman
1.10.

When trying to manually sync a repo I see the following errors for
Actions::Katello::Repository::Sync RuntimeError: There was an issue with
the backend service pulp: Pulp message bus connection issue.

My product pages don’t load at all and when I try to get info from
hammer product info --id 29 I get Katello::Resources::Candlepin::Product:
401 Unauthorized {“displayMessage”:“Invalid oauth unit
or secret”,“requestUuid”:“64aba749-0384-409a-a32c-098a6f7064a1”} (GET
/candlepin/products/71)

Not really sure how I got in this state but could use some help. I’m in
#theforeman regularly if it’s easier to help there. @jgarr

Thanks


You received this message because you are subscribed to a topic in the
Google Groups “Foreman users” group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/foreman-users/K6X4Eq-bOY4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

Thanks for the reply, Justin…

Until now it seems we are the only unlucky users to experience this
intermittent issue that starts to become very frustrating… Thinking to
post a bug report on Issues - Katello - Foreman.

··· On Wednesday, March 30, 2016 at 3:54:37 PM UTC-4, Justin Garrison wrote: > > I don't actually remember with this specific issue. I don't have any > outstanding issues on my Satellite server and my Katello system got > reinstalled. > > Sorry > > > -- Justin > > On Wed, Mar 30, 2016 at 11:51 AM, dcp <dcp...@gmail.com > > wrote: > >> Hello Justin, >> >> Did you get any help with this Pulp message bus connection issue ? >> I have the same problem, and cannot use pulp-related features >> (products/repos/sync) in foreman/katello. >> katello-service restart may help for a few minutes, but the problem comes >> back very quickly.: >> >> [app] [W] Pulp message bus connection issue. [app] [E] RuntimeError: >> There was an issue with the backend service pulp: Pulp message bus >> connection issue. >> >> There seems to be a relevant Satellite bug that was fixed ( >> https://bugzilla.redhat.com/show_bug.cgi?id=1236096) in Satellite >> version 6.1.1, but we still see this error with the latest katello 2.4 / >> foreman 1.10. >> >> >> >>> When trying to manually sync a repo I see the following errors for >>> Actions::Katello::Repository::Sync RuntimeError: There was an issue with >>> the backend service pulp: Pulp message bus connection issue. >>> >>> My product pages don't load at all and when I try to get info from >>> `hammer product info --id 29` I get Katello::Resources::Candlepin::Product: >>> 401 Unauthorized {"displayMessage":"Invalid oauth unit >>> or secret","requestUuid":"64aba749-0384-409a-a32c-098a6f7064a1"} (GET >>> /candlepin/products/71) >>> >>> Not really sure how I got in this state but could use some help. I'm in >>> #theforeman regularly if it's easier to help there. @jgarr >>> >>> Thanks >>> >>> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "Foreman users" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/foreman-users/K6X4Eq-bOY4/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> foreman-user...@googlegroups.com . >> To post to this group, send email to forema...@googlegroups.com >> . >> Visit this group at https://groups.google.com/group/foreman-users. >> For more options, visit https://groups.google.com/d/optout. >> > >

If either of you can go ahead and file an issue with this, it would be
greatly appreciated and we can look into it more. Include any related error
messages in /var/log/messages

For good measure try running these commands in order:

systemctl restart qpidd

systemctl stop pulp_celerybeat.service pulp_resource_manager.service
pulp_workers.service

systemctl start pulp_celerybeat.service pulp_resource_manager.service
pulp_workers.service

systemctl restart qpidd

John Mitsch
Red Hat Engineering
(860)-967-7285
irc: jomitsch

··· On Thu, Mar 31, 2016 at 12:37 PM, dcp wrote:

Thanks for the reply, Justin…

Until now it seems we are the only unlucky users to experience this
intermittent issue that starts to become very frustrating… Thinking to
post a bug report on
Issues - Katello - Foreman.

On Wednesday, March 30, 2016 at 3:54:37 PM UTC-4, Justin Garrison wrote:

I don’t actually remember with this specific issue. I don’t have any
outstanding issues on my Satellite server and my Katello system got
reinstalled.

Sorry

– Justin

On Wed, Mar 30, 2016 at 11:51 AM, dcp dcp...@gmail.com wrote:

Hello Justin,

Did you get any help with this Pulp message bus connection issue ?
I have the same problem, and cannot use pulp-related features
(products/repos/sync) in foreman/katello.
katello-service restart may help for a few minutes, but the problem
comes back very quickly.:

[app] [W] Pulp message bus connection issue. [app] [E] RuntimeError:
There was an issue with the backend service pulp: Pulp message bus
connection issue.

There seems to be a relevant Satellite bug that was fixed (
https://bugzilla.redhat.com/show_bug.cgi?id=1236096) in Satellite
version 6.1.1, but we still see this error with the latest katello 2.4 /
foreman 1.10.

When trying to manually sync a repo I see the following errors for
Actions::Katello::Repository::Sync RuntimeError: There was an issue with
the backend service pulp: Pulp message bus connection issue.

My product pages don’t load at all and when I try to get info from
hammer product info --id 29 I get Katello::Resources::Candlepin::Product:
401 Unauthorized {“displayMessage”:“Invalid oauth unit
or secret”,“requestUuid”:“64aba749-0384-409a-a32c-098a6f7064a1”} (GET
/candlepin/products/71)

Not really sure how I got in this state but could use some help. I’m in
#theforeman regularly if it’s easier to help there. @jgarr

Thanks


You received this message because you are subscribed to a topic in the
Google Groups “Foreman users” group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/foreman-users/K6X4Eq-bOY4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
foreman-user...@googlegroups.com.
To post to this group, send email to forema...@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.