[Katello] Candlepin DB out of sync

I upgraded to Katello 3.2.2 last week, but during the process had an issue
with the candlepin upgrade, since I had migrated my postgresql server to
another VM. I was able to manually run the steps, but unbeknownst to me,
the installer reset the candlepin.conf to point to localhost for postgres,
and also started up the postgresql service as well… so it's been writing
to the wrong candlepin database since the 22nd.

What is the best route to take here? I know I've probably changed things in
the db since I moved to the new DB host… would anything else break by my
just re-importing the candlepin database into the new host?

··· --

KAYAK

Neil Hanlon

Devops Engineer

+1 978 902 8171

Well this seems not good… I ended up just pointing it back to the DB on
the other server, but it looks like there are other issues…

Specifically, when a repo sync runs:

Dec 28 14:02:31 [redacted] pulp[5543]: pulp.server.event.http:ERROR:
Received HTTP 500 from HTTP notifier to https:
//[redacted]/katello/api/v2/repositories/sync_complete?token=zZh7D6xYaXaUj8FFfkvtq3ME6kvZegri.

That url gives:

{"displayMessage":"Couldn't find Katello::Repository with 'id'=sync_complete","errors":["Couldn't find Katello::Repository with 'id'=sync_complete"]}

I'm not sure where to go from here… My only guess is this is an inconsistency with the DB somewhere, or that something didn't get upgraded/migrated correctly in my katello upgrade from 3.1.x to 3.2.2.

Any help is greatly appreciated as when this happens my web UI freezes… sometimes requiring me to restart httpd.

I'm on IRC: NeilHanlon if you want to chat in realtime.

I also think this is related to the overall problem:

2016-12-28 14:14:37 f101b581 [app] [I] Processing by Katello::Api::Rhsm::
CandlepinProxiesController#upload_package_profile as JSON
2016-12-28 14:14:37 f101b581 [app] [I] Parameters: {"_json"=>"[FILTERED]",
"id"=>"432d04cc-d045-4b7f-9c92-2b4eb3dccd93"}

This appears to be what actually makes foreman/katello hang… i'm going to
ride it out and see if it works this time…

··· On Wednesday, December 28, 2016 at 2:19:14 PM UTC-5, nha...@kayak.com wrote: > > Well this seems not good... I ended up just pointing it back to the DB on > the other server, but it looks like there are other issues... > > Specifically, when a repo sync runs: > > Dec 28 14:02:31 [redacted] pulp[5543]: pulp.server.event.http:ERROR: > Received HTTP 500 from HTTP notifier to https: > //[redacted]/katello/api/v2/repositories/sync_complete?token=zZh7D6xYaXaUj8FFfkvtq3ME6kvZegri. > > > > > That url gives: > > {"displayMessage":"Couldn't find Katello::Repository with 'id'=sync_complete","errors":["Couldn't find Katello::Repository with 'id'=sync_complete"]} > > I'm not sure where to go from here... My only guess is this is an inconsistency with the DB somewhere, or that something didn't get upgraded/migrated correctly in my katello upgrade from 3.1.x to 3.2.2. > > > Any help is _greatly_ appreciated as when this happens my web UI freezes... sometimes requiring me to restart httpd. > > > I'm on IRC: NeilHanlon if you want to chat in realtime. > >

> I also think this is related to the overall problem:
>
>
> 2016-12-28 14:14:37 f101b581 [app] [I] Processing by Katello::Api::Rhsm::
> CandlepinProxiesController#upload_package_profile as JSON
> 2016-12-28 14:14:37 f101b581 [app] [I] Parameters: {"_json"=>"[FILTERED]",
> "id"=>"432d04cc-d045-4b7f-9c92-2b4eb3dccd93"}
>
> This appears to be what actually makes foreman/katello hang… i'm going to
> ride it out and see if it works this time…

Is there any error or stack trace in production.log when the 500 happens?

··· On 12/28/2016 02:26 PM, nhanlon@kayak.com wrote:

On Wednesday, December 28, 2016 at 2:19:14 PM UTC-5, nha...@kayak.com wrote:

Well this seems not good… I ended up just pointing it back to the DB on
the other server, but it looks like there are other issues…

Specifically, when a repo sync runs:

Dec 28 14:02:31 [redacted] pulp[5543]: pulp.server.event.http:ERROR:
Received HTTP 500 from HTTP notifier to https:
//[redacted]/katello/api/v2/repositories/sync_complete?token=zZh7D6xYaXaUj8FFfkvtq3ME6kvZegri.

That url gives:

{“displayMessage”:“Couldn’t find Katello::Repository with ‘id’=sync_complete”,“errors”:[“Couldn’t find Katello::Repository with ‘id’=sync_complete”]}

I’m not sure where to go from here… My only guess is this is an inconsistency with the DB somewhere, or that something didn’t get upgraded/migrated correctly in my katello upgrade from 3.1.x to 3.2.2.

Any help is greatly appreciated as when this happens my web UI freezes… sometimes requiring me to restart httpd.

I’m on IRC: NeilHanlon if you want to chat in realtime.

I was able to nail down what I believe was causing this last week with the
help of someone in the irc channel.

Unbeknownst to me, the pulp.conf for Apache was not written correctly, so
api calls for pulp requests were being passed to foreman directly instead
of pulp. I had to rerun the upgrade and manually trigger a restart of
Apache once pulp.conf was successfully written. All seems well so far.

··· On Jan 3, 2017 8:53 AM, "Chris Duryee" wrote:

On 12/28/2016 02:26 PM, nhanlon@kayak.com wrote:

I also think this is related to the overall problem:

2016-12-28 14:14:37 f101b581 [app] [I] Processing by Katello::Api::Rhsm::
CandlepinProxiesController#upload_package_profile as JSON
2016-12-28 14:14:37 f101b581 [app] [I] Parameters:
{"_json"=>"[FILTERED]",
“id”=>“432d04cc-d045-4b7f-9c92-2b4eb3dccd93”}

This appears to be what actually makes foreman/katello hang… i’m going
to
ride it out and see if it works this time…

Is there any error or stack trace in production.log when the 500 happens?

On Wednesday, December 28, 2016 at 2:19:14 PM UTC-5, nha...@kayak.com > wrote:

Well this seems not good… I ended up just pointing it back to the DB
on

the other server, but it looks like there are other issues…

Specifically, when a repo sync runs:

Dec 28 14:02:31 [redacted] pulp[5543]: pulp.server.event.http:ERROR:
Received HTTP 500 from HTTP notifier to https:
//[redacted]/katello/api/v2/repositories/sync_complete?token=
zZh7D6xYaXaUj8FFfkvtq3ME6kvZegri.

That url gives:

{“displayMessage”:“Couldn’t find Katello::Repository with
’id’=sync_complete”,“errors”:[“Couldn’t find Katello::Repository with
’id’=sync_complete”]}

I’m not sure where to go from here… My only guess is this is an
inconsistency with the DB somewhere, or that something didn’t get
upgraded/migrated correctly in my katello upgrade from 3.1.x to 3.2.2.

Any help is greatly appreciated as when this happens my web UI
freezes… sometimes requiring me to restart httpd.

I’m on IRC: NeilHanlon if you want to chat in realtime.


You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.