I started the installation early this morning and its been churning now
for about 12 hours. This is a bigger install than the test install I
ran the upgrade on yesterday, but 12 hours is a bit much.
Since there was no load on the machine, I started digging. I've found
this in /var/log/messages repeating over and over again :
Jun 23 17:49:33 katello pulp: pulp.server.db.connection:ERROR:
(14193-49952) Could not connect to MongoDB at localhost:27017:
Jun 23 17:49:33 katello pulp: pulp.server.db.connection:ERROR:
(14193-49952) Cannot connect to database default :
Jun 23 17:49:33 katello pulp: pulp.server.db.connection:ERROR:
(14193-49952) Unknown option max_pool_size
Jun 23 17:49:33 katello pulp: pulp.server.db.connection:ERROR:
(14193-49952) … Waiting 1 seconds and trying again.
Mongodb is up and running, and the ports are open :
I believe this turned out to be an old version of pulp-server causing
the problem. Not sure how that wasn't upgraded with the rest, but
updating it to the latest seems to have resolved the issue.
Still chugging along on the upgrade, but it's well on it's way not with
no errors popping up.
···
On 6/23/16 5:53 PM, Jason 'XenoPhage' Frisvold wrote:
> Greetings,
>
> I started the installation early this morning and its been churning now
> for about 12 hours. This is a bigger install than the test install I
> ran the upgrade on yesterday, but 12 hours is a bit much.
I'll toss up some supplemental info, as I had the same experience, I
started the upgrade on a Friday afternoon, and came back in on the
following Tuesday to find it still sitting at the Pulp upgrade step. I
performed an strace on the corresponding rake process, and found it was
performing a slew of stat, link, and unlink operations for RPM file
references in /var/lib/pulp. On a hunch that it was performing this
process on every single directory that correlates with a Content View, I
used the cvmanager utility from GitHub ( https://github.com/evgeni/katello-cvmanager/blob/master/
) to clean up all Content Views beyond the three most recent versions for
all of our Products, and my ensuing attempt to upgrade took a mere two to
three hours. Case and point, if you're like me and have a lot of Content
Views, then this is likely negatively impacting the upgrade process and
it's suggested to clean them up prior to completing it. I'll add that my
Pulp directory / volume is ~270G in size, just in case someone cares to
compare upgrade completion times based on that attribute. Hope this info
helps somebody out there!
Good to hear, let us know if you run into any more issues with the upgrade.
-John
John Mitsch
Red Hat Engineering
(860)-967-7285
irc: jomitsch
···
On Fri, Jun 24, 2016 at 7:03 PM, Jason 'XenoPhage' Frisvold < xenophage@godshell.com> wrote:
On 6/23/16 5:53 PM, Jason ‘XenoPhage’ Frisvold wrote:
Greetings,
I started the installation early this morning and its been churning now
for about 12 hours. This is a bigger install than the test install I
ran the upgrade on yesterday, but 12 hours is a bit much.
I believe this turned out to be an old version of pulp-server causing
the problem. Not sure how that wasn’t upgraded with the rest, but
updating it to the latest seems to have resolved the issue.
Still chugging along on the upgrade, but it’s well on it’s way not with
no errors popping up.
how long did it take in the end? I am upgrading our production katello
instance from 2.4.4 to 3.0, and the pulp_migrate step has taken 6 hours so
far - I see no errors in the logs and pulp-manage-db seems to be running,
taking CPU and memory time up. Our production system has a much larger pulp
repository than the test instance I've done this on, but is there any
advice on how long this step should take? (more a question for the Katello
devs I guess!) It is 4pm here now, but the last logs in /var/log/messages
from pulp are below (from 10:15am)
/var/lib/pulp is 176G vs the 42GB on the test instance
Thanks,
Dylan
···
On Tuesday, June 28, 2016 at 1:27:37 AM UTC+12, John Mitsch wrote:
>
> Jason,
>
> Good to hear, let us know if you run into any more issues with the upgrade.
>
> -John
>
> John Mitsch
> Red Hat Engineering
> (860)-967-7285
> irc: jomitsch
>
> On Fri, Jun 24, 2016 at 7:03 PM, Jason 'XenoPhage' Frisvold < > xeno...@godshell.com > wrote:
>
>> On 6/23/16 5:53 PM, Jason 'XenoPhage' Frisvold wrote:
>> > Greetings,
>> >
>> > I started the installation early this morning and its been churning now
>> > for about 12 hours. This is a bigger install than the test install I
>> > ran the upgrade on yesterday, but 12 hours is a bit much.
>>
>> I believe this turned out to be an old version of pulp-server causing
>> the problem. Not sure how that wasn't upgraded with the rest, but
>> updating it to the latest seems to have resolved the issue.
>>
>> Still chugging along on the upgrade, but it's well on it's way not with
>> no errors popping up. :)
>>
>> --
>> ---------------------------
>> Jason 'XenoPhage' Frisvold
>> xeno...@godshell.com
>> ---------------------------
>>
>> "Any sufficiently advanced magic is indistinguishable from technology."
>> - Niven's Inverse of Clarke's Third Law
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Foreman users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to foreman-user...@googlegroups.com .
>> To post to this group, send email to forema...@googlegroups.com
>> .
>> Visit this group at https://groups.google.com/group/foreman-users.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
haha - it must have heard me, it just finished and carried on to the next
steps, go figure! So 176GB of pulp data takes roughly 6 hours (suppose
it'll be different based on hardware etc!)
Dylan
···
On Friday, July 1, 2016 at 4:03:29 PM UTC+12, Dylan Baars wrote:
>
> Hi Jason,
>
> how long did it take in the end? I am upgrading our production katello
> instance from 2.4.4 to 3.0, and the pulp_migrate step has taken 6 hours so
> far - I see no errors in the logs and pulp-manage-db seems to be running,
> taking CPU and memory time up. Our production system has a much larger pulp
> repository than the test instance I've done this on, but is there any
> advice on how long this step should take? (more a question for the Katello
> devs I guess!) It is 4pm here now, but the last logs in /var/log/messages
> from pulp are below (from 10:15am)
>
> Jul 1 10:15:49 wellkatello.niwa.local pulp: py.warnings:WARNING:
> (21830-39488) multi=True)
> Jul 1 10:15:49 wellkatello.niwa.local pulp: py.warnings:WARNING:
> (21830-39488)
> Jul 1 10:15:49 wellkatello.niwa.local pulp: pulp.server.db.manage:INFO:
> Migration to pulp_rpm.plugins.migrations version 27 complete.
> Jul 1 10:15:49 wellkatello.niwa.local pulp: pulp.server.db.manage:INFO:
> Applying pulp_rpm.plugins.migrations version 28
>
> /var/lib/pulp is 176G vs the 42GB on the test instance
>
> Thanks,
> Dylan
>
Could you also quote the size of your MongoDBs? I'd like to report all of
this data to the Pulp team.
Thanks,
Eric
···
On Fri, Jul 1, 2016 at 12:10 AM, Dylan Baars wrote:
haha - it must have heard me, it just finished and carried on to the next
steps, go figure! So 176GB of pulp data takes roughly 6 hours (suppose
it’ll be different based on hardware etc!)
Dylan
On Friday, July 1, 2016 at 4:03:29 PM UTC+12, Dylan Baars wrote:
Hi Jason,
how long did it take in the end? I am upgrading our production katello
instance from 2.4.4 to 3.0, and the pulp_migrate step has taken 6 hours so
far - I see no errors in the logs and pulp-manage-db seems to be running,
taking CPU and memory time up. Our production system has a much larger pulp
repository than the test instance I’ve done this on, but is there any
advice on how long this step should take? (more a question for the Katello
devs I guess!) It is 4pm here now, but the last logs in /var/log/messages
from pulp are below (from 10:15am)
Also, if you were to look at /var/log/messages for the parts during the
upgrade, are you able to pickout which particular Pulp migration step
seemed to take the longest?
···
On Fri, Jul 1, 2016 at 7:26 AM, Eric D Helms wrote:
Jason and Dylan,
Could you also quote the size of your MongoDBs? I’d like to report all of
this data to the Pulp team.
Thanks,
Eric
On Fri, Jul 1, 2016 at 12:10 AM, Dylan Baars baarsd@gmail.com wrote:
haha - it must have heard me, it just finished and carried on to the next
steps, go figure! So 176GB of pulp data takes roughly 6 hours (suppose
it’ll be different based on hardware etc!)
Dylan
On Friday, July 1, 2016 at 4:03:29 PM UTC+12, Dylan Baars wrote:
Hi Jason,
how long did it take in the end? I am upgrading our production katello
instance from 2.4.4 to 3.0, and the pulp_migrate step has taken 6 hours so
far - I see no errors in the logs and pulp-manage-db seems to be running,
taking CPU and memory time up. Our production system has a much larger pulp
repository than the test instance I’ve done this on, but is there any
advice on how long this step should take? (more a question for the Katello
devs I guess!) It is 4pm here now, but the last logs in /var/log/messages
from pulp are below (from 10:15am)
sorry for the slow reply. Size of /var/lib/pulp is 176GB - is that what
you're after re mongoDB size?
As far as /var/log/messages goes - its "Applying
pulp_rpm.plugins.migrations version 28" that takes so long
Jul 2 09:14:07 wellkatello.niwa.local pulp: pulp.server.db.manage:INFO:
Migration to pulp_rpm.plugins.migrations version 23 complete.
Jul 2 09:14:07 wellkatello.niwa.local pulp: pulp.server.db.manage:INFO:
Applying pulp_rpm.plugins.migrations version 24
Jul 2 09:14:07 wellkatello.niwa.local pulp: pulp.server.db.manage:INFO:
Migration to pulp_rpm.plugins.migrations version 24 complete.
Jul 2 09:14:07 wellkatello.niwa.local pulp: pulp.server.db.manage:INFO:
Applying pulp_rpm.plugins.migrations version 25
Jul 2 09:14:07 wellkatello.niwa.local pulp: pulp.server.db.manage:INFO:
Migration to pulp_rpm.plugins.migrations version 25 complete.
Jul 2 09:14:07 wellkatello.niwa.local pulp: pulp.server.db.manage:INFO:
Applying pulp_rpm.plugins.migrations version 26
Jul 2 09:14:07 wellkatello.niwa.local pulp: py.warnings:WARNING:
(25904-01664)
/usr/lib/python2.7/site-packages/pulp_rpm/plugins/migrations/0026_remove_distribution_field.py:23:
DeprecationWarning: update is deprecated. Use replace_one, update_one or
update_many instead.
Jul 2 09:14:07 wellkatello.niwa.local pulp: py.warnings:WARNING:
(25904-01664) multi=True
Jul 2 09:14:07 wellkatello.niwa.local pulp: py.warnings:WARNING:
(25904-01664)
Jul 2 09:14:07 wellkatello.niwa.local pulp: pulp.server.db.manage:INFO:
Migration to pulp_rpm.plugins.migrations version 26 complete.
Jul 2 09:14:07 wellkatello.niwa.local pulp: pulp.server.db.manage:INFO:
Applying pulp_rpm.plugins.migrations version 27
Jul 2 09:14:07 wellkatello.niwa.local pulp: py.warnings:WARNING:
(25904-01664)
/usr/lib/python2.7/site-packages/pulp_rpm/plugins/migrations/0027_remove_checksum_type_field.py:21:
DeprecationWarning: update is deprecated. Use replace_one, update_one or
update_many instead.
Jul 2 09:14:07 wellkatello.niwa.local pulp: py.warnings:WARNING:
(25904-01664) multi=True)
Jul 2 09:14:07 wellkatello.niwa.local pulp: py.warnings:WARNING:
(25904-01664)
Jul 2 09:14:08 wellkatello.niwa.local pulp: pulp.server.db.manage:INFO:
Migration to pulp_rpm.plugins.migrations version 27 complete.
Jul 2 09:14:08 wellkatello.niwa.local pulp: pulp.server.db.manage:INFO:
Applying pulp_rpm.plugins.migrations version 28
Jul 2 15:46:39 wellkatello.niwa.local pulp: pulp.server.db.manage:INFO:
Migration to pulp_rpm.plugins.migrations version 28 complete.
Jul 2 15:46:39 wellkatello.niwa.local pulp: pulp.server.db.manage:INFO:
Applying pulp_rpm.plugins.migrations version 29
Jul 2 15:46:39 wellkatello.niwa.local pulp: py.warnings:WARNING:
(25904-01664)
/usr/lib/python2.7/site-packages/pulp_rpm/plugins/migrations/0029_remove_rpm_unused_fields.py:23:
DeprecationWarning: update is deprecated. Use replace_one, update_one or
update_many instead.
Jul 2 15:46:39 wellkatello.niwa.local pulp: py.warnings:WARNING:
(25904-01664) multi=True
Jul 2 15:46:39 wellkatello.niwa.local pulp: py.warnings:WARNING:
(25904-01664)
Jul 2 15:46:40 wellkatello.niwa.local pulp: py.warnings:WARNING:
(25904-01664)
/usr/lib/python2.7/site-packages/pulp_rpm/plugins/migrations/0029_remove_rpm_unused_fields.py:28:
DeprecationWarning: update is deprecated. Use replace_one, update_one or
update_many instead.
Jul 2 15:46:40 wellkatello.niwa.local pulp: py.warnings:WARNING:
(25904-01664) multi=True
Jul 2 15:46:40 wellkatello.niwa.local pulp: py.warnings:WARNING:
(25904-01664)
Jul 2 15:46:40 wellkatello.niwa.local pulp: pulp.server.db.manage:INFO:
Migration to pulp_rpm.plugins.migrations version 29 complete.
Jul 2 15:46:40 wellkatello.niwa.local pulp: pulp.server.db.manage:INFO:
Applying pulp_rpm.plugins.migrations version 30
Jul 2 15:46:40 wellkatello.niwa.local pulp: py.warnings:WARNING:
(25904-01664)
/usr/lib/python2.7/site-packages/pulp_rpm/plugins/migrations/0030_remove_errata_unused_fields.py:23:
DeprecationWarning: update is deprecated. Use replace_one, update_one or
update_many instead.
Jul 2 15:46:40 wellkatello.niwa.local pulp: py.warnings:WARNING:
(25904-01664) multi=True
Jul 2 15:46:40 wellkatello.niwa.local pulp: py.warnings:WARNING:
(25904-01664)
Jul 2 15:46:42 wellkatello.niwa.local pulp: pulp.server.db.manage:INFO:
Migration to pulp_rpm.plugins.migrations version 30 complete.
Happy to supply the whole log if its helpful!
Dylan
···
On Saturday, July 2, 2016 at 12:50:07 AM UTC+12, Eric Helms wrote:
>
> Also, if you were to look at /var/log/messages for the parts during the
> upgrade, are you able to pickout which particular Pulp migration step
> seemed to take the longest?
>
> On Fri, Jul 1, 2016 at 7:26 AM, Eric D Helms > wrote:
>
>> Jason and Dylan,
>>
>> Could you also quote the size of your MongoDBs? I'd like to report all of
>> this data to the Pulp team.
>>
>> Thanks,
>> Eric
>>
>> On Fri, Jul 1, 2016 at 12:10 AM, Dylan Baars > > wrote:
>>
>>> haha - it must have heard me, it just finished and carried on to the
>>> next steps, go figure! So 176GB of pulp data takes roughly 6 hours ;-)
>>> (suppose it'll be different based on hardware etc!)
>>>
>>> Dylan
>>>
>>> On Friday, July 1, 2016 at 4:03:29 PM UTC+12, Dylan Baars wrote:
>>>>
>>>> Hi Jason,
>>>>
>>>> how long did it take in the end? I am upgrading our production katello
>>>> instance from 2.4.4 to 3.0, and the pulp_migrate step has taken 6 hours so
>>>> far - I see no errors in the logs and pulp-manage-db seems to be running,
>>>> taking CPU and memory time up. Our production system has a much larger pulp
>>>> repository than the test instance I've done this on, but is there any
>>>> advice on how long this step should take? (more a question for the Katello
>>>> devs I guess!) It is 4pm here now, but the last logs in /var/log/messages
>>>> from pulp are below (from 10:15am)
>>>>
>>>> Jul 1 10:15:49 wellkatello.niwa.local pulp: py.warnings:WARNING:
>>>> (21830-39488) multi=True)
>>>> Jul 1 10:15:49 wellkatello.niwa.local pulp: py.warnings:WARNING:
>>>> (21830-39488)
>>>> Jul 1 10:15:49 wellkatello.niwa.local pulp:
>>>> pulp.server.db.manage:INFO: Migration to pulp_rpm.plugins.migrations
>>>> version 27 complete.
>>>> Jul 1 10:15:49 wellkatello.niwa.local pulp:
>>>> pulp.server.db.manage:INFO: Applying pulp_rpm.plugins.migrations version 28
>>>>
>>>> /var/lib/pulp is 176G vs the 42GB on the test instance
>>>>
>>>> Thanks,
>>>> Dylan
>>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Foreman users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to foreman-user...@googlegroups.com .
>>> To post to this group, send email to forema...@googlegroups.com
>>> .
>>> Visit this group at https://groups.google.com/group/foreman-users.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
>>
>> --
>> Eric D. Helms
>> Red Hat Engineering
>> Ph.D. Student - North Carolina State University
>>
>
>
>
> --
> Eric D. Helms
> Red Hat Engineering
> Ph.D. Student - North Carolina State University
>
(suppose