When trying to create ~50 hosts at a time or more via the API, we're
running into deadlocks in the MySQL database, like so:
···
-----
Mysql2::Error: Deadlock found when trying to get lock; try restarting
transaction: INSERT INTO `hosts` (`architecture_id`, `build`, `certname`,
`comment`, `compute_profile_id`, `compute_resource_id`, `created_at`,
`disk`, `domain_id`, `enabled`, `environment_id`, `hostgroup_id`,
`image_file`, `image_id`, `installed_at`, `ip`, `last_compile`,
`last_freshcheck`, `last_report`, `location_id`, `mac`, `managed`,
`medium_id`, `model_id`, `name`, `operatingsystem_id`, `organization_id`,
`otp`, `owner_id`, `owner_type`, `provision_method`, `ptable_id`,
`puppet_ca_proxy_id`, `puppet_proxy_id`, `puppet_status`, `realm_id`,
`root_pass`, `serial`, `source_file_id`, `subnet_id`, `type`, `updated_at`,
`use_image`, `uuid`) VALUES (1, 0, NULL, NULL, NULL, 6, '2014-06-25
12:15:06', NULL, 1, 1, 1, 17, NULL, 20, NULL, '54.193.205.141', NULL, NULL,
NULL, NULL, NULL, 1, NULL, NULL, 'host.domain.com', 6, NULL, NULL, 9,
'User', NULL, NULL, 1, 1, 0, NULL, NULL, NULL, NULL, NULL, 'Host::Managed',
'2014-06-25 12:15:06', NULL, 'i-5f309101') (ActiveRecord::StatementInvalid)
-----
I checked the database and it's not particularly heavily loaded nor suffers
from poor performance otherwise from what I can see(disclaimer, I'm no DBA).
But as far as I know, the program that talks to the API basically just
fires one POST per host, and I guess this is simply a case of timing,
creating many hosts at once and a few will deadlock.
My questions then are for one, what’s the proper way to do this? Will the
API accept several hosts one after another in one call? If so I guess
that’s the proper way to do this?
Or is one call per host the right way and this is something that shouldn’t
be happening to begin with?
Can you check how many ruby processes are handling the requests?
We use optimistic locking, which can eventually lead to these
situations. But the server should survive this and should only miss few
calls, not all.
Interesting finding for sure. We need a test case. Do you have the
snippet that causes this?
LZ
···
On Thu, Jun 26, 2014 at 05:45:29AM -0700, Johan Sunnerstig wrote:
> Hello,
>
> When trying to create ~50 hosts at a time or more via the API, we're
> running into deadlocks in the MySQL database, like so:
> -----
> Mysql2::Error: Deadlock found when trying to get lock; try restarting
> transaction: INSERT INTO `hosts` (`architecture_id`, `build`, `certname`,
> `comment`, `compute_profile_id`, `compute_resource_id`, `created_at`,
> `disk`, `domain_id`, `enabled`, `environment_id`, `hostgroup_id`,
> `image_file`, `image_id`, `installed_at`, `ip`, `last_compile`,
> `last_freshcheck`, `last_report`, `location_id`, `mac`, `managed`,
> `medium_id`, `model_id`, `name`, `operatingsystem_id`, `organization_id`,
> `otp`, `owner_id`, `owner_type`, `provision_method`, `ptable_id`,
> `puppet_ca_proxy_id`, `puppet_proxy_id`, `puppet_status`, `realm_id`,
> `root_pass`, `serial`, `source_file_id`, `subnet_id`, `type`, `updated_at`,
> `use_image`, `uuid`) VALUES (1, 0, NULL, NULL, NULL, 6, '2014-06-25
> 12:15:06', NULL, 1, 1, 1, 17, NULL, 20, NULL, '54.193.205.141', NULL, NULL,
> NULL, NULL, NULL, 1, NULL, NULL, 'host.domain.com', 6, NULL, NULL, 9,
> 'User', NULL, NULL, 1, 1, 0, NULL, NULL, NULL, NULL, NULL, 'Host::Managed',
> '2014-06-25 12:15:06', NULL, 'i-5f309101') (ActiveRecord::StatementInvalid)
> -----
> I checked the database and it's not particularly heavily loaded nor suffers
> from poor performance otherwise from what I can see(disclaimer, I'm no DBA).
> But as far as I know, the program that talks to the API basically just
> fires one POST per host, and I guess this is simply a case of timing,
> creating many hosts at once and a few will deadlock.
>
> My questions then are for one, what's the proper way to do this? Will the
> API accept several hosts one after another in one call? If so I guess
> that's the proper way to do this?
> Or is one call per host the right way and this is something that shouldn't
> be happening to begin with?
>
> Regards
> Johan
>
> --
> You received this message because you are subscribed to the Google Groups "Foreman users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to foreman-users+unsubscribe@googlegroups.com.
> To post to this group, send email to foreman-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/foreman-users.
> For more options, visit https://groups.google.com/d/optout.
> Can you check how many ruby processes are handling the requests?
>
> We use optimistic locking, which can eventually lead to these
> situations. But the server should survive this and should only miss few
> calls, not all.
>
> Interesting finding for sure. We need a test case. Do you have the
> snippet that causes this?
>
> LZ
>
> > Hello,
> >
> > When trying to create ~50 hosts at a time or more via the API, we're
> > running into deadlocks in the MySQL database, like so:
> > -----
> > Mysql2::Error: Deadlock found when trying to get lock; try restarting
>
interesting… this is a mysql error, not a rails error, I wonder what is
the threshold.
maybe an easy solution would to recover from deadlocks by waiting and
trying again?
Ohad
···
On Thu, Jun 26, 2014 at 5:14 PM, Lukas Zapletal wrote:
> On Thu, Jun 26, 2014 at 05:45:29AM -0700, Johan Sunnerstig wrote:
I checked the database and it’s not particularly heavily loaded nor
suffers
from poor performance otherwise from what I can see(disclaimer, I’m no
DBA).
But as far as I know, the program that talks to the API basically just
fires one POST per host, and I guess this is simply a case of timing,
creating many hosts at once and a few will deadlock.
My questions then are for one, what’s the proper way to do this? Will the
API accept several hosts one after another in one call? If so I guess
that’s the proper way to do this?
Or is one call per host the right way and this is something that
shouldn’t
be happening to begin with?
One easy solution I was thinking of was to simply add a one second sleep
before every call.
That is, unless the API can handle one big chunk of hosts all in one call.
This particular error came from a test we ran yesterday where we brought up
2*50 hosts, i.e two bursts of 50 hosts each. I'd say we got ~5 of these per
burst.
Oh and this is Foreman 1.5 on Debian 6/ruby 1.8.7(yeah, there's an upgrade
planned here).
Johan
···
On Thursday, June 26, 2014 4:19:31 PM UTC+2, ohad wrote:
>
>
>
>
> On Thu, Jun 26, 2014 at 5:14 PM, Lukas Zapletal > wrote:
>
>> Can you check how many ruby processes are handling the requests?
>>
>> We use optimistic locking, which can eventually lead to these
>> situations. But the server should survive this and should only miss few
>> calls, not all.
>>
>> Interesting finding for sure. We need a test case. Do you have the
>> snippet that causes this?
>>
>> LZ
>>
>> On Thu, Jun 26, 2014 at 05:45:29AM -0700, Johan Sunnerstig wrote:
>> > Hello,
>> >
>> > When trying to create ~50 hosts at a time or more via the API, we're
>> > running into deadlocks in the MySQL database, like so:
>> > -----
>> > Mysql2::Error: Deadlock found when trying to get lock; try restarting
>>
>
> interesting.. this is a mysql error, not a rails error, I wonder what is
> the threshold.
>
> maybe an easy solution would to recover from deadlocks by waiting and
> trying again?
>
> Ohad
>
I am unable to reproduce this with hammer CLI, because to stress Foreman
one need to use plain REST API.
for X in $(seq 1 25); do NAME=$(openssl rand -hex 6); MAC=$(openssl rand
-hex 6 | sed 's/(…)/\1:/g; s/.$//'); (hammer host create
–environment-id 1 --architecture-id 1 --domain-id 1 --puppet-proxy-id 1
–operatingsystem-id 1 --root-password dummy123456789
–partition-table-id 7 --medium-id 7 --mac "$MAC" --name "host$NAME" &);
done
I think adding some delay would solve this issue. I wonder if we can
reproduce the same behavior with Postgres. I guess so.
LZ
···
On Thu, Jun 26, 2014 at 07:34:55AM -0700, Johan Sunnerstig wrote:
>
>
> On Thursday, June 26, 2014 4:19:31 PM UTC+2, ohad wrote:
> >
> >
> >
> >
> > On Thu, Jun 26, 2014 at 5:14 PM, Lukas Zapletal > > wrote:
> >
> >> Can you check how many ruby processes are handling the requests?
> >>
> >> We use optimistic locking, which can eventually lead to these
> >> situations. But the server should survive this and should only miss few
> >> calls, not all.
> >>
> >> Interesting finding for sure. We need a test case. Do you have the
> >> snippet that causes this?
> >>
> >> LZ
> >>
> >> On Thu, Jun 26, 2014 at 05:45:29AM -0700, Johan Sunnerstig wrote:
> >> > Hello,
> >> >
> >> > When trying to create ~50 hosts at a time or more via the API, we're
> >> > running into deadlocks in the MySQL database, like so:
> >> > -----
> >> > Mysql2::Error: Deadlock found when trying to get lock; try restarting
> >>
> >
> > interesting.. this is a mysql error, not a rails error, I wonder what is
> > the threshold.
> >
> > maybe an easy solution would to recover from deadlocks by waiting and
> > trying again?
> >
> > Ohad
> >
>
>
> One easy solution I was thinking of was to simply add a one second sleep
> before every call.
> That is, unless the API can handle one big chunk of hosts all in one call.
>
> This particular error came from a test we ran yesterday where we brought up
> 2*50 hosts, i.e two bursts of 50 hosts each. I'd say we got ~5 of these per
> burst.
>
> Oh and this is Foreman 1.5 on Debian 6/ruby 1.8.7(yeah, there's an upgrade
> planned here).
>
> Johan
>
> --
> You received this message because you are subscribed to the Google Groups "Foreman users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to foreman-users+unsubscribe@googlegroups.com.
> To post to this group, send email to foreman-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/foreman-users.
> For more options, visit https://groups.google.com/d/optout.
It would be interesting to know whether anyone else has tried similar
things in the past, regardless of whether they ran into this issue or not.
FYI the machines we're creating are Amazon EC2 instances, for the most part
we don't create more than a few at once which is probably why we haven't
ran into this previously.
Johan
···
On Friday, June 27, 2014 10:23:04 AM UTC+2, Lukas Zapletal wrote:
>
> I am unable to reproduce this with hammer CLI, because to stress Foreman
> one need to use plain REST API.
>
> for X in $(seq 1 25); do NAME=$(openssl rand -hex 6); MAC=$(openssl rand
> -hex 6 | sed 's/\(..\)/\1:/g; s/.$//'); (hammer host create
> --environment-id 1 --architecture-id 1 --domain-id 1 --puppet-proxy-id 1
> --operatingsystem-id 1 --root-password dummy123456789
> --partition-table-id 7 --medium-id 7 --mac "$MAC" --name "host$NAME" &);
> done
>
> I think adding some delay would solve this issue. I wonder if we can
> reproduce the same behavior with Postgres. I guess so.
>
> LZ
>
> On Thu, Jun 26, 2014 at 07:34:55AM -0700, Johan Sunnerstig wrote:
> >
> >
> > On Thursday, June 26, 2014 4:19:31 PM UTC+2, ohad wrote:
> > >
> > >
> > >
> > >
> > > On Thu, Jun 26, 2014 at 5:14 PM, Lukas Zapletal > > > wrote:
> > >
> > >> Can you check how many ruby processes are handling the requests?
> > >>
> > >> We use optimistic locking, which can eventually lead to these
> > >> situations. But the server should survive this and should only miss
> few
> > >> calls, not all.
> > >>
> > >> Interesting finding for sure. We need a test case. Do you have the
> > >> snippet that causes this?
> > >>
> > >> LZ
> > >>
> > >> On Thu, Jun 26, 2014 at 05:45:29AM -0700, Johan Sunnerstig wrote:
> > >> > Hello,
> > >> >
> > >> > When trying to create ~50 hosts at a time or more via the API,
> we're
> > >> > running into deadlocks in the MySQL database, like so:
> > >> > -----
> > >> > Mysql2::Error: Deadlock found when trying to get lock; try
> restarting
> > >>
> > >
> > > interesting.. this is a mysql error, not a rails error, I wonder what
> is
> > > the threshold.
> > >
> > > maybe an easy solution would to recover from deadlocks by waiting and
> > > trying again?
> > >
> > > Ohad
> > >
> >
> >
> > One easy solution I was thinking of was to simply add a one second sleep
> > before every call.
> > That is, unless the API can handle one big chunk of hosts all in one
> call.
> >
> > This particular error came from a test we ran yesterday where we brought
> up
> > 2*50 hosts, i.e two bursts of 50 hosts each. I'd say we got ~5 of these
> per
> > burst.
> >
> > Oh and this is Foreman 1.5 on Debian 6/ruby 1.8.7(yeah, there's an
> upgrade
> > planned here).
> >
> > Johan
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "Foreman users" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to foreman-user...@googlegroups.com .
> > To post to this group, send email to forema...@googlegroups.com
> .
> > Visit this group at http://groups.google.com/group/foreman-users.
> > For more options, visit https://groups.google.com/d/optout.
>
> --
> Later,
>
> Lukas "lzap" Zapletal
> irc: lzap #theforeman
>
···
----- Original Message -----
> It would be interesting to know whether anyone else has tried similar
> things in the past, regardless of whether they ran into this issue or not.
> FYI the machines we're creating are Amazon EC2 instances, for the most part
> we don't create more than a few at once which is probably why we haven't
> ran into this previously.
>
> Johan
>
> On Friday, June 27, 2014 10:23:04 AM UTC+2, Lukas Zapletal wrote:
> >
> > I am unable to reproduce this with hammer CLI, because to stress Foreman
> > one need to use plain REST API.
> >
> > for X in $(seq 1 25); do NAME=$(openssl rand -hex 6); MAC=$(openssl rand
> > -hex 6 | sed 's/\(..\)/\1:/g; s/.$//'); (hammer host create
> > --environment-id 1 --architecture-id 1 --domain-id 1 --puppet-proxy-id 1
> > --operatingsystem-id 1 --root-password dummy123456789
> > --partition-table-id 7 --medium-id 7 --mac "$MAC" --name "host$NAME" &);
> > done
> >
> > I think adding some delay would solve this issue. I wonder if we can
> > reproduce the same behavior with Postgres. I guess so.
> >
> > LZ
> >
> > On Thu, Jun 26, 2014 at 07:34:55AM -0700, Johan Sunnerstig wrote:
> > >
> > >
> > > On Thursday, June 26, 2014 4:19:31 PM UTC+2, ohad wrote:
> > > >
> > > >
> > > >
> > > >
> > > > On Thu, Jun 26, 2014 at 5:14 PM, Lukas Zapletal > > > > wrote:
> > > >
> > > >> Can you check how many ruby processes are handling the requests?
> > > >>
> > > >> We use optimistic locking, which can eventually lead to these
> > > >> situations. But the server should survive this and should only miss
> > few
> > > >> calls, not all.
> > > >>
> > > >> Interesting finding for sure. We need a test case. Do you have the
> > > >> snippet that causes this?
> > > >>
> > > >> LZ
> > > >>
> > > >> On Thu, Jun 26, 2014 at 05:45:29AM -0700, Johan Sunnerstig wrote:
> > > >> > Hello,
> > > >> >
> > > >> > When trying to create ~50 hosts at a time or more via the API,
> > we're
> > > >> > running into deadlocks in the MySQL database, like so:
> > > >> > -----
> > > >> > Mysql2::Error: Deadlock found when trying to get lock; try
> > restarting
> > > >>
> > > >
> > > > interesting.. this is a mysql error, not a rails error, I wonder what
> > is
> > > > the threshold.
> > > >
> > > > maybe an easy solution would to recover from deadlocks by waiting and
> > > > trying again?
> > > >
> > > > Ohad
> > > >
> > >
> > >
> > > One easy solution I was thinking of was to simply add a one second sleep
> > > before every call.
> > > That is, unless the API can handle one big chunk of hosts all in one
> > call.
> > >
> > > This particular error came from a test we ran yesterday where we brought
> > up
> > > 2*50 hosts, i.e two bursts of 50 hosts each. I'd say we got ~5 of these
> > per
> > > burst.
> > >
> > > Oh and this is Foreman 1.5 on Debian 6/ruby 1.8.7(yeah, there's an
> > upgrade
> > > planned here).
> > >
> > > Johan
> > >
> > > --
> > > You received this message because you are subscribed to the Google
> > Groups "Foreman users" group.
> > > To unsubscribe from this group and stop receiving emails from it, send
> > an email to foreman-user...@googlegroups.com .
> > > To post to this group, send email to forema...@googlegroups.com
> > .
> > > Visit this group at http://groups.google.com/group/foreman-users.
> > > For more options, visit https://groups.google.com/d/optout.
> >
> > --
> > Later,
> >
> > Lukas "lzap" Zapletal
> > irc: lzap #theforeman
> >
>
> --
> You received this message because you are subscribed to the Google Groups
> "Foreman users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to foreman-users+unsubscribe@googlegroups.com.
> To post to this group, send email to foreman-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/foreman-users.
> For more options, visit https://groups.google.com/d/optout.
>
Thanks, I'll see if I can run another test with ~50 hosts and provide the
data in that bug.
Johan
···
On Monday, June 30, 2014 3:02:51 PM UTC+2, Tom McKay wrote:
>
>
> A while back I ran modest test, six threads hitting production install
> (postgres), and hit "PGError: ERROR: deadlock detected"
>
> http://projects.theforeman.org/issues/5990
>
> ----- Original Message -----
> > It would be interesting to know whether anyone else has tried similar
> > things in the past, regardless of whether they ran into this issue or
> not.
> > FYI the machines we're creating are Amazon EC2 instances, for the most
> part
> > we don't create more than a few at once which is probably why we haven't
> > ran into this previously.
> >
> > Johan
> >
> > On Friday, June 27, 2014 10:23:04 AM UTC+2, Lukas Zapletal wrote:
> > >
> > > I am unable to reproduce this with hammer CLI, because to stress
> Foreman
> > > one need to use plain REST API.
> > >
> > > for X in $(seq 1 25); do NAME=$(openssl rand -hex 6); MAC=$(openssl
> rand
> > > -hex 6 | sed 's/\(..\)/\1:/g; s/.$//'); (hammer host create
> > > --environment-id 1 --architecture-id 1 --domain-id 1 --puppet-proxy-id
> 1
> > > --operatingsystem-id 1 --root-password dummy123456789
> > > --partition-table-id 7 --medium-id 7 --mac "$MAC" --name "host$NAME"
> &);
> > > done
> > >
> > > I think adding some delay would solve this issue. I wonder if we can
> > > reproduce the same behavior with Postgres. I guess so.
> > >
> > > LZ
> > >
> > > On Thu, Jun 26, 2014 at 07:34:55AM -0700, Johan Sunnerstig wrote:
> > > >
> > > >
> > > > On Thursday, June 26, 2014 4:19:31 PM UTC+2, ohad wrote:
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Jun 26, 2014 at 5:14 PM, Lukas Zapletal > > > > > wrote:
> > > > >
> > > > >> Can you check how many ruby processes are handling the requests?
> > > > >>
> > > > >> We use optimistic locking, which can eventually lead to these
> > > > >> situations. But the server should survive this and should only
> miss
> > > few
> > > > >> calls, not all.
> > > > >>
> > > > >> Interesting finding for sure. We need a test case. Do you have
> the
> > > > >> snippet that causes this?
> > > > >>
> > > > >> LZ
> > > > >>
> > > > >> On Thu, Jun 26, 2014 at 05:45:29AM -0700, Johan Sunnerstig wrote:
> > > > >> > Hello,
> > > > >> >
> > > > >> > When trying to create ~50 hosts at a time or more via the API,
> > > we're
> > > > >> > running into deadlocks in the MySQL database, like so:
> > > > >> > -----
> > > > >> > Mysql2::Error: Deadlock found when trying to get lock; try
> > > restarting
> > > > >>
> > > > >
> > > > > interesting.. this is a mysql error, not a rails error, I wonder
> what
> > > is
> > > > > the threshold.
> > > > >
> > > > > maybe an easy solution would to recover from deadlocks by waiting
> and
> > > > > trying again?
> > > > >
> > > > > Ohad
> > > > >
> > > >
> > > >
> > > > One easy solution I was thinking of was to simply add a one second
> sleep
> > > > before every call.
> > > > That is, unless the API can handle one big chunk of hosts all in one
> > > call.
> > > >
> > > > This particular error came from a test we ran yesterday where we
> brought
> > > up
> > > > 2*50 hosts, i.e two bursts of 50 hosts each. I'd say we got ~5 of
> these
> > > per
> > > > burst.
> > > >
> > > > Oh and this is Foreman 1.5 on Debian 6/ruby 1.8.7(yeah, there's an
> > > upgrade
> > > > planned here).
> > > >
> > > > Johan
> > > >
> > > > --
> > > > You received this message because you are subscribed to the Google
> > > Groups "Foreman users" group.
> > > > To unsubscribe from this group and stop receiving emails from it,
> send
> > > an email to foreman-user...@googlegroups.com .
> > > > To post to this group, send email to forema...@googlegroups.com
> > > .
> > > > Visit this group at http://groups.google.com/group/foreman-users.
> > > > For more options, visit https://groups.google.com/d/optout.
> > >
> > > --
> > > Later,
> > >
> > > Lukas "lzap" Zapletal
> > > irc: lzap #theforeman
> > >
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups
> > "Foreman users" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an
> > email to foreman-user...@googlegroups.com .
> > To post to this group, send email to forema...@googlegroups.com
> .
> > Visit this group at http://groups.google.com/group/foreman-users.
> > For more options, visit https://groups.google.com/d/optout.
> >
>
Just as a reference in case anyone lands here via Google or some such, we
eventually solved this problem by migrating off of MySQL and onto
PostgreSQL.
Johan
···
On Tuesday, July 1, 2014 3:56:11 PM UTC+2, Johan Sunnerstig wrote:
>
> Thanks, I'll see if I can run another test with ~50 hosts and provide the
> data in that bug.
>
> Johan
>
> On Monday, June 30, 2014 3:02:51 PM UTC+2, Tom McKay wrote:
>>
>>
>> A while back I ran modest test, six threads hitting production install
>> (postgres), and hit "PGError: ERROR: deadlock detected"
>>
>> http://projects.theforeman.org/issues/5990
>>
>> ----- Original Message -----
>> > It would be interesting to know whether anyone else has tried similar
>> > things in the past, regardless of whether they ran into this issue or
>> not.
>> > FYI the machines we're creating are Amazon EC2 instances, for the most
>> part
>> > we don't create more than a few at once which is probably why we
>> haven't
>> > ran into this previously.
>> >
>> > Johan
>> >
>> > On Friday, June 27, 2014 10:23:04 AM UTC+2, Lukas Zapletal wrote:
>> > >
>> > > I am unable to reproduce this with hammer CLI, because to stress
>> Foreman
>> > > one need to use plain REST API.
>> > >
>> > > for X in $(seq 1 25); do NAME=$(openssl rand -hex 6); MAC=$(openssl
>> rand
>> > > -hex 6 | sed 's/\(..\)/\1:/g; s/.$//'); (hammer host create
>> > > --environment-id 1 --architecture-id 1 --domain-id 1
>> --puppet-proxy-id 1
>> > > --operatingsystem-id 1 --root-password dummy123456789
>> > > --partition-table-id 7 --medium-id 7 --mac "$MAC" --name "host$NAME"
>> &);
>> > > done
>> > >
>> > > I think adding some delay would solve this issue. I wonder if we can
>> > > reproduce the same behavior with Postgres. I guess so.
>> > >
>> > > LZ
>> > >
>> > > On Thu, Jun 26, 2014 at 07:34:55AM -0700, Johan Sunnerstig wrote:
>> > > >
>> > > >
>> > > > On Thursday, June 26, 2014 4:19:31 PM UTC+2, ohad wrote:
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > On Thu, Jun 26, 2014 at 5:14 PM, Lukas Zapletal > > > > > > wrote:
>> > > > >
>> > > > >> Can you check how many ruby processes are handling the requests?
>> > > > >>
>> > > > >> We use optimistic locking, which can eventually lead to these
>> > > > >> situations. But the server should survive this and should only
>> miss
>> > > few
>> > > > >> calls, not all.
>> > > > >>
>> > > > >> Interesting finding for sure. We need a test case. Do you have
>> the
>> > > > >> snippet that causes this?
>> > > > >>
>> > > > >> LZ
>> > > > >>
>> > > > >> On Thu, Jun 26, 2014 at 05:45:29AM -0700, Johan Sunnerstig >> wrote:
>> > > > >> > Hello,
>> > > > >> >
>> > > > >> > When trying to create ~50 hosts at a time or more via the API,
>> > > we're
>> > > > >> > running into deadlocks in the MySQL database, like so:
>> > > > >> > -----
>> > > > >> > Mysql2::Error: Deadlock found when trying to get lock; try
>> > > restarting
>> > > > >>
>> > > > >
>> > > > > interesting.. this is a mysql error, not a rails error, I wonder
>> what
>> > > is
>> > > > > the threshold.
>> > > > >
>> > > > > maybe an easy solution would to recover from deadlocks by waiting
>> and
>> > > > > trying again?
>> > > > >
>> > > > > Ohad
>> > > > >
>> > > >
>> > > >
>> > > > One easy solution I was thinking of was to simply add a one second
>> sleep
>> > > > before every call.
>> > > > That is, unless the API can handle one big chunk of hosts all in
>> one
>> > > call.
>> > > >
>> > > > This particular error came from a test we ran yesterday where we
>> brought
>> > > up
>> > > > 2*50 hosts, i.e two bursts of 50 hosts each. I'd say we got ~5 of
>> these
>> > > per
>> > > > burst.
>> > > >
>> > > > Oh and this is Foreman 1.5 on Debian 6/ruby 1.8.7(yeah, there's an
>> > > upgrade
>> > > > planned here).
>> > > >
>> > > > Johan
>> > > >
>> > > > --
>> > > > You received this message because you are subscribed to the Google
>> > > Groups "Foreman users" group.
>> > > > To unsubscribe from this group and stop receiving emails from it,
>> send
>> > > an email to foreman-user...@googlegroups.com .
>> > > > To post to this group, send email to forema...@googlegroups.com
>> > > .
>> > > > Visit this group at http://groups.google.com/group/foreman-users.
>> > > > For more options, visit https://groups.google.com/d/optout.
>> > >
>> > > --
>> > > Later,
>> > >
>> > > Lukas "lzap" Zapletal
>> > > irc: lzap #theforeman
>> > >
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> Groups
>> > "Foreman users" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> an
>> > email to foreman-user...@googlegroups.com.
>> > To post to this group, send email to forema...@googlegroups.com.
>> > Visit this group at http://groups.google.com/group/foreman-users.
>> > For more options, visit https://groups.google.com/d/optout.
>> >
>>
>
I'm running into this issue as well, only on Postgres.
I'm writing an async client using Python, which talks to the Foreman Rest
API. If I try to create large numbers of hosts in parallel, I see all
connections being simultaneously made to passenger on the server side, but
often most of the creates will fail. I've had it happen with as few as 10
hosts - 8 failed and 2 were successful. The logs will show entries like
this:
DETAIL: Process 13727 waits for ShareLock on transaction 14539; blocked by
process 13725.
Process 13725 waits for ExclusiveLock on tuple (1,24) of relation
18856 of database 18716; blocked by process 13727.
I'm using a development VM for this, and it's small, ~2GB of RAM, running
Foreman plus the database (I just did the standard install off the Foreman
site). If I drop the number of passenger processes down to 2, I don't see
the issue. If I raise it to 7 or more, I have the problem. Maybe part of
my issue is due to the small server, but it looks like the true bottleneck
is the database. Has anyone seen anything similar?
···
On Thursday, November 27, 2014 at 6:20:01 AM UTC-8, Johan Sunnerstig wrote:
>
> Just as a reference in case anyone lands here via Google or some such, we
> eventually solved this problem by migrating off of MySQL and onto
> PostgreSQL.
>
> Johan
>
> On Tuesday, July 1, 2014 3:56:11 PM UTC+2, Johan Sunnerstig wrote:
>>
>> Thanks, I'll see if I can run another test with ~50 hosts and provide the
>> data in that bug.
>>
>> Johan
>>
>> On Monday, June 30, 2014 3:02:51 PM UTC+2, Tom McKay wrote:
>>>
>>>
>>> A while back I ran modest test, six threads hitting production install
>>> (postgres), and hit "PGError: ERROR: deadlock detected"
>>>
>>> http://projects.theforeman.org/issues/5990
>>>
>>> ----- Original Message -----
>>> > It would be interesting to know whether anyone else has tried similar
>>> > things in the past, regardless of whether they ran into this issue or
>>> not.
>>> > FYI the machines we're creating are Amazon EC2 instances, for the most
>>> part
>>> > we don't create more than a few at once which is probably why we
>>> haven't
>>> > ran into this previously.
>>> >
>>> > Johan
>>> >
>>> > On Friday, June 27, 2014 10:23:04 AM UTC+2, Lukas Zapletal wrote:
>>> > >
>>> > > I am unable to reproduce this with hammer CLI, because to stress
>>> Foreman
>>> > > one need to use plain REST API.
>>> > >
>>> > > for X in $(seq 1 25); do NAME=$(openssl rand -hex 6); MAC=$(openssl
>>> rand
>>> > > -hex 6 | sed 's/\(..\)/\1:/g; s/.$//'); (hammer host create
>>> > > --environment-id 1 --architecture-id 1 --domain-id 1
>>> --puppet-proxy-id 1
>>> > > --operatingsystem-id 1 --root-password dummy123456789
>>> > > --partition-table-id 7 --medium-id 7 --mac "$MAC" --name "host$NAME"
>>> &);
>>> > > done
>>> > >
>>> > > I think adding some delay would solve this issue. I wonder if we can
>>> > > reproduce the same behavior with Postgres. I guess so.
>>> > >
>>> > > LZ
>>> > >
>>> > > On Thu, Jun 26, 2014 at 07:34:55AM -0700, Johan Sunnerstig wrote:
>>> > > >
>>> > > >
>>> > > > On Thursday, June 26, 2014 4:19:31 PM UTC+2, ohad wrote:
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > > On Thu, Jun 26, 2014 at 5:14 PM, Lukas Zapletal < >>> lz...@redhat.com >>> > > > > > wrote:
>>> > > > >
>>> > > > >> Can you check how many ruby processes are handling the
>>> requests?
>>> > > > >>
>>> > > > >> We use optimistic locking, which can eventually lead to these
>>> > > > >> situations. But the server should survive this and should only
>>> miss
>>> > > few
>>> > > > >> calls, not all.
>>> > > > >>
>>> > > > >> Interesting finding for sure. We need a test case. Do you have
>>> the
>>> > > > >> snippet that causes this?
>>> > > > >>
>>> > > > >> LZ
>>> > > > >>
>>> > > > >> On Thu, Jun 26, 2014 at 05:45:29AM -0700, Johan Sunnerstig >>> wrote:
>>> > > > >> > Hello,
>>> > > > >> >
>>> > > > >> > When trying to create ~50 hosts at a time or more via the
>>> API,
>>> > > we're
>>> > > > >> > running into deadlocks in the MySQL database, like so:
>>> > > > >> > -----
>>> > > > >> > Mysql2::Error: Deadlock found when trying to get lock; try
>>> > > restarting
>>> > > > >>
>>> > > > >
>>> > > > > interesting.. this is a mysql error, not a rails error, I wonder
>>> what
>>> > > is
>>> > > > > the threshold.
>>> > > > >
>>> > > > > maybe an easy solution would to recover from deadlocks by
>>> waiting and
>>> > > > > trying again?
>>> > > > >
>>> > > > > Ohad
>>> > > > >
>>> > > >
>>> > > >
>>> > > > One easy solution I was thinking of was to simply add a one second
>>> sleep
>>> > > > before every call.
>>> > > > That is, unless the API can handle one big chunk of hosts all in
>>> one
>>> > > call.
>>> > > >
>>> > > > This particular error came from a test we ran yesterday where we
>>> brought
>>> > > up
>>> > > > 2*50 hosts, i.e two bursts of 50 hosts each. I'd say we got ~5 of
>>> these
>>> > > per
>>> > > > burst.
>>> > > >
>>> > > > Oh and this is Foreman 1.5 on Debian 6/ruby 1.8.7(yeah, there's an
>>> > > upgrade
>>> > > > planned here).
>>> > > >
>>> > > > Johan
>>> > > >
>>> > > > --
>>> > > > You received this message because you are subscribed to the Google
>>> > > Groups "Foreman users" group.
>>> > > > To unsubscribe from this group and stop receiving emails from it,
>>> send
>>> > > an email to foreman-user...@googlegroups.com .
>>> > > > To post to this group, send email to forema...@googlegroups.com
>>> > > .
>>> > > > Visit this group at http://groups.google.com/group/foreman-users.
>>> > > > For more options, visit https://groups.google.com/d/optout.
>>> > >
>>> > > --
>>> > > Later,
>>> > >
>>> > > Lukas "lzap" Zapletal
>>> > > irc: lzap #theforeman
>>> > >
>>> >
>>> > --
>>> > You received this message because you are subscribed to the Google
>>> Groups
>>> > "Foreman users" group.
>>> > To unsubscribe from this group and stop receiving emails from it, send
>>> an
>>> > email to foreman-user...@googlegroups.com.
>>> > To post to this group, send email to forema...@googlegroups.com.
>>> > Visit this group at http://groups.google.com/group/foreman-users.
>>> > For more options, visit https://groups.google.com/d/optout.
>>> >
>>>
>>
