Lots of "Mysql2::Error: Deadlock found when trying to get lock" under increased load

Let us know next week if this helped. I highly suggest upgrade to
1.15, it is a very solid release.

LZ

··· On Wed, Oct 4, 2017 at 11:04 PM, 'Konstantin Orekhov' via Foreman users wrote: > >> See the comment there, do you have this in your instance? If not git >> blame the commit and apply it. You have some older version I assume. > > > Yes, I'm running several 1.14.1 and 1.14.3 instances/clusters. Both had the > same issue with deadlocks. I've updated 2 of them with above patch and was > lucky enough to immediately observe a registration of at least 62 systems > went through w/o a single error. > I'll monitor things more, but so far this is huge steps forward. > > Thanks! > > -- > You received this message because you are subscribed to the Google Groups > "Foreman users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to foreman-users+unsubscribe@googlegroups.com. > To post to this group, send email to foreman-users@googlegroups.com. > Visit this group at https://groups.google.com/group/foreman-users. > For more options, visit https://groups.google.com/d/optout.


Later,
Lukas @lzap Zapletal

> Let us know next week if this helped. I highly suggest upgrade to
> 1.15, it is a very solid release.
>
>
Is this patch for MySQL a part of 1.15? As you suggested, I've taken it
from develop branch, so assumed it is not released yet.

Plus, there are other 2 things that worry me:

Do you think all of the above make it to 1.15.5?

> Let us know next week if this helped. I highly suggest upgrade to
> 1.15, it is a very solid release.
>

Are there any performance improvements for Smart-Proxy in 1.15, BTW?

Lately, in one of my busiest locations, we've started seeing a strange
issue when SmP stops responding on 8443 for API calls. The process itself
is running, logs messages are logged, just no response from it:

[root@spc01 ~]# systemctl start foreman-proxy

[root@spc01 ~]# date; curl --connect-timeout 30 -kSs
https://localhost:8443/bmc; date
Thu Oct 5 17:53:36 MST 2017
curl: (7) Failed connect to localhost:8443; Connection refused
Thu Oct 5 17:53:36 MST 2017

It does take ~30 seconds to start up in our env because of large DHCP
dataset, during which the connection would be refused.

[root@spc01 ~]# date; curl --connect-timeout 30 -kSs
https://localhost:8443/bmc; date
Thu Oct 5 17:53:49 MST 2017
curl: (28) NSS: client certificate not found (nickname not specified)
Thu Oct 5 17:54:19 MST 2017

Then it starts working for a very short period of time (above) and then
stops (below).

[root@spc01 ~]# date; curl --connect-timeout 30 -kSs
https://localhost:8443/bmc; date
Thu Oct 5 17:54:24 MST 2017
curl: (28) Operation timed out after 30001 milliseconds with 0 out of 0
bytes received
Thu Oct 5 17:54:54 MST 2017

So far there's nothing in proxy.log that helps me identify the issue. I
can't replicate it at will no matter what I do - had a bunch clients
hitting different APIs for a couple of days, nothing.
Then today the above happens and the only thing that helped me is to move
SmP from one node to another (I really wish DHCP SmP would allow for
active/active horizontal scaling instead of just being limited to a single
node).
Strace is useless as it only give this on tracing "ruby foreman-proxy"
process:

[root@spc03 ~]# strace -p 12526
strace: Process 12526 attached
futex(0x184e634, FUTEX_WAIT_PRIVATE, 1, NULL^Cstrace: Process 12526 detached
<detached …>

I tried https://github.com/tmm1/rbtrace, but it is so heavy that it
actually pretty much kills SmP by itself.

Do you have any suggestions on ways to troubleshoot this? I have DEBUG
enabled with these values:

:log_buffer: 4000
:log_buffer_errors: 2000

Also, is a way to move SmP from WebBrick to Apache/Passenger if that makes
sense at all? If so, any docs? Any other ways to increase the performance
as it does feel like a performance issue to me.

Thanks!

I've not seen that, no - I've CC'd someone who might know :wink:

Greg

··· On Thu, 2017-10-05 at 18:35 -0700, 'Konstantin Orekhov' via Foreman users wrote: > > Let us know next week if this helped. I highly suggest upgrade to > > 1.15, it is a very solid release. > > > Are there any performance improvements for Smart-Proxy in 1.15, BTW? > > Lately, in one of my busiest locations, we've started seeing a > strange issue when SmP stops responding on 8443 for API calls. The > process itself is running, logs messages are logged, just no response > from it: > > [root@spc01 ~]# systemctl start foreman-proxy > > [root@spc01 ~]# date; curl --connect-timeout 30 -kSs https://localhos > t:8443/bmc; date > Thu Oct 5 17:53:36 MST 2017 > curl: (7) Failed connect to localhost:8443; Connection refused > Thu Oct 5 17:53:36 MST 2017 > > It does take ~30 seconds to start up in our env because of large DHCP > dataset, during which the connection would be refused. > > > [root@spc01 ~]# date; curl --connect-timeout 30 -kSs https://localhos > t:8443/bmc; date > Thu Oct 5 17:53:49 MST 2017 > curl: (28) NSS: client certificate not found (nickname not specified) > Thu Oct 5 17:54:19 MST 2017 > > Then it starts working for a very short period of time (above) and > then stops (below). > > [root@spc01 ~]# date; curl --connect-timeout 30 -kSs https://localhos > t:8443/bmc; date > Thu Oct 5 17:54:24 MST 2017 > curl: (28) Operation timed out after 30001 milliseconds with 0 out of > 0 bytes received > Thu Oct 5 17:54:54 MST 2017 > > So far there's nothing in proxy.log that helps me identify the issue. > I can't replicate it at will no matter what I do - had a bunch > clients hitting different APIs for a couple of days, nothing. > Then today the above happens and the only thing that helped me is to > move SmP from one node to another (I really wish DHCP SmP would allow > for active/active horizontal scaling instead of just being limited to > a single node). > Strace is useless as it only give this on tracing "ruby foreman- > proxy" process: > > [root@spc03 ~]# strace -p 12526 > strace: Process 12526 attached > futex(0x184e634, FUTEX_WAIT_PRIVATE, 1, NULL^Cstrace: Process 12526 > detached > > > I tried https://github.com/tmm1/rbtrace, but it is so heavy that it > actually pretty much kills SmP by itself. > > Do you have any suggestions on ways to troubleshoot this? I have > DEBUG enabled with these values: > > :log_buffer: 4000 > :log_buffer_errors: 2000 > > Also, is a way to move SmP from WebBrick to Apache/Passenger if that > makes sense at all? If so, any docs? Any other ways to increase the > performance as it does feel like a performance issue to me. > > Thanks! > -- > You received this message because you are subscribed to the Google > Groups "Foreman users" group. > To unsubscribe from this group and stop receiving emails from it, > send an email to foreman-users+unsubscribe@googlegroups.com. > To post to this group, send email to foreman-users@googlegroups.com. > Visit this group at https://groups.google.com/group/foreman-users. > For more options, visit https://groups.google.com/d/optout.


IRC / Twitter: @gwmngilfen
Diaspora: gwmngilfen@joindiaspora.com

Hello,

please use git to find out which branches it landed in, I believe the
MySQL facter patch is 1.15+ only.

For 1.15.5 you need to talk with release engineer of this version
which is Daniel, if the changes are small enough I see no reason not
to include them. I think it's too late for 1.15.5 tho, maybe .6.

For smart proxy, there was a patch done by Dmitri who redesigned DHCP
parser, it's much more capable and faster now. I think this landed in
1.16 RC1, yeah: Refactor #19441: Rewrite isc dhcpd config parser to support various nested blocks - Smart Proxy - Foreman
(https://github.com/theforeman/smart-proxy/commit/21813c6cde0d2be10747682f1a001a7c0bd3ffb9)

I did not hear about unresponsive smart-proxy processes, can you check
system limits (open handles etc)? SELinux? Firewall. Any proxy plugins
enable? Then file a redmine bug, haven't seen that.

··· On Fri, Oct 6, 2017 at 3:35 AM, 'Konstantin Orekhov' via Foreman users wrote: > >> Let us know next week if this helped. I highly suggest upgrade to >> 1.15, it is a very solid release. > > > Are there any performance improvements for Smart-Proxy in 1.15, BTW? > > Lately, in one of my busiest locations, we've started seeing a strange issue > when SmP stops responding on 8443 for API calls. The process itself is > running, logs messages are logged, just no response from it: > > [root@spc01 ~]# systemctl start foreman-proxy > > [root@spc01 ~]# date; curl --connect-timeout 30 -kSs > https://localhost:8443/bmc; date > Thu Oct 5 17:53:36 MST 2017 > curl: (7) Failed connect to localhost:8443; Connection refused > Thu Oct 5 17:53:36 MST 2017 > > It does take ~30 seconds to start up in our env because of large DHCP > dataset, during which the connection would be refused. > > > [root@spc01 ~]# date; curl --connect-timeout 30 -kSs > https://localhost:8443/bmc; date > Thu Oct 5 17:53:49 MST 2017 > curl: (28) NSS: client certificate not found (nickname not specified) > Thu Oct 5 17:54:19 MST 2017 > > Then it starts working for a very short period of time (above) and then > stops (below). > > [root@spc01 ~]# date; curl --connect-timeout 30 -kSs > https://localhost:8443/bmc; date > Thu Oct 5 17:54:24 MST 2017 > curl: (28) Operation timed out after 30001 milliseconds with 0 out of 0 > bytes received > Thu Oct 5 17:54:54 MST 2017 > > So far there's nothing in proxy.log that helps me identify the issue. I > can't replicate it at will no matter what I do - had a bunch clients hitting > different APIs for a couple of days, nothing. > Then today the above happens and the only thing that helped me is to move > SmP from one node to another (I really wish DHCP SmP would allow for > active/active horizontal scaling instead of just being limited to a single > node). > Strace is useless as it only give this on tracing "ruby foreman-proxy" > process: > > [root@spc03 ~]# strace -p 12526 > strace: Process 12526 attached > futex(0x184e634, FUTEX_WAIT_PRIVATE, 1, NULL^Cstrace: Process 12526 detached > > > I tried https://github.com/tmm1/rbtrace, but it is so heavy that it actually > pretty much kills SmP by itself. > > Do you have any suggestions on ways to troubleshoot this? I have DEBUG > enabled with these values: > > :log_buffer: 4000 > :log_buffer_errors: 2000 > > Also, is a way to move SmP from WebBrick to Apache/Passenger if that makes > sense at all? If so, any docs? Any other ways to increase the performance as > it does feel like a performance issue to me. > > Thanks! > > -- > You received this message because you are subscribed to the Google Groups > "Foreman users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to foreman-users+unsubscribe@googlegroups.com. > To post to this group, send email to foreman-users@googlegroups.com. > Visit this group at https://groups.google.com/group/foreman-users. > For more options, visit https://groups.google.com/d/optout.


Later,
Lukas @lzap Zapletal

> please use git to find out which branches it landed in, I believe the
> MySQL facter patch is 1.15+ only.
>

Yes, I already found that and am planning on an upgrade in our lab instance.
BTW, even after applying a patch (on 1.14), which helped tremendously, from
time to time I still get some duplicate entries caused by an already
discovered system trying to send its discovery payload. For whatever reason
though, Foreman discovery plugin does not recognize it as the same host and
creates a new entry in a DB with different ID:

<https://lh3.googleusercontent.com/-ouz9YJ25K5M/WdvXTxpj4lI/AAAAAAAAAEE/1LcC1bcM2PUzzuH3gQtTQuT0Bx7mFHKTQCLcBGAs/s1600/Screen%2BShot%2B2017-10-09%2Bat%2B12.59.22%2BPM.png>

The host in question keeps of retrying, of course, and gets 422 "Name
already taken" over and over again. My question though is why a duplicate
was created instead of updating an existing host? It seems to me that this
host was not recognized properly. A complete debug log of a operation, that
I believe resulted in above duplicate entry, is here -


I did not see any 422 error before this transaction so I think this is it.
Although I did not see any long MySQL queries, the whole transaction still
took ~11 seconds to complete for some reason:

2017-10-07 01:26:16 e3da2f90 [app] [I] Completed 422 Unprocessable Entity
in 11291ms (Views: 0.4ms | ActiveRecord: 172.9ms)

> For smart proxy, there was a patch done by Dmitri who redesigned DHCP
> parser, it's much more capable and faster now. I think this landed in
> 1.16 RC1, yeah: Refactor #19441: Rewrite isc dhcpd config parser to support various nested blocks - Smart Proxy - Foreman
> <http://www.google.com/url?q=http%3A%2F%2Fprojects.theforeman.org%2Fissues%2F19441&sa=D&sntz=1&usg=AFQjCNGvX5FtIFwzkZEj8ilJdCjDl8c4oQ>
> (
> https://github.com/theforeman/smart-proxy/commit/21813c6cde0d2be10747682f1a001a7c0bd3ffb9)
>
>

welcomed change :slight_smile:

> I did not hear about unresponsive smart-proxy processes, can you check
> system limits (open handles etc)? SELinux? Firewall. Any proxy plugins
> enable? Then file a redmine bug, haven't seen that.
>

That's the problem - no smoking gun that I could find. No system resource
shortages logged, system itself is a rather beefy VM that does not even
sweat, no firewalls, selinux set to permissive mode. I only run 3 SmP -
bmc, dhcp and tftp.
On top of that, since I can't replicate this at will, I have to wait until
this issue manifests itself naturally.

And just to make it clear - it is not that SmP process becomes completely
unresponsive, but only an API-facing part. That's why I'm wondering if
moving away from Webrick to Apache or Nginx with Passenger is a possibility.
Another question along the same lines - is it possible to run each of the
smart-proxies as a separate processes (listening on different ports)
instead of one with several proxies and single port. For example, in this
particular situation even if one SmP was having an issue, it would not
affect the other 2, also it would also pinpoint the troubled proxy
simplifying troubleshooting efforts.

Thanks!
Konstantin.

··· From my side, any performance improvements for DHCP SmP is always a

Hey

I did not see any 422 error before this transaction so I think this is it.
> Although I did not see any long MySQL queries, the whole transaction still
> took ~11 seconds to complete for some reason:
>

Name has already been taken - this usually means that host (either
discovered or managed or unmanaged) of that name "macXXXXXXXXX" already
exist. Same mac address? You can change easily how discovered hosts are
being named, by default it is "mac" + MAC address, you can change that to
random number or any different fact you want. See settings and our
documentation. Try to add a random number at the end if that helps.

>
>
>> For smart proxy, there was a patch done by Dmitri who redesigned DHCP
>> parser, it's much more capable and faster now. I think this landed in
>> 1.16 RC1, yeah: Refactor #19441: Rewrite isc dhcpd config parser to support various nested blocks - Smart Proxy - Foreman
>> <http://www.google.com/url?q=http%3A%2F%2Fprojects.theforeman.org%2Fissues%2F19441&sa=D&sntz=1&usg=AFQjCNGvX5FtIFwzkZEj8ilJdCjDl8c4oQ>
>> (https://github.com/theforeman/smart-proxy/commit/21813c6cde
>> 0d2be10747682f1a001a7c0bd3ffb9)
>>
>
> From my side, any performance improvements for DHCP SmP is always a
> welcomed change :slight_smile:
>
>
>> I did not hear about unresponsive smart-proxy processes, can you check
>> system limits (open handles etc)? SELinux? Firewall. Any proxy plugins
>> enable? Then file a redmine bug, haven't seen that.
>>
>
> That's the problem - no smoking gun that I could find. No system resource
> shortages logged, system itself is a rather beefy VM that does not even
> sweat, no firewalls, selinux set to permissive mode. I only run 3 SmP -
> bmc, dhcp and tftp.
> On top of that, since I can't replicate this at will, I have to wait until
> this issue manifests itself naturally.
>
> And just to make it clear - it is not that SmP process becomes completely
> unresponsive, but only an API-facing part. That's why I'm wondering if
> moving away from Webrick to Apache or Nginx with Passenger is a possibility.
>

Proxy is a regular Sinatra app, so any Rack servere should do the trick
(Puma perhaps). I'd try that to see if it helps. Might be bug in Webrick,
try to downgrade or upgrade it.

Another question along the same lines - is it possible to run each of the
> smart-proxies as a separate processes (listening on different ports)
> instead of one with several proxies and single port. For example, in this
> particular situation even if one SmP was having an issue, it would not
> affect the other 2, also it would also pinpoint the troubled proxy
> simplifying troubleshooting efforts.
>

We don't support that, unfortunately.

>
> Name has already been taken - this usually means that host (either
> discovered or managed or unmanaged) of that name "macXXXXXXXXX" already
> exist. Same mac address? You can change easily how discovered hosts are
> being named, by default it is "mac" + MAC address, you can change that to
> random number or any different fact you want. See settings and our
> documentation. Try to add a random number at the end if that helps.
>

Well, no, that's the same host. It was discovered at some point back (8
days ago as you can see in the screenshot above). Then it got
rebooted/crashed, PXE-booted again and started sending its discovery
payload again, but gets 422. Usually, if it is the same host, Foreman just
updates existing entry with new facts, or at least, the time of last
report. But in the case above, that did not happen as for some reason
Foreman created another record in a DB (with a different ID) for the same
host. Unless I remove both of the records, 422s will continue to happen. I
was hoping that a gist I provided gives you enough info on what could have
caused that double-record situation for the same host.

Here are my DB entries for the above host:

[root@spc03 ~]# mysql -u foreman -p$DB_PASS foreman -e "SELECT * from hosts
where type = 'Host::Discovered' and NAME = 'mac90e2baea5d58'\G;"
*************************** 1. row ***************************
id: 446735
name: mac90e2baea5d58
last_compile: NULL
last_report: 2017-10-02 05:49:35
updated_at: 2017-10-02 05:49:46
created_at: 2017-09-15 22:44:42
root_pass: NULL
architecture_id: NULL
operatingsystem_id: NULL
environment_id: NULL
ptable_id: NULL
medium_id: NULL
build: 0
comment: NULL
disk: NULL
installed_at: NULL
model_id: 6
hostgroup_id: NULL
owner_id: 10
owner_type: User
enabled: 1
puppet_ca_proxy_id: NULL
managed: 0
use_image: NULL
image_file: NULL
uuid: NULL
compute_resource_id: NULL
puppet_proxy_id: NULL
certname: NULL
image_id: NULL
organization_id: NULL
location_id: NULL
type: Host::Discovered
otp: NULL
realm_id: NULL
compute_profile_id: NULL
provision_method: NULL
grub_pass:
global_status: 0
lookup_value_matcher: NULL
discovery_rule_id: NULL
salt_proxy_id: NULL
salt_environment_id: NULL
pxe_loader: NULL
*************************** 2. row ***************************
id: 456978
name: mac90e2baea5d58
last_compile: NULL
last_report: 2017-10-10 16:04:20
updated_at: 2017-10-10 16:04:20
created_at: 2017-10-07 07:13:19
root_pass: NULL
architecture_id: NULL
operatingsystem_id: NULL
environment_id: NULL
ptable_id: NULL
medium_id: NULL
build: 0
comment: NULL
disk: NULL
installed_at: NULL
model_id: NULL
hostgroup_id: NULL
owner_id: NULL
owner_type: NULL
enabled: 1
puppet_ca_proxy_id: NULL
managed: 0
use_image: NULL
image_file: NULL
uuid: NULL
compute_resource_id: NULL
puppet_proxy_id: NULL
certname: NULL
image_id: NULL
organization_id: NULL
location_id: NULL
type: Host::Discovered
otp: NULL
realm_id: NULL
compute_profile_id: NULL
provision_method: NULL
grub_pass:
global_status: 0
lookup_value_matcher: NULL
discovery_rule_id: NULL
salt_proxy_id: NULL
salt_environment_id: NULL
pxe_loader: NULL

Oh now I understand, looks like a new bug. I am not able to reproduce
with FDI 3.4.1 and develop Foreman. The code responsible for finding
existing record is here:

Theoretically if you changed configuration of bootif fact in settings,
this could happen. But I assume you haven't. Was it also the same FDI
version, just in case facter changed facts?

If you visit the original discovery host in the UI, do you see
Interfaces list? What MAC address is there? Is it detected as the
primary interface there?

Do you still have both records in the DB? Can you run rake console and
try these statements? Or insert Rails.logger.info statements there to
see the code flow.

··· On Tue, Oct 10, 2017 at 8:01 PM, 'Konstantin Orekhov' via Foreman users wrote: > >> >> Name has already been taken - this usually means that host (either >> discovered or managed or unmanaged) of that name "macXXXXXXXXX" already >> exist. Same mac address? You can change easily how discovered hosts are >> being named, by default it is "mac" + MAC address, you can change that to >> random number or any different fact you want. See settings and our >> documentation. Try to add a random number at the end if that helps. > > > Well, no, that's the same host. It was discovered at some point back (8 days > ago as you can see in the screenshot above). Then it got rebooted/crashed, > PXE-booted again and started sending its discovery payload again, but gets > 422. Usually, if it is the same host, Foreman just updates existing entry > with new facts, or at least, the time of last report. But in the case above, > that did not happen as for some reason Foreman created another record in a > DB (with a different ID) for the same host. Unless I remove both of the > records, 422s will continue to happen. I was hoping that a gist I provided > gives you enough info on what could have caused that double-record situation > for the same host. > > Here are my DB entries for the above host: > > [root@spc03 ~]# mysql -u foreman -p$DB_PASS foreman -e "SELECT * from hosts > where type = 'Host::Discovered' and NAME = 'mac90e2baea5d58'\G;" > *************************** 1. row *************************** > id: 446735 > name: mac90e2baea5d58 > last_compile: NULL > last_report: 2017-10-02 05:49:35 > updated_at: 2017-10-02 05:49:46 > created_at: 2017-09-15 22:44:42 > root_pass: NULL > architecture_id: NULL > operatingsystem_id: NULL > environment_id: NULL > ptable_id: NULL > medium_id: NULL > build: 0 > comment: NULL > disk: NULL > installed_at: NULL > model_id: 6 > hostgroup_id: NULL > owner_id: 10 > owner_type: User > enabled: 1 > puppet_ca_proxy_id: NULL > managed: 0 > use_image: NULL > image_file: NULL > uuid: NULL > compute_resource_id: NULL > puppet_proxy_id: NULL > certname: NULL > image_id: NULL > organization_id: NULL > location_id: NULL > type: Host::Discovered > otp: NULL > realm_id: NULL > compute_profile_id: NULL > provision_method: NULL > grub_pass: > global_status: 0 > lookup_value_matcher: NULL > discovery_rule_id: NULL > salt_proxy_id: NULL > salt_environment_id: NULL > pxe_loader: NULL > *************************** 2. row *************************** > id: 456978 > name: mac90e2baea5d58 > last_compile: NULL > last_report: 2017-10-10 16:04:20 > updated_at: 2017-10-10 16:04:20 > created_at: 2017-10-07 07:13:19 > root_pass: NULL > architecture_id: NULL > operatingsystem_id: NULL > environment_id: NULL > ptable_id: NULL > medium_id: NULL > build: 0 > comment: NULL > disk: NULL > installed_at: NULL > model_id: NULL > hostgroup_id: NULL > owner_id: NULL > owner_type: NULL > enabled: 1 > puppet_ca_proxy_id: NULL > managed: 0 > use_image: NULL > image_file: NULL > uuid: NULL > compute_resource_id: NULL > puppet_proxy_id: NULL > certname: NULL > image_id: NULL > organization_id: NULL > location_id: NULL > type: Host::Discovered > otp: NULL > realm_id: NULL > compute_profile_id: NULL > provision_method: NULL > grub_pass: > global_status: 0 > lookup_value_matcher: NULL > discovery_rule_id: NULL > salt_proxy_id: NULL > salt_environment_id: NULL > pxe_loader: NULL > > -- > You received this message because you are subscribed to the Google Groups > "Foreman users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to foreman-users+unsubscribe@googlegroups.com. > To post to this group, send email to foreman-users@googlegroups.com. > Visit this group at https://groups.google.com/group/foreman-users. > For more options, visit https://groups.google.com/d/optout.


Later,
Lukas @lzap Zapletal

>
> Theoretically if you changed configuration of bootif fact in settings,
> this could happen. But I assume you haven't. Was it also the same FDI
> version, just in case facter changed facts?
>

Yes, the same system with the same discovery proxy. I'm not using official
FDI, but a netbooted Ubuntu14.04 (Casper-based) with old foreman-proxy
version 1.9.2, smart_proxy_discovery_image-1.0.5.gem and discover-host from
around the same time. I had trouble bringing in later versions of
foreman-proxy to u14.04 because of dependency on ruby 2.x. If the fact that
I run such old versions on a client side is an issue in your opinion, I can
start the work on migrating to u16.04. But so far I have not seen an
indication that this is an issue - duplicate entries happen not too often
and most of the time it is working fine (and I have several thousands of
hosts going through this), especially now with that MySQL patch you gave me
earlier.

> If you visit the original discovery host in the UI, do you see
> Interfaces list? What MAC address is there? Is it detected as the
> primary interface there?
>

Yes, all of the data is in place for both duplicate entries and look
absolutely identical to me:

<https://lh3.googleusercontent.com/-Qjn-PntubGE/Wd5qYnQIQGI/AAAAAAAAAEU/nkbpCKnoFdwqGGRF-tWkEXyNK_SFRkq_gCLcBGAs/s1600/Screen%2BShot%2B2017-10-11%2Bat%2B11.57.26%2BAM.png>

<https://lh3.googleusercontent.com/-UJCurA1GVa4/Wd5qdFYOUaI/AAAAAAAAAEY/1f-YHzKHI1oNjbEj6_DZ8MAokTDO2zZqwCLcBGAs/s1600/Screen%2BShot%2B2017-10-11%2Bat%2B11.56.28%2BAM.png>

> Do you still have both records in the DB? Can you run rake console and
> try these statements? Or insert Rails.logger.info statements there to
> see the code flow.
>

I do still have those in a DB, yes. But I don't have that host running a
discover proxy anymore :frowning:
But if I find more duplicate entries, I don't quite understand what you
want me to do - try which statements? Or insert logger statements where
exactly?

Ideally I would like to see DEBUG and SQL logs for that transactions, but
thats flooding your production server.

In the console with both records presnet do something like:

::Nic::Managed.where(:mac => "MA:CA:DDRESS::", :primary => true)

··· On Wed, Oct 11, 2017 at 9:08 PM, 'Konstantin Orekhov' via Foreman users < foreman-users@googlegroups.com> wrote:

Theoretically if you changed configuration of bootif fact in settings,
this could happen. But I assume you haven’t. Was it also the same FDI
version, just in case facter changed facts?

Yes, the same system with the same discovery proxy. I’m not using official
FDI, but a netbooted Ubuntu14.04 (Casper-based) with old foreman-proxy
version 1.9.2, smart_proxy_discovery_image-1.0.5.gem and discover-host
from around the same time. I had trouble bringing in later versions of
foreman-proxy to u14.04 because of dependency on ruby 2.x. If the fact that
I run such old versions on a client side is an issue in your opinion, I can
start the work on migrating to u16.04. But so far I have not seen an
indication that this is an issue - duplicate entries happen not too often
and most of the time it is working fine (and I have several thousands of
hosts going through this), especially now with that MySQL patch you gave me
earlier.

If you visit the original discovery host in the UI, do you see
Interfaces list? What MAC address is there? Is it detected as the
primary interface there?

Yes, all of the data is in place for both duplicate entries and look
absolutely identical to me:

https://lh3.googleusercontent.com/-Qjn-PntubGE/Wd5qYnQIQGI/AAAAAAAAAEU/nkbpCKnoFdwqGGRF-tWkEXyNK_SFRkq_gCLcBGAs/s1600/Screen%2BShot%2B2017-10-11%2Bat%2B11.57.26%2BAM.png

https://lh3.googleusercontent.com/-UJCurA1GVa4/Wd5qdFYOUaI/AAAAAAAAAEY/1f-YHzKHI1oNjbEj6_DZ8MAokTDO2zZqwCLcBGAs/s1600/Screen%2BShot%2B2017-10-11%2Bat%2B11.56.28%2BAM.png

Do you still have both records in the DB? Can you run rake console and
try these statements? Or insert Rails.logger.info statements there to
see the code flow.

I do still have those in a DB, yes. But I don’t have that host running a
discover proxy anymore :frowning:
But if I find more duplicate entries, I don’t quite understand what you
want me to do - try which statements? Or insert logger statements where
exactly?


You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.


Later,
Lukas @lzap Zapletal

> In the console with both records presnet do something like:
>
> ::Nic::Managed.where(:mac => "MA:CA:DDRESS::", :primary => true)
>
>
OK, thanks, Lukas! As soon as I get that duplicate entries show up again,
I'll run above and provide a result here. After patching for MySQL query
issue, I don't see this happening very often (which is a good thing).

> OK, thanks, Lukas! As soon as I get that duplicate entries show up again,
> I'll run above and provide a result here. After patching for MySQL query
> issue, I don't see this happening very often (which is a good thing).
>

Hmm, I've got 2 duplicates, but I can't seem to run what you asked for:

[root@spc02 foreman]# pwd
/usr/share/foreman

[root@spc02 foreman]# rake --trace console
rake aborted!
cannot load such file – apipie/middleware/checksum_in_headers
/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in require&#39; /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:inrequire'
/usr/share/foreman/config/application.rb:5:in &lt;top (required)&gt;&#39; /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:inrequire'
/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in require&#39; /usr/share/foreman/Rakefile:1:in<top (required)>'
/usr/share/gems/gems/rake-0.9.6/lib/rake/rake_module.rb:25:in load&#39; /usr/share/gems/gems/rake-0.9.6/lib/rake/rake_module.rb:25:inload_rakefile'
/usr/share/gems/gems/rake-0.9.6/lib/rake/application.rb:604:in
raw_load_rakefile&#39; /usr/share/gems/gems/rake-0.9.6/lib/rake/application.rb:89:inblock in
load_rakefile'
/usr/share/gems/gems/rake-0.9.6/lib/rake/application.rb:160:in
standard_exception_handling&#39; /usr/share/gems/gems/rake-0.9.6/lib/rake/application.rb:88:inload_rakefile'
/usr/share/gems/gems/rake-0.9.6/lib/rake/application.rb:72:in block in run&#39; /usr/share/gems/gems/rake-0.9.6/lib/rake/application.rb:160:instandard_exception_handling'
/usr/share/gems/gems/rake-0.9.6/lib/rake/application.rb:70:in run&#39; /usr/bin/rake:37:in<main>'
[root@spc02 foreman]#

What am I missing here?

Hmm is this Debian? Should work.

LLZ

··· On Tue, Oct 24, 2017 at 2:50 AM, 'Konstantin Orekhov' via Foreman users wrote: > >> OK, thanks, Lukas! As soon as I get that duplicate entries show up again, >> I'll run above and provide a result here. After patching for MySQL query >> issue, I don't see this happening very often (which is a good thing). > > > Hmm, I've got 2 duplicates, but I can't seem to run what you asked for: > > [root@spc02 foreman]# pwd > /usr/share/foreman > > [root@spc02 foreman]# rake --trace console > rake aborted! > cannot load such file -- apipie/middleware/checksum_in_headers > /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require' > /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require' > /usr/share/foreman/config/application.rb:5:in `' > /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require' > /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require' > /usr/share/foreman/Rakefile:1:in `' > /usr/share/gems/gems/rake-0.9.6/lib/rake/rake_module.rb:25:in `load' > /usr/share/gems/gems/rake-0.9.6/lib/rake/rake_module.rb:25:in > `load_rakefile' > /usr/share/gems/gems/rake-0.9.6/lib/rake/application.rb:604:in > `raw_load_rakefile' > /usr/share/gems/gems/rake-0.9.6/lib/rake/application.rb:89:in `block in > load_rakefile' > /usr/share/gems/gems/rake-0.9.6/lib/rake/application.rb:160:in > `standard_exception_handling' > /usr/share/gems/gems/rake-0.9.6/lib/rake/application.rb:88:in > `load_rakefile' > /usr/share/gems/gems/rake-0.9.6/lib/rake/application.rb:72:in `block in run' > /usr/share/gems/gems/rake-0.9.6/lib/rake/application.rb:160:in > `standard_exception_handling' > /usr/share/gems/gems/rake-0.9.6/lib/rake/application.rb:70:in `run' > /usr/bin/rake:37:in `' > [root@spc02 foreman]# > > What am I missing here? > > -- > You received this message because you are subscribed to the Google Groups > "Foreman users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to foreman-users+unsubscribe@googlegroups.com. > To post to this group, send email to foreman-users@googlegroups.com. > Visit this group at https://groups.google.com/group/foreman-users. > For more options, visit https://groups.google.com/d/optout.


Later,
Lukas @lzap Zapletal

Please use foreman-rake (I assume this is a packaged .deb install).

··· On Mon, Oct 23, 2017 at 05:50:52PM -0700, 'Konstantin Orekhov' via Foreman users wrote: > [root@spc02 foreman]# rake --trace console


Michael Moll

>
> Please use foreman-rake (I assume this is a packaged .deb install).
>
>
This is CentOS7 install and foreman-rake did work. Here's the result:

[root@spc01 ~]# cd ~foreman
[root@spc01 foreman]# foreman-rake console
Successfully encrypted field for Setting::Auth oauth_consumer_key
Successfully decrypted field for Setting::Auth oauth_consumer_key
Successfully decrypted field for Setting::Auth oauth_consumer_key
Successfully decrypted field for Setting::Auth oauth_consumer_key
Successfully decrypted field for Setting::Auth oauth_consumer_key
Successfully encrypted field for Setting::Auth oauth_consumer_secret
Successfully decrypted field for Setting::Auth oauth_consumer_secret
Successfully decrypted field for Setting::Auth oauth_consumer_secret
Successfully decrypted field for Setting::Auth oauth_consumer_secret
Successfully decrypted field for Setting::Auth oauth_consumer_secret
/usr/share/foreman/lib/tasks/console.rake:6: warning: already initialized
constant ARGV
For some operations a user must be set, try User.current = User.first
Loading production environment (Rails 4.2.5.1)
Failed to load console gems, starting anyway
irb(main):001:0> ::Nic::Managed.where(:mac => "b4:99:ba:aa:4b:64", :primary
=> true)
=> #<ActiveRecord::Relation [#<Nic::Managed id: 619163, mac:
"b4:99:ba:aa:4b:64", ip: "10.8.161.191", type: "Nic::Managed", name:
"macb499baaa4b64", host_id: 458555, subnet_id: nil, domain_id: nil, attrs:
{"netmask"=>"255.255.255.0", "mtu"=>"1500", "network"=>"10.8.161.0",
"speed"=>"1000", "duplex"=>"full", "port"=>"Twisted Pair",
"auto_negotiation"=>"true", "wol"=>true}, created_at: "2017-10-20
03:44:00", updated_at: "2017-10-20 03:44:02", provider: nil, username: nil,
password: nil, virtual: false, link: true, identifier: "eth0", tag: "",
attached_to: "", managed: true, mode: "balance-rr", attached_devices: "",
bond_options: "", primary: true, provision: true, compute_attributes: {},
execution: true, ip6: nil, subnet6_id: nil>]>
irb(main):002:0>

However, just as in my previous example, DB has 2 different IDs with that
MAC:

[root@spc01 ~]# mysql -u foreman -p$DB_PASS foreman -e "SELECT * FROM hosts
WHERE type = 'Host::Discovered' and NAME = 'macb499baaa4b64'\G;"
*************************** 1. row ***************************
id: 430926
name: macb499baaa4b64
last_compile: NULL
last_report: 2017-09-30 06:56:07
updated_at: 2017-09-30 06:56:09
created_at: 2017-03-17 14:09:15
root_pass: NULL
architecture_id: NULL
operatingsystem_id: NULL
environment_id: NULL
ptable_id: NULL
medium_id: NULL
build: 0
comment: NULL
disk: NULL
installed_at: NULL
model_id: 7
hostgroup_id: NULL
owner_id: 10
owner_type: User
enabled: 1
puppet_ca_proxy_id: NULL
managed: 0
use_image: NULL
image_file: NULL
uuid: NULL
compute_resource_id: NULL
puppet_proxy_id: NULL
certname: NULL
image_id: NULL
organization_id: NULL
location_id: NULL
type: Host::Discovered
otp: NULL
realm_id: NULL
compute_profile_id: NULL
provision_method: NULL
grub_pass:
global_status: 0
lookup_value_matcher: NULL
discovery_rule_id: NULL
salt_proxy_id: NULL
salt_environment_id: NULL
pxe_loader: NULL
*************************** 2. row ***************************
id: 458555
name: macb499baaa4b64
last_compile: NULL
last_report: 2017-10-25 16:47:08
updated_at: 2017-10-25 16:47:09
created_at: 2017-10-20 03:44:00
root_pass: NULL
architecture_id: NULL
operatingsystem_id: NULL
environment_id: NULL
ptable_id: NULL
medium_id: NULL
build: 0
comment: NULL
disk: NULL
installed_at: NULL
model_id: NULL
hostgroup_id: NULL
owner_id: NULL
owner_type: NULL
enabled: 1
puppet_ca_proxy_id: NULL
managed: 0
use_image: NULL
image_file: NULL
uuid: NULL
compute_resource_id: NULL
puppet_proxy_id: NULL
certname: NULL
image_id: NULL
organization_id: NULL
location_id: NULL
type: Host::Discovered
otp: NULL
realm_id: NULL
compute_profile_id: NULL
provision_method: NULL
grub_pass:
global_status: 0
lookup_value_matcher: NULL
discovery_rule_id: NULL
salt_proxy_id: NULL
salt_environment_id: NULL
pxe_loader: NULL
[root@spc01 ~]#

<https://lh3.googleusercontent.com/-wBk_VPoo0Fs/WfDA4guCc6I/AAAAAAAAAEs/ZIrmiIJEh64IGPxH0dz0y3aqO90F7WDdgCLcBGAs/s1600/Screen%2BShot%2B2017-10-25%2Bat%2B9.49.21%2BAM.png>

Ok this confirms it. Bug #21479: Multiple discovered hosts with same MAC address - Discovery - Foreman we will
fix later.

We don't have an unique index on DB level, just in Rails level and a second
NIC with same MAC can sneak in. The relevant code in core is:

validate :mac_uniqueness, :if =&gt; Proc.new { |nic| nic.managed? &amp;&amp;

nic.host && nic.host.managed? && !nic.host.compute? && !nic.virtual? &&
nic.mac.present? }

which will not trigger for Discovery at all (host is not managed). In
discovery we try to search for existing host and if not found, we will
create new discovered host. This does not work correctly, we have turned
off validator for some reason:

host.save(:validate =&gt; false) if host.new_record?

So the validation for uniqueness won't hit.

··· On Wed, Oct 25, 2017 at 6:50 PM, 'Konstantin Orekhov' via Foreman users < foreman-users@googlegroups.com> wrote:

Please use foreman-rake (I assume this is a packaged .deb install).

This is CentOS7 install and foreman-rake did work. Here’s the result:

[root@spc01 ~]# cd ~foreman
[root@spc01 foreman]# foreman-rake console
Successfully encrypted field for Setting::Auth oauth_consumer_key
Successfully decrypted field for Setting::Auth oauth_consumer_key
Successfully decrypted field for Setting::Auth oauth_consumer_key
Successfully decrypted field for Setting::Auth oauth_consumer_key
Successfully decrypted field for Setting::Auth oauth_consumer_key
Successfully encrypted field for Setting::Auth oauth_consumer_secret
Successfully decrypted field for Setting::Auth oauth_consumer_secret
Successfully decrypted field for Setting::Auth oauth_consumer_secret
Successfully decrypted field for Setting::Auth oauth_consumer_secret
Successfully decrypted field for Setting::Auth oauth_consumer_secret
/usr/share/foreman/lib/tasks/console.rake:6: warning: already initialized
constant ARGV
For some operations a user must be set, try User.current = User.first
Loading production environment (Rails 4.2.5.1)
Failed to load console gems, starting anyway
irb(main):001:0> ::Nic::Managed.where(:mac => “b4:99:ba:aa:4b:64”,
:primary => true)
=> #<ActiveRecord::Relation [#<Nic::Managed id: 619163, mac:
“b4:99:ba:aa:4b:64”, ip: “10.8.161.191”, type: “Nic::Managed”, name:
“macb499baaa4b64”, host_id: 458555, subnet_id: nil, domain_id: nil, attrs:
{“netmask”=>“255.255.255.0”, “mtu”=>“1500”, “network”=>“10.8.161.0”,
“speed”=>“1000”, “duplex”=>“full”, “port”=>“Twisted Pair”,
“auto_negotiation”=>“true”, “wol”=>true}, created_at: “2017-10-20
03:44:00”, updated_at: “2017-10-20 03:44:02”, provider: nil, username: nil,
password: nil, virtual: false, link: true, identifier: “eth0”, tag: “”,
attached_to: “”, managed: true, mode: “balance-rr”, attached_devices: “”,
bond_options: “”, primary: true, provision: true, compute_attributes: {},
execution: true, ip6: nil, subnet6_id: nil>]>
irb(main):002:0>

However, just as in my previous example, DB has 2 different IDs with that
MAC:

[root@spc01 ~]# mysql -u foreman -p$DB_PASS foreman -e “SELECT * FROM
hosts WHERE type = ‘Host::Discovered’ and NAME = ‘macb499baaa4b64’\G;”
*************************** 1. row ***************************
id: 430926
name: macb499baaa4b64
last_compile: NULL
last_report: 2017-09-30 06:56:07
updated_at: 2017-09-30 06:56:09
created_at: 2017-03-17 14:09:15
root_pass: NULL
architecture_id: NULL
operatingsystem_id: NULL
environment_id: NULL
ptable_id: NULL
medium_id: NULL
build: 0
comment: NULL
disk: NULL
installed_at: NULL
model_id: 7

    hostgroup_id: NULL
        owner_id: 10
      owner_type: User
         enabled: 1

puppet_ca_proxy_id: NULL
managed: 0
use_image: NULL
image_file: NULL
uuid: NULL
compute_resource_id: NULL
puppet_proxy_id: NULL
certname: NULL
image_id: NULL
organization_id: NULL
location_id: NULL
type: Host::Discovered
otp: NULL
realm_id: NULL
compute_profile_id: NULL
provision_method: NULL
grub_pass:
global_status: 0
lookup_value_matcher: NULL
discovery_rule_id: NULL
salt_proxy_id: NULL
salt_environment_id: NULL
pxe_loader: NULL
*************************** 2. row ***************************
id: 458555
name: macb499baaa4b64
last_compile: NULL
last_report: 2017-10-25 16:47:08
updated_at: 2017-10-25 16:47:09
created_at: 2017-10-20 03:44:00

       root_pass: NULL
 architecture_id: NULL

operatingsystem_id: NULL
environment_id: NULL
ptable_id: NULL
medium_id: NULL
build: 0
comment: NULL
disk: NULL
installed_at: NULL
model_id: NULL
hostgroup_id: NULL
owner_id: NULL
owner_type: NULL
enabled: 1
puppet_ca_proxy_id: NULL
managed: 0
use_image: NULL
image_file: NULL
uuid: NULL
compute_resource_id: NULL
puppet_proxy_id: NULL
certname: NULL
image_id: NULL
organization_id: NULL
location_id: NULL
type: Host::Discovered
otp: NULL
realm_id: NULL
compute_profile_id: NULL
provision_method: NULL
grub_pass:
global_status: 0
lookup_value_matcher: NULL
discovery_rule_id: NULL
salt_proxy_id: NULL
salt_environment_id: NULL
pxe_loader: NULL
[root@spc01 ~]#

https://lh3.googleusercontent.com/-wBk_VPoo0Fs/WfDA4guCc6I/AAAAAAAAAEs/ZIrmiIJEh64IGPxH0dz0y3aqO90F7WDdgCLcBGAs/s1600/Screen%2BShot%2B2017-10-25%2Bat%2B9.49.21%2BAM.png


You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.


Later,
Lukas @lzap Zapletal

Ok. Is there anything I could do now to workaround this? The only thing worked for me so far was to periodically go through discovered hosts and remove duplicate entries.

Thanks!
Konstantin.

··· > On Oct 26, 2017, at 07:18, Lukas Zapletal wrote: > > Ok this confirms it. http://projects.theforeman.org/issues/21479 we will fix later. > > We don't have an unique index on DB level, just in Rails level and a second NIC with same MAC can sneak in. The relevant code in core is: > > validate :mac_uniqueness, :if => Proc.new { |nic| nic.managed? && nic.host && nic.host.managed? && !nic.host.compute? && !nic.virtual? && nic.mac.present? } > > which will not trigger for Discovery at all (host is not managed). In discovery we try to search for existing host and if not found, we will create new discovered host. This does not work correctly, we have turned off validator for some reason: > > host.save(:validate => false) if host.new_record? > > So the validation for uniqueness won't hit. > > >> On Wed, Oct 25, 2017 at 6:50 PM, 'Konstantin Orekhov' via Foreman users wrote: >> >>> >>> Please use foreman-rake (I assume this is a packaged .deb install). >>> >> >> This is CentOS7 install and foreman-rake did work. Here's the result: >> >> [root@spc01 ~]# cd ~foreman >> [root@spc01 foreman]# foreman-rake console >> Successfully encrypted field for Setting::Auth oauth_consumer_key >> Successfully decrypted field for Setting::Auth oauth_consumer_key >> Successfully decrypted field for Setting::Auth oauth_consumer_key >> Successfully decrypted field for Setting::Auth oauth_consumer_key >> Successfully decrypted field for Setting::Auth oauth_consumer_key >> Successfully encrypted field for Setting::Auth oauth_consumer_secret >> Successfully decrypted field for Setting::Auth oauth_consumer_secret >> Successfully decrypted field for Setting::Auth oauth_consumer_secret >> Successfully decrypted field for Setting::Auth oauth_consumer_secret >> Successfully decrypted field for Setting::Auth oauth_consumer_secret >> /usr/share/foreman/lib/tasks/console.rake:6: warning: already initialized constant ARGV >> For some operations a user must be set, try User.current = User.first >> Loading production environment (Rails 4.2.5.1) >> Failed to load console gems, starting anyway >> irb(main):001:0> ::Nic::Managed.where(:mac => "b4:99:ba:aa:4b:64", :primary => true) >> => #<ActiveRecord::Relation [#"255..255.255.0", "mtu"=>"1500", "network"=>"10.8.161.0", "speed"=>"1000", "duplex"=>"full", "port"=>"Twisted Pair", "auto_negotiation"=>"true", "wol"=>true}, created_at: "2017-10-20 03:44:00", updated_at: "2017-10-20 03:44:02", provider: nil, username: nil, password: nil, virtual: false, link: true, identifier: "eth0", tag: "", attached_to: "", managed: true, mode: "balance-rr", attached_devices: "", bond_options: "", primary: true, provision: true, compute_attributes: {}, execution: true, ip6: nil, subnet6_id: nil>]> >> irb(main):002:0> >> >> However, just as in my previous example, DB has 2 different IDs with that MAC: >> >> [root@spc01 ~]# mysql -u foreman -p$DB_PASS foreman -e "SELECT * FROM hosts WHERE type = 'Host::Discovered' and NAME = 'macb499baaa4b64'\G;" >> *************************** 1. row *************************** >> id: 430926 >> name: macb499baaa4b64 >> last_compile: NULL >> last_report: 2017-09-30 06:56:07 >> updated_at: 2017-09-30 06:56:09 >> created_at: 2017-03-17 14:09:15 >> root_pass: NULL >> architecture_id: NULL >> operatingsystem_id: NULL >> environment_id: NULL >> ptable_id: NULL >> medium_id: NULL >> build: 0 >> comment: NULL >> disk: NULL >> installed_at: NULL >> model_id: 7 >> >> hostgroup_id: NULL >> owner_id: 10 >> owner_type: User >> enabled: 1 >> puppet_ca_proxy_id: NULL >> managed: 0 >> use_image: NULL >> image_file: NULL >> uuid: NULL >> compute_resource_id: NULL >> puppet_proxy_id: NULL >> certname: NULL >> image_id: NULL >> organization_id: NULL >> location_id: NULL >> type: Host::Discovered >> otp: NULL >> realm_id: NULL >> compute_profile_id: NULL >> provision_method: NULL >> grub_pass: >> global_status: 0 >> lookup_value_matcher: NULL >> discovery_rule_id: NULL >> salt_proxy_id: NULL >> salt_environment_id: NULL >> pxe_loader: NULL >> *************************** 2. row *************************** >> id: 458555 >> name: macb499baaa4b64 >> last_compile: NULL >> last_report: 2017-10-25 16:47:08 >> updated_at: 2017-10-25 16:47:09 >> created_at: 2017-10-20 03:44:00 >> >> root_pass: NULL >> architecture_id: NULL >> operatingsystem_id: NULL >> environment_id: NULL >> ptable_id: NULL >> medium_id: NULL >> build: 0 >> comment: NULL >> disk: NULL >> installed_at: NULL >> model_id: NULL >> hostgroup_id: NULL >> owner_id: NULL >> owner_type: NULL >> enabled: 1 >> puppet_ca_proxy_id: NULL >> managed: 0 >> use_image: NULL >> image_file: NULL >> uuid: NULL >> compute_resource_id: NULL >> puppet_proxy_id: NULL >> certname: NULL >> image_id: NULL >> organization_id: NULL >> location_id: NULL >> type: Host::Discovered >> otp: NULL >> realm_id: NULL >> compute_profile_id: NULL >> provision_method: NULL >> grub_pass: >> global_status: 0 >> lookup_value_matcher: NULL >> discovery_rule_id: NULL >> salt_proxy_id: NULL >> salt_environment_id: NULL >> pxe_loader: NULL >> [root@spc01 ~]# >> >> >> >> >> -- >> You received this message because you are subscribed to the Google Groups "Foreman users" group. >> To unsubscribe from this group and stop receiving emails from it, send an email to foreman-users+unsubscribe@googlegroups.com. >> To post to this group, send email to foreman-users@googlegroups.com. >> Visit this group at https://groups.google.com/group/foreman-users. >> For more options, visit https://groups.google.com/d/optout. > > > > -- > Later, > Lukas @lzap Zapletal > -- > You received this message because you are subscribed to a topic in the Google Groups "Foreman users" group. > To unsubscribe from this topic, visit https://groups.google.com/d/topic/foreman-users/xGZvRqHbWcA/unsubscribe. > To unsubscribe from this group and all its topics, send an email to foreman-users+unsubscribe@googlegroups.com. > To post to this group, send email to foreman-users@googlegroups.com. > Visit this group at https://groups.google.com/group/foreman-users. > For more options, visit https://groups.google.com/d/optout.

Try to change to :validate => true but beware there might be dragons. I do
not remember why we set this.

LZ

··· On Thu, Oct 26, 2017 at 6:20 PM, 'Konstantin Orekhov' via Foreman users < foreman-users@googlegroups.com> wrote:

Ok. Is there anything I could do now to workaround this? The only thing
worked for me so far was to periodically go through discovered hosts and
remove duplicate entries.

Thanks!
Konstantin.

On Oct 26, 2017, at 07:18, Lukas Zapletal lzap@redhat.com wrote:

Ok this confirms it. Bug #21479: Multiple discovered hosts with same MAC address - Discovery - Foreman we will
fix later.

We don’t have an unique index on DB level, just in Rails level and a
second NIC with same MAC can sneak in. The relevant code in core is:

validate :mac_uniqueness, :if => Proc.new { |nic| nic.managed? &&

nic.host && nic.host.managed? && !nic.host.compute? && !nic.virtual? &&
nic.mac.present? }

which will not trigger for Discovery at all (host is not managed). In
discovery we try to search for existing host and if not found, we will
create new discovered host. This does not work correctly, we have turned
off validator for some reason:

host.save(:validate => false) if host.new_record?

So the validation for uniqueness won’t hit.

On Wed, Oct 25, 2017 at 6:50 PM, ‘Konstantin Orekhov’ via Foreman users < > foreman-users@googlegroups.com> wrote:

Please use foreman-rake (I assume this is a packaged .deb install).

This is CentOS7 install and foreman-rake did work. Here’s the result:

[root@spc01 ~]# cd ~foreman
[root@spc01 foreman]# foreman-rake console
Successfully encrypted field for Setting::Auth oauth_consumer_key
Successfully decrypted field for Setting::Auth oauth_consumer_key
Successfully decrypted field for Setting::Auth oauth_consumer_key
Successfully decrypted field for Setting::Auth oauth_consumer_key
Successfully decrypted field for Setting::Auth oauth_consumer_key
Successfully encrypted field for Setting::Auth oauth_consumer_secret
Successfully decrypted field for Setting::Auth oauth_consumer_secret
Successfully decrypted field for Setting::Auth oauth_consumer_secret
Successfully decrypted field for Setting::Auth oauth_consumer_secret
Successfully decrypted field for Setting::Auth oauth_consumer_secret
/usr/share/foreman/lib/tasks/console.rake:6: warning: already
initialized constant ARGV
For some operations a user must be set, try User…current = User.first

Loading production environment (Rails 4.2.5.1)
Failed to load console gems, starting anyway
irb(main):001:0> ::Nic::Managed.where(:mac => “b4:99:ba:aa:4b:64”,
:primary => true)
=> #<ActiveRecord::Relation [#<Nic::Managed id: 619163, mac:
“b4:99:ba:aa:4b:64”, ip: “10.8.161.191”, type: “Nic::Managed”, name:
“macb499baaa4b64”, host_id: 458555, subnet_id: nil, domain_id: nil, attrs:
{“netmask”=>“255.255.255.0”, “mtu”=>“1500”, “network”=>“10.8.161.0”,
“speed”=>“1000”, “duplex”=>“full”, “port”=>“Twisted Pair”,
“auto_negotiation”=>“true”, “wol”=>true}, created_at: “2017-10-20
03:44:00”, updated_at: “2017-10-20 03:44:02”, provider: nil, username: nil,
password: nil, virtual: false, link: true, identifier: “eth0”, tag: “”,
attached_to: “”, managed: true, mode: “balance-rr”, attached_devices: “”,
bond_options: “”, primary: true, provision: true, compute_attributes: {},
execution: true, ip6: nil, subnet6_id: nil>]>
irb(main):002:0>

However, just as in my previous example, DB has 2 different IDs with that
MAC:

[root@spc01 ~]# mysql -u foreman -p$DB_PASS foreman -e “SELECT * FROM
hosts WHERE type = ‘Host::Discovered’ and NAME = ‘macb499baaa4b64’\G;”
*************************** 1. row ***************************
id: 430926
name: macb499baaa4b64
last_compile: NULL
last_report: 2017-09-30 06:56:07
updated_at: 2017-09-30 06:56:09
created_at: 2017-03-17 14:09:15
root_pass: NULL
architecture_id: NULL
operatingsystem_id: NULL
environment_id: NULL
ptable_id: NULL
medium_id: NULL
build: 0
comment: NULL
disk: NULL
installed_at: NULL
model_id: 7

    hostgroup_id: NULL
        owner_id: 10
      owner_type: User
         enabled: 1

puppet_ca_proxy_id: NULL
managed: 0
use_image: NULL
image_file: NULL
uuid: NULL
compute_resource_id: NULL
puppet_proxy_id: NULL
certname: NULL
image_id: NULL
organization_id: NULL
location_id: NULL
type: Host::Discovered
otp: NULL
realm_id: NULL
compute_profile_id: NULL
provision_method: NULL
grub_pass:
global_status: 0
lookup_value_matcher: NULL
discovery_rule_id: NULL
salt_proxy_id: NULL
salt_environment_id: NULL
pxe_loader: NULL
*************************** 2. row ***************************
id: 458555
name: macb499baaa4b64
last_compile: NULL
last_report: 2017-10-25 16:47:08
updated_at: 2017-10-25 16:47:09
created_at: 2017-10-20 03:44:00

       root_pass: NULL
 architecture_id: NULL

operatingsystem_id: NULL
environment_id: NULL
ptable_id: NULL
medium_id: NULL
build: 0
comment: NULL
disk: NULL
installed_at: NULL
model_id: NULL
hostgroup_id: NULL
owner_id: NULL
owner_type: NULL
enabled: 1
puppet_ca_proxy_id: NULL
managed: 0
use_image: NULL
image_file: NULL
uuid: NULL
compute_resource_id: NULL
puppet_proxy_id: NULL
certname: NULL
image_id: NULL
organization_id: NULL
location_id: NULL
type: Host::Discovered
otp: NULL
realm_id: NULL
compute_profile_id: NULL
provision_method: NULL
grub_pass:
global_status: 0
lookup_value_matcher: NULL
discovery_rule_id: NULL
salt_proxy_id: NULL
salt_environment_id: NULL
pxe_loader: NULL
[root@spc01 ~]#

https://lh3.googleusercontent.com/-wBk_VPoo0Fs/WfDA4guCc6I/AAAAAAAAAEs/ZIrmiIJEh64IGPxH0dz0y3aqO90F7WDdgCLcBGAs/s1600/Screen%2BShot%2B2017-10-25%2Bat%2B9.49.21%2BAM.png


You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.


Later,
Lukas @lzap Zapletal


You received this message because you are subscribed to a topic in the
Google Groups “Foreman users” group.
To unsubscribe from this topic, visit https://groups.google.com/d/
topic/foreman-users/xGZvRqHbWcA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.


Later,
Lukas @lzap Zapletal