Hook appears to fire even when Foreman record creation fails

Support
1

We have some automation built around our Foreman server which adds records
to Foreman once a virtual machine is deployed in our cloud. I noticed a
strange event today, where we had a build whose record failed to add to
Foreman, yet the create hook still fired and added the host to DNS.

2015-03-16 00:03:06 – Foreman fails to create record: 500 Error
2015-03-16 00:03:10 – Create hook fires

The actual error I'm seeing is related to MySQL. It doesn't appear to be
associated with load on the server, or a full filesystem.

Here is the sanitized output:

@@@@@ response @@@@@@@@@@@@@@@@
headers = {'status': '500 Internal Server Error', 'x-request-id':
'fea6d242d6a4b64a704d5ec2eb395dff', 'x-powered-by': 'Phusion Passenger
4.0.18', 'transfer-encoding': 'chunked', 'foreman_version': '1.7.1',
'connection': 'close', 'server': 'Apache/2.2.15 (Red Hat)', 'x-runtime':
'39.919467', 'x-ua-compatible': 'IE=Edge,chrome=1', 'cache-control':
'no-cache', 'date': 'Mon, 16 Mar 2015 07:03:06 GMT', 'apipie-checksum':
'7fa19449fc6e30fe48aee072de960dc3', 'content-type': 'application/json;
charset=utf-8', 'x-rack-cache': 'invalidate, pass', 'foreman_api_version':
'2'}
code = 500
reason = Internal Server Error
--------- data received ------------
{
"error": {"message":"Mysql2::Error: Got error -1 from storage engine:
INSERT INTO hosts (architecture_id, build, certname, comment,
compute_profile_id, compute_resource_id, created_at, disk,
domain_id, enabled, environment_id, grub_pass, hostgroup_id,
image_file, image_id, installed_at, ip, last_compile,
last_freshcheck, last_report, location_id, mac, managed,
medium_id, model_id, name, operatingsystem_id, organization_id,
otp, owner_id, owner_type, primary_interface, provision_method,
ptable_id, puppet_ca_proxy_id, puppet_proxy_id, puppet_status,
realm_id, root_pass, serial, source_file_id, subnet_id, type,
updated_at, use_image, uuid) VALUES (1, 0, NULL, '99', NULL, NULL,
'2015-03-16 07:03:13', NULL, 1, 1, 1, '$1$hash', 1, NULL, NULL, NULL,
'10.0.0.10', NULL, NULL, NULL, NULL, '08:ab:cd:00:00:b1', 1, 3, NULL,
'hostname.example.com', 1, NULL, NULL, 1, 'User', NULL, NULL, 1, 2, 3, 0,
NULL, '$1$hash', NULL, NULL, 1, 'Host::Managed', '2015-03-16 07:03:13',
NULL, NULL)"}
}

If Foreman fails to add the record to the database, then why does it fire
off the hook? Does anyone have an explanation as to why the MySQL error
would cause this to happen, or is it a bug with Foreman?

2

Create hooks are part of Foreman's orchestration that runs a series of
steps to set up the host, e.g. creating a VM, creating DHCP/DNS records
and so on. Eventually they culminate in the host being stored in the
database, so I guess what happened is that step failed and Foreman
eventually rolled back the whole transaction, and undid the steps.

If Foreman did a rollback, then foreman_hooks should call the create
hook again with a "destroy" argument.

No idea why MySQL would fail to add the record though, unless it's
trying to cryptically indicate there's some sort of unique index issue.

··· On 18/03/15 22:12, Kyle Flavin wrote: > > We have some automation built around our Foreman server which adds > records to Foreman once a virtual machine is deployed in our cloud. I > noticed a strange event today, where we had a build whose record failed > to add to Foreman, yet the create hook still fired and added the host to > DNS.

–
Dominic Cleal
Red Hat Engineering