[Katello 2.4] Upgrade to katello 2.4 broke realm-proxy for FreeIPA

Support
1

I had a CentOS 7 server running Katello 2.3 and also acting as a realm
proxy for the FreeIPA realm it was joined to.

I did some upgrades this weekend (CentOS 7.0 -> 7.2 and Katello 2.3 -> 2.4,
as well as FreeIPA 4.1.4 -> 4.2.0) and this morning when I tried to
provision a host, the realm proxy is not working properly.

When you configure all your options and hit "Submit" it goes through the
usual steps listing that it is uploading TFTP boot images, contacting
realm, provisioning VMWare guest etc and all of those appear to work.

The machine gets provisioned, but when it is done the kickstart process it
has not been joined to FreeIPA. Looking at the error log entries on the
host, the error given is that the one-time-join password was invalid.

I looked in the anaconda-ks.cfg file and confirmed on 2 test machines I
provisioned that they both had different passwords for joining FreeIPA.

However, when I logged into the FreeIPA web ui and did a search for those
hosts, they did not appear.

So it appears that Katello is telling us it properly contacted the realm
proxy and properly setup a host entry, when it did not actually do that.

Previous to Katello 2.4, the realm proxy step of provisioning would fail if
there were any errors. For example, if I tried to provision a host with
the same name as one that already existed, the realm proxy step would fail
and rollback all the other steps (TFTP etc).

Now with Katello 2.4 it looks like it is failing for some reason, but not
reporting that to the system, so the installation proceeds and just doesn't
work right rather than failing outright at the start and letting us know
there was an error during the realm step.

I ran tcpdump on the Katello server and I can confirm that it appears that
Katello server successfully gets a kerberos ticket from a FreeIPA DC during
this process, then there is a whole bunch of TLSv1.2 and LDAP traffic, but
I can't see what's actually going on as that stuff is encrypted so it's
very hard for me to see where the process is failing. IE, I have no idea
how to figure out whether the IPA server ever sends back some type of "host
created successfully" reply.

I have ran journalctl -f on the FreeIPA server to try to track down what
happens but nothing relevant seems to be getting logged during that process.

2

> From: "Nathan Peters" <nathanpeters008@gmail.com>
> To: "Foreman users" <foreman-users@googlegroups.com>
> Sent: Monday, January 11, 2016 5:10:52 PM
> Subject: [foreman-users] [Katello 2.4] Upgrade to katello 2.4 broke realm-proxy for FreeIPA
>
> I had a CentOS 7 server running Katello 2.3 and also acting as a realm
> proxy for the FreeIPA realm it was joined to.
>
> I did some upgrades this weekend (CentOS 7.0 -> 7.2 and Katello 2.3 -> 2.4,
> as well as FreeIPA 4.1.4 -> 4.2.0) and this morning when I tried to
> provision a host, the realm proxy is not working properly.
>
> When you configure all your options and hit "Submit" it goes through the
> usual steps listing that it is uploading TFTP boot images, contacting
> realm, provisioning VMWare guest etc and all of those appear to work.
>
> The machine gets provisioned, but when it is done the kickstart process it
> has not been joined to FreeIPA. Looking at the error log entries on the
> host, the error given is that the one-time-join password was invalid.
>
> I looked in the anaconda-ks.cfg file and confirmed on 2 test machines I
> provisioned that they both had different passwords for joining FreeIPA.
>
> However, when I logged into the FreeIPA web ui and did a search for those
> hosts, they did not appear.

If you're getting an OTP in the anaconda-ks.cfg, it talked to FreeIPA successfully,
as it's FreeIPA that does the password generating.

> So it appears that Katello is telling us it properly contacted the realm
> proxy and properly setup a host entry, when it did not actually do that.
>
>
>
> Previous to Katello 2.4, the realm proxy step of provisioning would fail if
> there were any errors. For example, if I tried to provision a host with
> the same name as one that already existed, the realm proxy step would fail
> and rollback all the other steps (TFTP etc).
>
> Now with Katello 2.4 it looks like it is failing for some reason, but not
> reporting that to the system, so the installation proceeds and just doesn't
> work right rather than failing outright at the start and letting us know
> there was an error during the realm step.

The provisioning process wouldn't fail because as far as Foreman knows
it succeeded. It got an OTP.

>
> I ran tcpdump on the Katello server and I can confirm that it appears that
> Katello server successfully gets a kerberos ticket from a FreeIPA DC during
> this process, then there is a whole bunch of TLSv1.2 and LDAP traffic, but
> I can't see what's actually going on as that stuff is encrypted so it's
> very hard for me to see where the process is failing. IE, I have no idea
> how to figure out whether the IPA server ever sends back some type of "host
> created successfully" reply.
>
> I have ran journalctl -f on the FreeIPA server to try to track down what
> happens but nothing relevant seems to be getting logged during that process.

I doubt you'll see anything on the foreman side, because it's making API calls
to FreeIPA and clearly getting something back, but do enable debug logging and look
at /var/log/foreman-proxy/proxy.log.

Also try enabling debug logging on FreeIPA, which is actually quite good. You'd
see everything that's happening in the traffic in those logs.

··· ----- Original Message -----

–
You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

3

It turns out this was a case of the Foreman properly creating the entry,
and the IPA server failing to replicate that change to other servers.

Hence why the logs appear to show the host successfully provisioned, but
when it boots, it cannot register because the key wasn't replicated to the
server the host contacted.