I am currently running Katello 3.7 on VMware. I would like to take a clone of the system (while shutdown) and from that create a new VM for testing. I do plan on giving the new VM a new IP, reset the MAC address and give it a new hostname. I am guessing I will need to re-create the certs in Foreman and Katello. What is the best way to do this for everything?
Are there any other things I need to do so the new VM can function correctly?
There is a tool called katello-change-hostname that should take care of all the certs for you, iirc @John_Mitsch should have further info regarding its usage.
Correct, you can use katello-change-hostname to update the hostname of your system and it will take care of updating the certs. Check katello-change-hostname --help for usage and let me know if you run into any issues. Hope this helps!
I finally got around to doing this. Had some issues when I ran the katello-change-hostname. The script ran for a while then complained that the reverse DNS was not pointed at the new hostname. So I changed the reverse dns to make it happy and ran the command it told me to run after I fixed it, (Sorry do not remember what that was.)
When it was all done I changed its IPs to the new IPs. Cleaned up DNS rebooted the new DEV host. Made sure it came up with new IPs. Then started the old PROD host. Made sure it came up with its old IPs and worked.
Now I am finally looking at the DEV host and when I started up katello-services I go the error:
E, [2019-01-22T10:46:45.730519 ] ERROR ā : Unable to load private SSL key. Are the values correct in settings.yml and do permissions allow reading?: No such file or directory - /etc/foreman-proxy/ssl_key.pem
E, [2019-01-22T10:46:45.730701 ] ERROR ā : Error during startup, terminating. No such file or directory - /etc/foreman-proxy/ssl_key.pem
So I started looking around and did the following grep: grep -ari /etc/
The result shows me every config, ssl file ā¦ all with the old hostname. The new hostname is no where to be found in /etc/ at all.
What is the deal.
Do you remember which step katello-change-hostname originally failed? It sounds like parts of the script didnāt run properly.
If you have a snapshot or some way of reverting to before the hostname change ran, it would be best to revert to that, make the DNS changes, and then re-run the script.
Otherwise, I would try to completely run the script again. You may need to change the system hostname back to the original one to ātrickā the script, I canāt recall which version of katello we added a check that the new hostname is different than the old one.
Itās worth noting that you donāt change the system hostname before running the katello-change-hostname. There is a check for this in newer katello versions. So if your hostname is foo.example.com and you want bar.example.com, you just run katello-change-hostname for bar.example.com without changing the system hostname with hostnamectl or similar. The script needs the system to be on the old hostname to find itās occurrences. Iām not sure if that was the case here, but just trying to rule things out.
Hopefully this helps
I would very much like to avoid changing the hostname / IP back to the PROD hostname / IP if possible. Here is the error I got:
[root@fm ~]# katello-change-hostname -u admin -p m4w4yRj247KEcAJK gd4dc1fmdev01.DOMAIN.NET
Checking hostname validity
Checking overall health of server
Checking credentials
WARNING This script will modify your system. You will need to re-register any foreman clients registered to this system after script completion. Foreman Proxies will have to be re-registered and reinstalled. If you are using custom certificates, you will have to run the foreman-installer again with custom certificate options after this script completes. Have you taken the necessary precautions (backups, snapshots, etcā¦)?
Proceed with changing your hostname? [y/n]
y
Updating default Foreman Proxy
Updating installation media paths
updating hostname in /etc/hostname
setting hostname
checking if hostname was changed
stopping services
removing old cert rpms
Unable to upload Package Profile
deleting old certs
backed up /var/www/html/pub to /var/www/html/pub/fm.DOMAIN.COM-20190115093612.backup
updating hostname in /etc/hosts
updating hostname in foreman installer scenarios
backing up last_scenario.yaml
removing last_scenario.yaml
re-running the installer
foreman-installer --scenario katello -v --disable-system-checks --certs-regenerate=true --foreman-proxy-register-in-foreman true
Reverse DNS failed. Looking up 10.25.4.41 gave fm.DOMAIN.NET, expected to match gd4dc1fmdev01.tdsops.net
restoring last_scenario.yaml
cleaning up temporary files
Your system does not meet configuration criteria
[ INFO 2019-01-15T09:38:02 verbose] Executing hooks in group pre_migrations
[ INFO 2019-01-15T09:38:02 verbose] All hooks in group pre_migrations finished
[ INFO 2019-01-15T09:38:02 verbose] Executing hooks in group boot
[ INFO 2019-01-15T09:38:02 verbose] All hooks in group boot finished
[ INFO 2019-01-15T09:38:02 verbose] Executing hooks in group init
[ INFO 2019-01-15T09:38:02 verbose] All hooks in group init finished
[ INFO 2019-01-15T09:38:02 verbose] Loading default values from puppet modulesā¦
[ INFO 2019-01-15T09:38:02 verbose] ā¦ finished
[ INFO 2019-01-15T09:38:02 verbose] Executing hooks in group pre_values
[ INFO 2019-01-15T09:38:02 verbose] All hooks in group pre_values finished
[ INFO 2019-01-15T09:38:02 verbose] Running installer with args [["āscenario", ākatelloā, ā-vā, āādisable-system-checksā, āācerts-regenerate=trueā, āāforeman-proxy-register-in-foremanā, ātrueā]]
[ INFO 2019-01-15T09:38:02 verbose] Installer finished in 2.897987076 seconds
Something went wrong with the Katello installer.
Please check the above output and the corresponding logs.
Once the issue is resolved you may complete the hostname change by running: āforeman-installer --scenario katello -v --disable-system-checksā
and completing the following steps:
You will have to install the new bootstrap rpm and reregister all clients and Foreman Proxies with subscription-manager
(update organization and environment arguments appropriately):
yum remove -y katello-ca-consumer*
rpm -Uvh http://gd4dc1fmdev01.DOMAIN.NET/pub/katello-ca-consumer-latest.noarch.rpm
subscription-manager register --org=āDefault_Organizationā --environment=āLibraryā --force
Then reattach subscriptions to the client(s) and run:
subscription-manager refresh
yum repolistOn all Foreman Proxies, you will need to re-run the foreman-installer with this command:
foreman-installer --foreman-proxy-content-parent-fqdn gd4dc1fmdev01.DOMAIN.NET
āforeman-proxy-foreman-base-url https://gd4dc1fmdev01.DOMAIN.NET
āforeman-proxy-trusted-hosts gd4dc1fmdev01.DOMAIN.NET
Short hostnames have not been updated, please update those manually.Failed āforeman-installer --scenario katello -v --disable-system-checks --certs-regenerate=true --foreman-proxy-register-in-foreman trueā with exit code 20
So when this happened I changed the reverse DNS to the DEV one. Then ran:
foreman-installer --scenario katello -v --disable-system-checks
I do not have have any other proxies except the one that is part of the Foreman / Katello base system. So I did not run anything else.
In the current state, is the actual system hostname (from hostname -f) changed to the new hostname?
Yes. The hostname and IP has been changed.
The biggest problem with changing it back is finding time to bring down the PROD host for a while.
The best way would be to change the hostname back to the original on the system and then re-run katello-change-hostname.
The only other option I can think of is to modify the script to hardcode your old hostname in it. This can be done on this line, changing to @old_hostname = "foo.example.com", using the name of your old hostname.
The file is found in /usr/share/katello/hostname-change.rb. If you do go this route, make a backup of the file first.
I made a backup of the ruby file and will try changing the @old_hostname as you suggested.
Ran it and got a failure. Here is the output:
[root@gd4dc1fmdev01 ~]# katello-change-hostname -u admin -p PASS gd4dc1fmdev01.DOMAIN.net
Checking hostname validity
Checking overall health of server
Checking credentials
WARNING This script will modify your system. You will need to re-register any foreman clients registered to this system after script completion. Foreman Proxies will have to be re-registered and reinstalled. If you are using custom certificates, you will have to run the foreman-installer again with custom certificate options after this script completes. Have you taken the necessary precautions (backups, snapshots, etcā¦)?
Proceed with changing your hostname? [y/n]
yUpdating default Foreman Proxy
Updating installation media paths
updating hostname in /etc/hostname
setting hostname
checking if hostname was changed
stopping services
removing old cert rpms
No Match for argument: fm.DOMAIN.com-apache*
No Match for argument: fm.DOMAIN.com-foreman-client*
No Match for argument: fm.DOMAIN.com-foreman-proxy*
No Match for argument: fm.DOMAIN.com-foreman-proxy-client*
No Match for argument: fm.DOMAIN.com-puppet-client*
No Match for argument: fm.DOMAIN.com-qpid-broker*
No Match for argument: fm.DOMAIN.com-qpid-client-cert*
No Match for argument: fm.DOMAIN.com-qpid-router-client*
No Match for argument: fm.DOMAIN.com-qpid-router-server*
No Match for argument: fm.DOMAIN.com-tomcat*
deleting old certs
mv: cannot stat ā/var/www/html/pub/*.rpmā: No such file or directoryFailed āmv /var/www/html/pub/*.rpm /var/www/html/pub/fm.DOMAIN.com-20190123092950.backupā with exit code 1
root@gd4dc1fmdev01:~[root@gd4dc1fmdev01 ~]# exit
exit
Could I do the following:
Iām guessing you donāt want to run the katello-change-hostname script right after cloning because you will run into the same DNS issues you originally saw?
The script is really designed to run on the system while it has the old hostname
No. I have no problems running it on the system right after cloning. My problem is that cloning takes about 2 - 3 hours (due to the size of the drives). Then there is the unknown time it will take to run the katello-change-hostname. And if it fails there is more time there. All this time the PROD host has to be shutdown. While the PROD host is down its disrupting the creation of new instances and patching being done by other.
I can get down time but I still need to keep it to as short as possible. I can also take a snapshot of the PROD vm (its all vmware). Then convert the snapshot to a clone. And before I bring up the new clone remove the networking or change the networking to a different VLAN or boot to ISO so I can change the IP before I really boot it. This way I can do all this work with out bringing down the PROD host,
Whichever strategy you wind up using, I definitely recommend taking a snapshot before running katello-change-hostname, this will make it easy to revert and fix issues before any modifications happen on the system. The script makes pretty significant changes to a system so the best way to fix errors is by reverting to a clean snapshot, making changes, and re-running the script.
Very true. I rely heavily on snapshots. Out of all the reasons to virtualize I find this one to be the most important in my opinion.
I ran it my way and while it still had the same original issue I was able to play around with /etc/hosts and get the process restarted. About 15 minutes latter it looked like it changed everything.
Hammer ping comes back fine. I rebooted the host and it came back fine. Logged in to the GUI and that worked. The only issue I see is that when I go to Administrator -> About I see 2 proxies. One for the old host name, one for the new hostname. Both have a green status.
Can I just delete the proxy for the old hostname?
If there is one for the new hostname and the hostname has been changed, I think its fine to delete the old one
It would not let me delete it saying it was required by the system with the same name. So I left it.
When I upgraded to 3.8 I noticed the new proxy got updated but the old one did not. The same is true when I upgraded to 3.10.
Since the whole point of this exercise was to create an instance where I could test upgrades and the upgrades did work I am satisfied.
Thank you for all your help.