I noticed that over the last year a number of new users were asking questions as part of a transition from Spacewalk. If this was a path you took to Foreman, I would be very interested in your experience.
Were there:
Any particular challenges that you faced?
Any particular concept that you wished was explained better?
Any obstacles you hit?
Anything you learned that you wish you knew before?
Any Foreman-related resources that were of particular use to you?
Please let me know 
I am one of them
Answers are in the order of your questions.
Biggest challenge was to get accustomed to the product. While many things are easily understood, some concepts like content views and content libraries are something entirely new. There is no documentation available for former Spacewalkerâs trying to make the transition.
I would have loved to use a migration documentation, or a how-to, in order to make the transition. While a few docs are available by searching online, they all fall short of fully explaining a simple migration process. To be more exact, they basically stop at the moment you have your repos synced and all added to one single CV. Which you will pretty quickly find out is a stupid idea. But thatâs way too late.
Biggest obstacle, I would say is the product stability. I know this sounds a bit harsch, and I ensure you its not meant like this, but I had to reinstall Foreman/Katello 3 times before I got it to work. Please keep in mind, this is a migration, not someone installing a new linux management solution in-house. So I mirrored all my repos and hit some roadblocks there, like DB corruption. Then setting up remote smart proxies is a nightmare, as the syncing fails for some repos (being looked at right now). Tons of 500 errors when downloading packages, hanging tasks ⌠you get the idea. Eventually after the third setup I knew what to avoid (more or less) and could move forward.
Oh yes, I would have loved to know beforehand how to deal with CVâs, to NOT subscribe your smart proxy to the Library (why does it even allow you to do that?) and that there is practically no automation of any kind to keep up with updates. For the latter, without errataâs (whose import donât work for CentOS 7 at the moment) there is no email about new packages, no info about any kind of changes. You need to manually click through all your content views, aggregated views to update and publish them. Clean out the old versions. This is a tedious, cumbersome and very time-consuming task. I would say, folks coming from Spacewalk need to know, that you are constantly required to manually perform actions in Foreman/Katello, otherwise your linux systems will not be up to date. I still havenât gotten my head around that and its one of the biggest obstacles to overcome.
The most help I found was in these forums. Folks in here are friendly, helpful and mostly quite immediate in responding. The documentation of Foreman/Katello is well written but doesnât really explain hands-on procedures. Its more like an explanation of each setting with a few how-tos, but the overall concept is hard to understand from it.
Last but not least. The product is great, migrating is the way to go. Spacewalk is dead and there is no alternative to Foreman/Katello IMHO. I appreciate the help of everyone in here, form the development side and the community.
My 2 cents.
Thanks for getting back to us.
I am just curious, do you remember which version was that? We got a lot of feedback about stability and in the past few years weâve been very focused on stability improvements. More to come as well. If it was a recent version, please elaborate on problems not mentioned below so we can add these into our TODOs.
Was that mongo by the way? Thatâs has been addressed too and big upgrade (MongoDB->Postgres) is ahead of us!
Yes, we have documented that already. If you still missed this, probably itâs the time to add a warning in the UI itself:
Do not assign the Library lifecycle environment to your Smart Proxy server because it triggers an automated Smart Proxy sync every time the CDN updates a repository. This might consume multiple system resources on Smart Proxies, network bandwidth between Foreman and Smart Proxies, and available disk space on Smart Proxies.
Sure, initially Foreman 2.2 and Katello 3.17. (or 3.16, canât be 100%).
Exactly, there was a problem with thread allocation during repo sync if I recall correctly. The DB was left in a bad state and it was fixed in a little update a few days later. Canât wait for Postgres 
Thats where I read about it, correct. But it was after I got into trouble 
We deployed Spacewalk a few years ago, and it has worked well for us. In the past year, I installed Foreman 2.x several times before it worked in any appreciable manner.
Once the âbasicâ install was done, getting Katello content management working was rather simple (thanks to a GREAT writeup I found on the Interwebs after much searching).
However, other features are not as simple to get working (the âdocumentationâ for Foreman is sorely lacking). Remote command execution took some wrangling to get working, (and I donât trust it) Foreman lacks the concept of âconfiguration channelsâ that Spacewalk has, and thatâs a huge deal-breaker for us. While Iâm find cranking out Ansible playbooks or using other methods, our web master likes having a web interface to his config files, plus the integrated revision tracking, syntax highlighting, and so forth. Same with our DBA and his db config files. With RHEL planning to drop Puppet from Satellite, Iâm curious to see what happens next with Foreman. That leaves either Foreman to keep Puppet, or forge ahead with Ansible as the One True Configuration Management Solution within the suite. Thatâs not necessarily good or bad.
Provisioning integration with Xen and ESXi (we use both) is nice, except you have to remember to change a setting so when you remove a system from Foreman, you donât remove it from the hypervisor forever. That default is a big mistake and should be the opposite.
As an aside, we recently (30 days ago) pulled an eval license from Redhat for Satellite 6.8, installed on top of RHEL 7.9. For stream-family products (my understanding), RHELâs stuff âjust worksâ, and though it lacks more features (no Xen support to be found, no support for dpkg/deb that I can see), it does offer better documentation. And for $300/year (beer budget), RHEL-supported Satellite would be a better option for some folks, Iâd imagine. Finally, given what RHEL did, vis-a-vis C8->C8Stream, I declined to buy the license. We arenât going to toss a quasi-beta into production as the replacement for C7.
Iâm more curious now how Foreman will roll out alongside the successor(s) to CentOS. Thatâs really the hitch for us, in terms of making decisions. But, the lack of certainty, combined with Foreman being one prickly collection of software, has left us with using Spacewalk 2.10 for probably the next 24-36 months. I pondered Ubuntu as a baseline, but I got buckets of pushback on that, and frankly itâs not a hill I want to die for.
Honestly, while SW 2.x is NOT perfect, it sure seems far more usable and fits a niche better for a smaller enterprise/.edu environment where I donât have to Control The World, but just need a simple way to manage 100+ systems and give users disinclined to CLI work a way to manage configuration channels.
Thank you for starting this thread. Foreman is probably overall a good tool. But I think in many ways, SW was a better tool.
While the incremental update feature was added partially to help issues like this, I understand it doesnât fully solve the issue. Perhaps it could be useful to you if you havenât tried it yet.
Iâve never used Spacewalk before, and Foreman+Katello was my first introduction into system management tools. It sounds great that Spacewalk automatically updates packages. Katello sync plans can help you get closer to this, but it gets trickier for automatically publishing content views. You would need to set it up yourself with some combination of hammer and cron probably. An older thread exists where a user voiced very similar concerns Update management with Katello 3.5
If you donât need to filter any packages (definitely a big âifâ!), you could have your hosts assigned to the Library lifecycle environment and the default content view. If you had a sync plan set up for the hostsâ repositories, the available packages would always be up to date.
On the subject of automation, it would be worth checking out Foreman Ansible Modules if youâre interested in Ansible at all.
It would be interesting to have an automated way to keep content views up to date. Something like a content view publish plan could perhaps meet the requirement and keep the code simple. Or, perhaps a content view setting where CV publishes happen automatically if related repositories are re-synced. Just food for thought.
I will look into that. Thank you for the advice.
Maybe I was not clear enough here, sorry for the confusion Spacewalk automatically showed all applicable updates/errata in the host view, it did not automatically apply it. So in Foreman/Katello words, that would be in the content host view. Right now you will not see anything there unless you published a new CV version. Makes sense, since that is the way its supposed to work. However, you donât know if you have to publish a new CV version, as you do not get any info about updated repos from upstream, no emails or any other notification nor can you make a âdiffâ of the current CV to the current repo state.
Exactly what I would be looking for. As the library is pretty much not used for anything (no clients subscribe to it, not synced to smart proxies) it should automatically import all changes from the repos, maybe via a scheduled job. So you can at least see if in your 10+ repos something has changed, pretty much the same as you see on an aggregated CV when another child view has been published.
I think even going automatically from Library to Testing would be acceptable in many cases, how else you going to test otherwise? Again, that could be configurable.
Especially some form of reporting outside of errata updates would be important. And of course the ability to import CentOS 7 errata
I completely understand your pain. Was the first kinda huh moment when finishing the Katello installation for me too.
I have since converted most of the config adoption to Ansible, as it allows for a much more controlled process of rolling out changes. It can be done at the right moment in time, just before a certain service is started, it can be combined with installing the correct RPMs and enable the right services. In SW these tasks were all separated and hard to control.
However, not every aspect is covered in read-to-use Ansible roles. I found some good ones for basic functions, but in many cases I had to write my own Ansible roles. Since you can check in the templates plus the yaml files into any kind of version control system, it becomes manageable in a way.
Having said that, a unique selling point for Foreman/Katello would be an Ansible role which can perform the basic task of deploying a file, fully integrated into the GUI of Foreman, so that probably 80% of the use cases to build your own Ansible role would be covered.
Yes, that was very much a âhuhâ moment for me as well, and a thorough deal-killer. I had just gotten things set up so high-level admins could use SW 2.10 to push updated zone files via a config channel and then bounce named via an action chain. Itâs the simplest, most brain-dead way for non-vi / non-cli /non-emacs users to get the job done.
Is that optimal in my world? Not always, but the presence of choice and simplicity often overrides âlatest greatest whiz-bangâ when dealing with people who would look at YAML and also have a âhuhwhu?â moment.
I suspect as CentOS 7 finally ages out and we move to (possibly) RockyLinux, Foreman will come back to the forefront as our lifecycle management tool of choice. By then, hopefully itâs a bit more polished and presentable in the areas where it still shows rough edges.
That said, no one should take this as suggesting Foreman is junk or garbage. Itâs quite an effort, with a lot of moving pieces. As a small-time FOSS coder myself, I appreciate some of the challenges of herding cats and trying to âjust make it workâ.
Gotcha, I misread
Aside from an automated publishing option, maybe some sort of icon next to the content view if it has out of date packages might be interesting. @katello Iâm curious if anyone else has any ideas.
Very true, itâs not the first time we hear that. However itâs possible to deploy a cron job with a script that would regularly pull template of type âscriptâ that could write those templates. But there is really no concept of a configuration channel or configuration in Foreman UI or API.
Good idea.
Iâm waiting for Katello to make the el8 install jump before I begin a test migration of Spacewalk to Katello/Foreman. Forewarning that this is for a home install and not in any way for enterprise.
+1âing emmitchellâs comment: the absence of configuration channels, configuration file tracking, and a configuration channel hierarchy is what gives me the most pause about the migration. Since this is a home setup, I canât tell you the number of times Iâve updated a config file on a server only to forget Iâd done it later on and have Spacewalk remind me (at which point Iâd backfill the changes). It really saved my bacon and made rebuilds and the migrations from CentOS 7 to CentOS 8 relatively painless.
We talked about this feature way back and IIRC the consensus was: configuration management tool is to solution to the problem, we should not be building another one. I can understand that Satellite 5 way of doing this was comfortable, right from the UI, no git involved, no writing of manifests, roles or modules.
Maybe itâs the time to reconsider this, I like the idea of building it on top of an existing configuration management tool (or tools). A plugin that would provide web UI and CLI to create configuration files and it would generate required ansible/puppet modules that could be easily imported into Foreman or Katello sounds nice. What exactly would you expect from such a tool? I am assuming ability to define a file (filename) and contents (as a ERB template) and associate this with hosts/hostgroups. Anything else?
The templates we ship by default are part of our repository.
If you think itâs useful for others, contributions are highly encouraged. This area is IMHO the easiest for users place for users to contribute back and it greatly helps other users.
Interesting, I was under the impression that those ansible roles are to control Foreman/Katello as a service, not to deploy files or performing actions on the client servers. I will check up on that. Thank you!
We already had some thousand RHEL-hosts running with Katello repositories/contentviews/activationskeys and moved some hundred SLES-VMs from SUSE Manager 3.2 to Katello 3.14 in 2020 successfully but with some problems still remaining:
Regarding your questions:
Any particular challenges that you faced?
Katello works and thinks different than SUSE Manager, so we had to reorg our brains from SUMA-speak to Katello-speak - done.
Any particular concept that you wished was explained better?
No, we already knew Katello due to managing RHEL with Katello
Any obstacles you hit?
The SCC plugin was/is not documented very well (this could be made better). Therefore we had much try and error to do 
After adding a âproductâ within the âSUSE Subscriptionsâ thereâs no chance to remove/unsubscribe it afterwards. Or we donât know due to missing documentation?!
Anything you learned that you wish you knew before?
We would have learned less, if there had been a better documentation
Any Foreman-related resources that were of particular use to you?
We needed to build some packages for katello-host-tools and some more for SLES15. It really would have been nice if these packages had already been built and would be/will be distributed by the foreman community:
katello-host-tools-3.5.1-2suse1500.noarch.rpm
katello-host-tools-tracer-3.5.1-2suse1500.noarch.rpm
python2-syspurpose-1.28.0-1.x86_64.rpm
subscription-manager-1.28.0-1.x86_64.rpm
subscription-manager-rhsm-1.28.0-1.x86_64.rpm
subscription-manager-rhsm-certificates-1.28.0-1.x86_64.rpm
Regarding the âcalculate errata statusâ we are missing a goferd-RPM for SLES at all
Just to add some small notes.
This is already on the devsâ list according to:
https://github.com/ATIX-AG/foreman_scc_manager/issues/37
This will be gone in the future anyway and replaced by Remote Execution, which will get a pull-based provider for those needing this communication direction in the future.
One thing Iâve found, and I canât tell if itâs a bug because of the repeatedly mentioned lack of documentation, but Pulp only syncs yum content to smart proxies so they lack the actual tree of files required to perform a Kickstart (squashfs.img etc.). This is a deal breaker for using Foreman in my environment for network-related reasons, besides all the other bugs and learning from trial and error that have to take place in order to get to the point of using Foreman from scratch, and the concerns about being able to back the thing up without just shutting it down and taking VM snapshots every day.