Lot of critisism on reddit.com

Community
21

This thread seems to discuss many things. I’d suggest split that into separate topics and potentially working groups that would discuss, suggest and commit to work on respective changes. At this point it seems everyone wants the change. This great thread reminds me some discussions we had in the past. To avoid repeating it again in 6 months, we need to take some action. I’ll reuse good summary from @John_Mitsch and will summarize information I’ve seen here or elsewhere about existing efforts so people can join if they feel they can help in some area.

  • documentation - I think people interested in this can join @lzap, @mcorr and @bbuckingham who are actively working on improvements in this area, while there may be work outside of foreman-documentation, this sounds like a group of interest already
  • installer - @ekohl experiments in this area, I know @ehelms spent some time thinking about this too, if people are interested, I suggest reach out to them, two possible paths forward are keep tech and refactor, change tech but also limit capabilities of the installer but there may be much more to explore
  • webui - any change in here will lead to a discussion about changing existing UI or build a new separately, I think we got to a point it’s worth PoC of new standalone UI and see how that would work and if we’re missing something, what’s possible to achieve in reasonable time and what the impact would be to existing plugins and tests, in fact @Ron_Lavi I believe suggests that at Upgrading to Patternfly 4 in Foreman, perhaps new thread should be created on this topic, it may be broken to more subtopics too (page redesign, separate repo, new tech, pf4 etc)
  • improve tests - there’s already a working group and suggestion at E2E and functional testing meeting notes, thanks @John_Mitsch, @amirfefer, @ekohl, and @Roman_Plevka, I believe any help is welcome, reach out if you’re interested
  • scope - I think @tbrisker did a great job of reducing things in 2.0. I read in this topic, some puppet functionality is also a good candidate. I know @tbrisker is long term working on reducing number of settings, simplifying org/locs. Recently we also talked about pages such as hw models, statistics, trends which are probably not used much. Let’s propose dropping/extracting them in community survery. I think reducing complexity also belongs to here, e.g. our over complicated permission system, multitenancy model etc can be simplified if we sacrifice some fucntionality. Tomer, if you want help with this, count me in. If you’d like to create some king of a group of interest, please let me know where I can join.

There’s a big chance I’ve missed some other ongoing efforts, please share what you’re working on, so we can see where we can cooperate and mvoe on. If you’re not involved yet but would like to join any of the topic above, hopefully the list of nicks helps.

6 Likes
22

This is really interesting and thanks for sharing how it has progressed, I will have to look into it more. I think continuing to develop a layer like this would be a great way to enhance the usability of Foreman. It doesn’t require a re-write and will give users a way to get up and running with Foreman without inducing decision fatigue and needing to know every corner of the app before you even install it. Perhaps it warrants a working group, even if its still in the investigation stage?

23

It’s indeed very positive if you look on it this way. We are on a good track, let’s keep focused.

1 Like
24

THIS. This. this. Thank you for confirmation - we are actually bringing these books upstream and into Foreman context so everyone can benefit, understand, plan deployment and contribute. @mcorr and @bbuckingham

4 Likes
25

Also, welcome to the forum. Share your experience here, so we can learn from it. Others can find your solutions. And eventually we can improve the project or its documentation.

27

Isn’t this, what we actually need? getstarted.theforeman.org - a small ui that outputs the required command for our installer to get started? I don’t think yet another installer and more code that has to be maintained by the community is what we need.
Sorry guys, the puppet modules aren’t the problem. If users know the correct parameters, it works great and sets Foreman up. I guess the pain points are, that users are overwhelmed by the amount of parameters. So let’s just hide them or give them a curated set of ready-to-use installer commands. Why can’t we tell users “Run this command and you’re done.”? Certainly not because our installer is puppet based. The technology isn’t the problem, it’s the workflows we don’t have.

2 Likes
28

Also, welcome to the forum. Share your experience here, so we can learn from it. Others can find your solutions. And eventually we can improve the project or its documentation.

Now that I have more experience and confidence with the toolkit, I begin to see where the project lacks in my opinion, but keep in mind that I use Katello along.

To me, content creation is a tedious process if it’s not automated ahead in time, and even automating it would be tedious as well (lots of hammer repository create in a bash file that easily become gigantic). It requires lots of planning, and to make sure parameters are meeting the requirements, and I had a big surprise in my learning experiments with the on-demand flag and a Fedora repository, prompting me to restart my lab from scratch (missing packages from upstream). I’m glad this parameter is now visible in the UI, but I still feel like a small warning would be really appropriate as some mirrors will not cache the full content even for CentOS. As an example of such repository, the Centos 7 update repo hosted by Belnet ( https://ftp.belnet.be/mirror/ftp.centos.org/7/updates/x86_64/Packages/ ).
It could lead to this kind of issues for organization that are lagging behind for any reason. Now that I know about that behaviour, it’s something that I’m perfectly fine with, but that was a tidbit that caused lots of questioning.

Also, I’m in the process of updating the Katello stack, and the very important part for the following is that I integrated it in the content management, so it has a specific composite content view from which I pull updates. The upgrade process from 3.12 (which is currently running) is more complicated by far, with the need to recreate every repositories. For a one-node installation, this isn’t smart, but that’s why I’ve got my own lab, to have sub-optimal setups and highlight some issue. This led me to take a look at the documentation, and a great idea to make the process smooth for those updates that require editing repository configuration would be to pull the repos RPM, and create repositories in the product from there. When creating a product, I can scan a URL, I can manually create it, but I can’t upload the rpm containing all the information and create it.

As an additional process I’m still pulling my hair on is Remote Execution and Ansible. When running ad-hoc jobs, I can clearly see that there’s a very specific structure, the idea is to pin a role to a hostgroup. How about running ad-hoc playbooks ? In the documentation ( Foreman :: Plugin Manuals ), the example is running a bash command in a shell, that one works easily. Rebooting hosts through Ansible however will return an error if executed as the SSH connection gets killed in the process, the playbook interprets it as an error. I recall having a playbook that would not return any error on those, but I can’t find a way to integrate it in the job execution. I use the platform I have setup because it works as I want it to, but there are small tasks for which I’d get back to basic Ansible playbooks, such as copying a file over. This could be integrated in Foreman and allow to pull file from the versioned content.

On the things that are great, I have it setup such as I can automatically deploy a virtual machine through the web UI (ignoring a small bug I’ve got in the UI), the machine installs itself and is updated then. Basically, injecting enough work in Katello would allow me to become an optional part of the machine deployment and configuration, perfect automation. Domain junction worked after a fair amount of tinkering (FreeIPA domain),

Sorry guys, the puppet modules aren’t the problem. If users know the correct parameters, it works great and sets Foreman up. I guess the pain points are, that users are overwhelmed by the amount of parameters. So let’s just hide them or give them a curated set of ready-to-use installer commands. Why can’t we tell users “Run this command and you’re done.”? Certainly not because our installer is puppet based. The technology isn’t the problem, it’s the workflows we don’t have.

Totally agree, the first time I had a look at Foreman, I spent a fair amount of time reading what parameter exist, noting on a notepad what was interesting, writing my command, and then it installed like a breeze.
A getstarted.theforeman.org would be really great, or even have a foreman-gen-install which would generate an foreman-install based on answers. That one would even work offline.
The problem I had was the amount of options available, which was really overwhelming at first.

As for the installation process and the full “get started” for Katello, I’ve seen this blog post recently ( Katello: Create Products, Repositories, Content Views, Lifecycle Environments, Activation Keys | Lisenet.com :: Linux | Security | Networking ).
Maybe having a way to pre-create some products would also be interesting, as most setups are performed on CentOS systems ? At least taking a look at the system on which would allow to pull the repository configuration (if it’s using a very specific mirror), pre-create the products, GPG keys, …

2 Likes
29

I think this is what we are all leaning towards, but the question is if that should be done through documentation or software.

30

If anyone wants to start this, I am willing to participate in delivering know-how for provisioning-related options. We need much more people involved in this (configuration, content, plugins) and also some fancy JS/HTML.

3 Likes
31

We actually have that, kinda. It asks you 500 questions and then you are done :slight_smile:

Here is how such a session would start, just for fun. This is how engineers create interfaces :slight_smile:

# foreman-installer -i
Resetting puppet server version param...
Welcome to the Satellite installer!
-----------------------------------

This wizard will gather all required information. You can change any parameter
to your needs.


Ready to start? (y/n)
y

Main Config Menu
1. [✓] Configure certs
2. [✓] Configure foreman
3. [✓] Configure foreman_cli
4. [✗] Configure foreman_compute_ec2
5. [✗] Configure foreman_compute_gce
6. [✗] Configure foreman_compute_libvirt
7. [✗] Configure foreman_compute_openstack
8. [✗] Configure foreman_compute_ovirt
9. [✗] Configure foreman_compute_rackspace
10. [✗] Configure foreman_compute_vmware
11. [✓] Configure foreman_plugin_ansible
12. [✗] Configure foreman_plugin_bootdisk
13. [✓] Configure foreman_plugin_discovery
14. [✓] Configure foreman_plugin_docker
15. [✓] Configure foreman_plugin_hooks
16. [✓] Configure foreman_plugin_openscap
17. [✓] Configure foreman_plugin_remote_execution
18. [✓] Configure foreman_plugin_tasks
19. [✓] Configure foreman_plugin_templates
20. [✓] Configure foreman_proxy
21. [✓] Configure foreman_proxy_content
22. [✓] Configure foreman_proxy_plugin_ansible
23. [✗] Configure foreman_proxy_plugin_dhcp_infoblox
24. [✗] Configure foreman_proxy_plugin_dhcp_remote_isc
25. [✓] Configure foreman_proxy_plugin_discovery
26. [✗] Configure foreman_proxy_plugin_dns_infoblox
27. [✓] Configure foreman_proxy_plugin_openscap
28. [✓] Configure foreman_proxy_plugin_pulp
29. [✓] Configure foreman_proxy_plugin_remote_execution_ssh
30. [✓] Configure katello
31. [✓] Configure puppet
32. Display current config
33. Save and run
34. Cancel run without Saving
Choose an option from the menu... 2

Module foreman configuration
1. Enable/disable foreman module, current value: true
2. Set admin_username, current value: "admin"
3. Set admin_password, current value: "changeme"
4. Set admin_first_name, current value: UNDEF
5. Set admin_last_name, current value: UNDEF
6. Set admin_email, current value: UNDEF
7. Set db_manage, current value: true
8. Set db_type, current value: "postgresql"
9. Set email_delivery_method, current value: UNDEF
10. Set email_smtp_address, current value: UNDEF
11. Set email_smtp_port, current value: 25
12. Set email_smtp_domain, current value: UNDEF
13. Set email_smtp_authentication, current value: "none"
14. Set email_smtp_user_name, current value: UNDEF
15. Set email_smtp_password, current value: UNDEF
16. Set locations_enabled, current value: true
17. Set organizations_enabled, current value: true
18. Set initial_organization, current value: "MyOrg"
19. Set initial_location, current value: "MyLoc"
20. Set ipa_authentication, current value: false
21. Set puppetrun, current value: false
22. Configure Advanced parameters
23. Reset a parameter to its default value
24. Back to main menu
Choose an option from the menu... 22

Group Advanced parameters (of module foreman)
1. Set foreman_url, current value: "https://fivesix.ipsix.lan"
2. Set unattended, current value: true
3. Set unattended_url, current value: UNDEF
4. Set authentication, current value: true
5. Set passenger, current value: true
6. Set passenger_ruby, current value: "/usr/bin/tfm-ruby"
7. Set passenger_ruby_package, current value: "tfm-rubygem-passenger-native"
8. Set plugin_prefix, current value: "tfm-rubygem-foreman_"
9. Set use_vhost, current value: true
10. Set servername, current value: "fivesix.ipsix.lan"
11. Set serveraliases, current value: ["foreman"]
12. Set ssl, current value: true
13. Set custom_repo, current value: true
14. Set repo, current value: "1.18"
15. Set configure_epel_repo, current value: false
16. Set configure_scl_repo, current value: false
17. Set selinux, current value: UNDEF
18. Set gpgcheck, current value: true
19. Set version, current value: "present"
20. Set plugin_version, current value: "present"
21. Set db_adapter, current value: UNDEF
22. Set db_host, current value: UNDEF
23. Set db_port, current value: UNDEF
24. Set db_database, current value: UNDEF
25. Set db_username, current value: "foreman"
26. Set db_password, current value: "ErjLWsWz4Qkx9bneuf5rSPGnp2ScUwz4"
27. Set db_sslmode, current value: UNDEF
28. Set db_root_cert, current value: UNDEF
29. Set db_pool, current value: 5
30. Set db_manage_rake, current value: true
31. Set app_root, current value: "/usr/share/foreman"
32. Set manage_user, current value: true
33. Set user, current value: "foreman"
34. Set group, current value: "foreman"
35. Set rails_env, current value: "production"
36. Set user_groups, current value: ["puppet"]
37. Set passenger_interface, current value: UNDEF
38. Set passenger_prestart, current value: true
39. Set passenger_min_instances, current value: 1
40. Set passenger_start_timeout, current value: 90
41. Set vhost_priority, current value: "05"
42. Set server_port, current value: 80
43. Set server_ssl_port, current value: 443
44. Set server_ssl_ca, current value:
"/etc/pki/katello/certs/katello-default-ca.crt"
45. Set server_ssl_chain, current value:
"/etc/pki/katello/certs/katello-server-ca.crt"
46. Set server_ssl_cert, current value:
"/etc/pki/katello/certs/katello-apache.crt"
47. Set server_ssl_certs_dir, current value: ""
48. Set server_ssl_key, current value:
"/etc/pki/katello/private/katello-apache.key"
49. Set server_ssl_crl, current value: ""
50. Set server_ssl_protocol, current value: UNDEF
51. Set client_ssl_ca, current value: "/etc/foreman/proxy_ca.pem"
52. Set client_ssl_cert, current value: "/etc/foreman/client_cert.pem"
53. Set client_ssl_key, current value: "/etc/foreman/client_key.pem"
54. Set keepalive, current value: true
55. Set max_keepalive_requests, current value: 10000
56. Set keepalive_timeout, current value: 5
57. Set oauth_active, current value: true
58. Set oauth_map_users, current value: false
59. Set oauth_consumer_key, current value: "wKHh22dY4qMzd5KPvPAMJAhmUBPcHk3X"
60. Set oauth_consumer_secret, current value: "S8etdxcYNPEADhyLw8yTLzGm5SoqKwjF"
61. Set http_keytab, current value: "/etc/httpd/conf/http.keytab"
62. Set pam_service, current value: "foreman"
63. Set ipa_manage_sssd, current value: true
64. Set websockets_encrypt, current value: true
65. Set websockets_ssl_key, current value:
"/etc/pki/katello/private/katello-apache.key"
66. Set websockets_ssl_cert, current value:
"/etc/pki/katello/certs/katello-apache.crt"
67. Set logging_level, current value: "info"
68. Set logging_type, current value: "file"
69. Set logging_layout, current value: "pattern"
70. Set loggers, current value: {}
71. Set telemetry_prefix, current value: "fm_rails"

… etc etc you get the idea :slight_smile:

Those passwords - don’t worry the VM is dead already. Just a testing machine.

32

I think this may be a good start and interim solution. But there are problems with our existing installer and even the puppet based approach that won’t go away with it.

  1. more complicated things will continue sufferring from installer CLI limitations (no webui wizards, --help showing too many things, no interaction between related parameters)
  2. we need to own the whole configuration of deployed service (e.g. httpd, rsyslog, postgresql, redis) not just things we want to install (foreman, smart proxy, pulp, …)
  3. lack of devs willing to write the puppet code, resulting in rather building separate scripts or just documenting manual reconfiguration

Of course rewrite does not help with at least the first one. But as explained at RFC: Make Foreman easy to deploy and maintain this shouldn’t be a rewrite. We need to have the installer do the minimum and make the rest of the configuration from Foreman itself. But I agree, until we get there, existing installer can be used.

33

I agree this is an issue and recently I’ve been thinking about how we can improve this. There’s a huge issue with many parameters that aren’t marked as advanced while they should be. In particular the Foreman Proxy and Puppet modules suffer from this. These pretty trivial changes but I don’t see anyone taking responsibility here.

I still see this as a feature, not a problem. We often talk about being opinionated and this is a case that I’m strongly in favor of that. This way we can ensure we have a supportable deployment.

That is an issue, but something that will always happen. For example, I wouldn’t want to submit patches if the installer was written in Ansible and yet others prefer some other tool. We’d see the same thing if we only shipped containers.

Traditionally what we’ve seen in open source is that someone writes an alternative, people like that alternative better than the initial and becomes the default. In other cases a consensus was reached a rewrite was needed. If another group of people are willing to build and maintain an alternative installer capable of replacing the current one, I’d be happy to hand it over.

That’s just very misleading because the first sentence there is:

In this part of the tutorial we will show how to set up Foreman authentication manually (without using installer option).

That’s just a bad example given 5.7.1 is the example with the installer. Given the fact there’s still instructions there referencing RHEL 6 (which we dropped in Foreman 1.13) suggests this is a good example of nobody updating documentation though.

1 Like
34

This should at least clean up the EL6 instructions:

35

As much as I don’t believe Puppet is the right tool to manage Foreman, I believe rewriting it is out of table. There’s no way, we cannot afford spending six months with this.

What makes sense it a “wizard” (in any form, preferably online) that would generate the installer command. That makes a lot of sense. Actually, the more I think about it the more I like the idea of it. Let me explain.

The reason why users naturally treat Foreman/Katello/Satellite deployments as “pet” servers even when they were mostly deployed by Puppet is the bad installation experience. All the details you had to find and lack of documentation make users believe that it’s hard to reinstall. And it is true, it’s not easy, particularly for newcomers. They will make everything to avoid doing that again.

If we have a wizard that makes this experience shorter and nicer, then people could start thinking - well maybe I am going to deploy new version next to the old one since it only took one hour. This is where users can actually leverage the benefit of Puppet. Maybe the reason why I see Puppet as the bad tool for the job stems from the fact that me or users do not think about quickly redeploying it - the thing which Puppet is good at.

Such a wizard I think should have these features:

  • Describes features rather than plugins or technical details. Solves: Less need of documentation and searching.
  • Is easy to add new questions/options and to maintain for all community users. Solves: We will actually have content in it.
  • Questions are independent, users can take any order or even finish it early. Solves: Impatient users actually finish it.
  • Has automated testing that will make sure it would never generate an invalid installer command. Solves: It would actually work.
  • Respects there are multiple versions of the project or the installer with different options. Solves: Old versions (probably we would just branch the wizard for each release).
  • Can be easily branded to downstream (Satellite, Orachino) so product owners are interested in investing more effort. Solves: Resources.

What if I start a small know-how engine in TypeScript? Would you be interested in delivering know-now (backend) or writing a fancy UI (frontend)? If you give heart it means you want to participate.

1 Like
36

And the ansible based installer linked here is exactly the alternative that may or may not work. I’m pretty sure there are people who are not interested in ansible at all.

I’m sorry, I thought we also have other means of external authentication document there, but turns out we have it in Satellite documentation only. GSS proxy setup was the one that was not added to our installer. This chapter was added in 1.5, but the installer got the feature right after in 1.6. Yeah, I don’t have a good upstream example.

37

That’s what I meant with getstarted.theforeman.org. As an alternative to a Wizard, small quickstart guides could also help a lot. I like how CoreOS has done them. Do we have small tutorials that get you started?

What we would need are tutorials like this:

  • How to get the Foreman app installed (this is what the wizard could solve).
  • How to setup Pxebooting on Bare Metal of Centos 8.
  • How to setup Pxebooting on Vmware of Centos 8.
  • How to setup image booting of Centos 8 on ovirt.
38

The huge issue with this are combinations. You want PXEBooting setup, but also Openstack and without Katello. But we have PXEBooting setup tutirial with Katello etc etc.

I have not made this up, this is actual struggle we have in the foreman documentation, old or new does not matter. Documentation must be opinionated to give you good instructions. This is where a wizard could help.

However wizard without documentation is useless, we need to integrate both. It could be interesting if wizard would actually output:

  • Installer command.
  • Documentation chapters which are relevant to start reading.
39

We have some scenarios documented, but I’d argue we mostly lack descriptions of these scenarios in the first place. Without those, you can’t really build any wizard.

I’d also like to make a distinction between Katello and Foreman. For Katello there never has been a clear picture showing the architecture. It’s old and imperfect, but at least Foreman starts the manual with a picture that shows the main components and how they communicate. If you don’t even know the components, how can you be expected to configure them?

40

Perhaps this is actually part of the issue here.
As a developer, i care a lot about the different components and how they are configured and communicate and so on.
As a user, I shouldn’t have to care - I want to be able to, for example, sync content, or provision vms, or any other workflow. I don’t have to know how the internals work. I might want to know at some point how to tweak some things, but that would be much further down the line. At the initial installation, i just want to be up and running within an hour, and not worry about a ton of internal details and configuration options.

2 Likes
41

In an ideal world I would agree, but from my experience you typically do not start on a green field and you have to consider an existing environment with all its specifications. I had environments were it took me multiple weeks of discussions and changed plans before we could find a way how to implement one way of provisioning. If we want to take this in account it is not only guiding the user through a wizard if he should not need to internals, it would also being perceptive of the environment.

For example we would have to tell him, there is already a DHCP server which perhaps already has a next-server setting pointing to somewhere else, so the user gets information on how to configure a smart-proxy to manage this two components. But what if he is not allowed to change them, how do we recommend him a different approach of provisioning?

1 Like