There has been a lot of discussion, both here on the community forum and outside of it, about re-architecting the UI for Foreman and plugins. Many ideas have been proposed, but these discussions tend to be theoretical and often branch into different directions. The goal of this post is to bring this topic for a larger discussion by identifying the problems with the existing UI, what should our goals be, and present the research I’ve done into this. Even after looking into it this week, there are still a lot of open questions and I would love to hear from other developers by opening up this discussion early.
First things first, why even change anything with the UI and it’s technologies?
- Currently, development is difficult on UI pages. It tends to be slow and we haven’t made much progress in the past 2-3 years of introducing React and other new technologies into our stack. We have discussed re-writing pages and now do a better job working with UX professionals, to not be able to implement these changes at a relatively quick pace is holding us back.
- Testing is difficult. With our UI stack being reliant on Rails, it is hard to do anything outside of unit/snapshot tests as you have to load the whole application to test anything from a functional perspective. This has made it hard to introduce testing technologies that help with e2e (or similar) testing, such as cypress.io, and use easy-to-configure CI services such as github actions. We do see UI bugs that seem like they could easily be caught by more comprehensive testing.
- API: For Foreman core, the UI doesn’t completely rely on the REST API that is user-facing. This means that the external API doesn’t always have the functionality that the UI has and there is no way to guarantee this. Having the UI only communicate through the API would ensure the REST API has the same functionality as the UI. This would help hammer and other supported API client tooling as well.
- Sharing and overwriting parts of the UI: We’ve struggled to work with plugins and overwrite parts of the UI. Deface is a technology that doesn’t exactly fix our needs. We’ve made improvements with this with slot+fill, but could perhaps even improve this further and come up with shared libraries and other ways to standardize the UI and plugin interaction.
- Staying flexible: The UI (even the new parts) being reliant on Rails means that it’s hard to stay as flexible as we could in the UI. We are limited to the tools ruby/rails provides for JS integration. Sometimes this limits us or makes it more difficult to take advantage of the JS ecosystem. As one example, we use webpack-rails, a no-longer-maintained gem.
I would love to hear from others who work on the UI about any other pain points they have felt.
What are the goals of this investigation and discussion? I’d like to start with a “what do we want?” approach and not limit our thinking around the existing technologies. Here are my goals for the investigation and what I was trying to target with research:
- The UI can be deployed and developed independently of the back-end: In general, decoupling the UI from the back-end. This would make it more testable, strictly enforce our API compatibility, and provide a better developer experience. This is not to say we couldn’t serve static assets through rails in a production environment, but my goal is that i can ‘npm start’ or ‘npm build’ (or an equivalent command) and have a fully functional and independent application that can pointed at a Foreman server.
- Leverage our existing work: No one wants to “re-write the re-write”. This is a common criticism when the discussion of making changes in our UI technologies comes up and it is a valid one. We should use our existing work that moved pages and parts of the UI to React. IMHO Having some refactoring to fit a new model is fine, but if we need to do full re-writes of pages, that is not productive to us overall. Ideally, the newer React pages and components are moved over as-is to a new architecture.
- Have any “new” UI pages work seamlessly with the existing UI: Let’s be very realistic here - some pages written in erb will likely live in our application for a long time. Going from an “old” page to a “new” page should feel and look the same to a user. This includes menus and authentication. We will want any new UI architecture to fit in comfortably with the old one. Let’s not assume that “this is temporary and we will move everything over soon”. Assume living with Rails-generated pages is here to stay for a while and any new technologies should co-exist with this, even if we have the goal of moving all the pages to a new UI model.
So what is holding us back from moving to an independent UI model? I started to look into this because we have been having a lot of theoretical discussions, and it turns out many approaches fall apart when you try to implement something. Here are the areas that make us “unique” and it is important we acknowledge these first as a solution will have to solve these issues.
- Plugins: This is challenging for a couple reasons
- We allow for a variable amount of plugins to be installed - This means we have to accommodate many combinations of plugins and ensure the UI functions well.
- Plugins can be installed after the initial installation - This means the UI has to be able to change in production. More on this later, but where we see an issue with this is any sort of compiled client-side assets. How would you change them when a plugin is introduced?
- Menus and chroming: the menu and chroming such as the top bar will have to be the same as the existing pages.
- Authentication: How does a user stay authenticated when they move to new UI pages?
- Global state: How do we keep the information of what organization the user has selected and share that across plugins? (as one example)
- Sharing components: How do we share components across Foreman and plugins?
- Supporting disconnected installs: We support disconnected (no external network) installs, which means no reliance on hosted dependencies.
- On-premise installation: We are not SaaS and many tools and workflows are designed around the hosted SaaS model. We need to account for this in our architecture and ensure the UI can be installed as we expect.
There are different ways to approach this, but given the goals of an independent UI and the challenges of plugin architecture, it makes some of the more common models will be hard for us to implement.
The independent UI model makes me look at SPA (yes, I’ve somehow managed to avoid this word until here ), meaning single-page-application, an application that handles routing internally and does not have to call back to the server to render every page. It turns out we rely on Rails engines quite a bit to handle a lot of plugin and dynamic logic for us, so there are a lot of challenges to moving the UI to be independent, especially with an SPA model.
The issue with this is having plugins, how do you extend an SPA? That made me look into managing multiple SPAs on the front-end. I was skeptical at first, but there are many benefits to this model and measures to make sure they co-exist in a non-bloated manageable manner. There are downsides as well that I will get into. This is what I spent the most time researching so here are my findings and I’ll go over some alternative approaches that I hope to research more as well:
Multiple SPA model
Disclaimer: This is not a proposal, just the research I’ve done into this approach that I am sharing, helping to identify the technical challenges of moving to an SPA model and patterns we may want to consider
This model is sometimes called ‘microfrontends’, which is annoyingly buzzwordy, but it really means serving multiple SPAs on the front-end and managing what shows where on the page and which routes display the SPA you want. There is a good write up on them here that goes into more detail. It also is an approach that the open-source Cloud Services for RHEL takes.
Using multiple SPAs on the front-end is appealing for a few reasons:
- It fits our plugin architecture
- Each team owns their own tech stack, meaning they can make their own decisions around testing, linting, CI, etc…
- Reliance between applications is minified and limited. The shared parts of the application can be managed and independent applications will not rely on each other.
A popular library that is used for this is single-spa, which I used to test this concept out and get a kind-of-functional proof of concept here: https://github.com/johnpmitsch/foreman-spa-microfrontend
I’m not pushing that we use the single-spa package if we go this direction, but it did let me test out the concept and identify some issues we would have using multiple SPAs.
The way single-spa works is it allows you to load an SPA on different parts of your application, on different routes and even within the same page. The SPAs can be written in different frameworks even, though I assumed we will all use React and standardize on a version.
Once I got it set up, I was able to create these applications:
- root-config: the single-spa config that manages routing
- foreman-chroming: This is an application that will always show and contain the menus and top bar (chroming) for the application. It will also be responsible for loading dependencies.
- foreman-core: An application serving foreman core pages
- katello: An application serving Katello pages.
These are all in the same repo to keep things easy, but in practice these would be separate repositories or repositories within a monorepo.
This worked well in my testing to always show chroming on the side and then load an application based on the route. You can try this out in the POC.
After that, I had some questions and tried to solve them best I can:
Will assets not double load if we are loading multiple React apps on a page?
- Potential Solution: only load assets in foreman-chroming, which is assumed to be loaded on every page. This can be shared through webpack externals: example here
- We could use the meta foreman-js packages and webpack externals to standardize the packages used for each application. The applications would have access to these package through webpack externals, so no common assets are double-loaded
How to dynamically load applications based on plugins being present?
- We don’t have a pre-determined amount of plugins. How can we handle having the root-config use plugins as a dependency if we don’t know how many there will be?
- Potential Solution: AFAICT, there isn’t support for having a dynamic amount of dependencies in your package.json file. So I used preinstall and postinstall scripts to edit the package.json file, add plugins based on a configurable file, and reset package.json back to the base state. I also have it set up environment variables that turn on when the plugin is present that could be used in the routing and application code. It does work well, but I’m open to other ideas
How can plugins fill in other pages across applications?
- This is a challenging one. Right now we allow plugins to fill in parts of the Foreman UI and I assume we want similar functionality.
- I haven’t solved this in the POC, but here are my thoughts and potential solutions.
- Can we adapt slot and fill to work in an independent UI model? This is a react-based approach that was added to foreman recently. Slot and fill is based on a global registry, so we would have to do something similar here, and that can cause more coupling between applications, which is discouraged. Perhaps the foreman-chroming application could be configured to support this as a source of truth for the slot and fill registry.
- Could we have a shared foreman-components library that plugins can edit, gating functionality behind environment variables? With environment variables turned on for each application, we could have a shared library that loads parts of components dynamically. For example, it could have a
HostFormcomponent, and Katello can add functionality to this that is gated behind an environment variable, which is set during the pre-install npm script that I mentioned previously. It also allows these changes to be reviewed organization-wide, assuming that
foreman-componentsis it’s own repo. This is a bit of a different model than we have now, but moving to a new UI architecture would allow us to approach extending functionality differently.
Menus: How do we use the same menu in both Rails pages and an independent UI?
- Potential Solution: we could separate this menu into its own npm package that takes certain input. This would be used by Rails and the independent UI. This could be used to show the menu and we could use an API call to get the menu entries (this would probably need to be refactored on the back-end to do so). It also could be cached so the UI isn’t calling for the menu items every page load.
- I haven’t fully fleshed out this idea so I am open to solutions: The criteria being the UI uses the same menu with Rails pages and new pages and we don’t duplicate code.
Routing: How do we route to both Rails and an independent UI application?
- I would like to look into using Apache to do this. We can use it to route specific paths to the independent UI app and from there it is handled by single-spa routing or another application-based router. All other routes go to rails.
How do we install plugins after the initial installation?
- This is unfortunately where this approach falls apart: An microfrontends app created in this manner bundles up assets with webpack and evaluates environment variables during compilation. So all logic is done client-side with static files, meaning no environment variables present with which you can base routing or what-shows-on-the-page.
- This means we will likely need some sort of server-side logic if we want to install a plugin that adds new pages to the UI or extends parts of the UI. So I think compiling and serving static assets will be difficult, unless there is some sort of overarching server-side logic that manages them.
Please jump in if I am missing anything, but this approach did fall short when trying to figure out how plugins are installed after installation. However, there are parts to this approach I really like, and it highlights challenging areas that we will find in other approaches. Hopefully it was informative.
There are still a lot of other open questions such as authentication, browser history, versioning, deployment, development workflows, etc… However, I think the plugin model is going to be the hardest to solve and IMHO is the biggest hurdle to moving to an independent UI model. So that issue is where I’m starting my evaluation of different approaches.
As they say, there are many ways to skin a cat, and there are all sorts of UI architecture models out there. Here are a few that I briefly looked into and would like to research further. Please share any that you have encountered as well.
- A server-side-rendered react application, separate from Rails, but using the server side rendering to dynamically handle plugins and other code. This can be based off configuration files that are updated by the installer. This would solve the “install additional plugins after initial installation” problem.
- I know we are already rendering React server-side with Rails, but the point is it decouples our backend and frontend.
- There could be potential in the microfrontend approach here or it could be a single SPA with certain points of extension, for example a directory you drop your plugin files into and the routing is based on a configuration file.
- Apache SSI to handle server logic. I haven’t fleshed this out, but apache having a way to implement server-side logic could allow us to dynamically server static files based on a plugin’s presence.
- Uber’s Fusion.js framework, a framework that supports React and apparently supports plugin architectures, allowing large applications to standardize parts and modularize parts of their application. I haven’t looked into it much, but it seems interesting and worth mentioning due to the plugin-able aspect.
A quick word on the API. To completely move to an independent UI, Foreman would have to have a fully functional external API. However, the current React pages are supported by the API, so the first step would be to move those over. That will be plenty of work even before we touch the API.
I bring this up because I see a lot of UI discussions go to a GraphQL vs REST api debate and that is not something we would have to solve immediately to create an independent UI, especially when talking about existing functionality. Unless I’m missing something obvious, let’s keep that conversation separate from this one
I would like to emphasize that I’m bringing this topic up for discussion in order to bring the conversations past the theoretical and actually investigate the technical hurdles. There has been no decision to change the Foreman UI and we are just exploring options. Perhaps moving to a fully independent UI won’t be for us, but I think it’s worthy of dicussion. Hopefully, developers find this useful and a good jumping off point for more discussion.
This was a huge brain dump so let me know if there are parts that aren’t clear or any questions. Thanks!