The journey to Single Page Application

Development
1

Hi,

Ever since we started using react, webpack etc, we often discussed if a migration to a SPA makes sense or not, or in other words, is it a goal we should set or not.

I personally was against that, as I thought its both a very large undertake, and also that its not as important to our users (e.g. they just want a good UX), and lastly, that maybe we won’t be able to rewrite the entire application in a timely fashion, or that it would be just a waste of efforts (e.g. why spend time rewriting things that work well).

but over the last year, I’ve started to change my mind, mainly because we have a better understanding of what it actually means, and also a realization that if we won’t set it up as a goal, it would be hard to provide a better experience to our users.

I’m not expecting that we would rewrite our entire UI, rather, I want to shift the gravity of our UI to SPA, for example, instead of expecting rails to generate the menus, navigation etc, we can let the browser maintain the session state, and only load partial content (via JSON api calls, or partial html response).

IMHO this will allow us to get a better UX quickly (e.g. reduce the full page reload times, keep data in our browser redux store etc) while keeping backward compatibility with current rails views and plugins.

I’ve started to map all needed work (it is an incomplete list) at Tracker #23446: Change foreman to SPA (Single Page Application) - Foreman

What do you think?
Thanks,
Ohad

1 Like
2

I think it does make sense to migrate Foreman to to a single page app.
As some of you already know we developed a single page frontend to Foreman. From a technical perspective, we used react (without redux) and graphql.
I’d like to share some (subjective) experiences we had:

  1. React development is much slower than simple HTML/ERB development. Dev times increase. Even more with redux.
  2. Response times for users improve a lot. The app feels a lot faster. And you can do crazier UX stuff like fancier widgets.
  3. Proper error handling with feedback for users is super important. If we’d do the frontend again, we would further improve this. We already added error codes (the request id) to responses. This helps with debugging issues.
  4. Graphql is great. As a developer you can specify what information / relations you need and don’t need from the API. This is key.
  5. You’ll love deploying static html files as an admin. It’s super simple. Deployment times are super fast and easy.
  6. It’s a lot cleaner to design a page with one technology than mixing technologies all the time (react inside erbs with jquery select fields).
  7. The webpack / javascript stack is a lot more complicated. Admins (read: Foreman users) are more likely to fix issues in an ERB template themselves or apply a patch then using webpacked ES6.

I’d suggest to create a controller for SPA content pretty early in the development process so that plugins can use that and migrate early. We should make sure we provide good patterns for basic “index” and “edit” pages. Maybe we can just move the Katello SPA infrastructure to Foreman core.

Is there an alternative for deface in the js world? If not we need to make sure that we provide enough hooks for plugins to extend the page. Plugins are super important for Foreman’s success and community (in my opinion) and we already burden plugin authors with a lot of maintenance work.

Do you mind explaining that a bit more? What can’t we do with ERB/HTML that we can only do with react? Are there some real-world examples so people can ralate better?

1 Like
3

can you provide some color to going without redux? did you use another state management solution (such as mobx) or it wasn’t required for your usage case?

That’s indeed what we are seeing, I would agree that if you need something fast, rails (and our layer of helpers on top) is probably the fastest way, and to some degree, that is to be expected after so many years. we’ve got to a point where most of our ui is the same just with different data.

The downside of course is that when you want to have a more rich UI / expereince, than it starts taking the same amount of time.

I think the key here, is to make the trivial things trivial and not to over complicate the hard stuff, for example, I believe that adding a chart, or adding a toast notification now is pretty trivial based on the react code that we have today?

Can you expand on this? are those exception error code style or something else?

That’s great to hear, I really hope to see the graphql pr kick back to live soon(hint hint)

Are you suggesting users have something like a continues deployment for the UI or just how much simple it is to upgrade once running in SPA? :slight_smile:

I fully agree - but are you suggesting that we should start a rewrite on the side? is there a better approch than what we have been doing so far?

You are right that editing the file on your server is way easier than finding the github repo and file location that effects the bug you are seeing, having said that, I do think that JS is becoming more and more popular, and today I would not expect anyone starting with html/css not to learn about es6 too…

TBH we have been trying to do exactly this, we slowly tried to build the components that make up a common foreman flow (index page with search, table and pagination, a form and potentially a show page) - we are actually not that far off, we already have components for forms, pagination, and have PR’s for tables etc. my hope is that soon we will have enough components that most folks can just migrate off to react without knowing too much about it.

Sadly I don’t think there is one (I’m happy to be corrected) I fully agree that plugins, or in other words being a platform is key factor in our value to users.

I avoided setting the SPA as a goal until now, mostly because I didnt (and still don’t) want to rewrite the application for the sake of rewrite, my goal was always to give us more tools to build better UI.
I believe that we are in the point of where we can pivot towards having a more SPA than rails application for the UI, I’m not saying that we need to replace screens, but rather to move things like logging in, routing, permission checks etc to the browser, while still getting HTML from the server for current pages, and avoiding server roundtrips for new pages built using react.
I think that once you have the common pattern (of index, edit and show pages) available in react, we could probably move parts of the application fairly quickly.

4

We tried to keep it simple and decided that we don’t need a state management solution at all.

I’ve had some issues with charts and decided to use “the old way” to make things work. I don’t remember the details, but rendering charts inside classic tabs didn’t work very well. There were a lot of ui glitches when switching tabs.

Basically, we show the API response code (the code that is logged in the rails logs) to the user. So a user can create an issue and reference the response code and then it’s just a grep CODE /var/log/foreman/production.log to find the request in the logs.

Yeah, we had to change priorities a bit. But Peter has it on his list.

Yeah, we have a travis based deployment to S3. That’s a lot of fun. I don’t know if that can be adapted to a “product”. But putting static files in an rpm should be pretty trivial as well. And you could, in theory, build such an rpm for every PR. Or even deploy a full canary build for every PR.

I was just trying to support your effort towards a SPA. The way we currently do it is fine for a migration but shouldn’t be the final goal imho. I believe we are at a point where the approach is starting to become a bottleneck. I don’t have a full overview, but we should be pretty close to making basic pages (most of the index pages for example) all in.

I fully agree. I was just trying to suggest that we provide the infrastructure for plugins to add “full SPA” pages to core early in the process.

5

I would love to see foreman as a single page application. For what it’s worth we are already going that route with the new katello react stuff. We are using react-router in our new pages and it’s working pretty well so far.

I think that depends on a developer’s primary language(s). As someone who has been fortunate enough to avoid erb and rails for the most part I think that react/redux is faster. And to Ohad’s point, react development within foreman will become much quicker as our components increase.

6

That’s what I was about to write too. I believe that it will get better once we have examples for of commonly used page types (like lists and forms) and a portfolio of re-usable components (which is already growing).

I personally feel that it’s indeed slower now, but on the other hand we’re adding a lot more tests for the UI code than we used to do. That’s something we should count it too. So by believe is it pays back in long term maintainability.

7

I tend to wait when it comes to new technologies. Yesterday, Fedora 28 came out and I will indeed wait for another two months with upgrades. My favourite 3rd party repos are not yet created, things are not ironed out and I don’t feel being an early adopter. I tend to trust the Rails community - if they say that ERB with TurboLinks is enough, I am fine working with what I have. Turbolinks Rails applications do work nicely across devices and are fast according to some youtube talks, it does require some amount of knowledge and dedication, like anything else. Personally, I prefer ERB with the opposite direction - less javascript. But that’s me, what matters is what’s best for the team.

Let’s not do this for UX performance. I am afraid Foreman is a kind of application where SPA won’t necessary help to improve loading times, we have resources being loaded from backend systems and some pages in Katello are exceptionally slow in some cases. SPA is good if you keep your browser tab opened, like your gmail. But if you are Foreman user who visits an instance three times a day, SPA can actually lead to the opposite - slow initial loading time. Remember Google Groups and its terrible loading time? That kind of experience is possible with the new Foreman, we must be very aware of this and avoid it. Google Groups are dead by the way.

Let’s set an initial loading time goal and never exceed it.

What matters these days is coding effectivity - if SPA can deliver in this regard, that’s more important than loading times if we keep the initial load on reasonable level (let’s set a goal in advance). Every development team or community is different, we must make sure most of us will be effective enough delivering patches, changes or understanding how things works. We need to keep this in mind when going forward. This is the biggest threat of all I think. We will need a lot of new components, tests, demos and documentation.

Let’s make sure we are all able to deliver quickly.

I like the idea of single API for both CLI and web, also I think that REST is weird idea and I always preferred RPC. If GraphQL solves some problems, sure I’d be very careful tho with Rails integration - to me it looks like we are skipping controllers in MVC pattern. This can easily lead to code duplication in web/cli layers. What works for a web app doesn’t need to work for us, we have the rule: if it has UI, it has also CLI. To me it looks like GraphQL will likely be a complementary API, not a replacement. But we don’t want revolution, but evolution. So part of our app in ERB, part in React. Then we would end up with controllers, API controllers and GraphQL code logic on clients. This smells.

Let’s make sure GraphQL works nicely with CLI and web portions.

My last involvement with JavaScript was a Java app with ZK AJAX UI library and my experience was good primarily because we were not reinventing the wheel - ZK is set of predefined components we composed the application from. I’d like to see us not reinventing the wheel in this area. Maybe I just don’t understand how we do the new UX, but I am under impression that we take PatternFly (CSS style/template) and create our own components from that (patternfly-react git repo is just one year old). Not ideal, this can slow things down.

Let’s create an extensive UI prototype of most Foreman pages in advance.

Important part is accessibility and 508, it used to be a “no javascript” dogma which changed these days. I am just leaving a remark we should keep accessibility in mind, I’d assume that Patternfly UX team is deeply involved in this already.

Let’s keep accessibility in mind.

Last but not least, if we pick a component for migration to some different stack, let’s actually do the most challenging one - new/edit host form/wizard rather than doing things like subnets. Because if it works for host form, it probably works for the rest.

Let’s start with the most challenging parts.

Long term, I’d love Foreman UX to be simplified and converging to a single stack. But last couple of years, I’ve only seen adding more technologies and it was rather frustrating. I am all for convergence, whatever this is, but I am afraid that evolution will be long and painful. So with this regard:

Let’s execute fast.

Once it’s all set, let’s set a goal of two Foreman releases for the transition period. We are on a roll, let’s not ruin current user experience.

1 Like
8

Nice write-up. I will not comment on everything that i agree with

I’m not sure about this one though. It seems that the UI and CLI/stable API has different requirements, where the UI needs to be flexible while the stable API needs to be well… stable. My impression from the lessons learned from Katello side is that trying to use one end-point for both leads to not serving well for neither of the purposes.

Therefore, I would much rather like to see the API for UI to grow side-by-side the stable API, and mainly for UI purposes. If then turns out it’s usable for stable API, let’s do that. If not, no hard feelings.

9

Ah interesting. My concern around this is more valid then if my understanding of GraphQL is correct - more logic being client side (so the same logic will be duplicated in API controller code). How are we going to approach this?

10

I’d like to hear more about that from Katello developers. I’m interested in what didn’t work for them and what was causing breakages.
Because in Foreman I think that we suffer from the opposite problem - some of our APIs that are used less frequently are often broken when you try to use them. My naive feeling is that using the API from UI would ensure we avoid such issues. But it probably causes other problems that I don’t see now.

Why do you think we’ll need to put more logic to the client side?
I’d expect the code from controllers to be placed in mutations so that we keep conveniency of the interface.

11

I didn’t know about mutations. That would work.

I think that REST concept is wrong from the day one. I have zero experience with GraphQL, but after few tutorials read this looks like a generic thing that aims to solve all pains of REST. I tend to prefer RPC style approach where everything (the contract) is under developer’s control - particularly I like Protocol Buffers (ProtoBuff) or similar solutions. At least it’s much more readable, mutators seem to be awkward to me.

12

I am an enormous fan of having a single API. The benefits are a single point of entry and test with no duplication of logic. Users benefit from the discoverability of watching network traffic as they use UI (this happens a surprising number of times for novice users wanting to write scripts). The separation of UI and API endpoints problems can be seen clearly in foreman where there are UI actions that are not possible via API (orgs/locs mismatches, for example).

That said, I don’t like the katello controllers in that there is too much logic built in at the entry point. If the API and UI controllers separate but shallower, sharing the core logic, I could see a lessening of benefits for a single API. The two APIs could simply be presentation layers for results from shared logic.

As a simple example, consider this code. In addition to the controller method don’t forget about all the pre-processing of authorization and finding resources such as organization. If there are two APIs, they both need to hit this same code and just render results differently.

I once tried to move hammer csv processing from the client to a server plugin. I found that I didn’t have entry methods equivalent to all the API endpoints; I effectively had to cut&paste controller logic. This is the source of my concern since what I was doing was effectively creating a new API endpoint (format CSV instead of json).

tl;dr Please don’t duplicate controller logic and also please make API controller logic shallower.

1 Like
13

My main concern less about performance and loading time.

I believe the first thing we should ask ourselves is:
How easy it is, for new developers, to read, follow and understand the code?

Our goal must be to make it easier as possible.

I believe the important benefit of an SPA (when it is done right!) are:

  1. healthy separation between the server and the client

  2. client-server communication done from an official documented protocol.

  3. It builds hierarchically, you got a single starting point you can easily find and it just links some blocks together, each block builds from other blocks in a super reusable way.
    You can follow the code from point to point without losing yourself into the spaghetti hell.
    How many times you asked yourself, what code is running before the other and who is loaded first?

  4. It’s a standalone app! You need a server to get data but you don’t actually depend on it blindly. You can actually run the app from anywhere.
    When talking about micro-service approach, imo, the client-side build should be 100% static and we can have a micro-service that should just deliver those static assets.

I believe our main mission should be to make the code simple, documented, easy to follow and easy to change.
Today I see SPA, not as a way to do things better. More as this is where you find yourself when you do things right.

1 Like
14

I’d be curious what other Katello devs think, as we’ve never really had a ‘retrospective’ about this aspect of Katello.

In my experience it has been a great benefit to the project. As you mention, it helps to ensure that apis work, but also helps ensure api/ui parity (and then to some degree cli). The one downside to our implementation is that the returned json fields (defined in rabl) are the same for UI responses and api responses. It would be nice if we could change the json response for just the UI, while leaving the api alone, possibly via a header. It would make things more flexible for UI development and help us to not break backwards compatibility on the api. I imagine the UI responses would be layered ontop of the api responses, but i’m sure thats a discussion for another day.

15

These are indeed very important aspects and I definitely see the value of all that you mentioned. At the same time I don’t agree that we put them above performance. If we rewrite the whole UI just for the sake of using new technology then there’s no value added for the end users and we only cause troubles to authors of existing plugins (even though the new interface is cleaner and better documented, they still need to rewrite it -> more work).
I think that we should definitely iterate towards a single stack of technologies and well documented internal APIs. All that step by step, starting with pages where the UX is painful now. When there’s enough components to rewrite a page into React, then let’s do it. But we must make sure the performance is the same or better.

1 Like
16

I very much agree with this. Let’s not forget that webpack is causing us a lot of pain with regards to plugins and is delaying 1.18.0 RC1. There is no black box integration test of any UI aspect in the pipeline. That means its constantly the cause of nightly breakages. While there are integration tests in the foreman git repository, they don’t reflect the way we install to user systems. It also doesn’t test with plugins. For that we need black box tests. https://github.com/SatelliteQE/robottelo does have the framework to test this so we probably need to select a subset of the tests to use them in our nightly pipeline.

Now I’m not against SPA and do see the benefits. I appreciate that it’s hard to keep the application working while changing it. I am arguing for more tests and usable nightlies. That should make branching a simple event rather than a long fight to fix many broken things.

1 Like
17

@Tomas_Strachota @ekohl, the point is, this process is going to be painful.

For the short term, we will harm performance as we continue doing when mixing all those technologies together and it will lead to a lot of confusions.

But for the long-term, it is going to make the development process much easier, I think mostly for plugins (they should have formal APIs to do things and tons of reusable building blocks).

As I honestly see it, it is going to be a curve when performance getting slowly worse and then a large improvement and I think we are in this process for a long time. don’t you think so?

@ekohl there are tons of powerful tools for integration testing and e2e-testing inside the nodejs stack. @amirfefer and I had many talks about it recently.

18

You are missing the point. I want black box testing. We see that the actual deployment is a different environment. The benefit of Robottelo is that RH QE is already using it for downstream. That means we can share resources. Since it’s black box testing, there’s no reuse value so using the nodejs stack isn’t a direct benefit. With my strong dislike for the nodejs stack I personally see it as a disadvantage but that’s my personal preference.

19

I have no preference to any outcome in this discussion, but I do want to address this point:

There is a clear preference in the the community survey data (both this year and last year) to address stability as a priority. Taking a path that makes things worse in the short term is a risk. That’s not to say we shouldn’t do it - I just want to make people aware of the data we have.

If your feeling is that we’re already on this curve, and the large payoff is coming, then great. But I’m not sure I can personally +1 a further degrade in the testing/nightlies/release schedule area.

20

Imo i prefer to keep the test-cases together with the source code and ask the developers to take responsibility over it. it’s a different discussion but let me just put this here: