1.16.0 was released in November 2017. That was the last known good version we have released.
1.16.1 and 1.17.0 were both released over a month ago and both had significant issues reported, that in my opinion should have led to very rapid releases of 1.16.2 and 1.17.1 to mitigate them.
1.18 has been branched a month ago and still no RC has been released, so no users have been testing it at all - we don’t even know if there are any significant issues in it.
We haven’t had a good release in half a year, and it almost feels like nobody is too concerned about it.
Why aren’t we developers dropping everything and getting these 3 releases fixed and out the door?
1.17 includes rails 5, which most likely broke more stuff that we still haven’t found - because most users who tried to upgrade to it got bitten and rolled back to older versions. Lets fix the crucial issues and get it out the door.
1.18 is currently blocked by webpack issues with plugins. Why aren’t we releasing an RC1 that doesn’t work with some plugins so we can at least get early feedback from users not using these plugins?
My personal plan right now is to work only on issues that are blocking releases. Please ping me here or on IRC on anything that needs maintainer action and I will look at it right away. Let’s bring back our community’s trust and focus on getting these releases out.
A ‘like’ just isn’t enough here, well said @tbrisker - I had a draft half-written that basically says the same thing, but you’ve beaten me to it.
This years’ survey data confirms that the 3 (ish) month schedule still works well for our community. Slipping it to 6 months does not seem like a good idea. If there’s anything non-code related I can do to help, I’m in - at the very least we’ll be bringing the OSU slaves online next week to give us more testing options.
Don’t forget many plugins were/are also broken and we didn’t really check it properly.
My impression is that in the past 2 weeks or so we started to scratch the surface of using webpack in a compiled form. I’d say an RC1 implies something is getting close to being done. I’d argue we could have a look at a beta1 which should be the same procedure but perhaps manage expectations more realistically.
The biggest problem is that once we branch we no longer get nightly foreman builds. We need those since they include the build process for plugins. i.e. we need Foreman on a build level to be done before we branch it in our current build process. That hasn’t happened yet.
It would be awesome if more people realized we need working nightlies so releasing becomes a trivial event. What would help a lot was better test coverage in our nightly pipeline. Currently there are no functional UI tests on actual installations. I would like to integrate something so we at least verify various pages load correctly which would test the webpack side.
Thanks for putting this together. Adding my thoughts below
I agree there are some critical issues, though I don’t think 1.17 would not be usable for most users. @Gwmngilfen do we have some data e.g. from RSS feed how many 1.17 instances we have running? Anyway I do agree that 1.17.1 should get out asap, we can keep few bugs for 1.17.2. I don’t think we should shorten release cycles unless there are real grave bugs, e.g. Foreman can’t be installed, critical CVE. 1 month schedule for z-stream seems reasonable to me and worked in past, every developer has the whole month to fix critical issues.
There are many deployements with Katello or REX or ansible. All of these would be affected, I’d prefer not to be overwhelmed by redmine issues that we need to triage and close as dups. RC stands for release candidate, releasing something we know does not work, does not feel right for RC. Also in past, we heard plugin maintainers very clear, that core should be in sync with working plugins. I think it’s better to fix it before RC1, especially since it looks we’re close to fix the issue.
I hope that we can take countermeasures in future, e.g. developers demoing on nightlies, regular upstream triage done by more devs, higher awareness of packaging pipeline, testing happy patch scenarios before the release etc. so that we avoid these problems, rather than trying to firefight or take shortcuts.
In general, any help with releases is greatly appreciated. Thanks for helping closing blockers. I think more hands for 1.19 release could also help.
This topic is very quiet for such an important issue, so a bump is in order
At the moment no, I’d forgotten that feature was in 1.17. I will add an item to my TODO to investigate it. @evgeni the blog is not covered by Fastly, right? That would complicate it…
I agree we can’t release an RC1 until we’re confident that it’s mostly working, but that time pressures are growing. We’re now at 3 releases in a row which have been delayed, that speaks to systemic issues rather than bad luck. I’d love to hear from more voices in this, especially from @dLobatog and @Ondrej_Prazak who have been the recent release managers - what’s causing you pain? (on top of the nightly issues as listed by @ekohl).
We have a plan for core triage which I encourage people to go respond in too. Can we do more to highlight pipeline issues? Would some integration to Discourse help (perhaps a status banner or something) if people are frequent visitors here? Do we need a new release manager to help Daniel and Ondrej?
Just a matter of knowing where to look. If I want to gather unique hits to the RSS feed (say by IP/user-agent or something) then I would expect the CDN would cache that pretty strongly, as it doesn’t change much. Thanks for the info though.
I then loaded that up and parsed it into unique IP-Date pairs, and filtered for lines matching “Foreman” in the UserAgent. The results:
sha1(IP)
Date
UserAgent ~ /Foreman/
534ae6792a959266567fc9539ab9e280d4a17dd2
2018-04-28
2
97c28cf55c34f2f65f4fa12cebb9cc8f151f3884
2018-04-28
4
9ba28e3fe95dab05252bdd49332f39eeca9deb0e
2018-04-28
2
9ba28e3fe95dab05252bdd49332f39eeca9deb0e
2018-05-05
6
9ba28e3fe95dab05252bdd49332f39eeca9deb0e
2018-05-12
6
9ba28e3fe95dab05252bdd49332f39eeca9deb0e
2018-05-19
4
e3c9d53f93daaec3e3d4740b5fcd8491a1a4710a
2018-05-05
2
e3c9d53f93daaec3e3d4740b5fcd8491a1a4710a
2018-05-12
2
e3c9d53f93daaec3e3d4740b5fcd8491a1a4710a
2018-05-19
2
eb4ba57d5e0b8cb6727cb247bca7cde715bd661f
2018-05-07
2
f01e3dc069884902d8fe3a0e02bd16db5cea69ef
2018-05-07
2
That’s out of 4,352 unique IP/Date pairs (or just 0.25% of the total). I did it this way to try and filter out the very frequent hits from the likes of Tiny Tiny RSS and the Slack RSS bot.
So, not many - but we can’t know anything about those who have firewalled, disabled, or altered the URL.
OK, I’m an idiot. Due to a typo in my script, I was only hitting the http logs and missing the https ones (the above was psuedo-code that would have actually worked). The real data is much better.
For unique IP/Date pairs, we get 13152 matching “Foreman” out of 24783, or 53%
For unique IPs alone, its 3815 / 8374 => 46%
If I match on “Foreman/1.17” to exclude dev installs, then we get 3.3k IPs - not bad! This assumes IPs are a reliable indicator of a unique user, of course . This data is just over a month (from 22nd April) so if IPs change, they’ll show up twice.
@dLobatog is the release manager for 1.16, so he would be able to answer your questions regarding 1.16.2. I know there are already a few of PRs against the 1.16-stable branch which are planned for the 1.16.2 release.
Yeah, I’d suggest to bump the version to address this. If you want to help, you can check in foreman-packaging what is currently used and open a PR to bump the version. Thanks!
- I had a draft half-written that basically says the same thing, but you’ve beaten me to it.
