Feedback wanted - migrate mailing lists to a forum?

Hi all

TL;DR lets migrate to Discourse (forum software, self-hosted). I hope you'll read the rest of this mail though :)

This is a big proposal. If you prefer audio/video to huge posts, I've also put some of my thoughts in the latest community demo:

https://youtu.be/QHzNIFjMpTM?t=2103 (12min watch)

# Not confirmed, only a proposal so far

Lets get the important bit out of the way first - this is a *proposal*, it's is not guaranteed. No final decision has been taken. I'm posting this to get feedback from the community about whether it's a good idea, and I really (really really) want to hear what you've got to say.

# What am I proposing?

I want to migrate our mailing lists to a self-hosted Discourse instance. We have that Discourse instance set up, you can go play with it right now (see below for details). I want everyone to try it out, and tell me your thoughts.

# Why do we need this?

Let me start with a graph:



This is the volume of mailing list posts (users, dev, and the totals) over the last 6 years, grouped by month. As you can see, that traffic has been declining for the last 2 years, and that's a trend I also see in the IRC data too. This worries me.

For some time now, I've been wondering if a mailing list is the right fit for our needs as a community. Times change, and I believe there's a significant group of people that would be willing to be part of our project, but do not want to join a list. In modern times, people expect a UI, and are more comfortable there. This is a thought I have heard repeated from other community members too, so I'm not alone.

Specifically though, why a forum? I'm proposing we use Discourse, so let me link to https://blog.discourse.org/category/use-cases/ and pull a few examples out (in no specific order)

* Easier for new users to get started withClean and simple interface * Better search and search engine indexing
* Post creation tries to be helpful, suggesting similar topics * Markdown support
* Category-level subscriptions
* Browser & phone push notifications for new posts
* Community digest by email & “unread” on the web (for those who can’t keep an eye on the web site all the time)
* Converting a post to a wiki post, so everyone can edit it * Group-level notifications

Broadly, a forum gives us both many new features out of the box, plus flexibility for the future. This post is long enough, but I went into much greater detail in my post to foreman-dev a few weeks ago, please do read it if it interests you (and maybe come to the AMA next week):

https://groups.google.com/d/msg/foreman-dev/xSS1mNKpRQg/uFSmDNK6CAAJ

# I like mailing lists though

Discourse does offer a "mailing list mode". I'm not going to try and sell this to you as a 1:1 replacement for a mailing list, but the main features in terms of creating new topics in by mail, receiving new topics by mail, and getting/replying to new posts by mail should all work. Threading is there too, so the basic functionality should work fine - but I'd love to test this more. Setting up mailing list mode is covered in my intro notes to Discourse notifications here:



Is it a perfect replacement? No. Is it a worthwhile tradeoff? You tell me :)

# How do I help test this?

Glad you asked. Start at: https://community.theforeman.org

Everyone who has ever mailed foreman-users or foreman-dev already has an account (importing the lists creates those users) but all outbound email is disabled by default so we don't spam people. To recover your account, simple log in with your mailing list email address, and then hit Recover Password.

If you're a mailing list lurker (never posted) then you won't have an account yet - never fear, signups are available, or you can log in with your GitHub account. Once logged in, you'll be shown my FAQ to help orient yourself, and you can visit the docs for configuring mail (see above)

# What's next?

Nothing immediately. Next week I will hold an Ask Me Anything on our YouTube channel, and you're welcome to come ask your questions. Obviously questions are welcome here too, and you can email me privately (or private message @gwmngilfen on Discourse :P) if you like.

After some time - exactly how much depends on how quickly we get feedback - we'll be able to take a look at the feedback so far and come to a decision. I absolutely do *not* want to rush this - I want to listen for a while first. It's a big change, and we need to get it right.

I very much hope you'll join me on the expedition to the future. Change isn't easy, but I genuinely believe we need this. Let me know your thoughts!
As promised, I've now scheduled this on our YouTube channel. You can join live on the day at:



This will be a good opportunity to put concerns forward, ask for details on possible migration paths, or just help get yourself oriented in Discourse so you can decide if you like it.

If you can't make it live, do leave questions in advance for me - I've already got plenty of material, but I'll be happy to have more :)

Cheers,
Greg


··· On 16/11/17 17:16, Greg Sutcliffe wrote:
Next week I will hold an Ask Me Anything on our YouTube channel, and
you're welcome to come ask your questions.
+1


··· -----Ursprüngliche Nachricht-----
Von:Greg Sutcliffe <greg@emeraldreverie.org>
Gesendet: Freitag 17 November 2017 15:38
An: Foreman Users Community <foreman-users@googlegroups.com>
Betreff: Re: [foreman-users] Feedback wanted - migrate mailing lists to a forum?

On 16/11/17 17:16, Greg Sutcliffe wrote:
Next week I will hold an Ask Me Anything on our YouTube channel, and
you're welcome to come ask your questions.

As promised, I've now scheduled this on our YouTube channel. You can
join live on the day at:



This will be a good opportunity to put concerns forward, ask for details
on possible migration paths, or just help get yourself oriented in
Discourse so you can decide if you like it.

If you can't make it live, do leave questions in advance for me - I've
already got plenty of material, but I'll be happy to have more :)

Cheers,
Greg

--
You received this message because you are subscribed to the Google Groups "Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

Greg,

I gave Discourse another try via e-mail but I have some issues:

1) Edits are not propagated to e-mail only users, I edited a post and never got an e-mail about this. If this is a feature (I don't get notification on my own edits), that's fine, but otherwise this creates a huge hole as you can miss content.
2) It seems you can't reply to yourself via email, this often happens if you need to correct yourself. This is because you need the reply token, otherwise reply will lead to a new thread - this is broken in many clients but some do have ability to correctly reply usually via Reply to List feature (Mutt, Thunderbird via plugin). Discourse let you opt-in to receive copies which makes your folder a mess and you still need to wait because of (4).
3) All emails contain huge button to visit the thread, this is not an email interface but notification with some ability to reply. Not a great reading experience. Maybe we can edit the footer and make this a very small text with just non-intrusive link?
4) Does not support text/plain emails, it just send both in MIME which is "good enough" according to devs. I would love to *have* all emails in plain text because integration with other things like mailing list archives, bots or simply gmail users who cannot turn off HTML viewing by default. Solving (3) might improve reading experience for poor gmail users in HTML tho.
5) Email response is slow, I know there is some polling, but this is simply limiting and web users are in advantage as they see the content earlier.
6) Your MUA is collecting "sent addresses" as these are randomized addresses. Perhaps not a big deal.
7) Similarly to (1) there are tons of other actions in the web UI without e-mail notifications - "mailing list" users are put aside. 8) It's a notification system not a mailing list interface, please do carefully communicate this towards users.

If we choose to migrate, we will see users moving towards web UI not because it's good with all the shiny features, but because mailing list mode is terrible experience.

Now, on a different topic, with possible migration we will likely loose 3rd party archives. Our groups are already being archived at several places, for example the world famous:

https://www.mail-archive.com/foreman-users@googlegroups.com/index.html

The way this works is usually there is a registered user that receives all e-mails creating such an archive. There are multiple similar services, GMANE perhaps the second most well known and Nabble is also in wide use. I want not to break having this read-only for-life archive which anybody can user or download. It can still serve as a read-only backup when the service is down (e.g. planned outages) and most importantly if we experience some mailing list integration misbehavior we can all look into the "archive" to see what is being sent and how it looks in text mode. This will be also here as an archive of all emails sent, because if someone edits an entry and there will be discussion around the edit, we can always look into the archive. At the end, it's how people using mailing list interface will read all content.

Theoretically anybody can setup such an integration, but some administration or contacting mail-archive people might be required in order to make sure the account does not get deleted and receives all (publicly visible) emails. Big question is how this will work with Discourse "MIME only HTML+plain" format. Therefore I would like to request this to be put on TODO list ideally with priority so we have continuous mailing list archive for the future. Details are here:

https://www.mail-archive.com/faq.html#newlist

In short, just "subscribe archive@mail-archive.com to the mailing list" and make sure the account does not get deleted (e.g. expiration after X years). But something tells me this will fails and emails coming from Discourse will not be readable in such an archive due to MIME/HTML thing, let's see.

Then, you showed how easy is to import content into Discourse, but I would like to read something about what is the plan if this whole project fails. I tried to google some info about how to migrate from it, but apart from some "data liberation" download links in weird format (non-mbox/maildir) I could not find anything. We should at least have a plan how to approach this and what our options are to move back to mailing list.

And lastly, I see lukas_zapletal1 accounts, I was likely posting under same name but different emails into the list. Is there some kind of merge? Would like to only have one account with proper name, proper nick and multiple emails owning all my posts. I briefly saw some discussion around but I understand it is not possible, there are some workarounds: https://meta.discourse.org/t/ability-to-merge-users/9220/75

··· On Thu, Nov 16, 2017 at 6:16 PM, Greg Sutcliffe <greg@emeraldreverie.org> wrote:
Hi all

TL;DR lets migrate to Discourse (forum software, self-hosted). I hope
you'll read the rest of this mail though :)

This is a big proposal. If you prefer audio/video to huge posts, I've
also put some of my thoughts in the latest community demo:

https://youtu.be/QHzNIFjMpTM?t=2103 (12min watch)

# Not confirmed, only a proposal so far

Lets get the important bit out of the way first - this is a *proposal*,
it's is not guaranteed. No final decision has been taken. I'm posting
this to get feedback from the community about whether it's a good idea,
and I really (really really) want to hear what you've got to say.

# What am I proposing?

I want to migrate our mailing lists to a self-hosted Discourse instance.
We have that Discourse instance set up, you can go play with it right
now (see below for details). I want everyone to try it out, and tell me
your thoughts.

# Why do we need this?

Let me start with a graph:



This is the volume of mailing list posts (users, dev, and the totals)
over the last 6 years, grouped by month. As you can see, that traffic
has been declining for the last 2 years, and that's a trend I also see
in the IRC data too. This worries me.

For some time now, I've been wondering if a mailing list is the right
fit for our needs as a community. Times change, and I believe there's a
significant group of people that would be willing to be part of our
project, but do not want to join a list. In modern times, people expect
a UI, and are more comfortable there. This is a thought I have heard
repeated from other community members too, so I'm not alone.

Specifically though, why a forum? I'm proposing we use Discourse, so let
me link to https://blog.discourse.org/category/use-cases/ and pull a few
examples out (in no specific order)

* Easier for new users to get started withClean and simple interface
* Better search and search engine indexing
* Post creation tries to be helpful, suggesting similar topics
* Markdown support
* Category-level subscriptions
* Browser & phone push notifications for new posts
* Community digest by email & “unread” on the web (for those who can’t
keep an eye on the web site all the time)
* Converting a post to a wiki post, so everyone can edit it
* Group-level notifications

Broadly, a forum gives us both many new features out of the box, plus
flexibility for the future. This post is long enough, but I went into
much greater detail in my post to foreman-dev a few weeks ago, please do
read it if it interests you (and maybe come to the AMA next week):

https://groups.google.com/d/msg/foreman-dev/xSS1mNKpRQg/uFSmDNK6CAAJ

# I like mailing lists though

Discourse does offer a "mailing list mode". I'm not going to try and
sell this to you as a 1:1 replacement for a mailing list, but the main
features in terms of creating new topics in by mail, receiving new
topics by mail, and getting/replying to new posts by mail should all
work. Threading is there too, so the basic functionality should work
fine - but I'd love to test this more. Setting up mailing list mode is
covered in my intro notes to Discourse notifications here:



Is it a perfect replacement? No. Is it a worthwhile tradeoff? You tell me :)

# How do I help test this?

Glad you asked. Start at: https://community.theforeman.org

Everyone who has ever mailed foreman-users or foreman-dev already has an
account (importing the lists creates those users) but all outbound email
is disabled by default so we don't spam people. To recover your account,
simple log in with your mailing list email address, and then hit Recover
Password.

If you're a mailing list lurker (never posted) then you won't have an
account yet - never fear, signups are available, or you can log in with
your GitHub account. Once logged in, you'll be shown my FAQ to help
orient yourself, and you can visit the docs for configuring mail (see above)

# What's next?

Nothing immediately. Next week I will hold an Ask Me Anything on our
YouTube channel, and you're welcome to come ask your questions.
Obviously questions are welcome here too, and you can email me privately
(or private message @gwmngilfen on Discourse :P) if you like.

After some time - exactly how much depends on how quickly we get
feedback - we'll be able to take a look at the feedback so far and come
to a decision. I absolutely do *not* want to rush this - I want to
listen for a while first. It's a big change, and we need to get it right.

I very much hope you'll join me on the expedition to the future. Change
isn't easy, but I genuinely believe we need this. Let me know your thoughts!

--
You received this message because you are subscribed to the Google Groups "Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.


--
Later,
  Lukas @lzap Zapletal
Heya!

Answers in-line, but I wanted to put a quick summary here for folks in a hurry. Most, if not all, of these questions are *technical* in nature - they are things we can alter, either by existing settings, or by altering the Discourse code. I see this as similar to our Redmine instance - we've taken a FOSS tool and customized it to our needs, both via code changes and plugins.

More generally, though, your post reads as a direct comparison to a mailing list, and as I said in my opening post, I don't think a mailing list is right for us anymore.

I'm not trying to mislead on this point. The mailing list mode that Discourse offers *is* different, yes, so a direct comparision is likely to find flaws. The real question is whether the *other* things it does are worth the cost. I know (from our chat on the dev list) that you don't agree with my conclusions about our needs as a community, so we're going to end up differing on whether it's worthwhile.

If we migrate, then some "getting used to" is going to be needed, for sure. A balance of tweaks vs change-acceptance will no doubt be found, some of which I expect to be altered *after* we migrate. We don't have to set *everything* in stone on day 1 (and that flexibility is one of the things I like). Some things may be possible to do per-user, which is even better (again, flexibility to interact with the community as each user wants to).

OK, specific answers:

1) Edits are not propagated to e-mail only users, I edited a post
andnever got an e-mail about this.
5) Email response is slow, I know there is some polling, but this is
simply limiting and web users are inadvantage as they see the
content earlier.
There is no polling now, that was temporary. Inbound email is instantly visible in the UI.

Outbound email is delayed 5 min so that people have a chance to spot typos, missed links etc in their posts (in the UI ofc) before the emails are sent. I really don't think 5 minutes is a big deal, a quick scan of our lists suggests average response time is Order(hours) - to claim it's a disadvantage seems a bit of a jump. It seems an acceptable tradeoff to get allow posts to be correct when sent in the first place, cutting down on "oops, forgot the link" style followups.

As for editing, there is a limit on the length of time you can edit your posts, currently that seems to default to 60 days (seems too high to me, for sure). I guess we could set it to the same 5 mins as above, so that no editing can be done after mails are sent, if this is a significant concern for people.

2) It seems you can't reply to yourself via email, this often
happens if you need to correct yourself.
This needs some context for those following along, and it's actually a security thing.

This only happens when you reply to yourself by hitting "Reply" to the mail in your Sent folder. In this case you'll be sending a second mail to "something@community.theforeman.org" instead of
"reply+{token}@community.theforeman.org", which by default will create a new thread.

I say "by default" because there *is* a setting to allow this, but the consequence is that it then allows sender-spoofing to occur. Discourse sends a different reply token to every user, so your reply-to address is different to mine even for the same post, and Discourse requires the "From" header to match the token. Unless you have my token, you can't spoof-post things pretending to be me. We can disable this, but that means sender spoofing is then possible. Details at [1].

So, we could enable this setting, but there are also other workarounds - enabling "Send me my own posts" in mailing list mode will mean you have something to reply to with a token (albeit after 5 minutes), and moderators can also merge split threads back together in the UI. As such, I'm inclined to leave it as it is, but we can flip that setting if it becomes a problem.

3) All emails contain huge button to visit the thread
I see you're already asking about this upstream :)

Templates are editable, yes. That said, they seem fine in my client [2], pretty much the same 3 lines Google Groups adds. Would love to see what other plaintext readers get.

4) Does not support text/plain emails,
Again [2], plaintext in Thunderbird seems OK here.

The plaintext version is the raw markdown from the post, which has some quirks, admittedly - I've opened [3] to discuss options there - but largely I've been getting nice emails from it. Could be MUA specific (I am reminded of the quote that there are no good mail readers, only some that suck less than others :P).

Now, on a different topic, with possible migration we will likely
loose 3rd party archives.
I have no idea if it was deliberate action by someone (wasn't me), but I saw that Nabble has already created an account on our instance [4]. Not sure where that ends up yet, though, will investigate. As you say, it's entirely solvable. I've added it to the Draft Migration Plan so it's not forgotten.

Then, you showed how easy is to import content into Discourse, but I
would like to read something about what is the plan if this whole
project fails.
This is a good example of where the phrase "roll-back" doesn't really work, but rather "roll-forward" is the right term. If at some later time we decide we don't like Discourse, we'll plan a new migration with the same care that we're planning this one.

With regard to the data, there are rake tasks to export whole categories, and every post is in the DB. The mbox format is not complex and neither is the DB - I am certain we can handle getting every post out into a suitable format. As a backstop, we'll also have the archives on Nabble or wherever that ends up. With so many copies of the data, I think it's not too big a concern (no more difficult than it was to get the data out of Google Groups, certainly).

I also think we wouldn't go back to Google groups, but move on to something where we could keep using the community.theforeman.org domain - this means email users don't have to update addressbooks.

And lastly, I see lukas_zapletal1 accounts, I was likely posting
under same name but different emails into the list. Is there some
kind of merge?
Correct, the importer doesn't know those different email addresses are the same person. Yes, they can be merged, and I've tested it, but only once we stop importing the latest list posts regularly (the importer recreates the users while scanning the archive). I'll be happy to take a list of such duplicates and merge them if/when we migrate.

Cheers,
Greg

[1] https://meta.discourse.org/t/reply-own-mail-creates-new-topic/74519 [2] https://imgur.com/a/Moe5l
[3]
[4] https://community.theforeman.org/admin/users/2186/nabble

··· On 21/11/17 15:09, Lukas Zapletal wrote:
Dne středa 22. listopadu 2017 11:59:16 CET, Greg Sutcliffe napsal(a):
Heya!

Answers in-line, but I wanted to put a quick summary here for folks in a
hurry. Most, if not all, of these questions are *technical* in nature -
they are things we can alter, either by existing settings, or by
altering the Discourse code. I see this as similar to our Redmine
instance - we've taken a FOSS tool and customized it to our needs, both
via code changes and plugins.

More generally, though, your post reads as a direct comparison to a
mailing list, and as I said in my opening post, I don't think a mailing
list is right for us anymore.

I'm not trying to mislead on this point. The mailing list mode that
Discourse offers *is* different, yes, so a direct comparision is likely
to find flaws. The real question is whether the *other* things it does
are worth the cost. I know (from our chat on the dev list) that you
don't agree with my conclusions about our needs as a community, so we're
going to end up differing on whether it's worthwhile.

If we migrate, then some "getting used to" is going to be needed, for
sure. A balance of tweaks vs change-acceptance will no doubt be found,
some of which I expect to be altered *after* we migrate. We don't have
to set *everything* in stone on day 1 (and that flexibility is one of
the things I like). Some things may be possible to do per-user, which is
even better (again, flexibility to interact with the community as each
user wants to).

OK, specific answers:

1) Edits are not propagated to e-mail only users, I edited a post
andnever got an e-mail about this.

5) Email response is slow, I know there is some polling, but this is

simply limiting and web users are inadvantage as they see the

content earlier.

There is no polling now, that was temporary. Inbound email is instantly
visible in the UI.

Outbound email is delayed 5 min so that people have a chance to spot
typos, missed links etc in their posts (in the UI ofc) before the emails
are sent. I really don't think 5 minutes is a big deal, a quick scan of
our lists suggests average response time is Order(hours) - to claim it's
a disadvantage seems a bit of a jump. It seems an acceptable tradeoff to
get allow posts to be correct when sent in the first place, cutting down
on "oops, forgot the link" style followups.

As for editing, there is a limit on the length of time you can edit your
posts, currently that seems to default to 60 days (seems too high to me,
for sure). I guess we could set it to the same 5 mins as above, so that
no editing can be done after mails are sent, if this is a significant
concern for people.
Yes please, editing feature feels odd to me. When you read the thread later, it can be very confusing. Especially if we don't get email notifications about the edit.


2) It seems you can't reply to yourself via email, this often
happens if you need to correct yourself.

This needs some context for those following along, and it's actually a
security thing.

This only happens when you reply to yourself by hitting "Reply" to the
mail in your Sent folder. In this case you'll be sending a second mail
to "something@community.theforeman.org" instead of
"reply+{token}@community.theforeman.org", which by default will create a
new thread.

I say "by default" because there *is* a setting to allow this, but the
consequence is that it then allows sender-spoofing to occur. Discourse
sends a different reply token to every user, so your reply-to address is
different to mine even for the same post, and Discourse requires the
"From" header to match the token. Unless you have my token, you can't
spoof-post things pretending to be me. We can disable this, but that
means sender spoofing is then possible. Details at [1].

So, we could enable this setting, but there are also other workarounds -
enabling "Send me my own posts" in mailing list mode will mean you have
something to reply to with a token (albeit after 5 minutes), and
moderators can also merge split threads back together in the UI. As
such, I'm inclined to leave it as it is, but we can flip that setting if
it becomes a problem.

3) All emails contain huge button to visit the thread

I see you're already asking about this upstream :)

Templates are editable, yes. That said, they seem fine in my client [2],
pretty much the same 3 lines Google Groups adds. Would love to see what
other plaintext readers get.

4) Does not support text/plain emails,

Again [2], plaintext in Thunderbird seems OK here.

The plaintext version is the raw markdown from the post, which has
some quirks, admittedly - I've opened [3] to discuss options there - but
largely I've been getting nice emails from it. Could be MUA specific (I
am reminded of the quote that there are no good mail readers, only some
that suck less than others :P).

Now, on a different topic, with possible migration we will likely
loose 3rd party archives.

I have no idea if it was deliberate action by someone (wasn't me), but I
saw that Nabble has already created an account on our instance [4]. Not
sure where that ends up yet, though, will investigate. As you say, it's
entirely solvable. I've added it to the Draft Migration Plan so it's not
forgotten.

Then, you showed how easy is to import content into Discourse, but I
would like to read something about what is the plan if this whole
project fails.

This is a good example of where the phrase "roll-back" doesn't really
work, but rather "roll-forward" is the right term. If at some later time
we decide we don't like Discourse, we'll plan a new migration with the
same care that we're planning this one.
Just to minimize the risk this could happen, I think it would be reasonable to run discourse and users mailing list in parallel at least for month. We could change all links on our web sites to mention only discourse, we could include the scheduled shutdown information to every reply on mailing list but just to be cautious, we should not drop the existing list until we're sure discourse works for us.


···
On 21/11/17 15:09, Lukas Zapletal wrote:
--
Marek


With regard to the data, there are rake tasks to export whole
categories, and every post is in the DB. The mbox format is not complex
and neither is the DB - I am certain we can handle getting every post
out into a suitable format. As a backstop, we'll also have the archives
on Nabble or wherever that ends up. With so many copies of the data, I
think it's not too big a concern (no more difficult than it was to get
the data out of Google Groups, certainly).

I also think we wouldn't go back to Google groups, but move on to
something where we could keep using the community.theforeman.org domain
- this means email users don't have to update addressbooks.

And lastly, I see lukas_zapletal1 accounts, I was likely posting
under same name but different emails into the list. Is there some
kind of merge?

Correct, the importer doesn't know those different email addresses are
the same person. Yes, they can be merged, and I've tested it, but only
once we stop importing the latest list posts regularly (the importer
recreates the users while scanning the archive). I'll be happy to take a
list of such duplicates and merge them if/when we migrate.

Cheers,
Greg

[1] https://meta.discourse.org/t/reply-own-mail-creates-new-topic/74519
[2] https://imgur.com/a/Moe5l
[3]
https://meta.discourse.org/t/plaintext-and-or-raw-emails-for-mailing-list-mo
de/74267 [4] https://community.theforeman.org/admin/users/2186/nabble
I am wholy in support of moving away from Google Groups to Discourse.

I dislike Google Groups, the interface is horrid and I find it difficult to get the information I want out of it. I go so far as to actively avoid partaking in community discussions in projects that use Google Groups as is forum platform.

A clean cut over to Discourse makes the most sense. I understand Marek's trepidation, but if you run both systems in parallel then you just wind up with spit brain. I know you can migrate a google group to discourse, I am not sure you can "merge" a group. But before one path or the other is taken it should be verified if you can ineed perform a merge.

Callum


···
Thanks for answers, as long as you will keep pushing changes upstream and try to solve some of my concerns, that'd be fine. The most annoying is the big blue button for me, I'd prefer some small light-gray link instead (if any). I understand that's only my personal preference, you could possibly push this upstream as user-configurable option. Not being able to reply to myself is not huge deal as you can merge threads which are mis-posted that's fine. Bringing possibility to edit posts down to 5 minutes seems like a reasonable solution, still allows you to quickly fix mis-formatting but makes sure people are not doing crazy edits afterwards.

Re nabble - I tried to integrate a test archive, created an account but it did not work - mails did not show up (although they were sent, I was receiving copies). For some reason Nabble ignored them. So I deleted the archive place and removed the testing account. At the end of the day, I'd prefer mail-archive.com or gmane.org as these are pretty much top archival sites. As an alternative (if integration fails), I'd be fine with integrating with any kind of mail archival tool locally and offering some downloads in mbox formats (e.g. via rsync or plain http). HTML interface is not needed, I just want to have mbox-based copies somewhere else than Discourse database (ideally on a different host). That's really all, mail-archive.com seemed like a good option (backup of backup).

Allright, I am not super excited but as long as I won't see the huge button for every single post and we lock down edits a bit more strict (5-10 minutes) and we will have some standard (mbox) archive somewhere else (*), I am good.

If integration with mail-archive.com fails, I can step in and help with setting up such an archive somewhere else.


··· On Wed, Nov 22, 2017 at 11:59 AM, Greg Sutcliffe <greg@emeraldreverie.org> wrote:
Heya!

Answers in-line, but I wanted to put a quick summary here for folks in a
hurry. Most, if not all, of these questions are *technical* in nature -
they are things we can alter, either by existing settings, or by
altering the Discourse code. I see this as similar to our Redmine
instance - we've taken a FOSS tool and customized it to our needs, both
via code changes and plugins.

More generally, though, your post reads as a direct comparison to a
mailing list, and as I said in my opening post, I don't think a mailing
list is right for us anymore.

I'm not trying to mislead on this point. The mailing list mode that
Discourse offers *is* different, yes, so a direct comparision is likely
to find flaws. The real question is whether the *other* things it does
are worth the cost. I know (from our chat on the dev list) that you
don't agree with my conclusions about our needs as a community, so we're
going to end up differing on whether it's worthwhile.

If we migrate, then some "getting used to" is going to be needed, for
sure. A balance of tweaks vs change-acceptance will no doubt be found,
some of which I expect to be altered *after* we migrate. We don't have
to set *everything* in stone on day 1 (and that flexibility is one of
the things I like). Some things may be possible to do per-user, which is
even better (again, flexibility to interact with the community as each
user wants to).

OK, specific answers:

On 21/11/17 15:09, Lukas Zapletal wrote:
1) Edits are not propagated to e-mail only users, I edited a post
andnever got an e-mail about this.

5) Email response is slow, I know there is some polling, but this is
simply limiting and web users are inadvantage as they see the
content earlier.

There is no polling now, that was temporary. Inbound email is instantly
visible in the UI.

Outbound email is delayed 5 min so that people have a chance to spot
typos, missed links etc in their posts (in the UI ofc) before the emails
are sent. I really don't think 5 minutes is a big deal, a quick scan of
our lists suggests average response time is Order(hours) - to claim it's
a disadvantage seems a bit of a jump. It seems an acceptable tradeoff to
get allow posts to be correct when sent in the first place, cutting down
on "oops, forgot the link" style followups.

As for editing, there is a limit on the length of time you can edit your
posts, currently that seems to default to 60 days (seems too high to me,
for sure). I guess we could set it to the same 5 mins as above, so that
no editing can be done after mails are sent, if this is a significant
concern for people.

2) It seems you can't reply to yourself via email, this often
happens if you need to correct yourself.

This needs some context for those following along, and it's actually a
security thing.

This only happens when you reply to yourself by hitting "Reply" to the
mail in your Sent folder. In this case you'll be sending a second mail
to "something@community.theforeman.org" instead of
"reply+{token}@community.theforeman.org", which by default will create a
new thread.

I say "by default" because there *is* a setting to allow this, but the
consequence is that it then allows sender-spoofing to occur. Discourse
sends a different reply token to every user, so your reply-to address is
different to mine even for the same post, and Discourse requires the
"From" header to match the token. Unless you have my token, you can't
spoof-post things pretending to be me. We can disable this, but that
means sender spoofing is then possible. Details at [1].

So, we could enable this setting, but there are also other workarounds -
enabling "Send me my own posts" in mailing list mode will mean you have
something to reply to with a token (albeit after 5 minutes), and
moderators can also merge split threads back together in the UI. As
such, I'm inclined to leave it as it is, but we can flip that setting if
it becomes a problem.

3) All emails contain huge button to visit the thread

I see you're already asking about this upstream :)

Templates are editable, yes. That said, they seem fine in my client [2],
pretty much the same 3 lines Google Groups adds. Would love to see what
other plaintext readers get.

4) Does not support text/plain emails,

Again [2], plaintext in Thunderbird seems OK here.

The plaintext version is the raw markdown from the post, which has
some quirks, admittedly - I've opened [3] to discuss options there - but
largely I've been getting nice emails from it. Could be MUA specific (I
am reminded of the quote that there are no good mail readers, only some
that suck less than others :P).

Now, on a different topic, with possible migration we will likely
loose 3rd party archives.

I have no idea if it was deliberate action by someone (wasn't me), but I
saw that Nabble has already created an account on our instance [4]. Not
sure where that ends up yet, though, will investigate. As you say, it's
entirely solvable. I've added it to the Draft Migration Plan so it's not
forgotten.

Then, you showed how easy is to import content into Discourse, but I
would like to read something about what is the plan if this whole
project fails.

This is a good example of where the phrase "roll-back" doesn't really
work, but rather "roll-forward" is the right term. If at some later time
we decide we don't like Discourse, we'll plan a new migration with the
same care that we're planning this one.

With regard to the data, there are rake tasks to export whole
categories, and every post is in the DB. The mbox format is not complex
and neither is the DB - I am certain we can handle getting every post
out into a suitable format. As a backstop, we'll also have the archives
on Nabble or wherever that ends up. With so many copies of the data, I
think it's not too big a concern (no more difficult than it was to get
the data out of Google Groups, certainly).

I also think we wouldn't go back to Google groups, but move on to
something where we could keep using the community.theforeman.org domain
- this means email users don't have to update addressbooks.

And lastly, I see lukas_zapletal1 accounts, I was likely posting
under same name but different emails into the list. Is there some
kind of merge?

Correct, the importer doesn't know those different email addresses are
the same person. Yes, they can be merged, and I've tested it, but only
once we stop importing the latest list posts regularly (the importer
recreates the users while scanning the archive). I'll be happy to take a
list of such duplicates and merge them if/when we migrate.

Cheers,
Greg

[1] https://meta.discourse.org/t/reply-own-mail-creates-new-topic/74519
[2] https://imgur.com/a/Moe5l
[3]

[4] https://community.theforeman.org/admin/users/2186/nabble

--
You received this message because you are subscribed to the Google Groups "Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.


--
Later,
  Lukas @lzap Zapletal
I have to admit I haven't done much testing with discourse but so far I like what I see.

The one thing I can say in favor of the mailing list is that I see the numbers go up in my inbox and that reminds me to go and check and stay updated.
I can achieve the same by using discourse in mailing list mode but I like the UI so much that I probably won't do that.
This can also be seen in the UI notifications, it's just getting used to looking in a different place.

I feel both taging and groups help deal with some of the issues I have with searching the mailing list.
If I'm writing the newsletter or just want to be updated with latest versions I search our announcement list for core releases and then the user list for plugins.

The graphic +1/-1 makes it easier then reading through all the responses, especially when some of them have quoted parts of the original email making it harder to find and count to find out what the majority of people want.

So from me it's a +1.


··· On Wed, Nov 22, 2017 at 6:05 PM, Callum Scott <scott.callum@gmail.com> wrote:

I am wholy in support of moving away from Google Groups to Discourse.

I dislike Google Groups, the interface is horrid and I find it difficult
to get the information I want out of it. I go so far as to actively avoid
partaking in community discussions in projects that use Google Groups as is
forum platform.

A clean cut over to Discourse makes the most sense. I understand Marek's
trepidation, but if you run both systems in parallel then you just wind up
with spit brain. I know you can migrate a google group to discourse, I am
not sure you can "merge" a group. But before one path or the other is
taken it should be verified if you can ineed perform a merge.

Callum

--
You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

Dne středa 22. listopadu 2017 11:59:16 CET, Greg Sutcliffe napsal(a):
Heya!

Answers in-line, but I wanted to put a quick summary here for folks in a
hurry. Most, if not all, of these questions are *technical* in nature -
they are things we can alter, either by existing settings, or by
altering the Discourse code. I see this as similar to our Redmine
instance - we've taken a FOSS tool and customized it to our needs, both
via code changes and plugins.

More generally, though, your post reads as a direct comparison to a
mailing list, and as I said in my opening post, I don't think a mailing
list is right for us anymore.

I'm not trying to mislead on this point. The mailing list mode that
Discourse offers *is* different, yes, so a direct comparision is likely
to find flaws. The real question is whether the *other* things it does
are worth the cost. I know (from our chat on the dev list) that you
don't agree with my conclusions about our needs as a community, so we're
going to end up differing on whether it's worthwhile.

If we migrate, then some "getting used to" is going to be needed, for
sure. A balance of tweaks vs change-acceptance will no doubt be found,
some of which I expect to be altered *after* we migrate. We don't have
to set *everything* in stone on day 1 (and that flexibility is one of
the things I like). Some things may be possible to do per-user, which is
even better (again, flexibility to interact with the community as each
user wants to).



OK, specific answers:

1) Edits are not propagated to e-mail only users, I edited a post
andnever got an e-mail about this.

5) Email response is slow, I know there is some polling, but this is

simply limiting and web users are inadvantage as they see the

content earlier.

There is no polling now, that was temporary. Inbound email is instantly
visible in the UI.

Outbound email is delayed 5 min so that people have a chance to spot
typos, missed links etc in their posts (in the UI ofc) before the emails
are sent. I really don't think 5 minutes is a big deal, a quick scan of
our lists suggests average response time is Order(hours) - to claim it's
a disadvantage seems a bit of a jump. It seems an acceptable tradeoff to
get allow posts to be correct when sent in the first place, cutting down
on "oops, forgot the link" style followups.

As for editing, there is a limit on the length of time you can edit your
posts, currently that seems to default to 60 days (seems too high to me,
for sure). I guess we could set it to the same 5 mins as above, so that
no editing can be done after mails are sent, if this is a significant
concern for people.

Yes please, editing feature feels odd to me. When you read the thread
later,
it can be very confusing. Especially if we don't get email notifications
about
the edit.

2) It seems you can't reply to yourself via email, this often
happens if you need to correct yourself.

This needs some context for those following along, and it's actually a
security thing.

This only happens when you reply to yourself by hitting "Reply" to the
mail in your Sent folder. In this case you'll be sending a second mail
to "something@community.theforeman.org" instead of
"reply+{token}@community.theforeman.org", which by default will create a
new thread.

I say "by default" because there *is* a setting to allow this, but the
consequence is that it then allows sender-spoofing to occur. Discourse
sends a different reply token to every user, so your reply-to address is
different to mine even for the same post, and Discourse requires the
"From" header to match the token. Unless you have my token, you can't
spoof-post things pretending to be me. We can disable this, but that
means sender spoofing is then possible. Details at [1].

So, we could enable this setting, but there are also other workarounds -
enabling "Send me my own posts" in mailing list mode will mean you have
something to reply to with a token (albeit after 5 minutes), and
moderators can also merge split threads back together in the UI. As
such, I'm inclined to leave it as it is, but we can flip that setting if
it becomes a problem.

3) All emails contain huge button to visit the thread

I see you're already asking about this upstream :)

Templates are editable, yes. That said, they seem fine in my client [2],
pretty much the same 3 lines Google Groups adds. Would love to see what
other plaintext readers get.

4) Does not support text/plain emails,

Again [2], plaintext in Thunderbird seems OK here.

The plaintext version is the raw markdown from the post, which has
some quirks, admittedly - I've opened [3] to discuss options there - but
largely I've been getting nice emails from it. Could be MUA specific (I
am reminded of the quote that there are no good mail readers, only some
that suck less than others :P).

Now, on a different topic, with possible migration we will likely
loose 3rd party archives.

I have no idea if it was deliberate action by someone (wasn't me), but I
saw that Nabble has already created an account on our instance [4]. Not
sure where that ends up yet, though, will investigate. As you say, it's
entirely solvable. I've added it to the Draft Migration Plan so it's not
forgotten.

Then, you showed how easy is to import content into Discourse, but I
would like to read something about what is the plan if this whole
project fails.

This is a good example of where the phrase "roll-back" doesn't really
work, but rather "roll-forward" is the right term. If at some later time
we decide we don't like Discourse, we'll plan a new migration with the
same care that we're planning this one.

Just to minimize the risk this could happen, I think it would be
reasonable to
run discourse and users mailing list in parallel at least for month. We
could
change all links on our web sites to mention only discourse, we could
include
the scheduled shutdown information to every reply on mailing list but just
to
be cautious, we should not drop the existing list until we're sure
discourse
works for us.

one time instead of having to end up checking in two different places. As Greg said before, it's going to be a roll forward and less of a rollback, so regardless of the failure (on the first day or after a year) we will need to figure out how to fix it from that point onward.

··· On Wed, Nov 22, 2017 at 1:42 PM, Marek Hulán <mhulan@redhat.com> wrote:
On 21/11/17 15:09, Lukas Zapletal wrote:


--
Marek

With regard to the data, there are rake tasks to export whole
categories, and every post is in the DB. The mbox format is not complex
and neither is the DB - I am certain we can handle getting every post
out into a suitable format. As a backstop, we'll also have the archives
on Nabble or wherever that ends up. With so many copies of the data, I
think it's not too big a concern (no more difficult than it was to get
the data out of Google Groups, certainly).

I also think we wouldn't go back to Google groups, but move on to
something where we could keep using the community.theforeman.org domain
- this means email users don't have to update addressbooks.

And lastly, I see lukas_zapletal1 accounts, I was likely posting
under same name but different emails into the list. Is there some
kind of merge?

Correct, the importer doesn't know those different email addresses are
the same person. Yes, they can be merged, and I've tested it, but only
once we stop importing the latest list posts regularly (the importer
recreates the users while scanning the archive). I'll be happy to take a
list of such duplicates and merge them if/when we migrate.

Cheers,
Greg

[1] https://meta.discourse.org/t/reply-own-mail-creates-new-topic/74519
[2] https://imgur.com/a/Moe5l
[3]
https://meta.discourse.org/t/plaintext-and-or-raw-emails-
for-mailing-list-mo
de/74267 [4] https://community.theforeman.org/admin/users/2186/nabble

--
You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

Thanks for answers, as long as you will keep pushing changes upstream
and try to solve some of my concerns, that'd be fine.
Thanks for being understanding, nothing is going to be perfect for everyone. Of course we'll do our best to get changes where we think they're needed, that's one of the great values of FOSS :)

The most annoying is the big blue button for me, I'd prefer some
small light-gray link instead (if any).
As I say, I only see that in HTML-view, are you definitely in plaintext mode?

I understand that's only my personal preference, you could possibly
push this upstream as user-configurable option.
I see you started a thread, I'm watching it for replies. If anything comes up that we can use, I'll act on it, sure.

Not being able to reply to myself is not huge deal as you can merge
threads which are mis-posted that's fine.
It also doesn't seem very common, looking through the archive, so manually fixing it seems like the way forward, for sure.

Bringing possibility to edit posts down to 5 minutes seems like a
reasonable solution, still allows you to quickly fix mis-formatting
but makes sure people are not doing crazy edits afterwards.
Done.

Re nabble - I tried to integrate a test archive, created an account
but it did not work - mails did not show up (although they were sent,
I was receiving copies).
Ah that was you, I did wonder :). Sad that it failed - but I'm totally ok with an archive of some kind, we can work out the details as part of the migration plan, for sure. Thanks for suggesting it.

Greg


··· On 23/11/17 14:16, Lukas Zapletal wrote:
Yes please, editing feature feels odd to me. When you read the thread
later, it can be very confusing. Especially if we don't get email
notifications about the edit.

Just to minimize the risk this could happen, I think it would be
reasonable to run discourse and users mailing list in parallel at
least for month.
want to minimize disruption to users, any future migration would be to something where we could re-use the "users|dev@community.theforeman.org" addresses. We'll never be returning to the Google Groups mailing list anyway.

On the above basis, I'm not sure what risk we're minimizing by running the two in parallel. The risk of Discourse not being right for us is still present, and the need to migrate to something new is still present.

We could change all links on our web sites to mention only
discourse, we could include the scheduled shutdown information to
every reply on mailing list but just to be cautious
finalize the import process until all the posts are migrated, so that's more work maintaining the importer, and also we can't merge duplicate users. The importer also blocks outgoing mail for about 6 minutes when it runs, so there's (small) chance that email notifies could fail if a post is created while the importer is running. This is all avoided if we can be done with the importing at migration-time.

So instead, I would suggest that we use [1] (second part about blocking), but instead we moderate (not block) all new posts, and make the very last visible post to the list be the final migration notice (as well as updating the group description of course). We can migrate moderated posts to Discourse behind the scenes, and tell the user where it went. These posts will never appear in the mailing list, so the importer isn't needed.

Even then, that's additional work (probably done by me, if we're being honest :P), so I'd only do that for a week or two at the most, just while people orient, and then block the Google group entirely.

we should not drop the existing list until we're sure discourse works
for us.
I don't know if it *ever* makes sense to delete the Group completely, as that removes the web archive too, but we can decide that much later on.

Hope that makes sense,
Greg

[1] https://support.google.com/groups/answer/2646833?hl=en&ref_topic=2458761

··· On 22/11/17 11:42, Marek Hulán wrote:>
The most annoying is the big blue button for me, I'd prefer some
small light-gray link instead (if any).

As I say, I only see that in HTML-view, are you definitely in plaintext
mode?
I am not actually, I migrated from Mutt to GMail recently and I have no option other than HTML here.

If folks would not mind, I'd like to change that to something reasonable. Here is what I propose: I will file a patch to Discourse and we can apply it onto our instance immediately as I expect longer discussion (perhaps an option or they might not like it at all).

··· --

Later,
  Lukas @lzap Zapletal
The most annoying is the big blue button for me, I'd prefer some
small light-gray link instead (if any).

As I say, I only see that in HTML-view, are you definitely in plaintext
mode?

I am not actually, I migrated from Mutt to GMail recently and I have
no option other than HTML here.

If folks would not mind, I'd like to change that to something
reasonable. Here is what I propose: I will file a patch to Discourse
and we can apply it onto our instance immediately as I expect longer
discussion (perhaps an option or they might not like it at all).

The big blue button is a bit annoying and just a simple link would work much better for me. Worth giving it a try.

-- Ivan


··· On Fri, 24 Nov 2017 at 10:01, Lukas Zapletal <lzap@redhat.com> wrote:


--

Later,
Lukas @lzap Zapletal

--
You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

I'm OK with that, customization is fine. I think the obvious starting place is tha macros used in the email templates. I'll take a look at where they're defined next week, maybe we can tweak it, or maybe there's even one defined that's better for us...

Greg


··· On 24/11/17 09:00, Lukas Zapletal wrote:
If folks would not mind, I'd like to change that to something
reasonable. Here is what I propose: I will file a patch to Discourse
and we can apply it onto our instance immediately as I expect longer
discussion (perhaps an option or they might not like it at all).
Yeah, I was thinking something very small and grey - opposite of distracting big blue button.

LZ


··· On Fri, Nov 24, 2017 at 3:44 PM, Ivan Necas <inecas@redhat.com> wrote:

On Fri, 24 Nov 2017 at 10:01, Lukas Zapletal <lzap@redhat.com> wrote:

The most annoying is the big blue button for me, I'd prefer some
small light-gray link instead (if any).

As I say, I only see that in HTML-view, are you definitely in plaintext
mode?

I am not actually, I migrated from Mutt to GMail recently and I have
no option other than HTML here.

If folks would not mind, I'd like to change that to something
reasonable. Here is what I propose: I will file a patch to Discourse
and we can apply it onto our instance immediately as I expect longer
discussion (perhaps an option or they might not like it at all).

The big blue button is a bit annoying and just a simple link would work much
better for me. Worth giving it a try.

-- Ivan

--

Later,
Lukas @lzap Zapletal

--
You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.


--
Later,
  Lukas @lzap Zapletal