We’re seeing high levels of spam at the moment. This post is about what’s happening and what action we should take.
Currently, one or more persons are making accounts at qq.com and using them to send mails to the #support and #development incoming addresses. We’re seeing about 40 mails per hour and this has been going on for some time. Discourse correctly rejects these mails, because the temporary email addresses don’t have a matching user here on the forum.
However - every mail gets a polite rejection from Discourse. This means that in the last month, we’ve sent over 50,000 rejections to this spam. Whilst it’s lovely that Discourse handles this for us, and we’ve not really noticed, it does mean we’ve gone way over our Mailgun allowance.
Fortunately, Mailgun does not operate hard limits, and the extra mails have cost us just $25 this month. However, we can’t ignore the situation, so I’d like to see what the community things we should do. There are three options:
Remove the incoming addresses for #support and #development. This is my preferred option - it doesn’t remove the ability to reply by email, only the ability to start a thread by email. I don’t think this is a heavily used feature, and therefore safe to remove, but perhaps I’m wrong - hence this poll
Block the qq.com domain at the Postfix level, dropping traffic at SMTP time. According to the DB we have just one valid user on this domain, who hasn’t been seen since Feb 2018, so this could be an acceptable route. It would block future users from this domain, of course.
Do nothing - the spam continues, but we don’t seem to be affected too badly. We’ve have to allocate budget for the Mailgun costs, of course, and track the problem to make sure it doesn’t get worse
Try to patch Discourse to not send this type of rejection mail. I’m not keen on patching Discourse, but we could start a conversation on Discourse Meta to see what could be done.
Given this is literally costing us by the day, I’m closing the poll in one week. If there’s a clear consensus before then, I’ll take the appropriate action. Thanks!
I don’t think domain blocking will be very effective - the spammers can easily set different origin domain and we’ll be back to square one, and possibly block actual users (no idea what that domain is, but seems like some sort of Chinese portal).
There are some tools to handle fake mails (such as DMARC) but those may also cause false positives for users with older or misconfigured mail servers, plus I’ve no idea if discourse supports them at all).
Dropping incoming addresses sounds ok if people don’t use them, but what prevents spammers from using the reply-to addresses as well? Is it possible to limit posting by mail to registered users?
However it is a good start to save 30 bucks. Discourse does seem to have a feature which prevents creating user accounts called “Screened emails”. Just use that and add the domain as a wildcard? Should work: “When someone tries to create a new account, the following email addresses will be checked and the registration will be blocked, or some other action performed.”
Technically nothing, as the From field is easily faked. However, every reply-to address is unique to both the target user and the post - that is, the email you get from this reply will have a different reply-to that the one Timo gets. They take the form of reply+c96f878306fe43f2908035fce7b9bfa5@community.theforeman.org and Discourse can map that ID to a user - if the reply isn’t from that user, it’s dropped silently at SMTP time. So the attacker needs to harvest these unique reply-to addresses somehow, and map them back to the correct users to bypass that. Tricky
Posting new threads by mail is limited to Trust Level 1 users, which is why we’re not seeing the spam on the forum. The issue is that Discourse is replying to the spam with a “StrangersNotAllowed” error email explaining why the post failed. I can’t see an option to stop sending those, hence why we have a problem - it might be a longer-term discussion with upstream about whether such an option is needed, but we need to do something now I think.
I’ve opened a thread on Meta to discuss with upstream, we can’t be the first people with this problem. However, I expect that to be a slow resolution, so let’s proceed with something short-term for now
The only issue is that spammers can start replying to threads instead creating new ones once they learn this. Chances are this is a targeted attack they read it as we write those lines. A long term solution is really needed, go and push on upstream while we decide on a short-term fix please.
They’ll need an account that has spent at least 20-30 mins browsing the forum to reply by mail. New users can’t. That’s short enough for ordinary people to never even notice, but long enough to not work for spammers who are registering hundreds of new accounts at a time
Due to how the mail-receiver works, shutting down the incoming address would only fix part of the problem - the Postfix daemon makes API calls to Discourse to determine if a post should be allowed. Thus, the spam was causing high levels of API traffic which is part of the instability we’ve seen this week.
Accordingly I have blocked qq.com at the SMTP layer, and added a swapfile to the VM as a temporary measure. Let’s hope things go back to normal now
By the way, I don’t see us having any kind of “captcha” during new registration process, can we add at least some dumb one to filter off mindless robots? Maybe a small plugin, something less intrusive like “how much is 5+4” that is not that of a problem to fill for newcomers.
I’m not sure that adds anything - we don’t have any issue with excessive signups anyway. My example above was illustration only, I’m not saying we actually see hundreds of account registrations. The spam we’ve had is speculative incoming email from unregistered addresses.
If we do ever see excessive spam registration, obviously we’ll want to take action, but for now I see no value in adding captcha to it.
Unfortunately we are now seeing spam from multiple domains. The previous measures cut our outgoing mail in half, but I don’t think it’s enough. I’m going to drop the inbound email addresses for #development and #support today, and hope that fixes it.
