We are excited to announce that Halon now provides official integration with Spamhaus Technology anti-spam & threat data feeds (IP & domain blocklists). Both companies worked together to ensure that this new functionality would be simple to deploy while also scaling all the way from smaller systems to large ISP’s with millions of users and complex email flows.
Did you open your mailbox this morning, only to find a few more of those obnoxious spam emails? Did you react with an irritated frown and a thought about that you are paying for some service to keep this shit out? Don’t worry, it’s perfectly normal. In fact, so did I. Read more.
Email is a major source of phishing and malware attacks. The Locky ransomware solely contributed to a 412% increase of malware emails in March compared to February, according to CYREN’s May 2016 cyberthreat report. While I believe that awareness and training is the most universally effective counter-measure, even that is really difficult, according to this recent study. We probably need a combination of training and technological advancements. One of the latter has to do with email authenticity. Can you trust an email’s sender adress? Generally no, but you can with DMARC. Read more.
Today’s leading spam filter technologies offer a very high degree of accuracy. In this blog I’ll describe the current state of spam classification, and propose a pretty innovative method that can significantly improve both senders’ and recipients’ satisfaction (as well as reducing the burden on administrators and support staff) by enabling senders to report false positives if they pass a CAPTCHA test. Let’s start by familiarising ourselves with the history of anti-spam.
The terminology that we normally use is
False positive, a blocked desired (legitimate) email (“ham”)
False negative, a missed spam that slipped through filters into a user’s mail box
Historically, spam filters had poor accuracy and low performance, and email was scanned after being accepted (probably as a consequence of the former). Finding themselves unable to reject email, they offered actions such as putting suspected spam in a junk folder, quarantine or by tagging the subject line.
This I believe, significantly damaged people’s trust in email as a reliable transport, simply because it makes legitimate (potentially important) email disappear.
The leading spam classification technologies today however, offers both high accuracy and performance. Many of them, including Cyren (that we use), uses fuzzy checksums (or “patterns”) to measure and classify email in a distributed, collaborative fashion. By constantly updating the hashing logic, anti-spam vendors are able to adopt as spammers evolve their tactics. By primarily looking at individual spam “outbreaks”, the false positive ratio is generally low in such systems. This is key, since people tend to be much less bothered by a few false negatives (missed spam) rather than having desired email blocked.
The high accuracy and performance also makes rejecting spam (rather than accepting it) a viable option. Rejecting spam is arguable superior to accepting and quarantining it, since the sender is informed about the email not being delivered to the recipient’s inbox. It reestablishes email as a reliable (transactionally safe) transport, while a copy of (the rejected) spam can still be retained in a quarantine of junk folder. Halon has advocated for this approach for a long time, and it’s a prerequisite for efficient feedback and reporting mechanisms like the one I’m going to describe now.
Using CAPTCHA to handle false positives
While I believe that our default approach of rejecting (giving a 500-error) spam with an informative error message (and storing a copy in a quarantine or junk folder) is superior to a traditional quarantine, there sure is room for improvement. For example, the sender needs to contact the recipient using some other mean (alternative email or phone, which they might not have), the quarantine might consume a significant amount of disk space, and the recipient might need to bother the support staff.
We’ve developed a self-service false-positive report and release project simply called sender-fp-release to address those shortcoming. As it says on its Github page, it allows senders to report false positives directly to the recipient after completing a reCAPTCHA.
In our experience, this system is a win for everybody;
The sender doesn’t need to manually contact the recipient, only verify a CAPTCHA
The recipient gets notified instantly, instead of having to browse through a junk folder
The helpdesk doesn’t need to do anything
Additionally, it saves disk space by only retaining spam for a short time (for example 1 day), unless the sender reports it. The retention time for reported email is extended (typically a week or two), giving the recipient plenty of time to release the email.
Many email providers such as web hosts, ESPs and even VPS providers are familiar with the consequences of being blacklisted; angry customers calling the support because of delayed or reject email, countless of hours tracking down abusive users and patiently trying to get of the blacklists.
Unlike many other anti-spam products marketing themselves as “turn-key” solutions, Halon provides a scriptable email gateway that works as a toolbox for hosting providers. It enables them to tailor the system to fit them perfectly using our high-level scripting language. For example, you can in a programmable fashion create rate limits of anything you like. If you can identify customers based on their sender domain (enforced by the sending email server), you can defer messages based on the customer’s current deliverability statistics such as script such as
if (rate("delivery-failures", $senderdomain, 0, 3600) > 999)
Defer("$senderdomain has more than 1000 failed deliveries during the last hour");
if (GetMailQueueMetric(["filter" => [ "senderdomain" => $senderdomain ]]) > 500)
Defer("$senderdomain has exeeded the max queue limit of 500 messages");
Although quite different from inbound spam, filtering outbound spam can be extremely effective with the right tools, because you know who the sender is. In order to create a maintenance-free system, you can even allow a low rate of spam (per customer) sail through, to minimise the impact of false positives.
There are however many other factors that can be weighted into the equation. We have compiled a short list of the most common and effective methods to combat outbound spam which includes (but isn’t limited to);
Using alternate IPs for suspected spam and source-hashing customers out through different IPs
Most of what we’ve discussed here works equally good in a fully transparent proxy installation, suitable for VPS providers that (for whatever reason) have chosen not to enforce the usage of an SMTP relay.
Skapa ett battre liv for dina medmanniskor och tjana pengar pa det
Vi erbjuder dig ett arbete pa fritiden, lon fran 90 EUR i timman
Fa 90 EUR kontant i handen for den forsta timmens arbete inom tre dagar
you’re certainly not alone (and not using our spam filters). At about 7 pm yesterday (Swedish time) someone thought it would be a good idea to send a massive burst of spam. It seems that for many of our customers, that single spam outbreak accounted for as much as 70-90% of the total traffic. It seems that all of them used “yahoo.nl” as sender domain, which (unsurprisingly) doesn’t use SPF.
Fortunately, the combination of Commtouch’s RPD and our own (Halon) outbreak signatures was able to block it entirely, from 6 pm.
We can see that a lot of this was also blocked at IP level. The “normal” amount of IP blocks is almost invisible in the graphs, compared to the spam outbreak. I’ve removed the axis of the graphs, but let me tell you this. One of our customers, which is a large hosting provider, blocked more than 4 million of those per hour. That sure is a pretty persistent spammer.
The Halon MTA is a flexible email operations and security platform.
It enables organisations that operate large-scale email services to offer competitive features by rapid implementation
and to lower maintenance costs through reliable deployment and reduced complexity.