Browse

Tag: tech

MSUI – What is it? Why do I need it? How do I get started?

Managed Service Providers (MSP) and System Integrators (SI) face a common challenge – administering email security as part of their service portfolio. As operating environments become more complex, MSPs and SIs need tools that can help them be more efficient and agile.

So let’s take a look at what MSUI is and how it will help your business.

Read more.

The DKIM replay attack, and how to mitigate

The DKIM replay attack is a way that spammers try to bypass spam filters by impersonating reputable organizations, exploiting the way that some mailbox providers use DomainKeys Identified Mail (DKIM) to build domain reputation. This can severely affect the reputation and deliverability of those organizations being impersonated. In this post we will look at DKIM replay; what it is, why it’s on the agenda, how it can be mitigated on both the sending and receiving side, and what’s happening next.

DKIM replay is nothing new. It is discussed in one section of the DKIM RFC 6376, and Word to the Wise provided recommendations back in 2014. But it looks to me as if the problem is growing. In December 2021, the privacy-focused mailbox provider ProtonMail experienced delivery problems to Gmail as a result of a DKIM replay attack, and we speak to email marketing companies that are fighting attacks more or less constantly. 

First some background. DKIM is a widely used technology to prevent email spoofing. It does so by adding a digital signature linked to the sender’s domain name, for each outgoing email message. This allows the receiver’s email server to check if an email claiming to be from a certain domain is in fact authorized by the owner of that domain name.

Email often travels in multiple hops: it is sometimes redistributed by for example mailing lists or forwarding rules. Therefore, DKIM is designed so that signed messages can be relayed (and consequently, replayed) by any server. It’s a feature of the standard. This feature is however becoming increasingly problematic, as so-called domain reputation is now playing a bigger part in large mailbox providers’ set of anti-spam measures.

What is domain reputation? Many of our readers have probably heard of IP reputation, which is the practice of keeping track of IP addresses that are distributing large volumes of spam. If an IP address is the originator of much spam, it will lead to a poor reputation, and email from that IP address is more likely to be treated as spam. Conversely, an IP address that is known to send much legitimate traffic and very little spam could have a better-than-neutral reputation, and email from that IP is unlikely to be treated as spam. Domain reputation is similar, but considering domain names instead of IP addresses. DKIM is an attractive source of domain information since it is proven by a digital signature. 

What’s the problem? The issue with DKIM and domain reputation is that the DKIM signature only guarantees that the email was once sent from the domain name owner’s server. The same email can be re-sent many times, from any source, and the signature is still valid. As mentioned above, this is a feature of DKIM, necessary for common use cases such as mailing lists to work.

DKIM replay attack. The so-called DKIM replay attack is used by spammers to bypass spam filters at popular mailbox providers known to utilize domain reputation (like Gmail). The spammer does so by sending a single spam to themselves from a provider with a good reputation, and then re-sending that DKIM-signed spam to tons of recipients using the spammer’s own infrastructure.

  1. Identify a popular mailbox provider known to use domain reputation (like gmail.com)
  2. Identify an email provider (let’s call it X) with a good reputation
  3. Sign up for an account at provider X, and send a single spam to yourself
  4. You will now have a spam message which is DKIM signed with X’s domain name
  5. Re-send (replay) that spam message to millions of recipients at @gmail.com, hoping that the good reputation of X’s domain name will result in those messages not being detected as spam
  6. If the domain reputation of X deteriorates, pause sending and wait for X’s reputation to improve before resuming sending

In several cases, the spammers sent many times the provider’s own volume worth of spam. During the Protomail incident in December 2021, the spammers sent 50 times Protonmail’s own volume at one point. 

What can be done? There are some countermeasures and good practices that can be applied by both senders (that DKIM sign) and receivers (that do domain reputation), but no silver bullet as of writing.

On the sending side, both mailbox providers (like telcos or web hosting) and senders (email marketing, ESPs) might be vulnerable, especially if they provide online sign-up. As long as user-generated email messages are DKIM signed with the organization’s domain name, there is a risk. These are some recommendations and mitigation ideas we’ve picked up:

  • Consider a separate domain for new customers if you allow unvetted sign-up.
  • Set an expiration (x=) as short as feasible to limit the time during which replay can be performed, preferably shorter for newly created or suspicious accounts. While the RFC states that the x= tag is not intended as a replay defense, it has proven an effective countermeasure. We did an analysis of the effective lifetime specified by x= for a few large ESPs and noticed that several are using lifetimes of a few days, and in some cases as little as hours or minutes.
  • Re-sign messages in the defer queue as they are being retried, as a way to reduce the expiration time (x=) further. See the example below of how to do that with Halon MTA.
  • Oversign important headers like Date, Subject, From, To, and CC to make sure that they cannot be added after being signed if they are missing from the original message.
  • Sign with the customer’s domain if possible, which is feasible for most email marketing and web hosting companies where customers bring their own domain name. Even if you also sign with your own (provider) domain, the receiver performing domain reputation is likely to use the customer domain if it aligns with the From header.
  • Add a user identity (i=) if you are unable to use a customer domain.
  • Signing the entire message and as many headers as possible to prevent the spammer from changing the message to be even more “spammy” after being signed by your server.
  • Make sure you have a DKIM rotation system in place so that you can easily rotate keys, in the unfortunate event that you need to do so quickly. That being said, the expiration time (x=) should be the primary way to control expiration.
// Re-sign deferred email as they are being retried in pre-delivery script
$mime = MailMessage::File($message["file"]["modified"]);
$dkim = $mime->signDKIM($sel, $domain, $key, ["return_header" => true]);
Try(["additional_headers" => ["DKIM-Signature: ".$dkim]]);

On the receiving side, it’s important that mailbox providers that use DKIM for domain reputation try to detect DKIM replay attacks. I think it’s fair to say that the way mailbox providers use DKIM for domain reputation is the source of the DKIM replay problem. Detecting DKIM replay attacks can be difficult, however. Identifying unusually large volumes of messages with the same signature could be part of the equation, which is also suggested by the RFC. It can be tricky however since legitimate mailing list volumes can be very large too.

I think that the DKIM replay issue highlights why industry groups like M3AAWG are very valuable. They bring organizations together to solve issues and make the internet safer, even if those organizations are competitors or have unaligned business motives. The next meeting is in London, this June. Please let us know if you are interested in collaborating!

Panel discussion on running your own email infrastructure

At the end of last year, Halon was a sponsor of the Winter 2021 Inbox Expo in Valencia, Spain. Amongst the many subjects discussed was the topic of building your own email infrastructure. 

While many organizations rely on outsourcing email sending to email service providers (ESPs), there are plenty of organizations that choose to run their own email infrastructure. During the event, representatives from Mailkit, Postmastery, and Inter7 participated in a panel discussion on the topic. During the session, they explored the landscape of email management, the resources required and some of the potential challenges faced when building your own private email infrastructure.

The panelists in Valencia included:

  • Jakub Olexa. Co-founder of Mailkit (and his new ESP service Omnivery) based in the Czech Republic.
  • Maarten Oelering. Co-founder of Postmastery, a dedicated email infrastructure consultancy established 8 years ago in the Netherlands.
  • Keith Kouzmanoff. Co-founder of Inter7 who specializes in supporting private MTA infrastructure for organizations including GoDaddy, Fastweb, and Playboy. Based in the United States.
Read more.

Halon’s year in review, looking back on 2021

Image by Ian Schneider

First of all, our industry has just celebrated a great anniversary. 2021 was the year that email turned 50 years! The MIT graduate Ray Tomlinson introduced the convention of the ‘@’ symbol to identify a message recipient on a remote computer system. In October 1971, he sent the first email. Today, 50 years later, email remains the most ubiquitous form of communication, and de-facto business communication medium. Email addresses are one of very few widely used, vendor-agnostic, independent addressing schemes, together with phone numbers of physical addresses. We believe that email aligns very well with the principal design of the open internet, and we don’t really see any viable alternatives around the corner. Improving email is at the very heart of everything we do. We’re continuously working hard to make email service providers all over the world more competitive, to avoid unhealthy domination by a few.

It’s also been another year with the pandemic lingering. While we have all been doing our best to enjoy our home offices, meeting each other whenever possible, and being safe, we must highlight how much we miss meeting up with our customers and other people in the email industry. We have been lucky to meet up with a few, but we cannot stress enough how much we’re looking forward to seeing everyone in person again. However, in terms of exchanging great ideas and knowledge within the email space, we’ve been happy to participate in several virtual and hybrid events such as it-sa, Black Hat, Cloud & Cyber Security Expo, Certezza, Inbox Expo, MailCon, Cloudfest, three M3AAWG meetings, and JPAAWG.

We have also had a busy year on the development side, with a focus on supporting DevOps processes, containers, and orchestration. This can improve both speed and reliability for service deployment, especially with larger teams. This effectively makes your email infrastructure more agile. A new plugin architecture that complements Halon script allows us and our customers to employ a greater level of componentization, made convenient through Visual Studio Code extensions and new package repositories. Developing in Halon script can be made more efficient with Docker Desktop templates, and debugging Halon script running within the MTA directly from Visual Studio Code by leveraging its breakpoint interface. Communication between components is designed to work with the dynamic scaling of orchestration tools, and we now provide Docker and Kubernetes templates.

On top of that, we have released a plugin for BIMI verification. It does the heavy lifting for mailbox providers looking to show brand images in their users’ inboxes. We have also compiled Halon MTA for the ARM processor architecture, evaluating performance and feasibility to run on for example AWS Graviton.

When looking back at the year we also need to highlight our growth – and, what a year it has been in terms of welcoming new Halonites to the Halon team! We’re beyond happy to have been welcoming so many new, ambitious and knowledgeable talents. It feels fantastic to be growing and to see just how much we can accomplish together.

These are just some of the things we have been up to in 2021 and we are very happy to call this another successful year. Now, we are beyond excited for the year ahead. It will be an eventful one with lots of things planned. Already now, we have lots of open positions in engineering, marketing, and sales. So, if you are interested in joining a fast-growing tech company, we would love to hear from you.

Halon 5.8 with container orchestration

Image by Phil Roeder

The snow is deep, Christmas is just around the corner, and so is our next quarterly Halon MTA release 5.8 codename “packy”. It comes packed with improvements for both senders and mailbox providers; most notable around packaging and container orchestration readiness with for example Docker and Kubernetes.

It all starts with added componentisation aided by our new RPM and APT/deb repositories. The MTA and its different components can now be installed and scaled separately. There is no need for a 1:1 mapping between for example MTA and rate limiting or content filtering instances. Communication between components is designed to work with the dynamic scaling of Kubernetes. In order to get started quickly, we provide templates for Docker and Kubernetes. The new repositories also includes pre-compiled plugins from the extension library for quick and easy installation. This is the natural evolution of the previous release’s DevOps orchestration focus, allowing you to deploy and scale Halon MTA in a way that is just as flexible and modular as the software itself. The result is a more economical and scalable deployment that can respond to changing demands instantaneously.

Our large-scale senders will appreciate our new performance-optimised HTTP submission API. It’s ideal for injecting pre-formatted RFC822 messages directly into the end-of-DATA script hook. It’s configured just like SMTP endpoints, and accepts options in JSON format:

$ curl --request POST \
   --header "X-API-Key: badsecret" \
   --form "[email protected];type=application/json" \
   --form "[email protected];type=message/rfc822" \
   https://mta.hostname:443/v1/submission

Speaking of sending performance; now that DKIM signing with EdDSA (ed25519) is becoming more popular, we’ve added capabilities for faster dual signing using the additional_signatures options to signDKIM(). Halon is one of very few to offer this optimisation, which provides a significant boost for one of the most CPU-intensive parts of email delivery.

The native plugin (C ABI) system has received many additions. We’ve added APIs for creating Halon script classes/objects, throw Halon script exceptions, executing Halon script callback functions and two new functions for encoding/decoding JSON directly from Halon script variables. You can see many of those new APIs hard at work in the halon-extra extension library on Github.

As usual, the release comes with tons of other improvements. It’s now possible to configure multiple queue spool paths, which can be useful when scaling queue storage performance in certain environments. The active queue policies have received multiple updates, including a shortcut for creating lists of items (like domains) in order to avoid repetition, as well as the ability to pass properties from matching policies to the post-delivery script which is useful when creating custom backoff rules. Analysing realtime process metrics and tuning performance-related settings is now easier with the halontop command.

We hope that you will enjoy this new quarterly release. Don’t hesitate to reach out if you have any questions or want to learn more about what we do!

The past, present and future; TLS for email

Image by Michele M. F.

Let’s start our quick history lesson by setting the scene; Madonna was topping the charts and Kevin Mitnick had just been arrested after nearly three years on the run from the authorities. It’s February 1995 and researchers at Netscape had just released SSL version 2.0, much of the work was credited to Taher Elgamal, often referred to as the father of SSL. SSL version 1.0 was being worked on before that, but was never published.

Things didn’t start out great; SSL 2.0 was quickly discovered to contain a few bad flaws making it vulnerable to man-in-the-middle attacks, amongst other things. It was also impractical to implement especially for web hosting providers since it only supported a fixed domain certificate. SSL 3.0 addressed the issues found in SSL 2.0, and has been in use since 1996. It wasn’t until the discovery of the POODLE attack in 2014 that things started looking grim for SSL 3.0, which was subsequently deprecated in 2015.

Read more.

Black Friday Cyber Monday: Make sure you deliver in time

Image by Mike Cohen

During the busiest shopping season, namely Black Friday Cyber Monday (BFCM), an uptick in email volumes is often a given, but there are a number of things you can do to make sure your email sending performs at its best without getting yourself caught up in the net that is cast to catch the spammers. Are you a brand marketer, or providing email sending services to marketers?

Read more.

Discussing email encryption at JPAAWG #4

Photo by Luca Sartoni

Halon is presenting at, and sponsoring, the 4th general meeting of JPAAWG. It’s an independent regional offshoot of the global Messaging, Malware and Mobile Anti-Abuse Working Group backed by long-time M3AAWG members IIJ and TwoFive. M3AAWG is where the industry comes together to work against botnets, malware, spam, viruses, DoS attacks and other online exploitation. It’s the largest industry association, with more than 200 members worldwide, bringing together all the stakeholders in the online community in a confidential, open forum.

I will be speaking about the latest email authentication technologies together with Hackim Farrell from Proofpoint; specifically transport encryption with DANE and MTA-STS. Until recently, there was no widely deployed technology for properly protecting email communication from eavesdropping. We have been advocating for improved email encryption for a long time at conferences and industry meetings all over the globe. We’ve nudged and encouraged, and is very happy to see that there’s now 3 million email domains with DANE, and that both Microsoft and Google has enabled MTA-STS for their consumer mailbox services.

Are you interested in taking part, and happen to know one of the members? If so, I recommend reaching out and applying to attend an upcoming meeting as their guest! If not, we’re happy to guide you through the process. Don’t hesitate to contact us if you want to know more about what we do at M3AAWG, or want to talk about email in general.

The great ARM wrestle; Graviton and RSA

With the introduction of both Apple Silicon (M1) and AWS Graviton, the use of ARM processors has really started to enter the mainstream desktop and server market. Long-term, things are looking really good for ARM in the datacenter, but are we there yet?

The ARM-based Apple Silicon has been praised for delivering good performance with low power usage and heat generation, which is perfect for a portable device. AWS Graviton 2 performs well in a performance to cost perspective as well. Even Microsoft is dipping its toes in the water, so obviously there’s something to this. So is it any good for our Halon MTA? Well, the short answer is that it depends.

Let’s start with the desktop side of things. With our Visual Studio Code extensions it’s very easy to run a local containerised MTA for testing purposes. Some of us here at Halon use Apple laptops with the new M1 chip, and everything works fine in x86 mode under Rosetta 2, but we of course wanted to test it natively! Did it work without issues? Absolutely. Was it exceptional? Yes, in a few ways it was. Benchmarking Halon MTA on Apple M1 against similar x86-based desktops and even Xeon servers reveals excellent per-core performance. DKIM signing and verification performance really stood out, beating the latest (and significantly more power hungry) Intel Xeon processors with the same numbers of cores. However, the workload that you throw at it when testing your configuration and scripts is honestly not enough to really make any change in performance noticeable. In the screenshot below, you can see Halon MTA running natively and containerised on an ARM-based Mac.

Given what you just read about Halon MTA running on Apple’s ARM chips, surely the ARM-based c6g (Graviton 2) instances must be perfect for running Halon MTA on Amazon AWS? Well, here it gets a little complicated. During our first benchmarks of c6g.2xlarge instances (ARM) against c5.2xlarge (traditional x86) we saw a steady performance increase of around 20%, while being roughly 20% cheeper. Fantastic! TLS speeds with AES-128 were also phenomenal, about 30% faster. But that was until we started testing more complex workflows.

When testing isolated DKIM signing with RSA 1024 and EdDSA (elliptic curve), the x86-based c5.2xlarge was about 20% faster than the ARM-based c6g.2xlarge. With RSA 2048, the difference was even bigger; with the x86-based instance being more than twice as fast. Consequentially, when benchmarking a typical email sending setup with DKIM signing (using RSA 2048) and queuing to EBS storage, the signing slowed down the ARM-based c6g.2xlarge instance to the point where it no longer made economical sense. This was verified using OpenSSL’s speed command testing RSA 2048 per-core performance, listed in the table below.

Machine Architecture Sign / s Verify / s
AWS c5.2xlargex861 68556 671
AWS c6g.2xlargeARM29310 720
Apple Mac Mini M1ARM1 76773 035

This might appear very strange, given the exceptional per-core RSA performance of Apple’s M1 chip. Evidently, it’s more about specific CPUs rather than the architecture itself. The following tweet suggests the Amazon AWS team are aware of the RSA performance issues.

It’s possible that this should be attributed to ARM’s Neoverse N1 platform on which Graviton 2 is based, because Cloudflare’s benchmarks of another N1-based CPU called Altra from Ampere seems to have similar RSA performance.

To summarise; ARM can be fast, and is almost certainly here to stay. Is it good for running an MTA server such as Halon? It could be, but it very much depends on the workload, configuration and the type of ARM chip. Fingers crossed for Graviton 3 addressing those RSA issues.

All tests where performed on AWS c5.2xlarge and c6g.2xlarge instances, as well as a Mac Mini with the M1 chip.

Adding BIMI to your mailbox service to improve security and user experience

Image by Valimail

BIMI (Brand Indicators for Message Identification) is an emerging industry standard which attempts to drive adoption of strong email authentication, to help fight phishing, spoofing, and fraudulent emails by displaying verified brand images in the inbox. Several mailbox providers, including Gmail, AOL and Fastmail, are now live with BIMI. Are you looking to increase the security and experience for the users of your mailbox service by implementing this new standard as well? With Halon MTA, you’re ready to get started. Our BIMI module does the heavy lifting; verifying the indicators for inbound email and attaching the images to the email for display in the users’ inbox. Currently there are very few MUAs with BIMI support, but if you develop your own webmail or app the BIMI group has guidance for how to display the verified logo in your user interface.

What BIMI is, and how it works

In short, BIMI displays verified brand images in users’ inboxes, visible even before messages are opened. It builds upon the existing email authentication standards (SPF, DKIM and DMARC), and incentives email senders to embrace those technologies by increasing their brand impression. According to Agari, only 24% of Fortune 500 sending domains are authenticated. BIMI improves visibility and engagement for the brands, while at the same time increasing security for users. It roughly consists of the following steps;

  1. The brand sending the email first needs to implement email authentication (enforcing DMARC) for their domain name.
  2. The brand then creates an appropriate image of their trademarked logo, optionally applies for a Verified Mark Certificate (VMC), stores it on a HTTPS location, and points at it with a BIMI DNS record.
  3. When a BIMI-enabled mailbox service (like Gmail) receives an email from the brand to a specific recipient, the mailbox service’s MTA first verifies that the email is in fact from the sending brand using email authentication like DMARC.
  4. If the email is successfully authenticated, the mailbox service’s MTA checks if the sending brand’s domain has BIMI, and if so, verifies that logo against the VMC certificate authorities or using some other reputation system.
  5. If the brand’s BIMI logo was successfully verified, it is attached to the email (for example as a header), before the email is shipped off the recipient’s inbox.
  6. Finally, the recipient looks at her inbox using the mailbox service’s webmail or app, which displays the brand’s logo next to the address and subject in the email listing.
Getting started with BIMI validation on Halon MTA

Halon was the first commercial MTA to add support for many great new security standards such as DMARC, EdDSA DKIM (elliptic curve), DANE and MTA-STS. To our knowledge, we’re the first to add BIMI validation as well. Our BIMI validation module is only 182 lines of Halon script, which I think is pretty impressive (especially considering that it includes a basic ASN.1 parser). To get started with BIMI on Halon MTA, just check out the BIMI module to your configuration repository, add the following to your end-of-DATA script hook. The module uses some X.509 functions from the unreleased Halon MTA 5.8, please let us know if you want to try it out. As for showing the images in your webmail or app, we’re happy to provide guidance and examples.

import $calist from "bimi/ca.crt";
import { bimi, bimi_vmc } from "bimi/bimi.hsl";
import { dmarc } from "dmarc/dmarc.hsl";

$dmarc = dmarc($mail, $senderip, $senderhelo, $senderdomain);
$bimi = bimi($mail, $dmarc);

// Verified Mark Certificate (VMC)
if ($bimi["record"]["a"]) {
	$bimi_vmc = bimi_vmc($bimi, $calist);
	if ($bimi_vmc["indicator"]) {
		$mail->addHeader("BIMI-Indicator",
			str_strip(array_join(pcre_match_all(#/(^(.{0,49})|(.{0,64}))/, 
					$bimi_vmc["indicator"])[1], "\r\n ")),
			["encode" => false]);
	} else
		echo "BIMI VMC error: $bimi_vmc; $bimi";
} else {
	if ($bimi["error"] and $bimi["class"] != "dmarc" and $bimi["class"] != "dns")
		echo "BIMI error: $bimi";
}

Thank you for reading, and don’t hesitate to reach out if you want to learn more!