Browse

Tag: tech

Adding BIMI to your mailbox service to improve security and user experience

Image by Valimail

BIMI (Brand Indicators for Message Identification) is an emerging industry standard which attempts to drive adoption of strong email authentication, to help fight phishing, spoofing, and fraudulent emails by displaying verified brand images in the inbox. Several mailbox providers, including Gmail, AOL and Fastmail, are now live with BIMI. Are you looking to increase the security and experience for the users of your mailbox service by implementing this new standard as well? With Halon MTA, you’re ready to get started. Our BIMI module does the heavy lifting; verifying the indicators for inbound email and attaching the images to the email for display in the users’ inbox. Currently there are very few MUAs with BIMI support, but if you develop your own webmail or app the BIMI group has guidance for how to display the verified logo in your user interface.

What BIMI is, and how it works

In short, BIMI displays verified brand images in users’ inboxes, visible even before messages are opened. It builds upon the existing email authentication standards (SPF, DKIM and DMARC), and incentives email senders to embrace those technologies by increasing their brand impression. According to Agari, only 24% of Fortune 500 sending domains are authenticated. BIMI improves visibility and engagement for the brands, while at the same time increasing security for users. It roughly consists of the following steps;

  1. The brand sending the email first needs to implement email authentication (enforcing DMARC) for their domain name.
  2. The brand then creates an appropriate image of their trademarked logo, optionally applies for a Verified Mark Certificate (VMC), stores it on a HTTPS location, and points at it with a BIMI DNS record.
  3. When a BIMI-enabled mailbox service (like Gmail) receives an email from the brand to a specific recipient, the mailbox service’s MTA first verifies that the email is in fact from the sending brand using email authentication like DMARC.
  4. If the email is successfully authenticated, the mailbox service’s MTA checks if the sending brand’s domain has BIMI, and if so, verifies that logo against the VMC certificate authorities or using some other reputation system.
  5. If the brand’s BIMI logo was successfully verified, it is attached to the email (for example as a header), before the email is shipped off the recipient’s inbox.
  6. Finally, the recipient looks at her inbox using the mailbox service’s webmail or app, which displays the brand’s logo next to the address and subject in the email listing.
Getting started with BIMI validation on Halon MTA

Halon was the first commercial MTA to add support for many great new security standards such as DMARC, EdDSA DKIM (elliptic curve), DANE and MTA-STS. To our knowledge, we’re the first to add BIMI validation as well. Our BIMI validation module is only 182 lines of Halon script, which I think is pretty impressive (especially considering that it includes a basic ASN.1 parser). To get started with BIMI on Halon MTA, just check out the BIMI module to your configuration repository, add the following to your end-of-DATA script hook. The module uses some X.509 functions from the unreleased Halon MTA 5.8, please let us know if you want to try it out. As for showing the images in your webmail or app, we’re happy to provide guidance and examples.

import $calist from "bimi/ca.crt";
import { bimi, bimi_vmc } from "bimi/bimi.hsl";
import { dmarc } from "dmarc/dmarc.hsl";

$dmarc = dmarc($mail, $senderip, $senderhelo, $senderdomain);
$bimi = bimi($mail, $dmarc);

// Verified Mark Certificate (VMC)
if ($bimi["record"]["a"]) {
	$bimi_vmc = bimi_vmc($bimi, $calist);
	if ($bimi_vmc["indicator"]) {
		$mail->addHeader("BIMI-Indicator",
			str_strip(array_join(pcre_match_all(#/(^(.{0,49})|(.{0,64}))/, 
					$bimi_vmc["indicator"])[1], "\r\n ")),
			["encode" => false]);
	} else
		echo "BIMI VMC error: $bimi_vmc; $bimi";
} else {
	if ($bimi["error"] and $bimi["class"] != "dmarc" and $bimi["class"] != "dns")
		echo "BIMI error: $bimi";
}

Thank you for reading, and don’t hesitate to reach out if you want to learn more!

Halon 5.7 brings DevOps to email engineering with Visual Studio Code

The Halon MTA is the most developer-friendly platform for email transport and security. With version 5.7 “catchy” we’re very happy to introduce an even better experience when using Halon together with Visual Studio Code and Docker Desktop.

Visual Studio Code is one of the most popular source code editor applications, made available for free and open source by Microsoft. It includes many features such as version control with Git, and has tons of available extensions. We have been using one of Visual Studio Code’s fundamental underpinnings, the Monaco code editor, in Halon’s web administration code editor since 2016. In 2018 we released a Visual Studio Code extension for working with both Halon script and YAML configuration files.

More and more customers embrace DevOps processes which combine version control (like Git), provisioning tools (like Chef, Puppet and Ansible) and containers (like Docker). This can improve both the speed and reliability of service development, especially with larger teams and deployments. In order to deliver the best possible experience regardless of which of those technologies our customers’ use, we’re now providing the different Halon components in smaller packages so that they can be integrated into your workflow. For example, you can now have the configuration packer and syntax highlighting in Visual Studio Code, connected to a local Docker instance running the Halon MTA and debugger. The container template enables you to set up a local development environment using Docker Desktop within minutes, complete with all MTA components.

Our extensions deliver syntax highlighting, code completion, built-in documentation, linting (static code analysis), debugging, staging and deployment. One of the most requested features is the debugger, leveraging Visual Studio Code’s breakpoint interface. It can be used on individual modules running inside the Halon script interpreter (hsh), or on a complete deployment by attaching to the MTA process (smtpd) of either a container or VM over SSH. Attaching to the MTA enables you to peek into the full connection and transaction context during not only development and staging, but also during for example troubleshooting of live traffic on production instances. Play the video below to see it in action!

What’s even more powerful is the combination of live debugging with what we call live staging; integrated blue-green testing which allows you deploy a new configuration (complete with all script and modules) for a small, select portion of traffic. It really highlights the unique capabilities which become possible because we develop the MTA, email scripting language and development environment hand-in-hand; as one product with a clear vision.

See the release notes and detailed changelog for a full list of changes. We hope that you will enjoy this new quarterly release. Don’t hesitate to contact us if you want to learn more about what we do, and how we enable email providers to build better services!

Not all successful DSN deploys are successful

Our latest release 5.5 adds support for the RFC 3461 DSN extension, and published a guide on how to get started. At first glance it may seem like a very nifty feature, and one may wonder why it’s not just enabled by default. Fact it, the DNS extension comes with a few peculiarities which need to be considered.

In many email environments, the receiver has multiple store-and-forward MTAs before the message reaches the recipient’s inbox. Each MTA in the path has a different purpose, and may reject the message (if it is spam, for example) rather than deliver it forward. This is what requires MTAs to send “failed” DSN notifications; when an MTA which isn’t your first (edge) rejects a message. The sender always has to be notified as per the transaction safety requirement of email.

One of the features provided by the DSN extension is that you may also request “delivered” notifications by the last MTA before the recipient’s inbox (or “relayed” notifications by the last MTA supporting the DSN extension). This lets the sender know when and if the message was delivered.

Thinking about it, this feature may be nice to have enabled in your MTA for outbound email. It would let you or your users know if and when a message is delivered; rather than only notifying you when and if delivery fails.

If you implement this feature for inbound traffic however, it would be one of your own MTA that would have to send those notifications out to anyone on the Internet that requested them. The notifications would go out from the last MTA in your inbound path. When messages are processed and routed internally, additional information that should not be visible to the sender are sometimes added. Allowing the sender to request a notification possibly containing some of this possibly sensitive information might not be ideal. This concern has be raised many times before, for example in the Exim documentation

Note: Supplying success-DSN messages has been criticised on privacy grounds; it can leak details of internal forwarding.

During our implementation and testing we enabled success notifications for some of our outbound traffic, to see which mailbox providers send such notifications. We were quite surprised to learn that Microsoft includes quite a lot of information regarding the recipients in their success notifications. We instantly reported this to the MSRC team but it was rejected as not being an issue 🙈

Amongst those hundred of lines of X-MS- headers you receive in their success notifications, I’m not sure if the last login time of the recipient is necessary. This is in fact the actual time when the recipient (eg. [email protected]) last checked their email:

X-MS-UserLastLogonTime: 10/1/2020 12:06:31 AM

Bottom line is; should you choose to implement this feature, make sure you do not add sensitive or unnecessary information to your success notifications that may expose your network internals or your customers’ personal information.

Halon 5.5 with DSN extension support

Winter is here, and so is our next quarterly Halon MTA release: 5.5 codename “mappy”. It’s slang for happy on a Monday, which is quite fitting! It comes packed with new features, functions and improvements which opens up even greater possibilities for building differentiated, efficient services.

Our large-scale senders will appreciate additional tools for IP warmup, ensuring deliverability by complying with receiver guidelines. You can now choose the order by which source IPs in a pool are chosen, as opposed to load balancing between all IPs. It allows you to use one IP primarily until it exceeds the threshold for daily warmup for a certain destination. Another feature which is especially useful for senders looking at migrating their email infrastructure to cloud environments (such as AWS) is outbound PROXY support. It enables you to use the powerful HAProxy load balancer rather than NAT, when sharing IPs between MTA instances.

Do you want to leverage the DSN extension to better track email delivery? If so, you’ll appreciate our fully-scriptable RFC 3461 implementation. It allows you to quickly and easily tailor the DSN behaviour exactly to your liking.

The Halon scripting language has been extended with new data storage classes, generator functions with yield, and several new functions for encoding. The strongly-typed Map() and Set() storage classes are performance-optimised alternatives to the more generic array type. Take our revamped DMARC implementation‘s public suffix list lookup for example. It parses a data file, then storing and caching the results in a Set() for efficient lookup of the slightly complex rules:

$rules = memory_fetch("public_suffix_list.dat", function ($k) {
		$rules = Set("string");
		$file = File("file://public_suffix_list.dat");
		if (!$file)
			return false;
		while (is_string($line = $file->readline()))
		{
			$line = str_strip($line);
			if ($line[0] == "/" or $line == "")
				continue;
			$rules->add($line);
		}
		memory_store("public_suffix_list.dat", $rules);
			return $rules;
	});

The integrated (VM) package has received several requested features. First of all, the built-in web server which hosts the web administration and JSON API can now be used to also host custom PHP scripts. It allows you to create additional API endpoints, which can write files used by the Halon script during email processing. One possible use case for this is to receive requests from legal interception systems. Secondly, it is now possible to add local DNS records to the built-in Unbound DNS resolver from the web administration. It can be useful when split-horizon DNS is unavailable, and you need to override some specific DNS record. Finally, it allows you to sort the built-in history on both finished and received time. As usual, the package is based on the latest version of FreeBSD (12.2, which was released a few weeks ago) and comes with updated third-party components.

We hope that you will enjoy this new quarterly release. Please see the release notes for more information, and don’t hesitate to reach out if you have any questions!

Would you like to “like” an email?

Social media and instant messaging (such as Slack) has made reactions like “thumbs up” 👍 and other emojis popular additions to everyday communication. Wouldn’t it be nice if you could quickly press like to confirm or endorse something proposed in an email, rather than having to write a reply?

Dave Crocker, well known for his work on email standards and a senior advisor for M3AAWG, recently submitted the draft “React: Indicating Summary Reaction to a Message”. It introduces email to the world of reaction emojis using an additional email header. It could be implemented in the email clients or webmail (MUAs), and doesn’t require any changes to the transport layer. It’s however an early draft with several considerations yet to be discussed, so I wouldn’t hold my breath just yet.

So, what could it look like? I don’t know, but it didn’t stop me from creating the mockup below!

Halon 5.4 with tailored bounces

Image by Dean Hochman

Sometimes you want to customise the bounce generation. It could be anything from translating bounces into another language or making them more user-friendly, to implementing standards such as SRS. In order to do so, you’ve had to override Halon’s default bounce generation. In our new release Halon 5.4 “bouncy”, we’ve added many options to the built-in bounce generator. This enables you to quickly tailor the way bounces look and work, without having to reimplement the default generator.

Queue([
  "delay" => 3600,
  "dsn_delayed" => true,
  "dsn" => [
    "readable_mimepart" => MIME()
      ->setType("text/html")
      ->setBody("<em>choo choo train is delayed</em>")
      ->toString(),
    "original_headers" => false,
    "subject_prepend" => "Not making it in time: ",
    "headers" => [
      "Foo: Bar"
    ]
  ]
]);

As usual, the release comes with many other improvements. The foreign function interface (FFI), which is used for loading external libraries into the MTA executable, can now export File classes as a C++ std::istream and X.509 resources as OpenSSL pointers. This powerful feature enables you to access the (potentially modified) email body as a virtual file. We’ve also added chunking (BDAT) for both sending and receiving, a zero-fill right shift bitwise operator, a data callback to the http() function, FFI function callbacks, and much more. Please see the release notes for a complete list of changes. We hope that you will enjoy this release. If you are new to Halon, don’t hesitate to contact us if you have any questions.

Why concurrency is important for large senders

Photo by Giuseppe Milo

Senders of large email volumes rely on the concept of multiple virtual queues for efficient delivery; separating email from different customers, to different destinations. This guarantees that for example one customer with a gigantic queue, or a destination which is tarpitting, doesn’t affect delivery of other traffic. Email transactions can take some time to complete, often as a result of content filtering at the destination. It is therefore necessary to support many simultaneous (concurrent) connections in each virtual queue, to support the throughput required for timely delivery. If each message takes an average of 1 second to deliver, 50 concurrent connections are needed to deliver 50 email per second.

The total maximum concurrency needed is the number of virtual queues multiplied with the desired concurrency for each queue. Traditional MTAs uses a process- or thread-based connection model. Those models scales poorly, and consequentially struggles to support more than a few thousand concurrent connections per server. This limitation becomes a problem as the system is scaled up, leading to bottlenecks or low customer density per MTA instance.

The Halon MTA uses a modern event-based connection model, sometimes called asynchronous or non-blocking. It scales exceptionally well; supporting tens of thousands of concurrent connections. Multiple worker threads are used to leverage all available CPU cores. This eliminates bottlenecks and can reduce the number of instances needed, which translates to lowered costs and simplified management.

Being asynchronous have benefits for receiving and processing inbound traffic as well. Consider the scenario where thousands of slow, or even idle, clients are connected to an MTA. Traditional MTAs would have to defer new connections from potentially legitimate senders, disturbing the service. Asynchronous MTAs like Halon are on the other hand be able to cope with a significantly large number of concurrent connections, and would handle the scenario above without breaking a sweat.

If you’re interested in learning more, don’t hesitate to reach out.

Halon 5.3 with new powerful queue API

Soap drop by Breic

Since the first Halon MTA release in 2008, we’ve had a text-based queue query syntax called HQL (a play on SQL) as part of our SOAP API. While it has served us well during all those years, it was time to move on to something more modern. The new 5.3 release (codename “buffy”) comes with a Protocol Buffers and JSON API which introduces a programmatic approach to queue operations.

The request and response body schemas are available on our Github page. For your convenience, the QueueList, QueueGroupByQueueUnload and QueueUpdate requests all have the same Condition argument. Those API calls can operate on both the active and defer queues, as well as messages on hold. That is why the condition argument both contains things like retry count, as well as resolved remote MX and IP. You can specify as many conditions as you like, and create logic-or expressions by specifying multiple conditions of the same type. There are exact matching, regular expressions, and intervals for integers and date/time. Needless to say; incredibly powerful.

The QueueGroupBy call returns the distribution based on the grouping parameters and intervals you choose; such as number of messages in various age buckets, grouped by recipient domain. This is useful for getting an overview of a large queue. Queries are blazingly fast, even with very large queues. All queue metadata (essentially the fields available as conditions) is loaded into memory, in order for the virtual sub-queues to work.

Halon 5.3 also comes with a new CLI called halonctl. It happens to be very useful when working with our API, as it can output the API request and response bodies for the command you run in JSON format. As you can se in the example below, the request body is printed first: 

$ halonctl queue update --bounce --state DEFER --jobid foobar --json-request --json
{
    "conditions": {
        "queues": [
            {
                "queue": "DEFER"
                ...

The CLI covers all the functionality of the product, and is a great complement to the web administration. Its configuration management sub-commands are useful for integrating Halon MTA instances into provisioning, deployment and CI/CD toolchains such as Puppet or Chef, where running commands is easier than making API calls.

Halon 5.3 comes with many other great improvements; such as connection pooling, a more efficient queue quota function, a new on-disk queue format and an Iconv() class for internationalisation conversion. Please see the release notes for a complete list of changes. We hope that you will enjoy this release as much as we do! If you are new to Halon, don’t hesitate to contact us, or dig into all our documentation that is available publicly on our website.

Halon 5.2 with ultra-flexible queuing

Photo by Karen Roe

We’re very proud to announce the upcoming 5.2 release “polly” which introduces a powerful queue policy engine. First and foremost, the queue and SMTP client’s network layer is now asynchronous. This allows an instance to handle tens of thousands of parallel connections. In combination with the reworked connection concurrency limits, this allows dynamic creation of a virtually unlimited number of independent sub-queues. This is useful for senders that need to separate email streams so that those that move slowly or get stuck don’t block others.

As usual, we made it flexible enough to fit any email service provider’s needs. Rather than having a fixed set of parameters and rollup/grouping options for establishing the sub-queues (with their respective thresholds), we allow you to define what constitutes a unique entry. You can choose any combination of fields, and group/rollup entries using regular expressions or wildcard. In the example below, we limit the concurrent per source IP and remote MX, and also rollup all Google’s MX entries into the same entry. The default concurrency is 5, except Google that gets 10.

- fields:
  - localip
  - remotemx:
      gsuite:
        - '*.google.com'
        - '*.googlemail.com'
        - '*.smtp.goog'
  conditions:
  - if:
      remotemx: '#gsuite'
    then:
      concurrency: 10
      rate: 50
  default:
    concurrency: 5
    rate: 10


Sometimes rollup per MX doesn’t cut it. There are several Microsoft Office365 locations (clusters), but the customer MX doesn’t reveal which they are on. To set a certain threshold for Office365 locations, we can rollup and match per MX, but limit per IP, as per the example below. Note that there’s no default threshold; it only affects Office365.

- fields:
  - localip
  - remotemx:
      o365:
       - '*.protection.outlook.com'
  - remoteip
  conditions:
  - if:
      remotemx: '#o365'
    then:
      concurrency: 10
      rate: 30

Thresholds and suspensions can be modified on the fly without reloading the configuration via
API, CLI, web administration or the MTA itself through this Halon script:

// If we have more than 10 failures per minute, lower rate for 5 minutes
$mx = $arguments["attempt"]["connection"]["remotemx"];
$code = $arguments["attempt"]["result"]["code"];
if ($mx and $code >= 400 and !rate("mx-fail", $mx, 10, 60, ["sync" => false]))
    cache ["ttl" => 300]
        PickupPolicy(
            ["localip", "remotemx"],
            ["remotemx" => $mx],
            ["rate" => [10, 60]], 300);

The reworked queue naturally comes with many new tools and APIs for interacting with the new functionality. This includes more subtle improvements, like the ability to view the queue’s shape by message age. By pressing an interval, you can dig into the specific messages, which are grouped by fields of your choice.

The new shared memory script functions and API opens up several possibilities. You can script statistic counters, which can then be read periodically over the API. Another use case is pre-loading data into the MTA over the API, rather than fetching and caching from within the script.

Finally, we now offer the ability to call methods in external shared libraries using our foreign function interface (FFI) class.

Check out the full changelog on GitHub for more information, and familiarise yourself with the important changes outlined in the release notes document before upgrading.

Implementing SMTP LANG with proxy script

The upcoming Halon 5.1 release introduces a new SMTP server proxy script. It’s configured to be executed before specific (or all) SMTP commands, even command which isn’t recognised by the SMTP server. In this blog we’ll describe how to implement our proposed SMTP LANG extension using this new script hook. First of all, we announce the LANG extension and the languages supported in the HELO script:

$announce =  [...$arguments["extensions"], "LANG SE"];
Accept(["extensions" => $announce]);

Next, we configure to proxy script to hook onto the new “BREV”, “INNEHÅLL” and “HEJDÅ” commands:

Finally, we setup the proxy script to translate the international (Swedish, in this case) commands with their standard SMTP respective:

$cmd = str_upper($arguments["command"]);
if ($cmd[0:11] == "BREV FRÅN:") { // MAIL FROM
    $realcmd = "MAIL FROM:" + $arguments["command"][11:];
    Pass(["command" => $realcmd]);
}
if ($cmd == "HEJDÅ") { // QUIT
    $codes = ["code" => 221, "enhanced" => [2, 0, 0]];
    Reply("Ha det bra", ["reply_codes" => $codes, "disconnect" => true]);
}
// etc...

The proxy script and the corresponding xtext functions can be used for many other things, like custom XCLIENT implementations. More information on the 5.1 will be released later this month.