Browse

Blog

Halon 5.3 with new powerful queue API

Soap drop by Breic

Since the first Halon MTA release in 2008, we’ve had a text-based queue query syntax called HQL (a play on SQL) as part of our SOAP API. While it has served us well during all those years, it was time to move on to something more modern. The new 5.3 release (codename “buffy”) comes with a Protocol Buffers and JSON API which introduces a programmatic approach to queue operations.

The request and response body schemas are available on our Github page. For your convenience, the QueueList, QueueGroupByQueueUnload and QueueUpdate requests all have the same Condition argument. Those API calls can operate on both the active and defer queues, as well as messages on hold. That is why the condition argument both contains things like retry count, as well as resolved remote MX and IP. You can specify as many conditions as you like, and create logic-or expressions by specifying multiple conditions of the same type. There are exact matching, regular expressions, and intervals for integers and date/time. Needless to say; incredibly powerful.

The QueueGroupBy call returns the distribution based on the grouping parameters and intervals you choose; such as number of messages in various age buckets, grouped by recipient domain. This is useful for getting an overview of a large queue. Queries are blazingly fast, even with very large queues. All queue metadata (essentially the fields available as conditions) is loaded into memory, in order for the virtual sub-queues to work.

Halon 5.3 also comes with a new CLI called halonctl. It happens to be very useful when working with our API, as it can output the API request and response bodies for the command you run in JSON format. As you can se in the example below, the request body is printed first: 

$ halonctl queue update --bounce --state DEFER --jobid foobar --json-request --json
{
    "conditions": {
        "queues": [
            {
                "queue": "DEFER"
                ...

The CLI covers all the functionality of the product, and is a great complement to the web administration. Its configuration management sub-commands are useful for integrating Halon MTA instances into provisioning, deployment and CI/CD toolchains such as Puppet or Chef, where running commands is easier than making API calls.

Halon 5.3 comes with many other great improvements; such as connection pooling, a more efficient queue quota function, a new on-disk queue format and an Iconv() class for internationalisation conversion. Please see the release notes for a complete list of changes. We hope that you will enjoy this release as much as we do! If you are new to Halon, don’t hesitate to contact us, or dig into all our documentation that is available publicly on our website.

Just as cool as building it from scratch – why an email platform is your new best friend

For those of you working with larger amounts of traffic, which solution is better; building an email platform from scratch, or using an email security and delivery platform (ESDP) service? In this blog post, we describe both solutions so you can determine which one suits your company best.

Read more.

Why implementing an email platform will go much quicker than you think

Thinking about implementing a new email platform, but keep postponing the process since it seems way too time-consuming? Here we describe the whole process to show you know how fast it can be done!

Read more.

Halon 5.2 with ultra-flexible queuing

Photo by Karen Roe

We’re very proud to announce the upcoming 5.2 release “polly” which introduces a powerful queue policy engine. First and foremost, the queue and SMTP client’s network layer is now asynchronous. This allows an instance to handle tens of thousands of parallel connections. In combination with the reworked connection concurrency limits, this allows dynamic creation of a virtually unlimited number of independent sub-queues. This is useful for senders that need to separate email streams so that those that move slowly or get stuck don’t block others.

As usual, we made it flexible enough to fit any email service provider’s needs. Rather than having a fixed set of parameters and rollup/grouping options for establishing the sub-queues (with their respective thresholds), we allow you to define what constitutes a unique entry. You can choose any combination of fields, and group/rollup entries using regular expressions or wildcard. In the example below, we limit the concurrent per source IP and remote MX, and also rollup all Google’s MX entries into the same entry. The default concurrency is 5, except Google that gets 10.

- fields:
  - localip
  - remotemx:
      gsuite:
        - '*.google.com'
        - '*.googlemail.com'
        - '*.smtp.goog'
  conditions:
  - if:
      remotemx: '#gsuite'
    then:
      concurrency: 10
      rate: 50
  default:
    concurrency: 5
    rate: 10


Sometimes rollup per MX doesn’t cut it. There are several Microsoft Office365 locations (clusters), but the customer MX doesn’t reveal which they are on. To set a certain threshold for Office365 locations, we can rollup and match per MX, but limit per IP, as per the example below. Note that there’s no default threshold; it only affects Office365.

- fields:
  - localip
  - remotemx:
      o365:
       - '*.protection.outlook.com'
  - remoteip
  conditions:
  - if:
      remotemx: '#o365'
    then:
      concurrency: 10
      rate: 30

Thresholds and suspensions can be modified on the fly without reloading the configuration via
API, CLI, web administration or the MTA itself through this Halon script:

// If we have more than 10 failures per minute, lower rate for 5 minutes
$mx = $arguments["attempt"]["connection"]["remotemx"];
$code = $arguments["attempt"]["result"]["code"];
if ($mx and $code >= 400 and !rate("mx-fail", $mx, 10, 60, ["sync" => false]))
    cache ["ttl" => 300]
        PickupPolicy(
            ["localip", "remotemx"],
            ["remotemx" => $mx],
            ["rate" => [10, 60]], 300);

The reworked queue naturally comes with many new tools and APIs for interacting with the new functionality. This includes more subtle improvements, like the ability to view the queue’s shape by message age. By pressing an interval, you can dig into the specific messages, which are grouped by fields of your choice.

The new shared memory script functions and API opens up several possibilities. You can script statistic counters, which can then be read periodically over the API. Another use case is pre-loading data into the MTA over the API, rather than fetching and caching from within the script.

Finally, we now offer the ability to call methods in external shared libraries using our foreign function interface (FFI) class.

Check out the full changelog on GitHub for more information, and familiarise yourself with the important changes outlined in the release notes document before upgrading.

Reduce risk by switching email delivery platform

If you’re operating an email delivery platform that’s growing in traffic but isn’t operating at its full potential, you might want to look for other options. There are of course challenges associated with switching platforms, but in the end, it can prove to be the best solution for your company.

Read more.

Reduce development time and costs with an MTA platform

The scripting on a platform created specifically for email enables you to create more with much less code. If you’re accustomed to a home-brew system based on open source components and decide to evaluate a comprehensive and scriptable email platform, you’re gonna find yourself spending less time on development and more time on value-adding strategies.

Read more.

Implementing SMTP LANG with proxy script

The upcoming Halon 5.1 release introduces a new SMTP server proxy script. It’s configured to be executed before specific (or all) SMTP commands, even command which isn’t recognised by the SMTP server. In this blog we’ll describe how to implement our proposed SMTP LANG extension using this new script hook. First of all, we announce the LANG extension and the languages supported in the HELO script:

$announce =  [...$arguments["extensions"], "LANG SE"];
Accept(["extensions" => $announce]);

Next, we configure to proxy script to hook onto the new “BREV”, “INNEHÅLL” and “HEJDÅ” commands:

Finally, we setup the proxy script to translate the international (Swedish, in this case) commands with their standard SMTP respective:

$cmd = str_upper($arguments["command"]);
if ($cmd[0:11] == "BREV FRÅN:") { // MAIL FROM
    $realcmd = "MAIL FROM:" + $arguments["command"][11:];
    Pass(["command" => $realcmd]);
}
if ($cmd == "HEJDÅ") { // QUIT
    $codes = ["code" => 221, "enhanced" => [2, 0, 0]];
    Reply("Ha det bra", ["reply_codes" => $codes, "disconnect" => true]);
}
// etc...

The proxy script and the corresponding xtext functions can be used for many other things, like custom XCLIENT implementations. More information on the 5.1 will be released later this month.

What integrated blue-green testing is, and how it benefits your email service

Deployment of changes or new features can be challenging, time-consuming and risky. Many service providers don’t have the infrastructure for production testing.

One great way of quickly and safely rolling out changes is blue-green deployment. Halon provides built-in, integrated traffic splitting that we call live staging. It’s a unique method that allows you to try out new code and configuration on a production host for only a selected part of your traffic, selected by random or IP address. This creates two virtual environments. The code editor and tooling makes it easy to test the working copy before you decide to deploy.

Read more.

Three ways to reduce the operating costs for your email service

If you are using an on-prem, open source based solution for your MTA, you have the advantage of low initial investment costs with no license cost. But later on, your operating costs can start to become quite severe. Factors that can become quite expensive for an email service provider is server rents or costs for hardware, as well as various costs for updates, maintenance and development. Additional support costs can reside depending on the setup. You may also experience issues with complicated pathways, and various components may be coupled together in complex ways.

Read more.

Speaking at M3AAWG #45 in San Francisco

As a supporting member, we’re happy to be participating in our 7th meeting on February 18-21. M3AAWG meetings are an exceptional opportunity to discuss the latest in messaging security with other professionals in a focused environment of working sessions and educational panels.

We would be delighted if you joined the transport encryption session that I’m speaking at, on Wednesday. Also, if you want to meet up, just get in touch! We’re all around; product, sales and engineering.