We’re very proud to announce the upcoming 5.2 release “polly” which introduces a powerful queue policy engine. First and foremost, the queue and SMTP client’s network layer is now asynchronous. This allows an instance to handle tens of thousands of parallel connections. In combination with the reworked connection concurrency limits, this allows dynamic creation of a virtually unlimited number of independent sub-queues. This is useful for senders that need to separate email streams so that those that move slowly or get stuck don’t block others.

As usual, we made it flexible enough to fit any email service provider’s needs. Rather than having a fixed set of parameters and rollup/grouping options for establishing the sub-queues (with their respective thresholds), we allow you to define what constitutes a unique entry. You can choose any combination of fields, and group/rollup entries using regular expressions or wildcard. In the example below, we limit the concurrent per source IP and remote MX, and also rollup all Google’s MX entries into the same entry. The default concurrency is 5, except Google that gets 10.

- fields:
  - localip
  - remotemx:
      gsuite:
        - '*.google.com'
        - '*.googlemail.com'
        - '*.smtp.goog'
  conditions:
  - if:
      remotemx: '#gsuite'
    then:
      concurrency: 10
      rate: 50
  default:
    concurrency: 5
    rate: 10

Sometimes rollup per MX doesn’t cut it. There are several Microsoft Office365 locations (clusters), but the customer MX doesn’t reveal which they are on. To set a certain threshold for Office365 locations, we can rollup and match per MX, but limit per IP, as per the example below. Note that there’s no default threshold; it only affects Office365.

- fields:
  - localip
  - remotemx:
      o365:
       - '*.protection.outlook.com'
  - remoteip
  conditions:
  - if:
      remotemx: '#o365'
    then:
      concurrency: 10
      rate: 30

Thresholds and suspensions can be modified on the fly without reloading the configuration via
API, CLI, web administration or the MTA itself through this Halon script:

// If we have more than 10 failures per minute, lower rate for 5 minutes
$mx = $arguments["attempt"]["connection"]["remotemx"];
$code = $arguments["attempt"]["result"]["code"];
if ($mx and $code >= 400 and !rate("mx-fail", $mx, 10, 60, ["sync" => false]))
    cache ["ttl" => 300]
        PickupPolicy(
            ["localip", "remotemx"],
            ["remotemx" => $mx],
            ["rate" => [10, 60]], 300);

The reworked queue naturally comes with many new tools and APIs for interacting with the new functionality. This includes more subtle improvements, like the ability to view the queue’s shape by message age. By pressing an interval, you can dig into the specific messages, which are grouped by fields of your choice.

The new shared memory script functions and API opens up several possibilities. You can script statistic counters, which can then be read periodically over the API. Another use case is pre-loading data into the MTA over the API, rather than fetching and caching from within the script.

Finally, we now offer the ability to call methods in external shared libraries using our foreign function interface (FFI) class.

Check out the full changelog on GitHub for more information, and familiarise yourself with the important changes outlined in the release notes document before upgrading.