Browse

Tag: tech

Seamless migration to 64-bit

AMD announced what would become x86-64 in 1999, and the first CPU was shipped in 2003.

A few years later, we released the first version of our email gateway. At that time, many 1U servers and network appliances used the new “Pentium M” processor, thanks to its energy efficiency. It was 32-bit. That’s how our 10-year-long 32-bit legacy began…

A few months ago, we got tired of having to build, test and release two different binary images of each release; one 32-bit (called i386) and another 64-bit (called amd64). To see what kind of systems were out there, we added the following code to the product, to detect 64-bit CPUs (even though the CPU was running 32-bit software) and displayed a warning on the first page of the web admin, if a 32-bit CPU was detected.

bool cpu_has_64bit_support() {
        uint32_t regs[4];
        do_cpuid(0x80000000, regs);
        if (regs[0] < 0x80000001)
                return false;
        do_cpuid(0x80000001, regs);
        if (regs[3] & 0x20000000)
                return true;
        return false;
}

Only a handful customers got back to us, saying that they say the warning, asking what was up. Some had 64-bit CPUs, but were running virtual machines on old hosts (for example VMware ESX3) which had a per-VM setting for enabling 64-bit. Other were still running old network appliances with Pentium M, and we encouraged them to go virtual instead (as most companies today have a virtual machine environment).

Satisfied about the results, we researched the different methods for migrating all users to 64-bit. First of all, we made sure that the 32-bit FreeBSD’s first-strage boot loader (that we use for updating in our dual-partition scheme) could load a 64-bit kernel/loader, which it could, fortunately. Then, we had to decide when to convert all the machine-dependent data (PostgreSQL database, RRD, etc). We finally decided to do the migration in “one step”; doing the data conversion during boot of the 64-bit release (only one 64-bit release will contain the migration code; 3.2-r10). By shipping that specific “migration” release with 32-bit compatibility libraries and a 32-bit version of PostgreSQL and other programs, we can dump the data to SQL, XML, etc and then to an import. Once migrated, the customer need to update once more, in order to get to the latest 64-bit release. We make sure that no systems are “bricked” by doing the cpu_has_64bit_support() check in the updated program before downloading the update.

A few weeks ago, we finally released the “32-to-64-bit migration release” (in the form of 3.2-r10) on a request basis, and it will released for everyone (with a CPU that detects as 64-bit) in a week or so.

New Halon release with added RESTfulness

Many of our customers uses the HSL scripting language’s unparalleled flexibility to design and build ambitious systems, integrations and services. It could be anything from automatically changing spamming accounts’ passwords to downloading and caching black/whitelists from external systems.

As usual, the changelog is packed with many other improvements. For example, GetMailQueueMetric() that can be used to implement quotas, and many filtering arguments to the API for more granular admin permissions.

The email gateway itself is controlled using a beautiful SOAP API. However, for making calls from the gateway to external systems, most people seem to prefer REST, and we’ve therefore added many options (such as method, headers and CA validation) to the http() function, and JSON support.

Bounces (sometimes called DSN or NDR) include the original subject in their subject. It makes them easier to find (by searching for subjects) and also for the user as he can easy see which message wasn’t delivered.

Tracing abusive users gets a lot easier now that we’ve introduced the ability to both search and view messages’ SASL username (used to submit the message).

Halon 3.2 based on FreeBSD 10 in the pipeline

We’ve been using beta and release candidate versions of FreeBSD 10 for some time, so that we were prepared to ship a major update based on FreeBSD 10, shortly after it’s announced. It has received lot’s of testing, with only a few issues (reported upstream). FreeBSD 10 brings numerous improvements, and we’re very pleased with its stability as well.

Some of the improvements that our customers will notice and enjoy are;

  • LDNS and Unbound as replacement for BIND (as resolver and DNS cache)
  • Hyper-V para-virtualization, bringing gigabit networking, VLANs, accelerated SCSI disk support, and much more.
  • Virtio (KVM) para-virtualization, for greatly improved network and disk performance
  • Support for VMware’s VMXNET3 driver
  • Support for Xen’s PVHVM mode, an hybrid virtualisation mode
  • Improved AES-NI support bringing higher encryption performance
  • Ability to grow mounted filesystems, allowing us to automatically resize the storage disk
  • Using the innovative clang as compiler

In addition to this, we’ve overhauled large portions of our systems (this is a major release, after all), including (but not limited to);

It’ll be available within short, when it has passed its testing phase.

Boosting performance with Halon 3.1-r4

We’ve just released 3.1 revision 4 of our e-mail gateway, with the main feature being a targeted performance boost.

Earlier optimisations were mainly focused on making parts of the system faster, whereas our recent efforts focused on certain common scenarios. Because the system consists of a large number of components (anti-spam, anti-virus, SMTP, database, etc) we found that pretty large performance gains could be achieved by redistributing resources (CPU, memory, I/O) to where they made the most difference. By focusing on the most common configurations, we’ve added “shortcuts” or “short circuit” that has made certain setups twice as fast.

One such scenario is the reject performance; by skipping steps that would anyway be discarded by the rejection. One example is the transaction safety (ACID compliance) that requires a buffer/cache synchronisation, which is bypassed if we detect that the message will anyway be rejected. Another example is the stat() function where multiple counts are batched (1+1+1=3) in fixed intervals, significantly reducing the overhead.

That’s just a few of the over 100 changes that has been made. We hope our high-traffic customers enjoy the additional capacity that this release will bring.

Just in time for christmas, ho ho ho.

Considerations regarding DMARC forensic reports

DMARC is a great step forward for email, as it finally enables us to trust an email’s From address.

Because of the way that most mailing lists are currently implemented, we believe that email receivers who implement DMARC verification (mail hosting companies in particular) should avoid sending forensic DMARC reports (ruf=).

The problem can be replicated by

  1. Someone requests to have forensic reports sent to their domain (adding DMARC with ruf=)
  2. He then sends a message to a mailing list
  3. The message is distributed to all subscribers
  4. The subscribers’ mail servers might respond with a forensic report
  5. The report contains the subscriber’s address

This results in information leakage. Lets say somebody is subscribed to a mailing list without participating, believing that his membership isn’t easily disclosed. Therefore, as of today, you probably don’t want to enable the sending of forensic reports when implementing DMARC verification. This is what a forensic report could contain:

Received: from example.com (localhost [127.0.0.1])
	by mailinglist.example.com with ESMTP id ...
	for <SUBSCRIBERS-MAIL-ADDRESS>; Fri, 13 Dec 2013 09:02:45 -0700 (MST)

Are we disclosing a vulnerability? No. The mailing list problem is well understood and discussed by the DMARC community. For example, most DMARC adopters don’t send forensic reports for various reasons. Also, because mailing lists typically violates the DMARC policy, there’s an ongoing effort to change mailing list software so that these issues are resolved. The purpose of this article is simply to raise awareness among potential DMARC adopters, such as mail hosting providers. The concept of forensic reports is powerful, but you need to be aware of this issue.

During some testing that we did, we found a few major hosting companies that did send forensic reports on their user’s behalf, thus leaking information about their users’ mailing list subscriptions.

Anyway, please start using DMARC, for everyone’s sake.

  • As for verification and sending reports;
    • Everyone should enable DMARC verification
    • High volume hosting companies in particular should contribute by sending aggregated reports
  • As for signing and receiving reports;
    • Most organisations could benefit from activating DMARC in monitoring mode (p=none, rua=mailto:…)
    • Organisations which identity is subject to spoofing such as phishing (institutional domains, banks, web services) should enable DMARC fully (p=reject)
    • Never-sending domains (which aren’t used to send mail) should enable DMARC (p=reject)

Sending aggregated reports from a Halon system

Building upon our library libdkim++ we had one of the first on-premise e-mail security products with DMARC validation. It’s however important that mail servers (high volume receivers, such as hosting providers, in particular) contribute by sending validation reports.

Instead of re-inventing the wheel, we’ve evaluated OpenDMARC‘s reporting tool. We’ve been using it for some time, and decided to write an integration script which you can download from GitHub. The installation is pretty straight forward by following our guide; which boils down to aggregating logs using remote syslog, and process/report them daily using logrotate.

Why should you start sending reports? Well, first of all it’s one of DMARC’s strengths (which is reflected in in’s name; Domain-based Message Authentication, Reporting and Conformance). For example, reporting is crucial for DMARC’s “monitoring mode”, which is helpful as a first step of deploying DMARC. Further, it enables senders to analyse what’s spoofed in their name. As quoted from dmarc.org;

DMARC addresses these issues, helping email senders and receivers work together to better secure emails, protecting users and brands from painfully costly abuse.

We encourage all our high-volume customers to evaluate DMARC reporting. Feel free to contact us if you have any questions!

Making PHP’s SOAP client asynchronous

Most web projects uses client-side JavaScript to handle concurrency, because of its asynchronous and callback-based nature. In some cases however, it’s desirable to achieve concurrency on the server side. One example could be the processing of sensitive data, that has to be filtered before handed to the client. In our case, it was a data size issue that triggered us to implement a concurrent SOAP client. We develop a hosting-grade anti-spam appliance, and an open source end-user web interface that allows users to interact with a cluster of anti-spam systems. Imagine a cluster with 10 nodes, and a user that views the 100 latest messages. Using JavaScript, that would require the client to download a listing of 10*100 messages, and with PHP it would result in 10 synchronous request.

Thanks to PHP’s flexibility, we could develop a cross-platform solution, without having to rely on a RPC multiplexer. We chose a select()ing method provided by CURL’s “multi” package, by extending the SoapClient class and implementing our own __doRequest.

You can download our helper script, and use it as

// Just demonstrating that it works with multiple functions, to multiple servers
$client1 = new SoapClientAsync('some-systems-wsdl', $options);
$client2 = new SoapClientAsync('another-systems-wsdl', $options);
$client1->someFunction($arguments);
$client1->anotherFunction($arguments);
$client2->anotherFunction($arguments);
soap_dispatch();
$result1 = $client1->someFunction($arguments);
$result2 = $client1->anotherFunction($arguments);
$result3 = $client2->anotherFunction($arguments);

We hope that our customers appreciate the speed boost that it gives their anti-spam system’s end-user web interfaces.

Halon 3.1 based on FreeBSD 9.2 and restructured UI

We’re just about to release a new version (codename “lucky”) of our e-mail security gateway (SP), based on the latest FreeBSD version (9.2) in an attempt to track new FreeBSD versions as closely as possible. Our users will get benefits directly out of the amazing work of the FreeBSD team;

  • Improved performance in many areas, increasing the throughput of the system
  • Support for KVM’s VirtIO, allowing for more efficient network and disk accesses
  • Numerous updates in drivers, services, external projects, etc

In addition to that, we’ve improved the web interface even more, making it more convenient for customers with large clusters and traffic volumes. For example, the rate control page now has thresholds, and persistent search filters. Another new feature allows you to create interval statistics (for example “daily”) using the the rate() function instead of stat(), with a script such as

rate("daily-stat", "delivered", 10000000, 3600*24);

and use that when creating pie chart such as

delivered=r("daily-stat,delivered")
blocked=r("daily-stat,total")-r("daily-stat,delivered")

Finally, the web interface’s menu is simplified, and if you’re running a cluster, your default page will be a cluster overview.
blog-sp31
The logic behind the new menu, is to avoid having two “Activity” menus. Instead, both clustered and non-clustered versions of those pages (reporting, mail tracking, logging and rate control) are accessed from the same menu. The “node selector” in the upper right corner is used to switch between specific nodes, or the cluster as a whole. All configuration (which is mainly clustered, except for network settings and node overrides) has its own place in the menu, indicated with the dark blue color. Configuration management such as revision management and plain-text editing is placed in the bottom of the Configuration menu.

We hope you will enjoy this new release, and appreciate all feedback we can get!

Sneak peek of Halon 3.0

We have some exciting news about our spam prevention series. The upgrade to FreeBSD 9 and overall refactoring was not the only treatment the SP series got this autumn and winter. We have collected feedback and performed evaluations of how our customers uses the web interfaces, trying to figure out what the best possible reporting and logging experience would be like. Read on to see what this has resulted in.

We have migrated to the new web interface from the security router series. That means a prettier UI, faster loading times, the ability to link directly to certain views using URLs with query strings, and better utilisation of your screen’s full width.

Let’s start with the mail tracking. The new UI provides some benefits of its own; displaying more information, auto-scaling all columns, and faster loading. We have combined the history, queue and quarantine within the same page. It’s pageable with a variable page size, so that you can view as many messages as you like per page. It has multi-select actions, for better queue management (viewing perhaps 1000 messages matching a certain search query, and bouncing them all). Finally, the “eye” icon brings up an inspector which you can use to view details for a message by just hovering items in the list.

The new log searcher is a lot faster than the previous, and can render thousands of lines without hogging your web browser. Most importantly, it can search multiple cluster nodes at the same time, viewing the number of hits (in real-time) per cluster node as a green badge. In that way, you can start a search for an IP address, and then ask someone to try sending the message again, and you will (when tailing in real-time) see a green badge on the cluster node which received the connection. Extremely handy.

The new reporting and graphs are based on the SR series code. That means a new statd which is fast, produces beautiful graphs, with real-time graphs, customisable legends, etc. Best of all is however that you can graph anything you like. To start with, you can create legends yourself; just look at the pie chart in the bottom right width the edit button clicked. You can even use math expressions to calculate values. Even cooler, you can use the new HSL stat() function in any flow, producing counters for whatever you like. There counters automatically becomes graphs and pie charts. I believe this is the most powerful reporting available in any mail security product ever. Perhaps any appliance.

Scripting, such as the system authentication script that allows for remote authentication and custom access levels, has become a lot better thanks to a great scripting editor with syntax highlighting and the ability to test the script using a “sandbox environment”.

The new web UI from the SR series doesn’t only bring nice real-time graphs, but also a true ANSI terminal.

We have made the already awesome clustering a lot easier to configure; with one “create cluster” guide joining two initial units, and one “add node” guide for adding a third, fourth, etc node to an existing cluster.

Halon 2.2.2.2 released with DNSSEC root trust

Today, on the 2nd of September we release 2.2.2.2. Neat, right?

Among the new features you’ll find the DNSSEC trusting the newly signed root anchor, administration user interface improvements and the usual stability and performance enhancements.

Now why would you care? Well, this could be your first step into the next generation of e-mail security. Why not start DKIM tagging when you’re at it?

There are small, yet useful features are well. Let’s say you want to implement a reporting rate control in your outgoing recipient flow, so that users or servers doesn’t send outgoing spam. In this example, we do this per-username ($saslusername, a pre-defined variable in the recipient flow) limiting the number of e-mail to 100 every hour, while sending at most one warning e-mail to the administrator about this every day and per user.

function myrate {
  if (rate("outbound", $saslusername, 100, 3600) == false) {
    $msg = "Rate "+$saslusername+" spam outbreak";
    if (rate("outbound-report", $saslusername, 1, 86400) == true) {
      mail("[email protected]", "[email protected]", "Rate", $msg);
    }
    Defer($msg);
  }
}

and then using it in code like

if ($saslauthed) {
  myrate();
  Accept();
}

Take care folks!