Browse

Tag: tech

Halon 4.4 “lofty” packed with small improvements

The 4.4 release “lofty” is all about fixing bugs, boost existing features, and improve performance and memory management in the Halon script engine. And like macOS “High Sierra”, it’s fully baked.

The unusually long changelog contains many small improvements. We’ve given the pre/post-delivery script a slight overhaul. It’s now possible to tailor the bounce behaviour via the the SetDNS() function. Additionally, we’ve added $action and $context, as well as functions to set MAIL/RCPT parameters. Finally, the SetSouceIP() enables you to choose an IPv4 and IPv6 address pair, which is a great when you want to provide customers with a private IPv4 and IPv6 or if you want to use diverse address pools.

The improved “Listen on” directive on the Server > SMTP listener page enables more fine-grained control over listen ports and IPs; such as listening on different ports for different IPs.

Quirks and fun trivia
  • We recently revised our LDAP implementation, and realised that our own syntax and mechanism for failover between hosts is rather superfluous, since OpenLDAP supports that natively. Consequently, we adopted the standard LDAP URI’s in our configuration, and existing configurations will be automatically migrated.
  • While we support the PROXY protocol (v1) that passes client source IP information from load balancers, we thought it was mostly as HAProxy thing. Apparently, it’s used by many other load balancers such as Amazon ELB, Citrix Netscaler, and F5 BIG-IP. Most of them implements the version 1 (which is human readable), but there is a second version of the protocol that’s binary-packed, and have a quite smart feature: its magic string (protocol identification) is \x0D\x0A\x0D\x0A\x00\x0D\x0A\x51\x55\x49\x54\x0A which translates into literal "\r\n\r\nQUIT\r\n", a string chosen specifically to case an error and disconnect against servers not supporting this protocol. Clever!
  • If you have a IPv6 only datacenter, but still want to process IPv4 clients, you can do so with a SIIT-DC gateway which uses IPv4-mapped-IPv6 addresses. In Halon, you can use SIIT-DC while still performing IPv4 reputation (such as DNSBL), by extracting and setting the IPv4 address in the CONNECT script. If that doesn’t make the point that we’re very scriptable, then what does?

Image from Tore Anderson’s SIIT-DC presentation

If you ever had problems signing in to a Halon using Firefox, it can be because a recent change in how “secure cookies” are handled. When signing in over HTTPS, we set the secure cookie flag, which forbids the cookie to be send over a unencrypted HTTP connection to the same host. That is all great, but if you then try to sign in over HTTP (for whatever reason) Firefox will not be able to login because there is already a cookie for that domain with the secure flag and it cannot be replaced, nor accessed. We addressed this by using different cookie names for HTTP and HTTPS. Regardless of this fix, you should not use HTTP when administering your Halon hosts.

Halon invited as speaker to ETIS Community Gathering 2017


Each year the ETIS Community Gathering brings together European telecommunication professionals to share knowledge and best practices in a trusted environment. The theme of the ETIS Community Gathering 2017 is ‘Shaping the Digital Ecosystem of the Future’.

Halon co-founder Jonas Falck will be speaking about DANE, SMTP STS and more, together with senior software engineer Erik Lax.

Erik Lax

The meeting is this year held on October 5-6 in Tallinn, Estonia. ETIS believes that Estonia, the first country to allow online voting in a general election, is a perfect place for a debate on the ‘Digital Ecosystem of the Future’, and we agree. It has the world’s fastest broadband speeds and holds the record for start-ups per person.

Its 1.3 Million citizens pay with their mobile phones, have their health records stored in the digital cloud, and file their annual tax return online in 5 min. Moreover Estonia will be holding the presidency of the EU council in the second half of 2017. Therefore ETIS invites relevant parties and start-ups to discuss lessons learned in e-Estonia.

One week with Scale Global in San Francisco

Last week brought Per Stenman and Anders Berggren of Halon to San Francisco and Palo Alto,
as a part of the Scale Global program.
Sunny California offered meetings with VC’s, entrepreneurs, coaches, and even a celebrity that we can not disclose.

Tech art and Andreesen HorowitzEntering AndreesenHorowitz

start-with-the-customerSharon Chang, Partner at AndreesenHorowitz

California eatingFood is necessary, so is instagramming.

Professor Huggy RaoHuggy Rao, Atholl McBean Professor of Organizational Behavior at Stanford Graduate School of Business.

SC MoattiSC Moatti, Managing Partner at Mighty Capital.

Tristan KromerTristan Kromer, Lean Startup Coach.

Cindy Alvarez and Anders BerggrenCindy Alvarez, Principal design researcher at Microsoft. Halon co-founder Anders Berggren visiting 500 Startups.

MeatNo trip to the US without getting a decent steak.

Maxime Prades and Gustaf AlströmerMaxime Prades, VP of Product Management at Algolia and Gustaf Alströmer, Partner at Y Combinator.

Rohit SharmaRohit Sharma, Partner at True Ventures.

Time-of-click protection against ransomware, malware and phishing

Time-of-click protection adds an extra layer of security to protect email users from accessing malicious content. Attacks including malware, ransomware and phishing are becoming more common and more sophisticated with every day, along with users keeping more sensitive information.

With an additional time-of-click protection, Halon will classify links in email every time it’s clicked, before allowing or denying the user to visit it. This means that if the scammer waits two minutes or two months with infecting the site, the user will still be protected when he or she chooses to click the link. It’s the extra layer of security that won’t allow you to visit infected websites by way of a link in an email protected by Halon.

Read more.

Meet “classy” and “cody”, Halon 4.2 and 4.3

We have done two new releases of Halon since last time we updated the blog with release matters. In Halon 4.1 “teamy”, released just before this summer, we introduced modules. A month later we followed up with 4.2 “classy” that added proper object orientation to the language (which works great in combination with modules). It spawned a few rewrites of our script examples (modules) to reflect this awesomeness. We initially added instance and class methods and variables (static), and in 4.3 “cody” we added the private keyword to functions and variables as well.

class HelloWorld
{
	private $name = "Dr Who?";
	constructor($name)
	{
		$this->name = $name;
	}
	function sayHello()
	{
		return "Hello ".$this->name";
	}
	static function ...()
	{
		...
	}
}

We’ve created a lot of modules and script examples. Some of those, such as the PostgreSQL and MongoDB modules, rely heavily on byte packed data structures. In order to better support those, we’ve added built-in functions such as pack() and unpack(). Upcoming modules and rewrites will also benefit from the new TLSSocket() class.

Here are some new additions to our module collection:

Other notable features from the changelog includes

  • FreeBSD 11.1 and new quarterly packages
  • sha2 hash functions
  • Added status and NDR codes to Reject, Defer and Deliver functions
  • SetTLS support CA name verification
  • DLP engine now support file hashes of SHA2-256 and SHA2-512
  • Added $sourceip variable to post-delivery script to easily determine which IP address that was used to send the mail
Geek out corner

One major change that only we can see and fully appreciate is the (both automated and manual) code migration to C++11 (and forward), using the truly awesome clang-tidy tool.

On another note; while we researched pack and unpack implementations by looking at other languages’ documentation (such as PHP, Perl and Python), we found a bug in PHP, which was fixed in 7.2, and backported to 7.1.9. The overall consensus of syntax and conventions amongst languages regarding how pack and unpack should work seems to reflect and mimic Perl.

Some scripting languages like JavaScript and HSL has the notion of class constructors but no destructors. The HSL memory model uses references counted automatic garbage collection to determine when objects should be removed.

“In a language with an automatic garbage collection mechanism, it would be difficult to deterministically ensure the invocation of a destructor, and hence these languages are generally considered unsuitable for RAII [Resource Acquisition Is Initialization]” – Wikipedia on destructors

MongoDB does unlike many other databases use little endian and not big endian (network byte order) in its wire protocol. This will let you send and receive data structures in native machine endian (for most people) since both x86 and amd64 use this convention. I highly recommend reading up on the fun historic trivia about endianness.

Want more in-depth info on the new releases? Get in touch with the support team.

Halon 4.1 “teamy” with scripting modules

We’ve shipped the 4.1 “teamy” release, and we’re happy to see that the majority of our users have already updated. The 4.0 release brought several major changes, and 4.1 includes many improvements to it. The web administration in general, and the script editor in particular, have received lots of attention. The new live staging functionality that was introduced in 4.0 now extends to queue (pre- and post-delivery) scripts as well, using the _stageid metadata field. You can employ even more aggressive caching now that the API includes a hslCacheClear() function, and we’re opened up a wide range of possibilities via the new setBody function in the DATA script’s MIME class, such as URL rewriting.

The most anticipated addition however, is a new language feature often referred to as modules. The Halon scripting language is highly email-centric. We strive towards keeping it as simple as possible, which is a great recipe for stability and high, predictable performance. At the same time, we’re designing it after the principle of least astonishment, and consequently it shares many characteristics with established languages. Having researched both namespaces (found in languages such as C++ and PHP) and modules (found in for example Python and Perl), we decided that modules was the best fit for our language and its users. It’s great when working in larger teams, since it introduces file-wide symbol (variable and function) scoping. Modules are regular script files, which becomes modules the moment you import them. It’s similar to include, except an imported file has its own scope. You explicitly choose which symbols to “import” to the parent file using the syntax

import { Foo as X, $bar as $y } from "test";

Variables are imported by reference, hence all changes to the variable in the module will be reflected by the imported variable. As modules have their own global scope, variables referenced in a module’s function by global or closure points at the module’s scope, as per the example below

$x = "hello";
function foo($bar) {
    global $x;
    echo $x;
    $bar();
}

imported by

import { foo } from "test";
$x = "world";
echo foo(function() closure ($x) { echo $x; });

which will echo

hello
world

We hope that the modules concept will be helpful in both smaller and larger projects, and we promise that much more is to come as Halon 4.2 “classy” hits the servers.

Halon and Spamhaus in email security partnership

We are excited to announce that Halon now provides official integration with Spamhaus Technology anti-spam & threat data feeds (IP & domain blocklists). Both companies worked together to ensure that this new functionality would be simple to deploy while also scaling all the way from smaller systems to large ISP’s with millions of users and complex email flows.

Read more.

How I fooled Microsofts Safe Link technology in 5 minutes

The Safe Link technology was recently launched by Microsoft through Office 365. The goal of this technology is to rewrite all URL’s in email to a URL classification service, so at the time of user-clicks it’s possible to reclassify a URL. This method is preferred as spammers more often replace the phishing URL’s site content after a message is being scanned, hence there is a need of reclassification later. Safe link is Microsoft’s “best-effort” to do so.

“For messages in HTML, Safe links identifies any link that uses the HREF attribute. For messages in plain text, Safe Link uses custom logic to identify any text resembling a URL.”

Microsoft.com

This method should work correctly in all MUA (email clients). From the web mail to your iPhone’s Mail app. However, replacing a URL in HTML as text is difficult. Just let me demonstrate how easy it is to fool Microsoft’s Safe Link:

<a x=">" href="http://badurl.com">click me</a>
      ^--- the regex? engine stops to detect the <a> tag here, and leaves the href unchanged.

Another obvious way to fool the Safe Link re-writer is to use a <form>-tag (it may not work in all email clients). You may be safe until spammers figure this out.

<form action="http://badurl.com"><input value="click me"></form>

If it’s this easy to fool, should it be done in another way or perhaps complemented with additional safeguards, preferably in the MUA (web mail, Outlook.app, etc)? I think so, and would have expected that Microsoft tried harder.

First suggestion; when rendering the email replace all links by asking the rendering engine what it has rendered

$("a").each(function () { /* all links are detected foolproof */ });

Second suggestion; Microsoft could surely use one of there own HTML parsers (like the one in the Edge engine) to detect where URL’s are located in the message in order to properly replace them, it’s probably better than a regex.

If customers are activating and paying for Safe Link they should be able to expect more value for their money and some more security.

In Halon you can do the same simple URL rewriting using this HSL code.

HSL instead of Safe Link

New features for the end-user interface

Have you checked out our open-source End-user interface on GitHub yet? It can be used either as is, modified to fit your needs, or only as an inspiration for your own code. Since last time we posted about it, we have updated it with a bunch of new features. Have a look and feel free to give us some feedback if you do decide to try it.

Archiving

Your Halon nodes archives can now be managed directly from the Messages tab in the interface. This feature allows the end-users to browse, preview, resend and download messages from the archives.

Rate limits

This feature makes it possible to monitor HSL rate limits from your Halon nodes in real-time (when logged in as an administrator). This can be very useful to get a quick glance at for example which users are sending large amounts of email (or suspect spam) or to discover other anomalies. It can also be used to reset rate limits for specific users.

Database partitioning

This feature improves search performance by splitting the message history into different partitions based on a customisable user ID.

Database based graphs

Before it was only possible to fetch the graphs from the Halon nodes directly but now it’s also possible to create graphs based on the data in the local database.

Generic datastore

The datastore is very useful for storing various kind of data that can then be fetched by the Halon nodes. Some examples of use cases for this could be overrides for rate limits, routing information and specific customer settings.

More themes

The interface is based on Bootstrap and uses a templating system called Twig which makes it very easy to switch between different themes and also make your own modification to them. We now provide multiple themes to choose from but you could also easily implement your own themes as well.

User administration

It’s now possible to create, edit and remove database users directly from the interface (when logged in as an administrator).

Live staging and SMTPUTF8 coming up in Halon 4.0

Halon 4.0 a.k.a. “forty” is around the corner, and it has a strong focus on developer friendliness and deployment. Let’s take a peek on what’s about to happen!

Most of the work has gone into the SMTP server. Most notably, it now supports running two different configurations, complete with compiled scripts, in parallel. We refer to it as “live staging”, since it allows you to try out new code on a production host, for only some select traffic, based on conditions, such as IP address. We’ve also introduced a new persistent connection $context variable to all the SMTP server’s scripts, including the new MAIL FROM script, which can be used to pass data between SMTP stages or even RSETs, like

$context[$messageid][$recipient] = lookup($recipient, ...

since $messageid is regenerated with each RSET. Another major feature is SMTPUTF8, which allows for international (and emoji!) email addresses.

The scripting language itself has a new resource value type, used by the Socket() networking class which, in turn, is used to implement various extensions such as a memcached client. Another new scripting features is bitwise operators, which you can see in action by viewing the TOTP example’s code.

In addition to the live staging, management is significantly more distinct and reassuring thanks to a checkout/commit pattern, and a more expressive configuration format. As you begin to make changes, already tested using the live staging, as larger commits with descriptive messages, browsing the configuration revision history will become much more enjoyable. It also supports larger teams much better.

Halon 4.0 user interface

As usual, it’s based on the latest FreeBSD version; 11.0 in this case. It features LLVM’s LLDB, Receive Side Scaling (RSS) in some Intel network drivers, unmapped IO in Xen and VirtIO, Amazon AWS SR-IOV networking, Tx/Rx multiqueue in VMware VMXNET3, and much more.

Check out the changelog for more detailed information, and try out the new web interface at demo.halon.io!