Browse

Tag: github

Halon 4.5 – gettin’ certy with it

The main focus in Halon 4.5 release is TLS, hence the name “certy”. Check out the the new features and functions and try them out. Also, the knowledge base is growing with a lot of good how-to’s to help you around.

TLS information has been made accessible in the Halon Platform scripting language, both on the receiving and sending side. Support for X.509 client certificates has been added, allowing you to both verify the sender identity in the SMTP server, as well as identify yourself when sending email through an SMTP client.

Experiment: we configured a busy email system to ask for a client certificate for all inbound connections, and found that approximate 5% of all traffic provides a client identity. Most of the traffic is from Gmail and Office356. We did not collect the percentage of domains, which we leave as an exercise for you.

$peercert = GetTLS();
$haspeercert = isset($peercert["peer_cert"]);
stat("peer-cert", ["yes" => $haspeercert, "no" => !$haspeercert]);

How to enable this feature and start authenticating clients was documented as KB article.

Implementation and facilitation of TLS reporting (tlsrpt) has begun. It is a new standard for reporting TLS failures, mainly focused on MTA-STS and DANE.

The TLSSocket() class now have a getpeercert() function and the ability to specify a client certificate. Now you see why we called it” certy”?

Support for custom SASL authentication mechanism has been added. This allows you to build authentication schemes such as OTP, OAUTHBEARER or CRAM-MD5, but also EXTERNAL to facilitate the client certificate features. The procedure is documented in our knowledge base along with two sample implementations.

If you haven’t found our knowledge base before, the KB is a place to find how-to’s. The dev team is expanding it as fast as we can, adding topics that customers have asked about.

Finally, I want to highlight the big effort we’ve done to simplify, modernize and overall improve the web administration. This is an ongoing project, and something that we’re paying a lot of attention to. We want to thank, and congratulate, the Bootstrap team for providing such a awesome framework. We managed to get the Bootstrap 4.0 release in, with just a few days of work.

You can read the full changelog on our GitHub of all the other features big and small.

Meet “classy” and “cody”, Halon 4.2 and 4.3

We have done two new releases of Halon since last time we updated the blog with release matters. In Halon 4.1 “teamy”, released just before this summer, we introduced modules. A month later we followed up with 4.2 “classy” that added proper object orientation to the language (which works great in combination with modules). It spawned a few rewrites of our script examples (modules) to reflect this awesomeness. We initially added instance and class methods and variables (static), and in 4.3 “cody” we added the private keyword to functions and variables as well.

class HelloWorld
{
	private $name = "Dr Who?";
	constructor($name)
	{
		$this->name = $name;
	}
	function sayHello()
	{
		return "Hello ".$this->name";
	}
	static function ...()
	{
		...
	}
}

We’ve created a lot of modules and script examples. Some of those, such as the PostgreSQL and MongoDB modules, rely heavily on byte packed data structures. In order to better support those, we’ve added built-in functions such as pack() and unpack(). Upcoming modules and rewrites will also benefit from the new TLSSocket() class.

Here are some new additions to our module collection:

Other notable features from the changelog includes

  • FreeBSD 11.1 and new quarterly packages
  • sha2 hash functions
  • Added status and NDR codes to Reject, Defer and Deliver functions
  • SetTLS support CA name verification
  • DLP engine now support file hashes of SHA2-256 and SHA2-512
  • Added $sourceip variable to post-delivery script to easily determine which IP address that was used to send the mail
Geek out corner

One major change that only we can see and fully appreciate is the (both automated and manual) code migration to C++11 (and forward), using the truly awesome clang-tidy tool.

On another note; while we researched pack and unpack implementations by looking at other languages’ documentation (such as PHP, Perl and Python), we found a bug in PHP, which was fixed in 7.2, and backported to 7.1.9. The overall consensus of syntax and conventions amongst languages regarding how pack and unpack should work seems to reflect and mimic Perl.

Some scripting languages like JavaScript and HSL has the notion of class constructors but no destructors. The HSL memory model uses references counted automatic garbage collection to determine when objects should be removed.

“In a language with an automatic garbage collection mechanism, it would be difficult to deterministically ensure the invocation of a destructor, and hence these languages are generally considered unsuitable for RAII [Resource Acquisition Is Initialization]” – Wikipedia on destructors

MongoDB does unlike many other databases use little endian and not big endian (network byte order) in its wire protocol. This will let you send and receive data structures in native machine endian (for most people) since both x86 and amd64 use this convention. I highly recommend reading up on the fun historic trivia about endianness.

Want more in-depth info on the new releases? Get in touch with the support team.

New features for the end-user interface

Have you checked out our open-source end-user interface on GitHub? It can be used either as it is, modified to fit your needs, or only as an inspiration for your own code. Since last time we posted about it, we have updated it with a bunch of goodies, have a look and feel free to give us some feedback if you decide to try it.

Archiving feature

Your Halon nodes archives can now be managed directly from the Messages tab in the end-user interface. This feature allows the end-users to browse, preview, resend and download messages from the archives.

Archive

 A page to monitor rate-limits in real-time

This feature makes it possible to monitor HSL rates from your Halon node(s) in real-time when logged in as an administrator. This can be very useful to get a quick glance at for example which users are sending large amounts of email or spam or to discover other anomalies. It can also be used to reset rate-limits for specific users.

Rate-limits

 Support for partitions when using database-log

This feature improves search performance by splitting the message history into different partitions based on a customisable userid.

 Support for database-log based graphs

Before it was only possible to fetch the graphs from the Halon nodes directly but now it’s possible to create graphs based on the database-log.database-stats

More themes

The end-user interface is based on Bootstrap and uses a templating system called Twig which makes it very easy to switch between different themes and also make your own modification to them. We now provided multiple themes to choose from in the settings file but you could also easily implement your own themes as well.

Generic datastore

The datastore is very useful for storing various kind of data that can be fetched by the Halon node(s).
Some examples could be overrides for rate-limits, routing information such as destination servers and specific domain or user settings.

Data store

An admin page to manage database users

It’s now possible to create, edit and remove database users directly from the web interface when logged in as an administrator.

User management

How to test SMTP servers using the command-line

In certain situations it can be very helpful to be able to quickly check if a SMTP server is online and reachable, has support for TLS and that it’s working, test user authentication and measure transaction delays and throughput. All of this and more can be done quickly using the command-line. Here’s your guide!

DNS lookup

The first step is to find out which SMTP server(s) is responsible for the domain that you want to test, if you already know this you can skip this step. There are several command-line tools that can be used for this but here I’m using nslookup as well as dig as examples.

# nslookup -type=mx example.local

(cut)
example.local mail exchanger = 10 vsp1.example.local.
example.local mail exchanger = 10 vsp2.example.local.
# dig example.local mx

(cut)
;; ANSWER SECTION:
example.local. 3600 IN MX 10 vsp1.example.local.
example.local. 3600 IN MX 10 vsp2.example.local.
(cut)

Verify connectivity

To verify if it’s possible to connect to the SMTP server you can use for example telnet or netcat.

# nc vsp1.example.local 25
# telnet vsp1.example.local 25

220 vsp1.example.local ESMTP

If you receive a SMTP banner similar to the one above you’re good to go, if you do not see any response from the server that can mean a lot of different things and troubleshooting this is beyond the scope of this article. A few examples of what could be wrong is that the server is down, that it’s rate-limiting your connections, that it’s behind a closed off firewall or that there are routing issues.

Test TLS

If you need to test TLS connections you can use the OpenSSL s_client tool for this. Below you can see one example of a server that is not supporting TLS and another one that does.

# openssl s_client -connect vsp1.example.local:25 -starttls smtp

(cut)
CONNECTED(00000003)
didn't found starttls in server response, try anyway...
139702030079656:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:774:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 177 bytes and written 325 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
# openssl s_client -connect vsp2.example.local:25 -starttls smtp

(cut)
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: 140C627F8364C06E74204CEBA31057415B1A24EC2E5D9B996C6F3277DB18F364
Session-ID-ctx:
Master-Key: C565114C052EDA50B176EA7962F415C3E2BD8A0FC62E243C592BB72164AAAE9625CE80EE81BF88FD8C480EAC4E20A74C
Key-Arg : None
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 300 (seconds)
TLS session ticket:
0000 - 73 04 63 bd 84 16 fd e4-a0 93 26 e0 71 3c 08 0b s.c.......&.qV)v.........
0070 - 19 08 d1 c2 14 ee da 69-cc 85 77 f6 13 39 3c f9 .......i..w..9<.
0080 - 8b 60 38 67 3c e9 f3 18-08 20 f6 1c 10 16 77 7f .`8gV.../..

Start Time: 1474028496
Timeout : 300 (sec)
Verify return code: 18 (self signed certificate)

Note that if you see messages like the ones below when inputing commands while connected using this tool it’s because it’s looking for lines beginning with an upper-case R or Q and either renegotiates or closes the TLS connection if it does. You can avoid this by either using a lower-case letter instead (when possible) or using the -quiet flag when connecting to the server.

(cut)
RCPT TO: <[email protected]>
RENEGOTIATING
(cut)
DATA
354 End data with .
Subject: I have some questions!
Question 1: ...
DONE

Test authentication

Setting up email clients just to verify that authentication is working can be a hassle, so it’s usually much quicker to test it using the command-line, but there are some things you need to be aware of.

First, you need to base64 encode the username and password that you want to test before connecting to the server. If you’re using a *NIX system you most likely have access to the base64 tool which you can use for this.

# echo -n "username" | base64
dXNlcm5hbWU=
# echo -n "password" | base64
cGFzc3dvcmQ=

After that, you most likely need to connect over TLS (see previous section for how to do this) for the server to allow you to authenticate. You can then send the AUTH LOGIN command and then first the username and then the password.

(cut)
250 8BITMIME
AUTH LOGIN
334 VXNlcm5hbWU6
dXNlcm5hbWU=
334 UGFzc3dvcmQ6
cGFzc3dvcmQ=
235 2.7.0 Authentication successful

Measure transaction delays

We at Halon have created a tool called smtpping which can be used to measure transaction delays as well as throughput of SMTP servers. Installation instructions for *NIX systems are provided on the link above. In the example below I’m measuring transaction delays against the mx record of the domain but you can also manually specify the server with the @ prefix (For an example of this see the next section).

# smtpping [email protected]

PING [email protected] ([vsp1.example.local]:25): 10300 bytes (SMTP DATA)
seq=1, connect=0.11 ms, helo=14.94 ms, mailfrom=15.06 ms, rcptto=38.12 ms, datasent=41.75 ms, quit=42.10 ms
seq=2, connect=0.15 ms, helo=15.12 ms, mailfrom=15.25 ms, rcptto=17.98 ms, datasent=21.80 ms, quit=22.02 ms
seq=3, connect=0.15 ms, helo=14.96 ms, mailfrom=15.15 ms, rcptto=17.23 ms, datasent=21.64 ms, quit=21.85 ms

--- 10.2.59.99 SMTP ping statistics ---
3 e-mail messages transmitted
connect min/avg/max = 0.11/0.14/0.15 ms
banner min/avg/max = 14.79/14.84/14.92 ms
helo min/avg/max = 14.94/15.00/15.12 ms
mailfrom min/avg/max = 15.06/15.15/15.25 ms
rcptto min/avg/max = 17.23/24.44/38.12 ms
data min/avg/max = 17.34/24.57/38.29 ms
datasent min/avg/max = 21.64/28.40/41.75 ms
quit min/avg/max = 21.85/28.66/42.10 ms

Measure troughput

Like I mentioned in the previous section about measuring transaction delays, it’s also possible to use our smtpping tool to measure throughput of an SMTP server. In the following example I’m using the -r flag to display the rate instead of the transaction delays and I also specify that I want to use 5 parallel worker processes instead on one. I also manually specify the server I want to check against rather than using the MX record by using the @ prefix.

# smtpping -r -w 0 -P 5 [email protected] @vsp1.example.local

0/s
134/s
203/s
187/s
202/s
194/s

Conclusion

So these were just a few examples of different things you can test on SMTP servers using command-line tools, but it’s of course possible to do even more than that. Just one example of that is our halonctl tool that can be used to manage our SMTP software, you can for example modify the configuration file, search the mail history, manage the mail queue and monitor the CPU and memory usage with it.

Using reCAPTCHA to handle spam misclassification

Today’s leading spam filter technologies offer a very high degree of accuracy. In this blog I’ll describe the current state of spam classification, and propose a pretty innovative method that can significantly improve both senders’ and recipients’ satisfaction (as well as reducing the burden on administrators and support staff) by enabling senders to report false positives if they pass a CAPTCHA test. Let’s start by familiarising ourselves with the history of anti-spam.

Release blocked email

Background

The terminology that we normally use is

  • False positive, a missed spam that slipped through filters into a user’s mail box
  • False negative, a blocked desired (legitimate) email (“ham”)

Historically, spam filters had poor accuracy and low performance, and email was scanned after being accepted (probably as a consequence of the former). Finding themselves unable to reject email, they offered actions such as putting suspected spam in a junk folder, quarantine or by tagging the subject line.

This I believe, significantly damaged people’s trust in email as a reliable transport, simply because it makes legitimate (potentially important) email disappear.

The leading spam classification technologies today however, offers both high accuracy and performance. Many of them, including Cyren (that we use), uses fuzzy checksums (or “patterns”) to measure and classify email in a distributed, collaborative fashion. By constantly updating the hashing logic, anti-spam vendors are able to adopt as spammers evolve their tactics. By primarily looking at individual spam “outbreaks”, the false positive ratio is generally low in such systems. This is key, since people tend to be much less bothered by a few false negatives (missed spam) rather than having desired email blocked.

The high accuracy and performance also makes rejecting spam (rather than accepting it) a viable option. Rejecting spam is arguable superior to accepting and quarantining it, since the sender is informed about the email not being delivered to the recipient’s inbox. It reestablishes email as a reliable (transactionally safe) transport, while a copy of (the rejected) spam can still be retained in a quarantine of junk folder. Halon has advocated for this approach for a long time, and it’s a prerequisite for efficient feedback and reporting mechanisms like the one I’m going to describe now.

Using CAPTCHA to handle false positives

While I believe that our default approach of rejecting (giving a 500-error) spam with an informative error message (and storing a copy in a quarantine or junk folder) is superior to a traditional quarantine, there sure is room for improvement. For example, the sender needs to contact the recipient using some other mean (alternative email or phone, which they might not have), the quarantine might consume a significant amount of disk space, and the recipient might need to bother the support staff.

We’ve developed a self-service false-positive report and release project simply called sender-fp-release to address those shortcoming. As it says on its Github page, it allows senders to report false positives directly to the recipient after completing a reCAPTCHA.

sender-fp-release

In our experience, this system is a win for everybody;

  • The sender doesn’t need to manually contact the recipient, only verify a CAPCHA
  • The recipient gets notified instantly, instead of having to browse through a junk folder
  • The helpdesk doesn’t need to do anything

Additionally, it saves disk space by only retaining spam for a short time (for example 1 day), unless the sender reports it. The retention time for reported email is extended (typically a week or two), giving the recipient plenty of time to release the email.

Release blocked email

If you believe that your spam handling could be improved, please take a look at the project, or maybe give it a spin.

End-user improvements and new integrations with cPanel and Odin

Although many customers prefer to use the Halon SMTP Platform as-is, most hosting providers want to implement end-user interfaces in order to offer a higher degree of customer self-service. Since the Halon SMTP platform is essentially a scriptable MTA (with many features such as anti-spam, signing, and much more) with an open source ecosystem, it makes a lot of sense to maintain the end-user interface as a GitHub project. It is designed as a boiler-plate, and our aim is that the code itself should be as simple and straight-forward as possible. It currently offers features such as indexed history, text logs, queue and quarantine management, black/whitelisting, spam settings and statistics.

In the previous weeks, we’ve added translation support using gettext, with English as the default language, and an extra translation in Swedish. If you want to maintain a translation in another language; please let us know! The language is automatically detected based on the browser’s HTTP_ACCEPT_LANGUAGE.

Rebranding is also made easier thanks to templates written in Smarty, which is also used for email notifications.

One of Halon’s strengths is seamless integration, something that we discussed in the previous blog entry. The same goes for the end-user interface, and we’ve done numerous integrations with common hosting platforms such as WHMCS, cPanel and Odin. The cPanel plugin is branched out into its own project, and offers session transfer for both admins and users (webmail). We recently went to Barcelona for an APS2 training hosted by Odin, and we’ve published an initial version of our Service Automation plugin. It currently offers session transfer for both admins and users, and we’re looking to extend its functionality in various ways.

We hope that you and your customers will enjoy the updated end-user experience! Do not hesitate to contact us if you want more information or help setting it up.

Making PHP’s SOAP client asynchronous

Most web projects uses client-side JavaScript to handle concurrency, because of its asynchronous and callback-based nature. In some cases however, it’s desirable to achieve concurrency on the server side. One example could be the processing of sensitive data, that has to be filtered before handed to the client. In our case, it was a data size issue that triggered us to implement a concurrent SOAP client. We develop a hosting-grade anti-spam appliance, and an open source end-user web interface that allows users to interact with a cluster of anti-spam systems. Imagine a cluster with 10 nodes, and a user that views the 100 latest messages. Using JavaScript, that would require the client to download a listing of 10*100 messages, and with PHP it would result in 10 synchronous request.

Thanks to PHP’s flexibility, we could develop a cross-platform solution, without having to rely on a RPC multiplexer. We chose a select()ing method provided by CURL’s “multi” package, by extending the SoapClient class and implementing our own __doRequest.

You can download our helper script, and use it as

// Just demonstrating that it works with multiple functions, to multiple servers
$client1 = new SoapClientAsync('some-systems-wsdl', $options);
$client2 = new SoapClientAsync('another-systems-wsdl', $options);
$client1->someFunction($arguments);
$client1->anotherFunction($arguments);
$client2->anotherFunction($arguments);
soap_dispatch();
$result1 = $client1->someFunction($arguments);
$result2 = $client1->anotherFunction($arguments);
$result3 = $client2->anotherFunction($arguments);

We hope that our customers appreciate the speed boost that it gives their anti-spam system’s end-user web interfaces.