In our last post, we provided an overview of the recent changes implemented by Gmail and Yahoo (collectively referred to as Yahoogle) regarding email-sending standards. These changes, which focus on authentication, one-click unsubscribe, and low spam complaints, aim to enhance the overall email experience for users by encouraging senders to adopt better practices.
This post will delve into the technical guidelines for preparing for these changes using Halon Engage. If you are a deliverability expert working with large-scale senders, email service providers or martech platforms, this is an essential read. By understanding and implementing these technical strategies, you can ensure you are compliant with the new standards, maintain a strong sender reputation, and optimize your email deliverability in an increasingly stringent environment.
It is important to be prepared beforehand and take a proactive approach to avoid any damage occurring. There are two key ways to be ready:
Gmail and Yahoo require authentication to be implemented to send emails to their infrastructure. However, in many cases, the domain control is not in the email provider's hands, and DNS issues can occur. Any issue will lead to authentication failure at Gmail and Yahoo, eventually triggering a block. This leads to a reputation drop and further complications.
To prevent these issues from happening, even before the delivery is attempted, Halon Engage can check if authentication is passing for the sending domain. If this check passes, delivery is attempted; otherwise, emails can be transferred to a different queue or sending IP pool, delivery can be paused, a flag can be raised, etc. It can also be used as a routing decision during migrations. This proactive approach provides an opportunity to safeguard the infrastructure without waiting for Gmail and Yahoo to block the infrastructure and then react.
Initial check: Perform an authentication check for SPF, DKIM, and DMARC before attempting delivery.
Action on failure: If the authentication check fails, configure the system to take appropriate actions such as:
This can be implemented in the HSL code by making a pre-emptive check in the “EOD” (end of DATA) hook point, before the message is queued.
if(!preemptive_check(MailMessage::String($mail->toString()), $transaction, $ippool)) {
// This check failed, decide what to do
}
The preemptive_check()function is provided in source code form, and can be tailored to suit. It uses the Halon DMARC module to verify both DKIM and SPF records are present and correct on the mail in transit, just like the recipient mailbox provider would do.
Do proactive checks such as this take the system some time to run? Yes, because DNS checks can have significant round-trip time. Depending on your priorities, you may not want to enable it for every email, in all email streams. Consider the following:
Since Halon Engage 12, the DNS resolving functions (dns_query, spf_query, and verifyDKIM) uses async IO, freeing up script threads to do other useful work during the DNS round-trip time
With the power of Halon script, you can choose to apply these checks only to specific mail streams.
Implementing a one-click unsubscribe option is a straightforward way to ensure compliance and improve user experience. This feature allows users to easily unsubscribe from mailstreams, thereby reducing spam complaints and improving engagement metrics.
Header configuration: Use Halon Engage to automatically add the necessary List-Unsubscribe header to all outgoing emails, satisfying the “one click unsubscribe” requirement with the List-Unsubscribe-Post header. This can be configured for upstream applications that feed messages specifically via the extended submission API and via all message sources including SMTP.
Here’s a simple example of the code you’d use in your End-of-Data (EOD) message hook, with unique tracking for each recipient:
// Queue message for all recipients
$id = $mail->snapshot();
foreach ($recipients as $recipient) {
enable_tracking($mail, $recipient["recipient"], $tracking);
$mail->queue($sender, $recipient["address"], $recipient["transportid"]);
$mail->restore($id);
}
The sent message will contain correct headers with a HTTPS tracking URL, such as:
List-Unsubscribe-Post: List-Unsubscribe=One-Click
List-Unsubscribe: <https://link.example.com/u/AVxxx-_mFu5c5sGQTYzbDoIwEET_ZZ-16ZaLlBDDrxSsFmhZAhViCP_uypNPk8ycMzvMUMIS7WpFfLdD7YynUxxx_Z9pAuUO0oxkjew3RED6LI9p48_TqHtxKdKsqiqJ3Kk_QbjnqxxxSip-a0_r-O4ws>
The halon-submission-tracking add-on can track opens, clicks, and unsubscribe events, providing data streams in an easy-to-consume JSON format. An unsubscribe looks like this:
{
"type": "unsubscribe",
"event_id": "3f01df67-b5a6-11ec-b8dd-000c292d8854",
"timestamp": "1649248582414",
"recipient": "john.doe@example.com",
"message_id": "<469c01df-b5a6-11ec-b8dd-000c292d8854@c568c9a1654f>",
"ip": "1.2.3.4",
"user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.3 Safari/605.1.15",
"additional": {}
}
The events can be batched and streamed to your data services in near real-time, such as ElasticSearch/Kibana or any other type of data warehouse. You can have multiple, simultaneous streams to different destinations.
Managing email queues effectively is crucial for maintaining high deliverability and protecting sender reputation, especially in a shared infrastructure environment. Typically, when one sender experiences issues and gets blocked, other senders on the same infrastructure are also impacted. This means that a single block can negatively affect all other senders sharing the infrastructure. However, with Halon Engage, we've reimagined queue management to provide granular control at any level of email using per-domain tenant queues.
Halon Engage allows you to manage email queues based on sending IP, destination domain/MX, sending domain, job id, or tenant id. This level of control ensures that delivery issues for one tenant do not cause a queue block for other tenants on the same shared sending IP. By isolating the impact of one sending domain's authentication failure, you can ensure that other domains remain unaffected, even when using the same infrastructure.
Here’s a simple policy file snippet. The “tenantid” values can be set up as the message is queued with great flexibility in HSL; the simplest is directly setting from the From: address domain. Here we see two tenants with their own specific concurrency and rate settings to Google, as well as overall default settings per-tenant.
Let’s review a scenario. The tenant “dance-studios.com” is a domain that’s still warming up i.e. establishing their reputation with Google (and also with the Halon platform owner). We provide a low concurrency setting of 1, and a low rate of 10 messages per minute. That ensures the new sender is only a small proportion of the overall traffic on a shared IP pool, perhaps.
The tenant “eddies-emporium.com” is at a more advanced reputation stage, so we allow them to have 10 concurrent connections and higher message rates.
The default, fully warmed-up tenants are granted up to 20 concurrent connections. However all tenants will respect the “backoff” rules. If messages bounce with specific warning codes, which may indicate list quality issues for example, that tenant will be suspended for a configurable period and will follow a “backoff” retry schedule until the problem is resolved. This protects both the tenant’s domain reputation and the overall platform / sending IP reputation.
policies:
- fields:
- jobid
- fields:
- localip
- fields:
- tenantid
- grouping
default:
concurrency: 20
properties:
backoff-retry-intervals: 600,1200,1800,3600
backoff-concurrency: 1
backoff-rate: 60/3600
backoff-ttl: 3600
backoff-disableable: true
backoff-suspendable: true
conditions:
- if:
tenantid:
- dance-studios.com
grouping:
- "&google"
then:
concurrency: 1
rate: 10/60
- if:
tenantid:
- eddies-emporium.com
grouping:
- "&google"
then:
concurrency: 10
rate: 1000/60
These settings work alongside other controls you may have, such as overall tunables per destination grouping (for Google, Yahoo and so on).
Here we see dance-studios.com has triggered that policy, and is being rate-limited - protecting both themselves and other senders.
But wait, there’s more! Let’s say you notice that Eddie’s Emporium is getting into trouble by sending on their 10 connections. In the Delivery Insights dashboard, you can directly create a dynamic rule, based on Tenant ID and Grouping, to reduce their concurrency for a while (in this case, 1 hour).
We can tag this rule with a name, so it shows up in the Delivery Insights for the messages.
In summary: Halon provides a powerful means to monitor and manage individual mail streams with the fine granularity expected by mailbox providers. You can work proactively by setting up specific policies, and reactively via the Delivery Insights dashboard.
Even with proactive measures in place, errors can still occur due to authentication failures or spam complaints. When they do happen, it's crucial to handle these errors effectively to mitigate their impact on your email infrastructure. Halon Engage provides robust reactive error handling capabilities to manage these situations efficiently.
◦ Verifying and correcting SPF, DKIM, and DMARC records
◦ Analyzing spam complaint data to identify patterns and underlying causes
◦ Implementing corrective actions, such as improving email content or adjusting sending practices.
Halon’s Bounce Patrol package provides a ready defined set of rules (known as “regexes”) for mapping responses from mailbox providers. For example, we recently updated this to handle new codes from Google, such as this:
/^4[25]1 4\.7\.2[3-7]/,badsender,&google
By leveraging the above comprehensive features of Halon Engage, you are well-equipped to navigate the new guidelines set forth by Gmail and Yahoo. These features not only ensure compliance but also enhance your ability to deliver relevant and engaging emails to your audience. Proactively embracing these changes will lead to a more secure and effective email marketing strategy, positioning your business for continued success in the evolving landscape of email deliverability.
With Halon Engage, you're not just prepared for today's standards; you're future-proofing your email infrastructure to adapt and thrive amidst ongoing and upcoming regulatory changes. This forward-thinking approach ensures that your email operations remain resilient, reliable, and ready for the future.