Acing Gmail and Yahoo's new sender guidelines

In our last post, we provided an overview of the recent changes implemented by Gmail and Yahoo (collectively referred to as Yahoogle) regarding email-sending standards. These changes, which focus on authentication, one-click unsubscribe, and low spam complaints, aim to enhance the overall email experience for users by encouraging senders to adopt better practices.

This post will delve into the technical guidelines for preparing for these changes using Halon Engage. If you are a deliverability expert working with large-scale senders, email service providers or martech platforms, this is an essential read. By understanding and implementing these technical strategies, you can ensure you are compliant with the new standards, maintain a strong sender reputation, and optimize your email deliverability in an increasingly stringent environment.

Be prepared

It is important to be prepared beforehand and take a proactive approach to avoid any damage occurring. There are two key ways to be ready:

Proactive authentication check

Gmail and Yahoo require authentication to be implemented to send emails to their infrastructure. However, in many cases, the domain control is not in the email provider's hands, and DNS issues can occur. Any issue will lead to authentication failure at Gmail and Yahoo, eventually triggering a block. This leads to a reputation drop and further complications.

To prevent these issues from happening, even before the delivery is attempted, Halon Engage can check if authentication is passing for the sending domain. If this check passes, delivery is attempted; otherwise, emails can be transferred to a different queue or sending IP pool, delivery can be paused, a flag can be raised, etc. It can also be used as a routing decision during migrations. This proactive approach provides an opportunity to safeguard the infrastructure without waiting for Gmail and Yahoo to block the infrastructure and then react.

Implementation plan

Initial check: Perform an authentication check for SPF, DKIM, and DMARC before attempting delivery.

Action on failure: If the authentication check fails, configure the system to take appropriate actions such as:

Transferring emails to a different queue
Pausing the delivery
Rejecting messages from being accepted by Halon
Raising a flag or sending an alert to the deliverability team

This can be implemented in the HSL code by making a pre-emptive check in the “EOD” (end of DATA) hook point, before the message is queued.

  if(!preemptive_check(MailMessage::String($mail->toString()), $transaction, $ippool)) {
           // This check failed, decide what to do
    }

The preemptive_check()function is provided in source code form, and can be tailored to suit. It uses the Halon DMARC module to verify both DKIM and SPF records are present and correct on the mail in transit, just like the recipient mailbox provider would do.

Do proactive checks such as this take the system some time to run? Yes, because DNS checks can have significant round-trip time. Depending on your priorities, you may not want to enable it for every email, in all email streams. Consider the following:

Since Halon Engage 12, the DNS resolving functions (dns_query, spf_query, and verifyDKIM) uses async IO, freeing up script threads to do other useful work during the DNS round-trip time
The DNS resolver leverages a local cache, so existent records are returned quickly
Returned results from the checking functions can also be cached, e.g. for a reasonable time such as 1 hour
For continuous checking of very high-traffic email streams, reactive real-time response to DMARC failure error codes from Google and Yahoo to trigger a suspension can be used instead.

With the power of Halon script, you can choose to apply these checks only to specific mail streams.

One-click unsubscribe

Implementing a one-click unsubscribe option is a straightforward way to ensure compliance and improve user experience. This feature allows users to easily unsubscribe from mailstreams, thereby reducing spam complaints and improving engagement metrics.

Implementation plan

Header configuration: Use Halon Engage to automatically add the necessary List-Unsubscribe header to all outgoing emails, satisfying the “one click unsubscribe” requirement with the List-Unsubscribe-Post header. This can be configured for upstream applications that feed messages specifically via the extended submission API and via all message sources including SMTP.

Here’s a simple example of the code you’d use in your End-of-Data (EOD) message hook, with unique tracking for each recipient:

// Queue message for all recipients
$id = $mail->snapshot();
foreach ($recipients as $recipient) {
    enable_tracking($mail, $recipient["recipient"], $tracking);
    $mail->queue($sender, $recipient["address"], $recipient["transportid"]);
    $mail->restore($id);
}

The sent message will contain correct headers with a HTTPS tracking URL, such as:

List-Unsubscribe-Post: List-Unsubscribe=One-Click
List-Unsubscribe: <https://link.example.com/u/AVxxx-_mFu5c5sGQTYzbDoIwEET_ZZ-16ZaLlBDDrxSsFmhZAhViCP_uypNPk8ycMzvMUMIS7WpFfLdD7YynUxxx_Z9pAuUO0oxkjew3RED6LI9p48_TqHtxKdKsqiqJ3Kk_QbjnqxxxSip-a0_r-O4ws>

The halon-submission-tracking add-on can track opens, clicks, and unsubscribe events, providing data streams in an easy-to-consume JSON format. An unsubscribe looks like this:


{
    "type": "unsubscribe",
    "event_id":  "3f01df67-b5a6-11ec-b8dd-000c292d8854",
    "timestamp": "1649248582414",
    "recipient": "john.doe@example.com",
    "message_id": "<469c01df-b5a6-11ec-b8dd-000c292d8854@c568c9a1654f>",
    "ip": "1.2.3.4",
    "user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.3 Safari/605.1.15",
    "additional": {}
}

The events can be batched and streamed to your data services in near real-time, such as ElasticSearch/Kibana or any other type of data warehouse. You can have multiple, simultaneous streams to different destinations.

Enhanced queue management

Managing email queues effectively is crucial for maintaining high deliverability and protecting sender reputation, especially in a shared infrastructure environment. Typically, when one sender experiences issues and gets blocked, other senders on the same infrastructure are also impacted. This means that a single block can negatively affect all other senders sharing the infrastructure. However, with Halon Engage, we've reimagined queue management to provide granular control at any level of email using per-domain tenant queues.

Per-domain tenant queues:

Halon Engage allows you to manage email queues based on sending IP, destination domain/MX, sending domain, job id, or tenant id. This level of control ensures that delivery issues for one tenant do not cause a queue block for other tenants on the same shared sending IP. By isolating the impact of one sending domain's authentication failure, you can ensure that other domains remain unaffected, even when using the same infrastructure.

Implementation plan

Queue segmentation: Set up separate queues for each sending domain or tenant ID. This involves configuring the system to assign emails to different queues based on their originating domain or tenant.
Real-time monitoring: Implement real-time monitoring to track the performance of each queue independently. This helps in quickly identifying and addressing issues specific to any domain or tenant.
Automated actions: Configure automated actions for queues that encounter issues. For instance, if a queue experiences a high rate of authentication failures, Halon Engage can be configured to automatically pause the queue. Additionally, the deliverability team can be notified and emails can be rerouted through an alternate infrastructure manually.
Failover mechanisms: Establish failover mechanisms to ensure continuity of service. If a particular queue is paused due to issues, emails can be automatically rerouted to another queue or infrastructure, minimizing disruption to service.

Benefits:

Enhanced resilience: Isolating issues to specific queues prevents widespread impact and maintains high deliverability for other tenants.
Improved control: Granular control over queues allows for more targeted troubleshooting and resolution of issues.
Scalability: The system can scale with the growth of email volume and the addition of new tenants, without compromising on deliverability or reputation.

Here’s a simple policy file snippet. The “tenantid” values can be set up as the message is queued with great flexibility in HSL; the simplest is directly setting from the From: address domain. Here we see two tenants with their own specific concurrency and rate settings to Google, as well as overall default settings per-tenant.

Let’s review a scenario. The tenant “dance-studios.com” is a domain that’s still warming up i.e. establishing their reputation with Google (and also with the Halon platform owner). We provide a low concurrency setting of 1, and a low rate of 10 messages per minute. That ensures the new sender is only a small proportion of the overall traffic on a shared IP pool, perhaps.

The tenant “eddies-emporium.com” is at a more advanced reputation stage, so we allow them to have 10 concurrent connections and higher message rates.

The default, fully warmed-up tenants are granted up to 20 concurrent connections. However all tenants will respect the “backoff” rules. If messages bounce with specific warning codes, which may indicate list quality issues for example, that tenant will be suspended for a configurable period and will follow a “backoff” retry schedule until the problem is resolved. This protects both the tenant’s domain reputation and the overall platform / sending IP reputation.

policies:
 - fields:
   - jobid
 - fields:
   - localip
 - fields:
   - tenantid
   - grouping
   default:
     concurrency: 20
     properties:
       backoff-retry-intervals: 600,1200,1800,3600
       backoff-concurrency: 1
       backoff-rate: 60/3600
       backoff-ttl: 3600
       backoff-disableable: true
       backoff-suspendable: true
   conditions:
     - if:
         tenantid:
         - dance-studios.com
         grouping:
         - "&google"
       then:
         concurrency: 1
         rate: 10/60
     - if:
         tenantid:
         - eddies-emporium.com
         grouping:
         - "&google"
       then:
         concurrency: 10
         rate: 1000/60

These settings work alongside other controls you may have, such as overall tunables per destination grouping (for Google, Yahoo and so on).

Here we see dance-studios.com has triggered that policy, and is being rate-limited - protecting both themselves and other senders.

But wait, there’s more! Let’s say you notice that Eddie’s Emporium is getting into trouble by sending on their 10 connections. In the Delivery Insights dashboard, you can directly create a dynamic rule, based on Tenant ID and Grouping, to reduce their concurrency for a while (in this case, 1 hour).

We can tag this rule with a name, so it shows up in the Delivery Insights for the messages.

In summary: Halon provides a powerful means to monitor and manage individual mail streams with the fine granularity expected by mailbox providers. You can work proactively by setting up specific policies, and reactively via the Delivery Insights dashboard.

Reactive error handling

Even with proactive measures in place, errors can still occur due to authentication failures or spam complaints. When they do happen, it's crucial to handle these errors effectively to mitigate their impact on your email infrastructure. Halon Engage provides robust reactive error handling capabilities to manage these situations efficiently.

Implementation plan

Error detection: Configure Halon Engage to monitor all Gmail and Yahoo-related errors (listed here).

Queue isolation: Immediately isolate the affected queue to prevent further damage. This ensures that the issue does not spread to other queues or affect overall deliverability.
Automatic rate-limiting: Automatically rate-limit the delivery of emails in the affected queue. This allows time to diagnose and resolve the issue without causing further problems.
Notification system: Set up a notification system to alert administrators and relevant stakeholders about the error. Notifications should include detailed information about the error and suggested next steps.
Error resolution workflow: Develop a workflow for resolving the error. This could involve:

◦ Verifying and correcting SPF, DKIM, and DMARC records

◦ Analyzing spam complaint data to identify patterns and underlying causes

◦ Implementing corrective actions, such as improving email content or adjusting sending practices.

Retry mechanism: Implement a retry mechanism to re-attempt delivery after resolving the issue. Ensure that retries are done in a controlled manner to avoid overwhelming the infrastructure.
Reporting and analytics: Utilize reporting and analytics tools to track error occurrences and resolutions. This data can provide insights into common issues and help in refining proactive measures.

Halon’s Bounce Patrol package provides a ready defined set of rules (known as “regexes”) for mapping responses from mailbox providers. For example, we recently updated this to handle new codes from Google, such as this:

/^4[25]1 4\.7\.2[3-7]/,badsender,&google

By leveraging the above comprehensive features of Halon Engage, you are well-equipped to navigate the new guidelines set forth by Gmail and Yahoo. These features not only ensure compliance but also enhance your ability to deliver relevant and engaging emails to your audience. Proactively embracing these changes will lead to a more secure and effective email marketing strategy, positioning your business for continued success in the evolving landscape of email deliverability.

With Halon Engage, you're not just prepared for today's standards; you're future-proofing your email infrastructure to adapt and thrive amidst ongoing and upcoming regulatory changes. This forward-thinking approach ensures that your email operations remain resilient, reliable, and ready for the future.