Message Deliverability: IP Blacklisting Event
Incident Report for Intercom
Resolved
What happened:

We recently experienced a spike in email volume across our customer base in what we believe to be GDPR-related outreach to users. While these communications were well-intentioned, a group of our customers sent email to inactive users, including spam trap addresses.

This led a spam monitoring organization called Spamhaus to flag Intercom in their blacklist. As a result, over the past two weeks, approximately 16% of our outbound emails were not delivered (14% to Microsoft domains, 1% to Apple domains, and the remainder to other domains). Emails going to any Google Apps, Gmail, Yahoo and AOL addresses were delivered as expected.

What we did about it:

Complete resolution of this issue has been the highest priority for our engineering teams for the last 2 weeks. We worked quickly to remove ourselves from the Spamhaus list and bolster our capabilities to detect irregularities and ensure email deliverability.

All of our sending IPs have now been delisted by Spamhaus as a result of extensive work on improving the quality of emails sent via Intercom. All emails to users and admins should be delivering as normal.

To prevent incidents like this happening again, we’re investing heavily in spam monitoring technology and detection resources, so we can identify irregular behaviour before it becomes an issue. This way we can help our customers maintain their reach and stay in good standing.

We are going to significantly increase the amount of engineering capacity dedicated to email deliverability, including more comprehensive checks and balances to ensure proper use of Intercom.

We appreciate that you entrust us to help you communicate with your customers and it’s not something we take lightly. These actions and others we’re taking should prevent similar incidents from happening again. If you have specific questions, do not hesitate to contact our Support team at team@intercom.io or via the Messenger.
Posted Jun 25, 2018 - 17:15 UTC
Monitoring
All of our sending IPs have now been delisted by Spamhaus as a result of extensive work on improving the quality of emails sent via Intercom.

All emails to users and admins should be delivering as normal. Delivery at Microsoft and Apple is returning to normal, but some smaller providers may take longer as their lists continue to update. Senders following best practices should be experiencing no deliverability issues.

Customers who don't follow best practices may still see impact in email deliverability. Please review the guidelines listed below for best practice advice on email sending.
Posted Jun 25, 2018 - 11:42 UTC
Update
After continuing to focus on improving the quality of emails using Intercom, we can confirm that more of our sending IP address and domain have been delisted from Spamhaus.

This means that customers following our best-practice advice around email sending will begin to see new emails sent via Intercom being delivered into email systems that use Spamhaus filtering. Previously sent emails that have failed to deliver will need to be re-sent.

We advise customers to follow our documentation below to help ensure good email deliverability. Following these guidelines will help increase the potential of your emails to be successfully delivered.
https://docs.intercom.com/onboard-and-engage-customers/ensure-your-emails-get-delivered/protect-your-app-from-high-bounce-rates

We also have some general guidelines below around sending emails that comply with our Terms of Service and Privacy Policy:
https://docs.intercom.com/pricing-privacy-and-terms/terms-of-service/intercoms-email-sending-guidelines#

Notifications (conversational email), new customers, very low volume senders, and customers who don't follow best practices may still see impact in email deliverability until all sending IPs have been removed from the Spamhaus listing.
Posted Jun 22, 2018 - 15:38 UTC
Update
We have had some of our email sending IPs delisted from Spamhaus, and continue to actively monitor our email sending statistics to measure the recovery. In effect, we are seeing improved delivery rates for transactional emails (password resets, CSV exports etc), as well as email messages and campaigns. We believe the work that we have been doing around enhancing our capabilities to detect and mitigate bad sending has been impactful towards the delisting.

We continue to monitor our sending statistics for notifications (conversational email), new accounts, and very low volume senders, which are still impacted at this time. We are still working to delist the IPs associated with these categories of sending with the highest priority.
Posted Jun 21, 2018 - 18:18 UTC
Update
Overnight, we have continued to monitor our email sending statistics. Although we still see impacted delivery to email providers using Spamhaus services, we are using these statistics to ensure that the quality of email that is being sent via Intercom remains high. We will keep working on resolution at the highest possible priority level.
Posted Jun 21, 2018 - 11:02 UTC
Update
We continue to work with our email vendor Sparkpost pending the delisting of our email sending IPs on the Spamhaus listings, but are still waiting on a response from Spamhaus.

Over the past 24 hours we have worked to monitor and verify the standards of customer sending behaviour through Intercom. Parallel to this, we have worked on standing up new sending infrastructure so that we can have it ready to go live once we have the all clear from Spamhaus, while also enhancing our tooling and proactive blocking of bad sending traffic.

We still cannot provide an ETA for resolution at this time as we await Spamhaus delisting, but we are prioritizing all work and vendor communications to get our email delivery back to expected levels.

We will continue to update at regular intervals or as status changes occur.
Posted Jun 20, 2018 - 16:29 UTC
Update
There have been no significant changes to the situation since our previous update. We are still awaiting on an update from Spamhaus and we continue to actively monitor our sending infrastructure for any improvements.
Posted Jun 20, 2018 - 09:35 UTC
Update
We are in talks with Spamhaus about our current blacklisting event, via our upstream email provider. We had previously tried to move traffic to new IPs and domains but had seen no improvement and the new infrastructure was also blacklisted by Spamhaus. We have now provided them with a list of certain IP addresses that we believe have our best traffic and show that we have reduced the amount of messages being sent that could potentially be marked as spam. We're asking if they will monitor these for improvement in sending quality. Going forward, we will be notifying them of any changes we plan to make to our sending infrastructure before doing so.

As of this update, we are still awaiting an update from Spamhaus about the health of our network, pending delisting.

Until we are delisted from Spamhaus, we will continue to see deliverability issues with email to Microsoft, Apple and other email providers who use Spamhaus feeds in their filtering services. We continue to monitor and verify good email delivering to other email providers such as Google Apps, Gmail, AOL and Yahoo, which are unaffected at this time.

We will continue to update at regular intervals or as status changes occur.
Posted Jun 19, 2018 - 10:48 UTC
Update
We continue to identify and clean up bad sending traffic on our network. We have an ongoing dialogue with Spamhaus, through our upstream email provider. We are awaiting a response but there is no change in status. We will update when we have more.
Posted Jun 18, 2018 - 19:08 UTC
Update
As the current situation for outbound emails from Intercom still shows degraded performance, we are giving a summary update of work we have undertaken so far.

All of our sending IP addresses and our main sending domain intercom-mail.com are listed on the Spamhaus blacklist. This happened due to poor performance from some senders around the 25th May and again around the 8th of June.

We can see approximately 16% of our outbound email is affected (14% to Microsoft domains, 1% to Apple, some low volume traffic to smaller domains). Emails going to any Google Apps, Gmail, Yahoo and AOL addresses are still delivering as expected.

We are working on a few tracks of work: cleaning up our traffic, escalating through our upstream email provider to Spamhaus and setting up new infrastructure to better isolate good senders from reputation issues and get transactional and notification emails working as expected.

As it stands, we have identified the bad traffic causing the issue and suspended email for poorly performing senders. We are currently waiting for Spamhaus to re-assess us for delisting, following these improvements. We don’t have an ETA for this at present.

We will update on all tracks of work as they progress, or by 18:00 UTC if there is no new information.
Posted Jun 18, 2018 - 11:22 UTC
Update
After more investigation, we can report that this issue is not impacting email delivery to Gmail, Google Apps, Yahoo, or AOL, which is most of our traffic.

We are still investigating delivery issues relating to Microsoft, Apple, and some small providers. We will continue to monitor the situation and we'll update with any developments as they occur.
Posted Jun 15, 2018 - 18:36 UTC
Update
We continue to work with our upstream email providers to resolve the IP reputation issue that is affecting email deliverability, admin notifications, password resets and signups with the highest priority. Due to this issue, some customers will see elevated error rates in the placement of email messages into inboxes. We will continue to update here as the situation develops.
Posted Jun 14, 2018 - 14:36 UTC
Update
We are still seeing elevated error rates in delivering some email messages. We are working with the relevant providers to resolve this reputation issue and will post updates as we have them.
Posted Jun 12, 2018 - 18:09 UTC
Identified
Due to an IP reputation issue, we have identified that some of our messages are not delivering as expected. This is affecting some of our outbound emails sent from Intercom causing higher bounce rates for messages to end users. We will update here as soon as this issue is resolved.
Posted Jun 11, 2018 - 11:15 UTC
This incident affected: Intercom message delivery (Admin notifications).