Courtesy update: There have been no further events or issues since the last update, ExchangeDefender operations are back to normal.
Thank you to all the clients and partners that opened the ticket and provided SMTP headers (or .eml/.msg files). During the first few minutes of the incident report we could not confirm or replicate the issue and we first attempted to confirm that the size of the message coming in is approximate to the size of the message going out of our network. This issue only occured on one node, one system, and was only caught by one partner in UK that manages a large email transaction business that can receive 100's of messages per second. We worked the issue backwards and located the problem within minutes.
Unfortunately, this required pausing mail flow in ExchangeDefender. All the messages have been processed and delivered, but with a delay. We did not want to risk corrupting more mail. Because this issue was occuring at the network service level (i.e., before the message is ever written to the disk) this was the only recourse and it's the first time this has happened in the company history. As a result, we will be adding a network service facility to do data duplication of incoming traffic so that it can be paused and replayed if a catastrophic event occurs between ExchangeDefender and target Inbox (more details on this soon).
Again, huge thanks to everyone that took a moment to provide SMTP header or .eml/.msg. You make it possible for us to resolve catastrophic issues within minutes instead of hours or days and we're eternally thankful for that and for your business.
Update 02/21/2024 15:11 PM 22 days
We are restoring inbound mail flow as of 10:00 AM.
We have identified an issue in a 3rd party content parsing engine on one of our inbound processing nodes. Affected systems & software have been pulled from production for further evaluation and troubleshooting. Mail flow will be slightly delayed for a few minutes and then resume normal operations.
Update 02/21/2024 14:44 PM 22 days
We are currently researching an urgent issue with Exchange recipients receiving messages with empty subject and empty body.
We still have not been able to reproduce or confirm the issue and at 9:35am EST we paused inbound mail processing to investigate.
We will keep you updated here.