At 8:12AM our Dallas DC firewall infrastructure suffered a catastrophic failure and failover wasn't instant across all services. Failure in one of the firewalls disconnected several private VPN connections that isolate/separate and protect the ExchangeDefender's internal services. We were forced to restart firewalls, which had to reestablish VPN connections, which required all the services to reload configurations simultaneously as routing work was started to move traffic around and away from Dallas. At about 8:24AM all services have been confirmed as green and back in services.
We have double checked every system and everything is back to normal.
Outage lasted approximately 10 minutes.
Update 02/13/2020 13:39 PM 7 days
Private VPN tunnels are back up and all the log, maillog, ips, services are verified to be back in sync.
Update 02/13/2020 13:30 PM 7 days
As of 8:21 the routing and almost all the applications have been restored. We are still working on what caused the problem and how to mitigate it automatically.
Update 02/13/2020 13:20 PM 7 days
We are working on a massive outage in Dallas. Service outages have been reported, we are working on it and will provide updates every 15 minutes as they become available.