Late summer of 2022, there was a major multi-day internet outage that happened in Canada that affected many businesses, small and large. Let’s discuss what happened and some of the steps you can take to protect your business if something similar happens to you.
In July of 2022, Canada’s Rogers Communications suffered a major internet outage. For those of you who haven’t heard of Rogers, it is a very large internet and wireless service provider, on the scale of ~ 20 billion US dollars and 26,000 employees. Rogers suffered an outage due to a packet storm on their network. It was caused by a software upgrade that accidentally removed all the traffic filters to the internet from their network. Without the filters, the Rogers network equipment became quickly overwhelmed and stopped functioning. Rogers serves about 30% of Canada’s internet capacity, so this was a big event and affected many people and businesses. It took about 2-3 days to fully restore services to all customers.
So, as a business delivering services to your customers, what would you have done in this case, if your internet service provider was down for multiple days? Businesses who had invested in SD-WAN and redundant ISPs were generally okay. However, those without redundancy, or without an SD-WAN platform to prioritize internet service, they unfortunately suffered catastrophic downtime. Some of them even had to rely on manual processes to keep their business running. They would never have thought service from a large and reliable vendor like Rogers would come to a halt like this, so the investment didn’t seem needed. Businesses with backup internet access generally made it through, and the ones that had SD-WAN did not experience any degradation in service, at all. This is because SD-WAN prioritizes real-time traffic, by application. Even if your redundant internet connection is based on inexpensive broadband, SD-WAN will automatically prioritize your most important applications, so that things like voice and point-of-sale (POS) have the capacity they need, while other less critical applications are limited to what’s left over. This is a significant benefit of SD-WAN and something to consider if real-time traffic is important to your business, because without it, your typical failover setup means a loss in all active communications which would need to be re-established once the backup connection becomes active.
SD-WAN has a couple of other key features as it relates to protection. For instance if your SD-WAN provider is a gateway or private middle mile service, your inbound traffic is also protected because inbound packets will be routed over the best available circuit. Also, if you’re using an SD-WAN provider with gateways or private middle miles, you’ll have built in DDoS protection. So SD-WAN provides a number of protections aside from just managing network resiliency.
Here’s the interesting takeaway. Even if you had a resilient design, and you had sufficient internet, and you had SD-WAN in place, your vendors and business partners may not. With the Rogers outage, a major credit card transaction system was taken offline, leaving their point of sale customers without the ability to accept credit cards. Another example was cell phone service being disrupted as well. We can imagine that many businesses with critical vendors and applications experienced an impact even if they themselves continue to have strong internet service. It might be far-fetched to try and identify whether or not your key business partners have network redundancy and utilize SD-WAN, but identifying the key providers and confirming with them may actually be a new priority element in your business continuity plan.
Beyond internet access, what other key services might be exposed to vendor interruption. Is voice critical to your business? Are you using Microsoft Teams for voice or collaboration? For example, what would happen if Microsoft had an issue or you had to quarantine Office 365 because of a cyber event?
If you’re curious how to protect your environments and learn more about our maintenance-free SD-WAN service, get in touch with one of our Managed SD-WAN specialists.