What are the Dangers of Failing to Test DCN Properly? Real-World Mishaps

According to Network Computing and the Meta Group, the hourly cost of an hour of downtime ranges from $90 thousand for media to $6.48 million for a brokerage service, with telecom hitting in the middle at $2 million.* You don’t have to look far to find dramatic examples of the consequences of inadequate testing, or perhaps no testing at all. Mishaps that occurred where proper testing was not implemented and where damage to the enterprise occurred, include:

  • April, 2013: A Bitcoin Service Site, prominently positioned in the lucrative bleeding edge cryptocurrency market–a digital monetary system outside traditional institutions. The Bitcoin services site Instawallet was hacked and suffered a seismic security breach.
    Damage: The site immediately went down and faced a deluge of claims against it. Bitcoin transactions being irreversible and anonymous make it virtually impossible to reclaim losses after theft. Unable to recover, the enterprise was deemed closed permanently by the following July.
  • In May 2012, NASDAQ experienced an embarrassing outage during the high-profile Facebook IPO. Occurring right at the opening of trading, it prevented brokers and investors from confirming or cancelling trades, while high-frequency robotrading amplified the outage failure. Law suits against NASDAQ, Facebook, and banks resulted, as well as an SEC investigation. NASDAQ’s brand injury was serious, impacting the investment community’s perspective on the fairness of trading practices.
  • Also in the spring of 2012, Visa’s network had a 45-minute outage that was due to a system update. Credit card transactions ceased and the risk of fraudulent transactions offline soared as customer satisfaction dropped. Exact financial impact is undisclosed, but standard metrics of failure at this scale are in the millions.
  • In the spring of 2011, the Sony PlayStation network was compromised, exposing the credit card numbers and other personal information of 77 million users, creating a 24-day outage. Sony spent $171 million in the first month dealing with the breach the company’s ultimate estimated loss is over $24 billion. The cost to credit card companies to issue replacement cards is estimated at $300 million.

Some IT leaders think these stories don’t apply to them as their enterprises and scale of infrastructure are different. However, the threat of failure extends beyond these global enterprise failures. Case in point: Sutter Health Key Regional Health Care Provider had a breach in October 2011 that compromised HIPAA data for over 4 million patients and patented data as well. A $1 to 4 billion class action lawsuit resulted from the security failure.

While NASDAQ insisted they tested before the launch of their initiative, the key was proper testing, which they did not perform with possible corner cases. Understanding the proper testing best practices and having the right qualified team to execute this testing is key to any IT initiative success.

Take this into consideration as well: Recent SANS research indicates that a large number of businesses will fail after being down for more than 72 hours. With the fast-moving competitive market today, any of the mishaps cited above are seismic enough to severely damage if not seriously cripple many enterprises today.


*Source: Network Computing, the Meta Group and Contingency Planning Research

comments powered by Disqus
× Spirent.com uses cookies to enhance and streamline your experience. By continuing to browse our site, you are agreeing to the use of cookies.