As I wrote yesterday, the outage at 8x8 wasn't likely their fault so i was quite pleased when their PR person, Joan Citelli, sent me this note today:
I wanted to provide you with an update/explanation of what caused 8x8’s service outage this past Tuesday.
In order to provide services to our customers from our three, redundant data centers, 8x8 utilizes several, non-contiguous Internet Protocol (IP) address blocks that are provided to us by Tier-1 Internet Service Providers (ISPs). Our service outage on January 12, 2010 was caused when an unaffiliated Tier-1 ISP interfered with the proper routing of all of these address blocks. While we were able to immediately restore service for some customers by broadcasting correct routing information via our backup data providers, the interference from the unaffiliated ISP continued for approximately four hours.
All 8x8 core software and networks continued to operate normally during the outage, but some customers were unable to reach our services. We are in the process of planning and implementing additional mechanisms to protect our customers from any similar routing interference in the future. We are also creating alternative information and communication mechanisms to more quickly communicate outage status and estimated time of repair information to our customers via our website, call centers and other communication channels should a similar issue affect our customers in the future.
Feel free to update your blog with this information.
Any bets on whom the offending culprits were? Level3 has been a supplier to most VoIP companies including 8x8 and most likely AT&T requested control of the IP address blocks as they are doing massive network work these days. Given how both companies play the media relations game, I would be surprised if we ever heard an "mea culpa."