Leonard Nimoy will be playing Spock in the next star trek movie… This makes one ask, will the set be wheel chair accessible, will someone please provide him a Rascal, or a Hoverround? I mean seriously, I think it’s time to let it go there Leonard. The movie is due out Christmas of 2008, lets hope he doesn’t pass away before the premiere ^.^
-Scott
Turns out (as we all expected) the drunk employee theory was bunk. The reason for the outage to the colo facility was that their emergency power systems did not work. 365Main has a HiTec flywheel UPS design, meaning they don’t use batteries. These big flywheels are able to sustain power to the facility for 11 seconds until the generators spool up.. obviously this didn’t quite work out. Perhaps there’s something to those good ol’ reliable battery backup systems that facilities like Equinix and ourselves use….
-S
It appears that a major data center in San Francisco which hosts some of the worlds most popular websites (Craigslist, LiveJournal, Technorati) suffered a major power systems failure when utility power was lost to the Center. There have been two initial reports first (the much more logical) report is that during a PE&G issue that cut power to the facility that backup systems did not engage properly *or* were not able to support the load. The other report (which is probably false) is that a employee got drunk and started on a rampage in Colo4 in the center. Persons that were in the Center during the outage report it as a utility failure and that generators did start but power was still lost.
The set up:
My business partner who resides in the SF bay area has a cable connection to comcast at his home. Until recently his connectivity to our colocation has been phenominal… that is until his IP changed. Apparently Comcast has different routing for different IP blocks; this proves to be a bit confusing as you would think it would be easier to load balance if all IPs were announcing the same and being routed the same. The MTR below shows the odd difference.
First his old IP which routed nicely through AT&T and had good stability (low packet loss / low jitter):
Packets Pings
Host Loss% Last Avg Best Wrst StDev
1. 209.104.4.225 0.0% 0.7 0.7 0.6 1.0 0.1
2. tus1.Login.COM 0.0% 0.4 0.4 0.3 0.5 0.0
3. phv-edge-03.inet.qwest.net 0.0% 7.6 20.7 7.1 138.5 41.4
4. phn-core-02.inet.qwest.net 0.0% 7.7 7.6 7.3 7.9 0.2
5. los-core-02.inet.qwest.net 0.0% 18.1 18.2 17.9 20.0 0.6
6. lap-brdr-01.inet.qwest.net 0.0% 18.0 22.3 18.0 43.8 9.2
7. 205.171.1.10 0.0% 20.9 20.9 20.6 22.6 0.6
8. tbr1.la2ca.ip.att.net 0.0% 32.7 36.9 32.4 72.7 12.6
9. tbr1.sffca.ip.att.net 0.0% 32.8 32.7 32.3 33.3 0.3
10. gar4.sffca.ip.att.net 0.0% 32.0 32.1 31.9 32.2 0.1
11. te-9-2-ar01.oakland.ca.sfba.comcast.net 0.0% 35.6 35.8 35.6 36.0 0.2
12. te-9-3-ur02.fairfield.ca.sfba.comcast.net 22.2% 36.6 36.6 36.4 36.7 0.1
13. te-9-3-ur01.fairfield.ca.sfba.comcast.net 0.0% 36.6 36.6 36.5 36.9 0.1
14. te-9-3-ur02.napa.ca.sfba.comcast.net 0.0% 37.5 37.6 37.4 38.5 0.3
15. GE-1-1-ur01.napa.ca.sfba.comcast.net 0.0% 37.4 37.3 37.2 37.5 0.1
16. ???
as you can see packets reach him 100% of the time (the packet loss on hop 12 is just a router shedding ICMP packets as it doesn’t show loss on the proceeding hops).
Now on one of his new IPs (this is no longer his IP and neither is the old).
Host Loss% Last Avg Best Wrst StDev
1. 209.104.4.225 0.0% 0.7 0.8 0.5 20.6 1.8
2. tus1.Login.COM 0.0% 0.4 0.6 0.3 23.9 2.2
3. so0-1-0.jr2.phx2.llnw.net 0.0% 13.4 13.6 13.3 33.7 2.1
4. ge3-1.fr4.phx2.llnw.net 0.0% 13.3 13.3 13.3 14.5 0.1
ge3-2.fr4.phx2.llnw.net
5. tge2-1.fr3.phx1.llnw.net 0.0% 13.3 13.7 13.3 21.8 1.5
6. ge-4-2-0.ar5.phx1.gblx.net 0.0% 23.3 26.5 23.2 110.3 13.9
ge-4-2-0.ar4.phx1.gblx.net
ge-3-2-0.ar4.phx1.gblx.net
ge-3-2-0.ar5.phx1.gblx.net
7. ge2-1-10G.ar4.LAX1.gblx.net 0.0% 27.2 35.0 23.2 234.4 39.5
te7-2-0GE.ar4.LAX1.gblx.net
8. 68.86.85.82 31.6% 24.7 24.6 23.9 32.9 1.2
9. Comcast-IP-Services-Sunnyvale.TenGigabitEthernet3-1.ar1.snv2.gblx.net 47.0% 33.1 39.4 33.0 198.0 24.8
10. 68.86.90.166 70.9% 34.7 35.6 34.6 53.2 3.2
11. te-9-2-ar01.oakland.ca.sfba.comcast.net 8.5% 39.3 39.8 39.0 69.8 3.1
12. te-9-3-ur02.fairfield.ca.sfba.comcast.net 14.5% 40.3 40.2 40.1 41.8 0.2
13. te-9-3-ur01.fairfield.ca.sfba.comcast.net 0.9% 40.2 40.2 40.1 41.3 0.1
14. te-9-3-ur02.napa.ca.sfba.comcast.net 2.6% 41.2 41.2 40.9 58.8 1.7
15. GE-1-1-ur01.napa.ca.sfba.comcast.net 0.9% 41.0 41.0 40.8 41.5 0.1
16. ???
You can see the horrible packet loss and 5ms more latency along with what looks like saturation between comcast and global crossing.
Come on Comcast you make billions a year, how about providing a quality service?
-Scott
My new company SPEAKservers LLC which is co-owned by Triphius and myself is starting to roll along nicely now. We’ve started to get regular inquiries and purchases :) If you are a avid online gamer I am sure you have heard of TeamSpeak, a online VoIP solution for games among other things. SPEAKservers sells these servers to the upper crust of online users who require mission critical uptime and connectivity.
Check out SPEAKservers 🙂
-Scott
