Ben J. Christensen » Production Problems

GMail Down for 100 Minutes

Ben Christensen — Wed, 02 Sep 2009 17:11:34 +0000

“Gmail’s web interface had a widespread outage earlier today, lasting about 100 minutes.”

See the following for details:

http://gmailblog.blogspot.com/2009/09/more-on-todays-gmail-issue.html

Summary of Bad Week for Datacenters

Ben Christensen — Mon, 13 Jul 2009 16:10:22 +0000

A summary of a bad week for Rackspace, Google and others:

http://www.datacenterknowledge.com/archives/2009/07/06/the-day-after-a-brutal-week-for-uptime/

Rackspace DFW Power Issues. Again.

Ben Christensen — Thu, 09 Jul 2009 21:38:16 +0000

Twice in the last two weeks the Rackspace DFW (Dallas/Fort-Worth) data center has lost power to sections of its facility.

Message from Rackspace CEO Lanham Napier

June 30, 2009

Rackspace community,

Yesterday afternoon at 3:15CDT our data center in Dallas experienced an interruption in power to portions of the facility. The interruption caused customer servers to lose power and go down.

…

Notice * July 7, 2009, 11:44 am CDT: Today a portion of our Dallas data center experienced a brief power interruption. Rackspace is aware of this issue and is currently investigating it. We will be sending out periodic updates as more information becomes available.

I commend their transparency through the use of Twitter and their blog, but rather disconcerting since it’s the datacenter I use and pay a premium for the amount of “redundancy” they are supposed to have, and the fact they had issues in November 2007 as well. More information here about that outage.

Google App Engine Down for 6 Hours

Ben Christensen — Thu, 09 Jul 2009 21:21:20 +0000

In my ongoing collection of “big guys having trouble”, Google’s “App Engine” services were down for 6 hours on July 2nd.

A scathing rant about Google’s lack of clarity on what went wrong and what’s being done about it – and therefore the lack of confidence in using the service.

Google’s posting about the issue are found here.

Netbeans Heap Profiler Works. Well.

Ben Christensen — Tue, 07 Jul 2009 20:23:57 +0000

Netbeans (which I never use) has seemingly come a long way since I last looked at it. The UI is certainly a lot nicer.

Today however, it’s the Heap Profiler that I’m happy with – cause it actually works unlike anything else I’ve tried today.

It loaded a 3.5GB heap in less than 30 seconds! (I had set my max heap for Netbeans to 12GB on my 16GB, 8-core machine).

Finally a tool for heap analysis that works and works well. And it’s elegant looking at the same time.

Memory Analyzer Can’t Handle Large Heaps

Ben Christensen — Tue, 07 Jul 2009 20:10:33 +0000

Despite the claims that Memory Analyzer works well with large heaps, the following screenshot is the evidence of my continued inability to have it parse a 3.5GB heap dump.

I have attempted JDK 5 and JDK 6, both 64-bit, with up to 14GB of memory allocated on an 8-core machine with 16GB of memory.

Note the memory bar at the bottom showing it’s using only 2121M out of 11879M – yet it still thinks it’s running out of memory.

The settings are:

-vmargs

-Xms12g

-Xmx14g

-XX:MaxPermSize=1G

-Dorg.eclipse.swt.internal.carbon.smallFonts

-XstartOnFirstThread

-vmargs

-Xms12g

-Xmx14g

-XX:MaxPermSize=1G

-Dorg.eclipse.swt.internal.carbon.smallFonts

-XstartOnFirstThread

Atlassian has Outage

Ben Christensen — Thu, 02 Jul 2009 18:35:33 +0000

I couldn’t access my hosted Confluence system this morning. An email just showed up confirming the issue.

“Earlier today, from 9:41 am to 10:17am PST, we experienced network issues that impacted our data center. At this time we do not know the exact root cause of the problem. “

— from an email I received

GMail down for 4 hours

Ben Christensen — Wed, 25 Feb 2009 04:43:05 +0000

Hosted Quickbooks Dead

Ben Christensen — Mon, 02 Feb 2009 18:02:59 +0000

More big guys having issues.

Google “Woops” – Human Error Marks All Sites As Malware

Ben Christensen — Mon, 02 Feb 2009 17:13:00 +0000

Human errors causes all sites to be marked as “malware”.

http://news.cnet.com/8301-1001_3-10153942-92.html?tag=mncol;txt