More speculation on one of the year’s biggest website outages

I have been amused the last few days at a stream of little web stories from big sites/publishers popping up in my Google Alerts originating with a SmartBear Software blog post on the biggest web outages of 2011.
For the adviser that is unaware SmartBear acquired AlertSite Inc., a website monitoring company of long standing back in April (some trivia for you, my former colleague Larry Selzer over at PC Magazine wrote one of the first reviews of AlertSite back in 2001).
Anyway, in addition to the likes of Netflix and Target was Bank of America, which took top honors with its October outage.
I speculated on the reasons for this outage at the time (see Mystery Inc.: Bank of America online outages a puzzler) and nothing definitive came of it then so I thought I would do a teensy bit more speculating by getting in touch with someone at SmartBear.
That someone is Ken Godskind, vice president of monitoring products at SmartBear.
We had a long and interesting conversation and I will definitely seek his counsel again in future.
Here is a shortened version of our conversation:

“First, their online banking is fantastic, however, that was an outage of a very long duration for any significant brand,” said Mr. Godskind at the beginning of our discussion.
“Stuff breaks though, people makes mistakes,” he added citing another big brand that had a major outage in April.
While Bank of America had two briefer outages earlier in the year, he said that it was this long and intermittent outage in October that really struck him and left him scratching his head.
We next talked for a spell about how things had changed in the website monitoring business since I first started tracking it a decade ago.
Back then a monitoring service like AlertSite could simply ping sites to make sure they were up and able to return the site to a web browser.
Fast forward to today and websites themselves have gotten far more complex. Now most sites are not static, they offer some level of interactivity and it is all about application delivery.
A service like AlertSite has to do much more than operate a few monitoring sites and Internet backbones, it has to actually measure that interactivity to make sure a site is actually working.
“We look at that mythical eighth layer of the OSI model,” Mr. Godskind said referring to how firms like his have to try and account for a much more sophisticated Internet today.
Theories: The timing of this happened around the time that Bank of America announced the $5-per-month debit card fee, which surely brought in a much higher load of traffic than was normally expected.
We agreed that it was unlikely to have been a denial of service attack as was speculated by some at the time.
“There was a fairly high correlation with outages when you expected user traffic and the site being up when there was much less traffic,” he said, the latter referring to times late at night or early morning.
We debated a range of other theories and ruled them out.
“So you could look at another potential theory here: was it an application upgrade? The results of which being that they [BofA] were slower handling the same amount of traffic,” he added.
Another possibility is that there was a hardware failure he said and that perhaps the bank had trouble chasing it down.
He explained that while he had not worked with Bank of America, other big companies he was familiar with had layers and sometimes layers upon layers of outsourced technology services.
And that would explain, in addition to the lack of transparency about providing explanations for such things — which is actually policy at Bank of America and other big institutions — why the outages were intermittent and ongoing.
They just had a difficult time tracking down what the problem was.
Perhaps some switch or router somewhere along the way or in-between their systems and someone else’s that was on the fritz and not yelling to them “hey over here, it’s me, I’m having problems.”

Ultimately though I agree with the rationale of Susan Adams over at Forbes about Brian Moynihan, Bank of America’s chief executive.
She added him to a Forbes list — “The Biggest CEO Screw-Ups of 2011,” for the whole $5-monthly-fee-to-use-their-debit-cards fiasco in September.
The deluge of web traffic that idiocy created probably contributed mightily to the bank’s longest stretch of intermittent outages in October.
That said it would not be a stretch to extrapolate and add a ledger line blaming him for their IT woes too.
Worry not for Mr. Moynihan, doubtless he will come out of it all quite well off.
Surely he will make out even better than his unwanted former minion Sallie (see “Great pay for terrible performance”).