Technical analysis of the bank america downtime.

Bank of Americas most recent downtime has stirred up the internet with all kinds of questions about what is really going on.  Everything from speculated ddos(distributed denial of service) attacks to some stating that Bank of America claims to be upgrading RAM.

Lets look at the issue from a technical standpoint and see what we can come up with.

This blog post is being updated in real time as I gather more information.  If this was a ram upgrade Bank of America would have had zero to minimal downtime(short spurts) as thier web servers operate in a load balanced/clustered setup.  This means that the website is hosted on many servers spread out so if one goes down, other servers take on the load of the down server.  So if they were increasing ram to prevent downtime it would be best to do this server by server,  letting the other servers pickup the load while each server gets upgraded.  So is this downtime caused by a ram upgrade?  I think not and if this is the case Bank of America should not be allowed to be a bank until they get a more competent I.T team.

So if that is not the case….is it in fact an attack?  Let’s look at the trace route.

root@web01 [~]# tracert -T www.bankofamerica.com
traceroute to www.bankofamerica.com (171.161.161.173), 30 hops max, 40 byte packets
 1  xxxxx
 2  xxxxx
 3  xxxxx
 4  xxxxx
 5  xxxxx
 6  ae-3.ebr3.Dallas1.Level3.net (4.69.132.78)  37.095 ms  35.937 ms  35.905 ms
 7  ae-4-90.edge5.Dallas1.Level3.net (4.69.145.202)  33.926 ms ae-2-70.edge5.Dallas1.Level3.net (4.69.145.74)  34.175 ms ae-3-80.edge5.Dallas1.Level3.net (4.69.145.138)  34.002 ms
 8  BANK-OF-AME.edge5.Dallas1.Level3.net (4.78.230.2)  35.708 ms  35.525 ms  35.518 ms
 9  171.161.191.248 (171.161.191.248)  35.961 ms  35.776 ms  35.467 ms
10  www.bankofamerica.com (171.161.161.173)  35.971 ms  36.069 ms  35.871 ms
11  www.bankofamerica.com (171.161.161.173)  36.157 ms  35.881 ms  35.964 ms
12  www.bankofamerica.com (171.161.161.173)  35.643 ms  35.382 ms  35.510 ms
13  www.bankofamerica.com (171.161.161.173)  35.869 ms  36.331 ms  38.235 ms
14  www.bankofamerica.com (171.161.161.173)  38.507 ms  35.909 ms  36.351 ms

Judging by this trace route I would have to say no this is not a DDOS attack.  How DDOS attacks work is they take the resources of many many computers and flood all those resources at a single target if this was the case we would not be getting a response back from Bank of Americas website (we could have also checked this with ping but if its a ddos its best to get a ping response from the upstream router to determine the attack size vrs pipe into the router)  or if we did get a response it would be very lagged,  these times are normal.  However one thing I did notice between this trace route and one I did earlier in the day is that I got more responses from Bank of America so it looks like they added more computers into the load balanced environment to combat whatever is going on.  It could mean they had a major surge of traffic today which is bad because its common knowledge to have resources to serve 75% more traffic than your average traffic and be able to serve 50% more than your average peak traffic.  And it’s unlikely a large organization like Bank of America would not have any idea of the amount of traffic it gets or to expect.

Could it be a targeted DOS attack? With the response times from the webserver being over 8-15 seconds(20-50 milliseconds being normal) It is very possible that some kind of resource starvation attack could be used to spike the cpu usage up to 100% which would produce very similar results to what is being seen.

Many customers have found ways to get around the issues by accessing the website via the nodes directly.  Connecting to the mobile site etc etc it is a mix and match as to what will work as users flood to newly discovered entrances into the online banking portion of the site and in turn only increase the load on the banking servers making it harder on the I.T staff to isolate problems.

Another speculation is a cyber attack/hack.  While this is very possible,  it is scary and unlikely that a bank would leave any part of its contaminated infrastructure on both the internet or on their own private network which makes me think that this is not a hack attempt/cyber terrorism attack on Bank of America.  However as you have seen in the post right under this one,  Cyber terrorism attacks are being brought to a whole new level of complexity and rather those attacks can succeed or not depends on the intelligence and training level of everybody who is a part of the company being attacked,  so I am not quite ready to write off that this is not a cyber terrorism attack off quite yet..

So what is the cause of the website being down?  It could be many different things and I would have to be onsite to figure it out.





More to come later as I do more tests.




Ill be fighting to legalize freedom tell the day I die.