Incident Reponse – Sony’s Meltdown

  • Clear here to download the original properly formatted document.
  • Incident response is vital to any network success.  We deal with many things on a daily basis whether it be attacks, intrusions, or disasters, but it is crucial that we respond to these incidents properly.  But what happens when we do not respond correctly or that there is a lot of confusion in how to respond?  In this piece I go in depth into the problems that occurred with Sony and their PlayStation Network due to very poor incident response plan.  I also challenge myself and conclude with methods that I would have done to improve the situation.

Attacks are something that we must learn to accept in today’s society.  We are so invested into our technological advancements and make it part of our lives that we tend to forget the dangers that come along with it.  It is very easy for a person to steal vital data, and destroy an entire system, which is what we have seen prominently a year ago.  Security is important, however, there is still a large number of people and companies that fail to acknowledge this importance.  Ignorance is a key word in all of this because whether you’re in charge of a large business, or just a home user, we are all susceptible to these attacks, but we do not want to believe that.  It’s easy to say you’re safe by putting up an improper firewall or install a virus scanner, but ignorance is bliss.  In today’s technological society, we must always be prepared.  Many people find this out the hard way, as do many large companies.

It was almost a year ago to this day (April 20, 2011) that Sony found themselves in hot water due to attacks on their system.  Sony is known for their business practices in various forms such as audio, video, and gaming.  This is a huge company with a rather widespread fan base all over the world.  Sony invests a lot into their gaming department where they currently have products such as the PlayStation 3, the PSP, and the new PS Vita.  All of these hardware devices play video games and other services, but most importantly is their online structure that allows users to play video games online, purchase digital downloads, stream movies, and chat with friends.  As is the case with most services, their online service requires the use of a username, password, and typically credit card information to purchase digital goods or an upgraded service.  In many cases, this information is stored on their servers for instant access.  The online portal is referred to as PSN (PlayStation Network), but due to Sony’s ignorance with security, it was hacked a year ago and the fallout from that was incredible.  PSN was down for nearly a month, users personal information were stolen, a lot of money was lost, and people started to distrust Sony as a brand.

So what happened?  At first Sony called it a network outage, which is something that happens from time to time.  The problem is that Sony knew they were hacked, but led the public to believe that it was just a normal outage for days.  Sony is very poor with their public relations, as something like this should have been reported much quicker.  An official quote from Sony at the time states, “An external intrusion on our system has affected our PlayStation Network and Qriocity services. In order to conduct a thorough investigation and to verify the smooth and secure operation of our network services going forward, we turned off PlayStation Network & Qriocity services on the evening of Wednesday, April 20th. Providing quality entertainment services to our customers and partners is our utmost priority. We are doing all we can to resolve this situation quickly, and we once again thank you for your patience. We will continue to update you promptly as we have additional information to share.” Seems a bit dry, does it not?  This started a lot of speculation as to who attacked their services, and the group “Anonymous” was ultimately blamed at first.  However, they claimed to have no part of it.

It took about seven days to get far more clear information on the situation.  Sony finally admitted that it was a massive hack with emails, passwords, and credit card information more than likely being stolen from the large 70 million user base.  A lot of the talk at first was that Sony was dealing with Denial of Service attacks due to outrage over Sony’s attempt to take down Geohot (a known hacker that exposed methods to override the PlayStation 3’s security offline).  However, it became clear that Sony was keeping secrets.  Eventually it was admitted that their security system was dismantled and important databases were breached.  These were accessed between April 17th to April 18th.  This timeline is important as it’s a good example of what not to do.  It’s hard to say for sure exactly what happened with Sony, but we do know that they had a very small security staff, and possible speculations were that due to Sony’s lazy efforts with security, a SQL injection attack may have been used.  The other guess was that since the PlayStation 3 was recently modified at that time with custom firmware, people were using it in different ways.  It could turn the device into a developer unit (to an extent), which would allow users to have access to trusted features such as access to their internal developer network which in turn could have allowed for a wide variety of opened gates to attack with.

And so began the near month long outage of the PlayStation Network.  If that were just the problem, then I’m sure Sony could have saved more face, however it became known that our personal data was not even encrypted.  This sensitive data that included names, emails, and credit cards were stored in plain files that anyone could read.  Encryption is a fundamental part of security, so why would Sony not have this?  It’s really just ignorance to be honest.  Sony isn’t known to be the most fan-friendly, but to not even protect the data with simple encryptions is baffling.  Of course we’re also talking about a company that used rootkits on their audio discs years ago.

In Sony’s defense, protecting any large network is an extremely daunting task, even with the best security.  But when you’re a very large company that depends on networks that your consumers use, you have to be much smarter and less ignorant to the fact.  This not only hurt the consumers, but it hit Sony in a major way.  Because of these attacks, and Sony’s foolishness, they were predicted to lose $3.2 billion, and had to spend $170 million to basically setup and properly execute a security system.  Do we have your attention now, Sony?  Luckily for Sony, there had not been many cases of personal information being used out in the wild.  They also set up a “Welcome Back” program for users who could use their Plus service free for a month, and gave consumers a few free games.  It was a nice gesture, but nothing at this point could fully regain the trust they once had.  Even a year later, many people now are timid to use their information on Sony’s network, most by disposable cards (a smart consumer move) to purchase digital goods from the service.  In my opinion, this was one of the bigger attacks I have seen in years that not only affected the PlayStation name, but also affected the growth of digital distribution.

As we can see, Sony’s response to this matter was very, very poor.  In my opinion, a good incident response would be to understand the network you are running in the first place… of course it would also be good to understand that your security is poor in the first place as well.  This is a company that should know better.  In this case, if the attacks were occurring between April 17th to April 19th then it would seem logical to immediately take notice and shut down servers.  There were nearly two full days of this activity occurring and no one noticed?  The services were not shut down until the 20th, how can this happen?  Proper incident response in my opinion would be to have noticed the suspicious activity on the first day, and perform security responses that should be in place.  Attempt to close the areas that the intrusions are occurring, if this means the network, then do it.  Consumers will always be upset, but if they understand that you reacted to save their information, and to prevent future issues with the service, we would understand.  But here’s the problem: two full days of activity occurring before anything was shut down.  And even after that, the public had no idea what was happening for another three days.  I believe April 23rd was when Sony made public that the services were shut down due to some intrusion attempts.  Yet, it was another five days after that when Sony admitted that they had been massively attacked, and that our information could have been stolen.  Seven days to notify their user base of this is unacceptable.  You’re always going to take flack, but a proper incident response would have been to understand the activity early, shut everything down quickly if it were the last resort, and notify the public within a day that this has happened, their information may have been stolen, and that we were doing everything we can to find all the details.

There were a lot of attacks last year, and even Valve, a very respected company in the gaming world, had their online digital service hacked.  The service called Steam, which allows PC users to purchase digital games and content had attacks occur on it.  Valve reacted very quickly to the situation, the same day, and let users know that there was an attack, and they were looking into it quickly.  They identified the problems (which turned out that the forums were hacked, and not the actual service), and let the public know exactly what was happening.  They even gave advice on what to do in this situation, and that their passwords must be changed (also to change their passwords if they are similar to other services).  People praised Valve for their dedication to their fans in quickly resolving an issue.  Sony could learn something here.

If I were in charge, this is how I would have handled this scenario with Sony.  The intrusions begun on April 17th, then we would know on April 17th that something needed to be done.  First off, I would have security setup that would allow me to see this quickly.  How about a nice IPS?  Granted, it’s harder to get specifics when Sony never let out how the attack came about, but we can take our guesses.  So, I’m having preventions already in place and to have staff notify me instantly of any unusual activity following the system of Possible, Probable, and Definite.  In that same day, I’m having my staff quickly learn all that they can and reporting to each other, and to me.  We then figure out the best methods to proceed with.  Are these attacks coming from an easy to cover exploit?  Let’s hurry up and close it, and then check the damages.  Is it much more that we have to shut down bigger parts of the network?  In the possibility that it was custom firmware for users with a PS3 accessing secret services, how do we not know where it’s coming from?  I understand that this task can be very, very difficult, but if this looks like the case then we must shut down this part of the service, even if it delays developers from getting content out there for days to weeks.  If all else fails, we shut down the entire system and carefully recount everything we know until we can identify manors to fix it.  Also, my incident response management would let the public know within a day that there was an attack on our systems, and that services would be down while we examine the situation.  If the situation shows that information was stolen, the public must know so they can possibly change passwords, check their billing statements, and even call their banks or credit card departments about possible charges.  I would have this done all within two days.  And one last statement on that, I would definitely have had all important information Encrypted!

Gene Spafford, a security professor at Purdue University, stated that Sony could have done far more to prevent this.  Spafford said “Sony was using outdated software on its servers — and knew about it months in advance of the recent security breaches that allowed hackers to get private information” and he also went on to mention that experts in the field were monitoring forums learned that outdated versions of the famous Apache Web Server was outdated and “was unpatched and had no firewall installed.” And that it was also “reported in an open forum monitored by Sony employees.”  So, basically, Sony knew that they were in trouble months before this attack, but did nothing about it.  Ego, Ignorance, or all of the above?  Mike Meikle, a CEO of the Hawkthorne Group said, “They really didn’t have a defined process to address data breaches.”  He also mentioned something that we are all too familiar with, that companies generally don’t have a process because security is an extra expense.

Why would a company like Sony not have updated and patched services?  Even crazier is the fact that no firewall was present at the time.  Why would they not have a protocol to follow?  If this were me, I would like to believe that all of this would have been included.  You always update these services to the latest versions due to the fact that outdated services usually have the disadvantage of having well-known vulnerabilities, and you always patch these services because many times this will cover up the holes in the system the best that it can be. Firewalls are very common tools to have now, I can’t imagine many companies not using one (even if it isn’t properly configured).

Security is there for a reason: to stop huge incidents like this from occurring.  It doesn’t mean you will be 100% protected all the time because that is false hope and only gets you in deeper water, but understanding that proper security measurements, while still problematic, can indeed prevent major outbreaks such as this.  What I see here is that even the simplest of security knowledge was not used.  Outdated and unpatched services, no firewalls, horrible reaction time, and bad public relations.  We have also learned that Sony clearly had no protocol in place for any incident like this.  This tells us that Sony just didn’t care.  Of course they didn’t want this to happen, and I’m sure they were sorry when they learned how much money they were going to lose, but how can you feel completely sorry for a company that had this type of attitude toward security?  It’s not good, it never is, but these kind of things are a wakeup call for many large businesses.  The incident response was about as bad as it could get it because when I think about what happened, I just think of a group of personnel sitting there unorganized and unsure what to do, or who to even report to.

The question begs though as to how Sony is doing a year later?  I’ve recently heard people talking about the one-year anniversary of the Sony attacks.  So, yes, this was a big deal for all types of people.  Right now Sony is doing much better.  They have gained some trust back from their users, but as I’ve mentioned, most are still afraid to just put their credit cards into their system.  Somehow, Sony has recuperated, but not to the fullest extent.  Yes, people are back playing their games through PSN, and purchasing digital downloads.  Sony practically rebuilt their security system from the ground up as to hope that this incident never occurs again.  It’s actually amazing as to how well a company can get back on their feet when they really need to, but we must also forget that people have short term memories.

Incident response and management is as important as any other factor of the security world.  We must have protocol, and we must follow order.  We need teams that understand what to do, and how to react.  Sony proved to the world that security is a necessity now more than ever as become more and more engrossed with an all-digital world.  This was a devastating blow to not only Sony, but to other companies who depend on digital distribution and online services.   It may have set us back a few steps, but perhaps that is necessary for security to take a few steps forward?  We can always hope.

Empson, R.(April 23, 2011)  Hack Attack: Sony Confirms PlayStation Network Outage Caused By ‘External Intrusion’. Retrieved from

Anthony, S.(April 27, 2011)  How the PlayStation Network was Hacked.  Retrieved from

Olivetti, J.(May 23, 2011)  Sony loses $3.2B, spends $170M in response to hacker attacks.  Retrieved from

Newman, J.(May 12, 2011)  Experts on PSN Hack: Sony Could Have Done More.  Retrieved from

Comments are closed.