Disaster Recovery – A detailed look at Verizon Wireless’ disaster recovery plans

images
  • Click here to download the original properly formatted document.
  • This piece explains the necessity of having a disaster recovery plan in place.  This is a huge ordeal due to the fact that disasters do not only occur from intrusions, but also naturally via floods, tornado’s, and even earthquakes.  All businesses should have these plans in place in case of unexpected circumstances to keep their services and data operational.  One of the best ways to understand disaster recovery is to focus on a real company that has implemented a great recovery plan.  Verizon Wireless is one of the largest cellular companies in the United States.   Disasters could be devastating to their business, so it is an absolute necessity that they have a great plan in place.

Verizon Communications is one of the largest communication networks in the
U.S.  They reach out to consumers with home broadband in the forms of DSL and
FIOS, cellular communication, mobile broadband in the forms of 3G and 4G, and even
offering cable television.  While they offer many different services, one could safely say
their main product is communication networks through landlines and cellular networks.
It is estimated that there are over 80 million subscribers to the Verizon Wireless network
which offers a variety of mobile phones from the classic style cell phones to the newestforms of Smartphones.

Verizon was technically born in 1983 under the name Bell Atlantic.  This was due
in part to the fact that AT&T actually had a widespread hold on communications, but
entered a breakup due to legal conflicts which created these “Baby Bell” companies.
Bell Atlantic was one of these and was mostly based in Philadelphia and the areas
surrounding it.  It would then grow a bit and eventually be involved with one of the
biggest mergers in business history with GTE.  In 2000, with the GTE merger, the name
was changed to Verizon Communications Inc.  Eleven years later, Verizon has indeed
become one of the largest communication networks in the U.S.  Even more interesting
is the fact that since its inception in 1983 due to the AT&T breakup, the two, which were
once together, now heavily and at times viciously compete against each other for sales
and subscribers across the U.S.  Competition is always a healthy thing for us, the
consumers, as they always try to outperform each other.

Businesses will be businesses, and competition will be fierce, but within each
business is the will to survive.  Survival of the fittest is a phrase thrown around an awful
lot, but within businesses, survival comes in many different forms.  Who carries the best
products, plans, and services?  Who’s right for me?  Who is going to get my money?
Those are common, but when it comes to surviving the actual elements, those who plan
will indeed be the fittest.  Of course we’re talking about Disaster Recovery and Business
Continuity.

DR and BC planning for any business is a necessity.  It allows businesses the
opportunity to recover from these things that aren’t exactly preventable.  In terms of a
company like Verizon, with heavy competition from AT&T, and the type of business they
are – communications, having the best DR and BC plans in place is pretty much an
absolute need.  We rely on communications day to day, minute to minute.  And while
the important part is to keep communications going through a crisis or disaster so all of
us can communicate and get the help we need, within competing businesses, having
the wrong plan could literally destroy a the other company.  Who’s going to be there
when I need them?  Whose service will be available or available as quickly as possible
when I need it the most?  It’s not just a matter of customer satisfaction it literally is a
matter of outperforming the other.  I hate to say that, because I think consumers should
come first, but in that world it is all about survival.  In turn, that’s actually good for us
because they both will compete for the best of everything – including BC and DR.
What’s important with a business like Verizon is to understand its DR and BC
planning.   To figure this out, we must understand what is important to them, and what
type of disasters a company like this should expect.  Since they do offer a variety of
services, I would like to focus more on their cellular communications service.  What we
have to keep in mind is that this isn’t just a single building, in order to provide a strong
product, cellular towers are setup all throughout the country which means that generally
all disasters have the ability to disrupt service, whether it’s regional or nationwide.

Threats & Disasters:

  • Flooding
  • Tornados
  • Hurricanes
  • Earthquakes
  • Fire
  • Theft
  • Power outages
  • Hackers
  • Vandalism
  • Network Outages
  • Terrorism

Possible Process or Equipment affected:

  • Cellular Towers
  • Data/Voice interruptions
  • OTA Updates / Phones
  • Wireless Bandwidth
  • Storage Servers
  • Sales

Risk Assessment:

Threat Likelihood Impact Rating Overall

Flooding                    Medium                      Low                             Low

Tornados                   Medium                      High                           High

Hurricanes                Low                             High                           Medium

Earthquakes             Low                             Medium                      Low

Fire                             Low                             Low                             Low

Theft                           Medium                      Medium                      Medium

Power Outages        High                           Low                             Low

Hackers                     Medium                      High                           High

Vandalism                 Medium                      Low                             Low

Network Outages     Medium                      High                           High

Terrorism                   Low                             High                           Medium

Risk assessing is always a guessing game.  Since I am mostly basing this off of
the wireless products from Verizon, I figured I’d try to nail down what would be the most
likely scenarios instead of just throwing everything out there.  Mostly, there would be the
threats of natural disasters to the towers and since they are spread throughout the
country, they do have the unfortunate situation of being hit with various threats.  This is
quite different than a company who has a building in one location where it is much
easier to guess the threats with the history of that location and obviously weather
patterns over the years.  With these towers, they just happen to be in about every
possible area where different disasters can occur.  This has to cause high anxiety on
the company, but luckily there is DR and BC.

It is easy to think that hurricanes would have an overall impact of High, but we
also have to think how often hurricanes happen.  The thing is, they do occur a few times
a year, mostly in the southern states, but the real bad ones such as Hurricane Katrina
are usually rare occasions.  And I examine my previous sentence by saying that most
occur in the south of the United States.  While these would obviously cause a high
impact rating, we also have to take into effect that not all the towers are located there.
Again, wide spread locations means one area might be affected while others are
running just fine.  It usually wouldn’t cause a nationwide outage.  You really have to
consider all these options when assessing risks.

For the most part I find that flooding, earthquakes, fires, power outages and
vandalism contain an overall low risk.  Obviously this doesn’t mean that they can’t
happen or won’t cause issues with the company, but I feel that their impact would be
relatively low and quick solutions would occur.  Flooding was happening all around the
country earlier this year, but did it affect cellular service much?  I don’t believe so and
we also have to realize that most towers are located in areas where floods would most
likely not hit them as easily.  Earthquakes are a bit tougher, but the threats of high-level
earthquakes do not happen all the time.  While it is something to consider, it’s also
something mostly happens in specific areas.  Fires at the towers are mostly a non-issue
to me.  Sure, something could go wrong – perhaps an explosion, but it most likely would
only affect that one tower.  It isn’t going to be widespread.  Power outages happen a lot,
but this is mostly based on the area and usually returns rather quickly.  The good thing
about power outages is that these towers have battery backups in case it happens.  As
for vandalism, people will be people and always try to damage equipment or spray paint
them.  With fences and watchers in effect, this occurrence is low.

I do find tornados, theft, and hackers to be the more interesting threats.
Tornados have the ability to pop up all around the country and their destruction can be
insane.  They are so unpredictable and besides the fact that they could tear these
towers down with ease, they could also wipe out phone lines, underground lines that
communicate with these towers, and large infrastructures supporting them.  Theft is
actually a bit more common then what one may think.  These towers sit in these places
and while they are watched and have fences and all that, people still manage to try to
steal copper from them.  We know the value of copper is extremely high right now and
the towers use copper lines.  Theft can also be grouped with personal data and user
information which would involve hackers.  Hackers will always try to break through the
systems whether it be the servers or through the wireless data/voice that are being
transmitted all day long.  These are very real and constant threats in my opinion.

Business Impact Analysis:

Cellular Towers

Sensitivity Level – High Sensitivity

Associated Risks - Outages due to weather damage, natural disasters, network hiccups, and people

Current Controls – Besides fixing damaged towers, Verizon has COWs (Cells on Wheels) to use at a damaged location temporarily.

Effectiveness – Due to a recent outage and a quick bounce back, this assessment is effective

Data/Voice Interruptions

Sensitivity Level – High Sensitivity

Associated Risks – Besides outages, chance for hacking and eavesdropping

Current Controls – Verizon’s CDMA Technology utilizes various levels such as digitally encoding and 128-bit encryption.  4G offers enhanced security through UICC, SIM, and non-invasive key storage/symmetric key authentication

Effectiveness – Nothing is completely safe, but Verizon has kept a good reputation with secured data and voice.  Current and new security technologies prove effective.

OTA Updates / Phones

Sensitivity Level – Mid Sensitivity

Associated Risks – Pushed updates OTA could cause problems with phones.  Bugs and vulnerabilities that went unnoticed could leave phones open for attacks or not allow smooth operations.

Current Controls – When an OTA update bugs the phone, it usually causes a lengthy wait on a fixed update.  Sometimes they will allow a return on phone for a working version.  This is a weak area.

Effectiveness – Not effective – Many complaints

Wireless Bandwidth

Sensitivity Level – Mid Sensitivity

Associated Risks – Congested networks due to an abundance of users crowding cells could slow the network down in regional areas.  Interrupted phone calls, slow data speeds, text messages not being pushed (Quite a regular occurrence on New Year’s Eve)

Current Controls – With the rollout of 4G and its extremely larger bandwidth and controls, this should open up a lot more road space.  As users migrate to 4G, 3G will become a little less congested which may equal it out.  However, throttling abusive users is a control set in place to free up bandwidth to others.

Effectiveness – This is somewhat effective as there aren’t a lot of complaints about bandwidth.  However, typically high events such as New Year’s can still cause congestion.  This isn’t bandwidth abuse, just an overabundance of information at once.  Perhaps the divide in 3G and 4G will solve any issues.

Storage Servers
Sensitivity Level – High Sensitivity

Associated Risks – Servers store private data of not only the Verizon Network, but of users personal data such as names, addresses, payment methods.  This information could be stolen internally or via hackers.

Current Controls – Servers are not stored in one single place with every single user.  Dependent on region which makes it difficult to access everything.  Proper security measures are in place such as passwords, encrypted data, security certificates.

Effectiveness – Again, nothing is safe and hackers have managed to break through the systems at times.  Consistently monitoring the servers and services and keeping up to date with current security models has kept most information safe.

Sales

Sensitivity Level – Mid Sensitivity

Associated Risks – Hackers, bad reputation, and failure to respond efficiently to disasters could cause sales to go down.

Current Controls – Verizon rarely has hiccups with their website for sales and account information.  They also offer physical stores around the country for consumers to test and purchase new phones and plans.  Website contains encryption, SSl, and is guaranteed by Verisign.

Effectiveness – Verizon continues to gain momentum as time progresses.  Quick responses to issues, and a general comfort level of the community ensure that sales continue.

Verizon proves to be mostly effective with their risk assessment on these specific
processes.  We have seen how they respond more recently to any cellular tower
damage by the tornados and storms that have passed through this Spring.  Their Cells
on Wheels, while not exclusive to them, is quite an interesting technology which allows
them to deploy cellular towers quickly and efficiently to keep wireless operations running
during disasters, or even to deploy them in an area where a big event is happening to
allow users the ability to use their phones where they otherwise would not be able to.
Besides the COWs, Verizon also offers COLTs (Cell on Light Trucks) which are similar
but are stationed to trucks. These are mostly used in case equipment has failed or been
damaged.  These COLTs have retractable masts, a microwave antenna which links it to
networks, power generators, and also a small office for teams to work in.  Even more
good news is the fact that they contain fuel, food, and water.  These are definitely
emergency oriented towers, and it’s a great plan when things go wrong.  Both COWs
and COLTs allow voice and data to pass through via their CDMA band.

Any communications worth a dime would include some sort of method to protect
all the data that is passing through these systems.  Having a network that would be
easily eavesdropped on during phone calls, or data retrieved from cell phones would
obviously cause a bad reputation.  No one would want to use them.  Verizon sets this
up quite well with their CDMA-based infrastructure which is no slouch.  When 3G
became widely available and commonly used, the risks became higher due to how
much more a user could do with their phones, and of course Smartphones.  We use
them at times like we use our computers – sending search results, typing in passwords
to access emails and Facebook, perhaps even buying through websites or the Android
marketplace. CDMA 3G was encrypted quite well with 128-bit and digital encoding.  4G
promises even higher levels of security with non-invasive key storage/symmetric key
authentication.  Security in these cases should be of high concerns and it seems that
Verizon understands that.  We can’t deny the fact that there are hackers out there doing
their best to get into these airwaves to eavesdrop and intercept vital information.

Verizon has had some issues with hackers getting into their storage servers.
The problem with this is that hackers are relentless and will do whatever they can.
There are no rules for them which make it quite difficult for security systems and
personnel.  Constant monitoring and latest security measures can only hope to keep
most out and contain most intruders.  Verizon definitely does not put security as a side
note like many companies do.  In fact, due to the fact that they offer security, DR, and
BC plans makes me believe that they truly understand and believe in network security
measures.  With this type of company, whether it is Verizon or AT&T, you better
understand that security is most important.  It can’t always be stopped, but if you can
keep a great track record and prevent over 90% of attempts, I’d say you’re doing a good
job.

OTA firmware updates and Wireless Bandwidth are another issue.  From
personal experience, it isn’t always the fault of Verizon as it is the company who created
the phone.  We have to remember that more than ever there are many processes with
these updates.  It has to not only work with the phone hardware, but the version of
Android it is running, and the networks that Verizon uses.  In actuality, it usually takes
quite awhile for these updates to hit due to the constant testing before Verizon will let
them loose.  Bugs happen, not everything is always seen, but they could be a bit
quicker at fixing the said issues after they happen.

Wireless bandwidth on the other hand is very difficult to manage.  It does
happen, but they have definitely been known to throttle users who they felt were
completely abuse data usages.  It is unlimited, but certain people will go over the
boundaries of what they should which causes the network to crawl when many others
are doing their daily things.  Besides investing in more towers to allow more freedom
possibly, it is for the most part an unnecessary expense.  With the rise of 4G and their
plans to have near-nationwide 4G coverage in 2013, this could definitely create that
balance as there will still be 3G users while 4G still takes off.  Eventually, most people
will probably move to 4G which could then congest it.  Perhaps this is why Verizon is
planning on moving to limited tier plans?

The next objective is to determine the teams that are or should be used in these
cases.  Any BC/DR plan will not work at all without teams put together in different
categories for maximum results.  Throwing together a bunch of people and telling them
to do something is only going to cause confusion and a waste of money put into the
plans.

Verizon has many teams, but the information about these teams are rather
difficult to find.  But it goes without saying that they do have the following teams ready to
go in case of emergencies:

Dick Price – Director of business continuance and emergency management

  • MERIT (Major Emergency Response Incident Team)
    • -Dennis DeRolf (MERIT Coordinator)
  • Crisis Management Team
  • Senior Management Team
  • Operations Assessment Team
  • IT Team
  • Administrative Support Team
  • Media Relations Team
  • Human Resources Team
  • Legal Affairs Team
  • Communications Team
  • Procurement Team

It is very difficult to find all of the team members for Verizon.  However, I did manage to pull together a small list of Team Leaders:

  • John Diercksen – Strategy, Development and Planning
  • Roger Gurnani – Chief Information Officer
  • Lowell McAdam – Chief Operating Officer
  • Anthony J. Melone – Chief Technology Officer
  • Marc C. Reed – Human Resources
  • Francis J. Shammo – Chief Financial Officer
  • Peter W. Thonis – Chief Communications Officer
  • John N. Doherty – Investor Relations
  • Daniel S. Mead – Chief Executive Officer
  • Thomas J. Tauke – Public Affairs, Policy and Communications

Teams play an important role in every DR/BC planning.  When a disaster occurs, the
teams are to respond in a quick manner in order to quickly handle the events that take
place.  All teams should consist of knowledgeable personnel that are able to perform
the tasks at the highest levels.  While it is difficult to find every team member here, it is
pretty safe to say with the way disasters were handled in the past, Verizon has the right
personnel in place.

As perfect as teams may seem the amount of disasters and the different ways
they affect the company, communities, and structures can’t be foreseen.  We have to
remember that while the business is doing their best to mitigate the problems to ensure
business continuity, they also have to deal with the aftermath of not only their own
business, but of the public.  Verizon is dedicated to helping the communities not only
keep important communication services through and after a disaster, but ensuring they
receive help through aid, food, water, and other common needs.  It’s a very large task,
but one that Verizon seems to excel at as they have dealt quite promptly with disasters
such as the September 11 attacks and Hurricane Katrina.  But even they need the
proper training to do this.

Training and testing are of vital importance to any good teams created.  While,
again, information is scarce of how Verizon operates, they clearly do have very well
trained members and do testing phases.  I understand the difficulty in doing as much
training and testing as should be.  However, if I were to create a schedule, it would look
something like this:

Testing

Every six months there will be testing to ensure that the current plans are maxed to their highest efficiency.

April – A week long field test training for the teams to see how they perform not only on their own merit, but with each other against many high-risk disaster scenarios.

October – A paper Walk-Through of our DR and BC plans along with how all should react in case of a disaster.  The walk-through will guide you all through realistic scenarios.

Repeat process

I know it is asking a lot for a company to do just one set of tests, but I think this is
where a lot of companies get it wrong.  Be thorough and get it right.  If you’re going to
put the money into DR/BC, and you should, then make sure that it’s going to work.  My
timeline does not seem farfetched.  This way the company is having each testing
method done once a year, but it is split up six months apart which I think gives them
enough time to plan for testing while allowing them to check that the plans are still
useful, current, and at highest efficiency levels.

In closing, Verizon does a lot of things outside of this paper to ensure continuous
business and to recover from disasters.  I’ve chosen what I felt are the more common
risks, and the higher impact levels.  I started this off by discussing competition a good
bit and I’m glad I did.  Going through this class, and this final project process has shown
me that having the best business continuity and disaster recovery plans is actually a
huge part of competition.  One would hope that all companies care about their
customers during disaster tragedies.  After all, we are the ones that choose to buy their
products and most likely we are going to go with the ones that have the best service,
and take care of us during those disasters.  Of course, their doing it to make sure they
survive as well, but this also has taught me that it isn’t only about taking care of their
own assets it’s also about keeping people happy.  And when people aren’t happy, let’s
just consider that another risk.  Verizon and AT&T will probably be fighting each other
for communication supremacy for a long time to come.  It’s nice to know, in a
roundabout way, that having the best BC/DR is actually a part of that competition.
We’re going to receive the best treatments we can get during the most horrible of times.

Verizon Wireless (2011) Year-Round Network Preparation.   Retrieved on June 6, 2011 from http://news.vzw.com/Emergency/Preparation.html

Verizon Wireless (2011) Strength.  Security.  Safety.   Retrieved on June 5, 2011 from http://b2b.vzw.com/govt/security.html

Price, C. (October 1, 2007) Verizon Launches Firmware Over-The-Air, Updates Many Phones.  Retrieved on June 6, 2011 from http://www.phonenews.com/verizon-launches-firmware-over-the-air-updates-many-phones-2324/

Spike, J. (May 11, 2011) Verizon uses cows to get areas in Alabama that were devastated by tornados back on line.  Retrieved on June 7, 2011 from http://www.broadbandexpert.com/blog/service-outages/verizon-uses-cows-to-get-areas-in-alabama-that-were-devastated-by-tornados-back-on-line/

Department of Management Services (NA) CDMA Security.  Retrieved on June 7, 2011 from http://www.dms.myflorida.com/suncom/suncom_products_and_pricing/wireless_services/wireless_data_services_aircard/features_options/verizon_wireless/cdma_security

Verizon Business (2011) Do you have a plan for the unplanned?  Retrieved on June 6, 2011 from http://www.verizonbusiness.com/resources/factsheets/fs_do-you-have-a-plan-for-the-unplanned_en_xg.pdf

Comments are closed.