Trial by fire
operations
BCP in action
With a critical systems failure putting its mainframe out of action
for a week, Monday 13 November, 2000 could have been one of the
blackest days ever experienced by Rl London. Instead the incipient
crisis was managed with only a few hours down time and with no
late settlement penalties whatsoever. Just two months after its
implementation, London's disaster recovery planning had already
begun to prove its worth. But as we know, Murphy doesn't do
things by halves. On the Thursday of the same week, in an entirely
unrelated incident, Rl Utrecht was also hit by a systems fail
ure. Again the rewards of good Business Continuity Planning
were plain to see. We find out what happened and note some
of the lessons to be learned.
Critical situation?
Paying its way
Early warning
IO What sNewS Issue 4-July/August 2001
I TYeople need to live and breathe this
X plan so that it becomes a natural re
sponse,' GFM head of operations Denise
Shiner told What's NewS when we re-
ported on the implementation of Rl Lon
don's contprehensive contingency response
package, the Business Continuity Plan
(BCP). She was describ-
ing the need for a series
of simulated disaster
scenarios to make sure
that the BCP made the
transition from plan to
practice as quickly as
possible. Before the first
exercise could be
arranged, however,
London's disaster re
covery provision was
being tested for real by
a full-scale crisis. London's Denise Shiner
London's John Spiteri
and technicians instructed to
begin migrating the systems
to the disaster recovery site at
St Matthews House in Old
Street, London. At the same
time the 1T staff tried to restore the envi
ronment as of close of business Friday 10
November. This effort failed due
to CPU failure. At 15:45 the de-
cision was taken to shift pro
duction to the Dealing Room
mainframe which had been suc-
cessfully booted up - and the
backlog was cleared with no set-
tlements missed and no penal
ties incurred. The DR main
frame then continued in
operation for the rest of the
week while technicians repaired
the CPUs of the crippled Tan
dem, ready for a return to ac
tion the following weekend.
At 09:08 on Monday 13 November the
Tandem mainframe in the Rl London of
fice suffered a critical CPU failure making
it impossible to process trading. At 12:30
the crisis management team was convened
'It would be difficult to ask for a more
conclusive demonstration of the value of
London's BCP and disaster recovery plan
ning,' says Johan van Waveren, global
coördinator BCP RI. 'Naturally we've cal-
culated what a week's lost business would
have cost us - not forgetting the threefold
multiplier effect of missed settlements,
penalties and reputational risk - and, nat
urally, the actual figure is confidential, but
let's just say we're happy to have lived-up
to our Triple A rating. Without minimiz-
ing the amount of hard work involved in
getting the systems migrated and the re
serve mainframe operational in that short
time, we now know that the system
works, and in fact, that it works relatively
smoothly.' John Spiteri, disaster recovery
manager at RI London, who was
responsible for the actual migration of the
systems even downplays the difficulty of
that operation. 'Really it was just a hic-
cough. Of course it would have been dif
ferent if there had been
physical damage, but as it
was, it was simply a ques-
tion of reconnecting sys
tems - all in a day's
work.'
But of course that doesn't
mean things couldn't be
improved. 'Looking back
on the London incident,
there was a clear problem
of escalation - of people
trying to solve problems thentselves rather
than notifying the BCP crisis management
team. The BCP should have come into op
eration at a much earlier stage,' comments
Van Waveren. The first CPU failures in
fact occurred on Sunday, but because the
Tandem is a fail-safe system and other
CPUs kicked-in to cover the load, the
gravity of the situation wasn't immedi-
ately appreciated. As a result of that
In early sumnier New York's dealing
room lost electricity for threc days
due to a malfunctioning electricity
switch. They invoked their BCP and
moved the dealing operations to the
New Jersey site. Et voila - business
continued. Hats off to BCP.
