10. The invisible bank

When customers scribble checks in the supermarket or slip cards into an ATM, they probably have no idea how these financial transactions are processed each day.  Nor do borrowers think about what happens to monthly loan payments.  When customers initiate these transactions, they get a receipt and probably think it is complete.

RON SAMFORD: 'He had the hardest job' / Photo: Russ Hoadley

RON SAMFORD: ‘He had the hardest job’ / Photo: Russ Hoadley

Unseen to them are the “back office” operations inside a bank, where the real transactions take place.  The  activities of this “invisible bank” are extensive, employing thousands of people who labor in a labyrinth of departments to make it all appear seamless to the public.  Some of these areas function around the clock in multiple shifts to keep things running smoothly.

All of the transactions they handle — including some via internet, telephone and mail — must be physically entered as debits and credits on the books of the company, then sometimes routed to other institutions for more processing to complete the cycle.

Systems and people

Part of this processing capability is lodged in computer systems, telephone and data networks, electronic linkages to service vendors — all relying on many, many layers of software applications.  Just as important, however, are the people – a small army – who expend enormous effort operating specialized high-speed machines – from “key-punch” and “proof” to mail- and check-handling and sorting.  Although at its core today banking is largely electronic, much of it remains rooted in the ancient activity of passing and processing information on paper.

Many of these operations at Hibernia – the applications and the people – were moved in haste from New Orleans to Shreveport, Houston and Dallas before, during and after Katrina and Rita.

As chief administrative officer, Ron Samford was responsible for all of these operational areas – the mainframe computer and the scores of “legacy” applications related to basic bank transactions.  Also, the telephone systems, intranet and internet connectivity, companywide email, deposit operations, the mailroom, wire transfer, security — the operational infrastructure for 300-plus branches.  At 53, Samford was the longest-tenured member of the company’s executive management committee, dating to 1992.

Top Technologist

Walter Walker reported to Samford and was the “chief technologist” for Hibernia, heading up all those layers of information technology that drove the invisible bank.  In this role, the 57-year-old executive was as close to a legend as the company had.  A longtime employee, Walker was a disciplined, resourceful manager.  He and his wife Colleen had two sons, 17 and 15. Before the storm, they evacuated to Baton Rouge.

Walker had extensive experience — in credit-card operations, bank mergers and technology platforms of all kinds.  He had created dependable and effective systems with support of Kirk Domingos, a shrewd old-guard executive team member who had retired a few years earlier.  He designed Hibernia’s Incident Management Team and its recovery guiding principles, and insisted on robust plans and mock-disaster drills.

Hibernia’s fast and flexible recovery — despite the scope of the calamity and the sacrifices of employees — was in no small measure due to the resilience Walker “baked into” the bank’s operational side.  Later, assessing the recovery, he said simply, “We had a plan for disaster, and the deviations we took were good ones.”  In the future the company needed to build more “redundancy … not just back-ups,” he concluded.  His only regret?  “That I didn’t get on those choppers myself and tell them to get my people.”

Scope of operations

There were very few things Samford and Walker’s people did  not operate directly or support indirectly.  Other areas of the bank depended on their hundreds of technology experts to keep computer systems and communication linkages running.

Samford came to Hibernia in 1992, as the company struggled out of a financial ditch after a broad collapse of commercial lending that damaged many institutions.  The board of directors then was under orders from its chief regulator (the Office of the Comptroller of the Currency) to repair the company’s balance sheet.  They hired a Florida banker, Steve Hansel, from Barnett Banks, Inc., to turn things around.  Hansel hired Samford and others to build a new executive team.

Beginning as controller, over the years Samford picked up increasing responsibility.  A native of Dallas and a die-hard Texas Tech fan, he graduated with a bachelor’s degree and MBA in public accounting and had a distinguished college football career.  As a CPA, he’d already had 11 years of experience with Price Waterhouse, including an assignment in London.  He came to Hibernia from Team Bank in Fort Worth.  Samford and his wife, Carol, married 28 years, had two daughters, 21 and 16.  They lived in New Orleans’ Garden District and evacuated first to Tyler, TX, to be with family, then to Dallas and eventually to Baton Rouge before returning home.

As they prepared for the storm, his family was influenced by a grim memory from September 2004, when Samford stayed at the Hibernia Center with his disaster recovery team during hurricane Ivan, but his family was caught for 24 hours in a massive traffic jam, trying to reach Tyler.  They did not want to repeat that.

WALTER WALKER: Chief technologist / Photo: Donn Young

WALTER WALKER: Chief technologist / Photo: Donn Young

Friday before Katrina, Samford was engaged in an all-day “deep dive” meeting with his staff and Capital One people, discussing technology issues related to the merger that was supposed to close in six days.

Walker who was to retire with the merger) also was present as was Bart Bragg, head of distributed services.  They were meeting with Gregor Bailar, Capital One’s chief information officer, and others from the Virginia credit card company.  As the day wore on, Bragg saw emails piling up in his BlackBerry from Ben Gautreaux and Greg Stelly, his two disaster recovery officers.  The messages got progressively more alarming as he thumbed through them.  “Urgently need to talk,” said one.

Friday shift

Friday did not start badly for Stelly.  It was his job to watch the storm and sound the alert if a dangerous turn should occur.  He grew more concerned when a 9 a.m. advisory indicated for the first time that the storm might track as far west as the Mississippi-Louisiana border, but he still felt “pretty comfortable” about the 75% probability that it would take a path up through the Florida panhandle.  Part of his job was to study data from a private forecasting firm, ImpactWeather, experienced in advising the oil industry in the Gulf of Mexico.  Its bulletins consistently placed the western edge of the “cone of probability” no closer than Gulfport, MS, about 70 miles east of New Orleans.  Their maps depicted this “cone” as a yellow swath representing pathways the storm might take, created by bringing together different forecasting models from government, commercial, media and academic sources.

FLORIDA LANDFALL: Friday map eases worry / Graphic: ImpactWeather

FLORIDA LANDFALL: Friday map eases worry / Graphic: ImpactWeather

Friday afternoon

As the day progressed, Stelly watched with increasing discomfort as the forecasters adjusted the track further west.  A dangerous shift was occurring.

In its 3 p.m. update, ImpactWeather – for the first time – showed New Orleans inside the potential strike zone.  Stelly’s stomach tightened.  “Alarms started to go off,” he remembered.

When he and Gautreaux connected with Bragg, Stelly sketched the deteriorating situation.  Bragg signaled to the others in the merger briefing that they needed to hear this.  He switched the call to speakerphone.  It did not take the group long to “declare a disaster,” as Walker recalled.  This official step was required to set in motion the company’s recovery plan.  If, as it now appeared, the storm might strike somewhere late Sunday or early Monday, they had little more than 48 hours to shift mission-critical operations into disaster recovery mode.

TARGETS NEW ORLEANS: Saturday map shows ominous shift / Graphic: ImpactWeather

TARGETS NEW ORLEANS: Saturday map shows ominous shift / Graphic: ImpactWeather

Disaster recovery mode

The Incident Management Team had a well-rehearsed plan to move important functions to safe areas.  In Shreveport, at the main banking office on Travis Street, a conference room adjacent to chairman Bo Campbell’s office had been pre-wired to act as command center for the technology team.  Most of the systems needed for basic functions could move there in a matter of hours.

Responsible for coordinating

At 42, Stelly was the company’s disaster recovery manager, having started his career in this work in 1998 with Regions Bank in Alabama.  A native of Lafayette, he returned to Louisiana in 2003, when he joined Hibernia to be its primary disaster recovery coordinator.  He and his wife, Karen, married 18 years, had two daughters, 12 and 8.  His family evacuated to Lafayette while Stelly first took up a post at the Hibernia Center, then evacuated to Shreveport, where he continued to help supervise recovery for a number of weeks before returning to New Orleans.

Bragg gave Stelly high marks.  “He basically coordinated the whole Incident Management Team process.  He was responsible for coordinating just about everything related to the disaster.”

GREG STELLY: Sends out disaster alert / Photo: Skip Federico

GREG STELLY: Sends out disaster alert / Photo: Skip Federico

Urgently, Stelly transmitted the “go” decision to a waiting group of Incident Management Team members.  Some old hands had been expecting the call and already had nudged their units toward “go” status.  When the order came, about 150 people packed critical work, equipment and clothing.  They rushed to take care of last-minute personal matters, loaded cars and began driving Friday night and Saturday morning the nearly 300 miles to Shreveport.

Getting hotel rooms

At the Hibernia Center, Stelly went into high gear.  He had to reserve as many hotel rooms as he could for the people swarming toward the Ark-La-Tex hub.  He also began to grab rooms in Houston and Dallas.  As planned, two people from Hibernia’s purchasing area – manager Mike Broussard and David Aucoin – helped call the hotels.  They had an advantage in Shreveport – the bank was a significant service provider to local casino hotels, which cooperated as much as they could.  By 10 p.m., they had enough space for the first wave, including lodging in New Orleans for the “stay team” people, a skeletong crew that would remain at the Hibernia Center to operate the mainframe and process deposits.

Around midnight, when he could do no more, Stelly wearily drove across Lake Pontchartrain to his Mandeville home where he pulled patio furniture into the garage and did whatever else he could to prepare.  Then he packed enough clothes for a 3-4 day stay at the Hibernia Center and went to bed.

~ ~ ~ ~ ~

‘Recycling’ clothing

Months later, Stelly remembered his clothing.  “I wore those three days worth of clothes the whole first week in Shreveport, ‘recycling’ everything including the socks.  We were so busy and focused on working that we never took the time to even think about taking clothes to the laundry.  That next Saturday (Sept. 3) or Sunday, I decided I had to take the time to buy some clothes because every time I sat down, I could smell my own feet, with my shoes on!”

“I told Ben (Gautreaux) I was leaving ‘early’ – sometime around 7:00 p.m. – to buy clothes.  He asked me to pick up a few things for him.  The next day, I joked with others that I’d done something I never dreamed I’d do in my career at Hibernia … I bought underwear for my boss!”

~ ~ ~ ~ ~

Saturday morning, Stelly’s family left for Lafayette, and he returned to New Orleans to re-claim his lonely post.  That night, he bedded down on a cot on the 7th floor of the Hibernia Center.  He had a television set tuned to weather updates.  He had trouble sleeping.

The picture gets worse

Sunday morning, satellite imagery displayed an even more dangerous picture of Katrina.  Not only was the storm headed straight for New Orleans, with an estimated landfall less than 24 hours away, it had intensified, as some had feared, to Category 5 with winds of 160-180 m.p.h.  Forces of this magnitude and the storm surge they create are hard to comprehend.  Category 5 winds howl in an otherworldly moan.   Such winds rip off roofs, explode structures, uproot trees and transform harmless objects into deadly projectiles.  The tsunami-like storm surge they produce is terrifying.  Before Katrina, no one – not even survivors of Betsy or Camille – fully understood what the Gulf of Mexico could do to coastal and inland areas.  They just knew it would be bad.

Stelly called Bragg to warn how hard this monster storm might be on the city.  After it hammered the area, he thought he probably would be cut off from the rest of the disaster team.  The two agreed Stelly should evacuate.  He left about 10:30 a.m., discovered I-10 was a “parking lot,” drove northward to Hattiesburg and Jackson, MS, looped west and eventually arrived in Shreveport early Monday morning, about 14 hours later.  Bleary-eyed, he reported to the command center for a while, then went to his hotel for nap and was back at the center about 4:30 a.m.

‘Prop heads’ to the rescue

Stelly’s focus shifted after getting to the Shreveport.  He not only needed to get workers settled into lodging, he also had to make sure workplaces were ready in the command center at the main office on Travis Street and on the second floor of Hibernia’s Bossier City branch on Benton Road.  He already had chairs, tables and wiring drops in both places, but all of the equipment – phones, faxes, computers, and printers – still had to be hooked up and tested.

Later, these areas would be overwhelmed by an influx of people well beyond what was planned, and additional space would have to be found on the fly.

While coordinating with different departments to finish the “build-out,” Stelly continued to stay in touch with Steve Hebert, who stayed behind in the Hibernia Center with the skeleton crew.  By now the center had switched to generator power, anticipating an electrical failure.

Operating from the kitchen

Meanwhile, in New Orleans, Samford had plane reservations for his family to leave Sunday morning on one of the last commercial flights out of New Orleans to Dallas.  However, by noon Saturday he changed his mind.  Despite qualms about getting stuck on the highway, the Samfords packed, and with their dog and a daughter (another was safely at college), they drove out.  They arrived at the Tyler, TX, home of Carol’s mother about 11 hours later.  Samford recalled contra-flow “worked extraordinarily well,” and it was a relatively uneventful ride.

BART BRAGG: Long hours at command center / Photo: Skip Federico

BART BRAGG: Long hours at command center / Photo: Skip Federico

‘Just a blur’

He took enough clothes for three days, his laptop and his BlackBerry (but forgot the charger).  After a few hours sleep, he began to get his bearings but recalled Sunday was “just a blur.”

‘Hardest job’

Incident Management Team reports “got worse and worse,” and Samford found that, when his BlackBerry inevitably died, he had only the phone at his mother-in-law’s home.  He sat at her kitchen counter, receiver glued to ear, as he coordinated far-flung elements of his group’s disaster response.  His boss, Herb Boydstun, looking back after the disaster, said, “Ron clearly had the hardest job.  I mean, he had all the property.  He had all the technology.”

Samford worked through Sunday, receiving reports from Hebert in New Orleans, from Bragg, Stelly, Gautreaux and others in Shreveport and elsewhere — if he could get through.  He agreed it was best to keep the Hibernia Center operating rather than risk moving the critical mainframe and sensitive data tapes to a Dallas center where another mainframe was on standby.  In hindsight, he believed that was the wrong decision.  He thought he should have insisted on buttoning up the New Orleans operations and got everything out before the storm hit.)  Samford finally went to bed about 3 a.m., Monday, and awoke a few hours later to watch on television as Katrina smashed ashore.  Through the day, he continued to get updates and had “a great sense of relief Monday afternoon.”  The banking infrastructure in New Orleans and surrounding parishes certainly had “taken some damage,” but the hits appeared to be body blows, not knockout shots.

Reports of levee breaks

Then, Samford began hearing fragmentary news reports of the levee breaks.  He was concerned but had no way of knowing how the flooding might affect the bank.  He assumed the storm-hardened Hibernia Center could continue operating.  “The enormity did not hit us right away,” he recalled.  Looking back, what stood out for him most in the first chaotic days were harrowing stories filtering in about some of his operations people:

  • John Bell, a Fidelity Information Services employee, who was “out-of-pocket” – unaccounted for – for about three days.
  • Dawn (Ricks) Horvath, a technology manager, who was trapped in her flooded Lakeview home with her husband and 18-month-old son.
  • Donna Barry, head of corporate design, who lost everything and had to care for a mother with Alzheimer’s, while staying with friends in Baton Rouge

A key role

Through all of this, operational readiness focused on one man – Bart Bragg.  Samford credited him with getting most critical systems up and running quickly.  “That was key,” he said.

Bragg, 47, had been with Hibernia seven years.  He was manager of distributed services. Among other things, he was responsible for the important relationship with Fidelity Information Services, Hibernia’s mainframe vendor.  He and his wife, Dottie, married for 23 years, had two children, 20 and 18.  The family, with their dog, evacuated first to Auburn, where they took hotel rooms Sunday and Monday.  Then he continued to Shreveport Tuesday, where he took charge of the systems recovery effort, working side-by-side with his systems engineers.  Because he was right there, his people were able to discuss and get approval of many solutions on the spot.  This was one of the keys to Hibernia’s quick recovery.  Bragg stayed in Shreveport until mid-October, when he returned to New Orleans and his family.

Bragg was a geologist and computer science engineer by training.  He was a relatively “new kid on the block,” having come to Hibernia from Entergy, the electric utility.  His managers were responsible for seven crucial areas:

  • Barton Black, administration and ATMs.
  • Conrad Bujard, project management.
  • Ben Gautreaux, technology risk, including disaster recovery.
  • Dawn (Ricks) Horvath, applications.
  • Mike Altman, telecommunications.
  • Robin Grunwald, network operations
  • Ted Saba, enterprise engineering.
COMMAND CENTER: Crowded boardroom is recovery HQ / Photo: Skip Federico

COMMAND CENTER: Crowded boardroom is recovery HQ / Photo: Skip Federico

When Bragg got Stelly’s warning call that Friday afternoon, he knew there was a lot resting on his shoulders.  The information technology area was responsible for the mainframe and supported 150 different computer applications, 585 ATMs and 314 branches.  The bank and all of its subsidiaries could not function without this backbone of computer services, which his people provided.

Like Stelly and Samford, Bragg first intended to go to the Hibernia Center during the storm, but the weather picture Sunday morning changed his mind.  He did not want to get trapped.

Driving to Auburn

TUCKED AWAY: 'Prop Heads' work wherever they can / Photo: Skip Federico

TUCKED AWAY: ‘Prop Heads’ work wherever they can / Photo: Skip Federico

Instead, with their 110-pound black lab “Patch,” the Braggs drove in drizzling rain to Auburn, about 365 miles.  Normally a 5-6 hour trip, it took 10.  They settled into a motel and Bragg dialed into IMT conference calls “almost every hour on the hour.”  After the storm, his first reaction was similar to others.  The city and Hibernia had “dodged a bullet … and from a technology standpoint, we got our people out early.”

He had about 50 people staged in Shreveport Saturday, ready for the business unit people, the loan services people and the deposit operations people who were coming in on Saturday and Sunday.  “We already had people up there from a systems standpoint to start recovery.”

NEVER ENDINGS: Duane Plauche and Larry Dillon plug away / Photo: Skip Federico

NEVER ENDINGS: Duane Plauche and Larry Dillon plug away / Photo: Skip Federico

Then, Tuesday morning he heard from Stelly.  “I just got a call from Steve Hebert, and they are taking on water in the Hibernia Center,”  Bragg was told.

“That’s bad.  That’s very bad,” he remembered thinking.

Mainframe decision

With these reports that New Orleans was flooding and that municipal water service was failing, Bragg realized he would be forced to relocate the mainframe computer to Dallas.  He also knew he had to get to Shreveport to coordinate what lay ahead.  He drove there Tuesday.

Wednesday morning, his people shut down the mainframe after spending hours backing up all the data on tapes.  While the transfer to Dallas was taking place, the entire bank – hundreds of branches – would be “flying blind.”

“There was a lot of stress,” Bragg recalled.  Working conditions at Travis Street and Benton Road created some of it.  Disaster team members huddled around a room full of PCs.  They never anticipated they might have to run the entire bank for an extended period from this site.  It was only supposed to be temporary, capable of accommodating a skeleton crew for a few days.  Past storm recoveries had been relatively brief and operations re-started in New Orleans within a few days.

Now the makeshift command center was so crowded it taxed air conditioning and electrical systems.  George Steudlein, a corporate design project engineer, recalled urgent requests to boost the utilities, “but they were already maxed out.”

To accommodate the small army, Bragg’s people rented extra PCs from a disaster-service vendor.  Between that initial order and later purchases, some 1,400 computers were acquired.  “We ordered about $3 million of equipment in the first few days …”

Not a pretty sight

It was not a pretty sight at Benton Road either.  At the command center, people took over hallways, other offices, any space they could find to set up servers, PCs, laptops.  At Benton Road, dozens more people in loan and deposit operations and other areas were shoehorned side-by-side at long tables.  They also commandeered every available space.  Folks who had nowhere to work simply set up their new PCs on the cardboard boxes in which they were shipped.

COMMITTED: Cynthia Thibodeaux and Bob Fremin / Photo: Skip Federico

COMMITTED: Cynthia Thibodeaux and Bob Fremin / Photo: Skip Federico

Mike Quinn, head of consumer lending and loan services and systems, had a vivid memory of it.

“They were literally, physically shoulder-to-shoulder.  Imagine working shoulder-to-should 10-12 hours a day in a cramped room, full of people, full of noise in September where it is hot as hell in Shreveport.

‘Grateful to be busy’

I walked in that room and people were smiling.  People were chatting.  They were working.  It was remarkable under those conditions.  These Hibernia employees had just rolled up their sleeves and said, ‘Well, we are not in New Orleans any more.  We are in Shreveport.  We have a job to do, and we are doing it’.  To me, it was like a vision.”  Systems engineer Chris Berthaut remembered — “You couldn’t make people leave those first few days.  Some people were grateful we were so busy.  There was a reason for us to be there.  It was exciting.  We did a lot of neat stuff as a group.”

~ ~ ~ ~ ~

Executive cubicles

While Shreveport workspaces were makeshift at best, top executives also were huddled around borrowed desks, tables and offices in Baton Rouge.  They made do with whatever they could find, first in the bank’s main office and later in cubicles at a mortgage division east of the city.

“I think it was a great experience for me to work in a cube,” recalled CEO Herb Boydstun.  “I thought we all worked quite well … I never felt like it hindered me from getting my job done.“  I think a lot of the reason for that is everybody has been pretty open and honest with each other.  Nobody … felt that they had to do things behind closed doors.  And I think a working atmosphere like that is more conducive to what we were doing – where everybody could hear what everybody else was saying all the time.”

Paul Bonitatibus agreed: “I sort of miss the camaraderie … I think everyone was making the best of what we had and we were recognizing the blessings in disguise.”

~ ~ ~ ~ ~

Indeed, some people slept in place.  “There were bodies sprawled everywhere,” recalled Jeremy Lala, another systems engineer.  Those working at Travis Street would relieve each other, grab an hour or two of sleep on the floor or, like Lala, slip next door into the chairman’s office and stretch out on his leather sofa.  Anthony Sciortino, one of the engineers’ newer teammates – he started the same day the bank announced its merger with Capital One – remembered Berthaut “lying on the floor covered by the Hibernia flag.”

INTENSE: Network analyst Chris Li concentrates / Photo: Skip Federico

INTENSE: Network analyst Chris Li concentrates / Photo: Skip Federico

“There was surprise and satisfaction” in what they were doing, Berthaut recalled.  “We did what we did for a reason.  And we tried not to sit in front of the TV.  That’s when our eyes would start to water.”

Everyone worked together

Bryan Bensel, another engineer and former U.S. Marine during Operation Desert Storm, thought the stress and adrenalin were similar to combat.  “You see things that seem humanly impossible.  Everyone worked together.  We would throw together quick ideas – solutions – and if anyone were free, they would jump in and help do it.  We commandeered stuff when we needed to.  By the end of the week, reality began to set in.  People would start to break down.”

“We were not going to let this hurricane rip things apart,” Berthaut concluded.

Shawn “Scoop” Foret, an engineer in charge of tape backups, realized how important IT was.  “We would hear from departments who really needed our help, and the thankfulness they expressed was … overwhelming.”  Another engineer, Skip Federico, recalled being impressed by the guys he was working with.  “They know they have lost a house, and they are here working their tails off.  It was inspiring to me.  If you stopped, and watched TV, you were helpless.  When you were working, you knew you were doing something that someone needed.”

Federico remembered the reaction of a co-worker who learned his Chalmette home was gone.  “No, man, now I’ve got a hot tub and a water bed,” he laughed.  He also recalled a help desk worker who just walked off to the side and cried quietly when she could not locate her grandmother in a nursing home.  “She dried her face and came right back, sat down and went to work answering calls.”

“We would cling to what we had,” Federico concluded.  “We knew we were making a contribution, and we wanted to help.  Our co-workers were like an extended family.  We did not want to leave them behind.”

Another teammate, Patrick Kadow, agreed.  “Thank God for humor and caffeine.  The spirit in that place was unbelievable.  The feeling was, ‘This is all I can do right now.  I can’t save my house, I can’t save the people in New Orleans, but I can do this.”

Kadow was in charge of a system that allowed people remote access to the company’s computers through an internet application known as Citrix and VPN.  He was swamped with requests from people who wanted to use it since they were scattered hither and yon.  Access to this sensitive gateway was usually very restricted.  Now, however, he got a go-ahead to help most people onto the system.  “It was very freeing.”

A LITTLE R&R: Families band together for cookouts / Photo: Skip Federico

A LITTLE R&R: Families band together for cookouts / Photo: Skip Federico

CEO’s visit

During a visit to Shreveport, Boydstun was overwhelmed.  “The hardest part (for me) was going out and seeing our people working in difficult circumstances, sitting on the floor, using cardboard boxes as desks, crammed in rooms, hurting from what had happened to them and their personal possessions and families back in New Orleans,” he recalled.

After a few days, the wives and families of the “propeller heads” started showing up at the bank, concerned to see if their loved ones were all right.  Back at the hotels where they were staying, the displaced families of recovery team members got acquainted, often for the first time, and began organizing cookouts.  “The wives and girlfriends formed a little community and would cook for us and go shopping for those who couldn’t get out of the office,” recalled Sciortino.  “They were instrumental in the recovery by preserving a sense of normalcy.”

‘They saved the bank’

Payroll manager Yvette Soniat, who was there with others in the recovery group, had deep admiration for the people at Travis Street and Benton Road.  “What surprised me was people’s resiliency … I’m not talking, in some cases, about people who are paid very well.  I am talking about some who might be at the entry-level of this organization … It was incredible.  They saved the bank.  They absolutely saved the bank.  It heartened me just to see the pride that these people had, even though everything they had personally might be gone.  There was still something to be proud of and to work for.  It was pretty cool.”

‘This does not feel right’

One person – not even a Hibernia employee – would never forget those first days.  Debbie Bertuccini was programming manager for Fidelity Information Services, the company that provided and serviced the mainframe computer.  She had a staff of about 40 programmers at the Hibernia Center who maintained, updated and repaired all of the bank’s major “legacy applications.”

A programmer since 1976, she worked most of her career either for Hibernia or for companies to which the bank outsourced its operations.  She probably knew as much as any single person could about the company’s major systems.  Bertuccini, 50, and her husband, Dean, had been married 30 years, and had two children, 21 and 19.  The couple rode out the storm in Mandeville, then drove first to Shreveport and then to Dallas.

Friday before Katrina, Bertuccini  recalled, “No one was really nervous about the storm.”  She arranged hotel rooms for her people working at the Hibernia Center, but “just like others, I did not think it was coming this way.”  She also bought food – a couple days worth of bread, cold cuts and snacks – for the breakroom refrigerator.  Saturday evening, she expected to attend a bridal shower, but friends “started to get worried.”  Despite this, Bertuccini worked until around mid-day Sunday.  Then, “Something told me, ‘This doesn’t feel right’.”

She left for Mandeville.  The drive across the Lake Pontchartrain Causeway, which normally was 25 minutes, took three hours.  At home, in a careless moment, she locked her keys in her car, along with all of her storm information – names and phone numbers for everyone.  Luckily, she found a key service to “pop the lock” and continued to dial into the frequent disaster conference calls.  That is, until Mandeville lost all communications – landlines and cellphones – about 2 a.m., Monday.  When that happened, “Well, I’m dedicated to work.  I tried to phone every 10 minutes.  I made myself a frantic mess.”

~ ~ ~ ~ ~

‘I never cried and prayed so much in my whole life.’

~ ~ ~ ~ ~

The Bertuccinis rode out the storm in Mandeville, but when Tuesday dawned, she told her husband, “I had to know what was going on.”  They headed for Dallas, driving along streets littered with trees, wires and debris.  Outside Hammond, Bertuccini’s cellphone beeped alive, and she learned that some of her people were in Shreveport, so they changed directions — and “stayed a month.”  Working from the Travis Street command center, she was going to be in the eye of another maelstrom — the recovery of the mainframe computer.

Shutting down the mainframe

“We decided to slowly shut down the machine in New Orleans.  Our plan was to take tapes out at the end of the night.”  However, even doing it overnight, there would be a problem of how to catch up “interday transactions” handled manually at branches and elsewhere that would need to be posted to accounts on the big computer.

They tried to transfer data across a special telephone line, but it didn’t work. The connection seemed to handle the data, but they worried the New Orleans computer might “come down’ during the transfers.  The amount of information was enormous — all of the bank’s records — approximately 560 MIPS (million instructions per second).  On magnetic tape, this amount would fill 40-50 cartridges.

That Wednesday night, Bertuccini followed the plight of colleagues stranded with the data tapes in a parking garage waiting for rescue.   “I never cried and prayed so much in my whole life.  I had worked with some of them for 25 years,” Bertuccini recalled.

Then the work really began

With the people and tapes finally in Dallas Thursday afternoon, “the work really began.”  The few people she had there began to recover and restore the data to the new machine.  They worked all night and predictably, “ran into some issues.”

Everyone hoped they would have the new mainframe up Friday.  The bank already had endured one full business day offline, and it was an awful experience for people in branches and on phones, unable to answer even basic questions.  Customers were anxious, fearful and angry.  This situation could not go on long.  The executive management team, fully briefed about the crisis, waited anxiously, knowing they were helpless to do anything.  The bank simply could not operate “blind,” unable to balance or reconcile its books.  With millions of transactions each day, records could become hopelessly snarled.  The bank’s financial integrity potentially was at stake.

Exhausted, they soldiered on

Bertuccini’s exhausted team continued to work through Friday, soldiering on with little break and almost no sleep.  By that night, 24 hours later, they were still trying to get the big computer to operate smoothly.

The issues were complex.  Reloading a mainframe – a huge CPU – with trillions of pieces of data is not a simple task.  Under normal circumstances, such machines hardly ever rest, operating almost 24 hours a day, every day, continually accepting new data, continually being updated.  If these machines required “rebooting,” the only thing similar to a personal computer was the word.  Some 700-800 different files of computer instructions had to be loaded, generally in a specific sequence.  Some of the files restored properly, others did not.  Some of the files were earlier versions.  There were “gaps” in the instructions or data that required laborious, creative “workarounds.”  The workarounds themselves were built on the fly, without testing.

~ ~ ~ ~ ~

‘We fixed things as they broke.’

~ ~ ~ ~ ~

 

Bertuccini and two colleagues – programming supervisor John Nunez and Jo Ann Katsorchis, a top-level programming analyst – sat side by side at PCs parked on a table in Shreveport.  They monitored every move by their cohorts in Dallas and communicated by phone every step of the way.  These three were the gurus who could diagnose, assess, re-program, fix and repair.  “We sat there hour after hour, making decisions about what to do next,” Bertuccini recalled.  “We fixed things as they broke.”  In Houston, three people were carrying the lion’s share of the start-up work on the CPU – Jimmy McAlpin, Mike O’Neal and Russell Martin.  As the hours ground by, they stayed at their seats and kept plugging away, driven by the knowledge of how important their work was.  “I almost felt lost … sometimes like a failure … even though I really was not.”

Prodding each other awake

Through the early morning hours Saturday, they kept at it, often having to jar one another awake when a screen changed or a new instruction was needed.  Once, Bertuccini remembers nudging Nunez.  His fingers were resting on the keys, his eyes closed.  Row after row of SSSs filled the computer screen.  “Mentally, we were flaked out.”

However, Bertuccini had unusual qualities, visible to others.  “She had a maternal instinct, a sense of ownership,” noted Chip Gremillion, head of Hibernia’s corporate products administration.  “She worries about everything, big and little, and pushes harder than anyone to get things done … and she’s very smart.”

Around 5 a.m., with a few final taps on the keyboard, the exhausted programming chief could see they had wrestled the devilishly difficult changeover to the mat.  “We did the best we could.”  Bertuccini, Nunez and Katsorchis had been at their terminals nearly non-stop for more than 48 hours.  “It seems comical now,” she reflected, “but it was stressful then.”

The Dallas mainframe began to hum and crunch.

Bertuccini got up and walked to a nearby hotel where her husband and children were staying.  They had come for the weekend to be with her.  She fell asleep for a couple of hours, then awoke and went with them for a bite to eat.  On the way, she asked if she could stop by the command center.  Her daughter revolted: “You can’t go, you can’t go,” she pleaded.  “You won’t come back.”  Bertuccini compromised.  With her kids as escorts, she paid a brief visit, and then allowed them to lead her to a nearby restaurant.  She decided to have a Margarita to unwind, and that is when the hours of sleeplessness hit.  She made her way back to the hotel for some blessed rest.  Saturday afternoon, she was up and back at work.

At the end of September, Bertuccini had to go to Dallas.  In October, she helped engineer another massive changeover, this time shifting the mainframe to a new location in Little Rock, AK.  She did not get back to New Orleans until mid-February, when she and her husband were staying with a friend and waiting to get a FEMA trailer.  When she looked back on those crucial 48 hours of recovery, she was philosophical:  “We did succeed.  We were not perfect, but the bank came up solid.  As crazy as it was, it was almost fun, even though it was real stressful … and we all worked so hard, we didn’t worry about our personal situations.”

Yet another move

As days wore on, more and more employees arrived in Shreveport, either finding their way on their own or being called there by managers who needed more help.  For Bragg and others on the IMT, this unplanned influx created a new issue.  They had to find more workspace, suitable for a longer period than they had expected.  They also had to get a lot more computers, telephones and other services installed.  “Everybody was yelling and hollering that they were just as important as the next person … We were just going in a lot of different directions, and we weren’t always being very efficient,” Bragg recalled.  “We were very harried and stressed out.  We knew we were doing some pretty good things, but we really were not feeling that way.”

Shreveport annex

Fortunately, there was a suitable building nearby.  They called it the Benton Road Annex and soon had it equipped with cubicles, computers, telephones, faxes and furniture for 250 people.  The first 40 loan services employees moved in Sept. 28.

That annex was not the only place where more offices were needed.  By early November, the operations area built and outfitted workspace for more than 1,400 employees in Shreveport, Dallas, Houston, Baton Rouge and other locations.

Second move of mainframe

Bragg also had another mainframe issue.  He, Samford and others wanted to make the company more resilient in future storms.  They decided the permanent home for the host computer should be Little Rock, AK.  Rather than bring the mainframe back to New Orleans from Dallas, they would transfer it permanently to this new location.  However, moving the data and functionality again, and so soon, was almost as big a hurdle as the first transfer.

“That was a huge coordination project, working with Fidelity and working with our own business units to get it moved and to make sure we did it smoothly so as to not impact (anyone) any more than we already had,” Bragg remarked.

It is pretty rare for a company to move a mainframe.  Often it can take a year or more of preparation.  To do it twice within a few months is practically unheard of.  But that is what Bragg’s team did — New Orleans to Dallas, then Dallas to Little Rock.  “That was probably one of the biggest single efforts that we worked on after we got most of the systems recovered.”

Move to Royal Ridge

If that were not enough, top management, with coaxing from Capital One, decided another important resiliency measure would be to transfer certain operations permanently from New Orleans to Dallas.  They settled on a facility previously developed by Capital One in an area called Royal Ridge.

This decision meant some 200-400 positions would have to relocate.  Many, if not most, of these were the jobs of the very same people who just had been displaced by the storms and were working long hours in Shreveport, Baton Rouge, Houston and elsewhere.  Some still did not know the condition of their New Orleans homes and were disconnected from family and friends.

~ ~ ~ ~ ~

Will it ever end?

With the move of the mainframe to Little Rock, for a brief time there was only one circuit at the new location connecting branches and call centers to it.  As luck would have it, before a second line could be installed, a telecommunications worker in Dallas accidentally disconnected it.

People raced to troubleshoot the problem, but it took nearly a day to trace down the problem.   “They found our Achilles tendon one week before it was going to be repaired, and it brought us to our knees again,” Ron Samford sighed.

~ ~ ~ ~ ~

Wrenching experience

It was going to be wrenching to break this news, and even harder for employees to decide whether to take relocation offers and begin new lives.

“For employees to have to do this after all they’d gone through the last four months of just a really emotional rollercoaster and a lot of work, that’s the big challenge … Where are they going to be?  People are still displaced.  People have children in school.  How do you get back to some sense of normalcy?”  Bragg reflected.

Public backlash?

When it was announced, CEO Herb Boydstun described the situation as it was.  The company needed to be more hurricane-proof.  In addition, the sad fact was that many employees could not return to New Orleans anyway.  They had no place to live.There was also the question of how New Orleans would react to news that Hibernia was moving jobs away.  Would there be a backlash?

Eventually, many employees stepped up to a new future in Dallas.  There was some attrition, but not much.  In new Orleans, the potential public uproar never occurred.

An odd result of the move was that it created many redundant workstations around the company.  Hundreds of recovery workers had been set up at new computers in Shreveport while their workstations back in New Orleans sat idle. Now they would be outfitted with a third workplace in Texas.

Getting back to New Orleans

For employees coming back to New Orleans, it was difficult as well, but in different ways.  Some managers said it was just as hard to come back as it was to move out.  Once in Shreveport, workers had gotten children enrolled in local schools.  They had made numerous living adjustments that were tricky to unwind.  When they returned, it was often to rental accommodations again.  And the school situation was much worse.

 

Leave a Reply

Your email address will not be published. Required fields are marked *