// Home / Enterprise / News

RBS: outsourcing not to blame for bank IT glitch

By Nicole Kobie

Posted on 26 Jun 2012 at 10:31

The CEO of RBS has said outsourcing is not to blame for the lengthy IT fault at the banking group.

RBS and Natwest say that their accounts are mostly back to normal after the almost week-long IT glitch, while accounts at Ulster Bank continue to be affected by the fault, with no solution expected before next week.

The incident has left many wondering how such a serious fault could last so long at a bank, and what the actual cause was - with reports suggesting a flawed upgrade in Computer Associates' software was the root of the problem.

A report in The Guardian said an update to CA 7, a workload automation system, went awry for three nights in a row. That meant the bank failed to properly process payments, leaving accounts out of date and the system struggling to catch up.

Things go wrong in technology - we have to learn the lessons from what went wrong here and try to make them less likely in the future

CA confirmed that RBS is a customer, and a spokeswoman said the software firm was "offering all assistance possible to help it resolve its technical issues which are highly unique to its environment." She added: "We do not comment on customer confidential issues."

The Unite union has pinned the blame on 30,000 IT job cuts at the banking giant since the financial crisis hit, with many roles replaced by offshoring. “Serious questions must be asked as to why constant job cuts are being made when there are clearly serious issues which need addressing by management," said David Fleming, Unite national officer.

Stephen Hester, CEO of RBS, said there was "no evidence" that overseas outsourcing was to blame for the fault, saying the firm's "UK backbone" of infrastructure had received significant investment.

"But things go wrong," he told Sky News. "Things go wrong in technology. We have to learn the lessons from what went wrong here and try to make them less likely in the future."

Customers at risk

The fault has led to delays with payments in and out of affected accounts, hitting customers in a variety of ways - including one being forced to spend the weekend in prison because he was unable to make bail.

The banking group has promised that no customers will be left "out of pocket" and is working with credit agencies to ensure missed payments don't put a black mark on customer credit ratings.

The Payments Council has advised all affected customers to visit their branch to sort out any problems and to keep any related documents in case they need to prove the bank is at fault.

The Financial Services Authority has said it expects a full report from the banking group about how the fault happened and how it treated customers during the incident.

RBS said its own customer account balances as well as those of Natwest customers had cleared overnight, aside from a few "specific sets of transactions". "We know this disruption was unacceptable and that many customers will still have questions and concerns," the bank said. "It is possible a small number of customers may experience delays as we return to a completely normal service."

"The full focus of our efforts will now be on delivering the same result for our Ulster Bank customers who continue to experience unacceptable delays to their accounts being updated," it said, adding it expects to restore full service by the start of next week, and will keep branches open late in the meantime.

Subscribe to PC Pro magazine. We'll give you 3 issues for £1 plus a free gift - click here

User comments

Outsourcing \ Offshoring

I realise that Mr Hester, as CEO of a major Financial Institution has to be careful with his words, but this was exaggerated care:
"...Stephen Hester, CEO of RBS, said there was "no evidence" that overseas outsourcing was to blame for the fault, saying the firm's "UK backbone" of infrastructure had received significant investment."

Taking this at face value, maybe it was some local 'outsourcing' that failed?
Outsourcing is a means of cutting costs much beloved of those in Finance. This is becuase even if it actually costs the same, or more, it cuts 'headcount' and looks good in the books.

Why is this important? Its becuse all those yummy bonuses are tied-in to predominantly accountancy-based metrics: like reducing costs like 'headcount'.

The truth is RBS has been caught-out, its much vaunted claims about 'Customer Care' revealed as the PR puff they actually are.

Of course technology can fail. That's not the issue. The issue is that their technology not only failed, but they had no viable recovery plan (no staff? no clue? who knows?).
Given the importance to millions of their customers of these systems, you'd think their Recovery Plan would be pretty impressive. It clearly isn't, but I'll bet it's cheap.....

RBS's salespeople... sorry Customer Care Staff doubtless justified their expensive Insurance Policies with words like: "You may never need this, but if anything goes wrong, you'll be glad you spent the money.." shame they didn't take their own advice.

By wittgenfrog on 26 Jun 2012

Read 'The Register'

... which seems to have got the best information about what the actual problem was, and how it was caused.

By JohnGray7581 on 26 Jun 2012

Unacceptable Testing Procedures?

What sort of testing procedures do RBS have? Why were they not able to back out of the upgrade when it failed? This is all simple stuff - someone should be sacked!

By SimplyGrey on 26 Jun 2012

Interesting most people aren't commenting here - either because it's not a tablet but an "unimportant mainframe" or because they're thinking, "There but for the grace of God..."

Re back-out - according to that article in The Register, it was the backout (of a minor software change) that caused the major problem. Given that from any one problem there will be dozens of back-out options (how much do you back out? etc.), it isn't actually simple stuff.

Re testing - again, assuming that The Register is right, the issue with testing a job scheduling system is that the only way you know that the right job identities have been entered is to run the schedule - which means running the production system. Which you clearly don't want to do in testing. Yes, you can mitigate a lot of it through parameterisation but you still can only run the test system which won't find the stupidly hard coded test system id where it should be parameterised.

RBS's Recovery Plan for a hardware failure probably is impressive. Recovery Sites to cope with software failures are pretty much of a moot point because Recovery Sites are normally kept in sync, so that if one set of software fails, well, the other will. If the Recovery Site software is not kept in sync then as soon as there's a hardware failure and you try to invoke Disaster Recovery, you're stuffed because you haven't got the right software loaded.

I suspect this is going to show that the ingrained knowledge of operations and scheduling staff is so much in their heads that handing it over to others is a long walk over a short pier. Worse for the original operations and scheduling staff, it's so unglamorous that any protests from them will have been dismissed as more complaints from those dinosaurs in mainframe scheduling. And no, I never was a mainframe scheduler!

By AdrianB on 26 Jun 2012

@AdrianB

Good points.

As you say these (presumably mainly legacy) systems are complex and need incredibly careful handling.

The Register article doesn't make too many specific points.
I'm still left with the strong feeling that RBS\Natwest have cut a few corners somewhere along the line.

By wittgenfrog on 26 Jun 2012

RACF

I used to manage the Mainframe testing platform at Natwest many, many years ago (pre RBS takeover). All I can say is that at that time they had the most OTT Release Management processes and some of the top CICS, IMS and JCL experts working for them. It's a long time since then but I'm amazed this has happened.
Yep, there for the grace of Ford.

By caulfiek on 26 Jun 2012

RACF

By caulfiek on 26 Jun 2012

@wittgenfrog

"I'm still left with the strong feeling that RBS\Natwest have cut a few corners somewhere along the line" I agree with you - probably the application knowledge has been left behind with reorganisations, early retirements and redundancies.

Incidentally, mainframe systems only look complex because it's (theoretically) possible to draw all the interfaces. If it's an event driven system, generally it's not possible to draw where the data comes from, so everyone thinks it's so much simpler, when it isn't.

By AdrianB on 26 Jun 2012

@caulfiek

I worked for NatWest around 20 years ago (pre RBS) but in Networking, not 'Mainframes'.

Their Change Control systems were incredibly precise and restrictive, which is as it should be where such sensitive systems are concerned!

Post RBS I don't know, but RBS 'leveraged' the efficiency (read cheapness) of their IT Management Systems as part of the buyout process.

By wittgenfrog on 27 Jun 2012

Leave a comment

You need to Login or Register to comment.