Sudden and severe Visa crash explained

In just 10.1 hours an incredible 2.4million UK payments failed.

Everything from supermarkets to bars to people trying to get a ticket saw their cards rejected.

Many retailers responded by only accepting cash, cash machines ran dry, some bars decided to offer free drinks to Visa customers, but one thing was exposed beyond all others - just how many of us are reliant on our cards.

“A third of all spending in the UK is processed by Visa. It’s deeply worrying, therefore, that such a vital part of the country’s payment infrastructure can fail so catastrophically," Nicky Morgan, Chair of the Treasury Committee, wrote in a letter to Visa Europe chief executive Charlotte Hogg.

“The consequences were sudden and severe. Many consumers and businesses were left stranded on Friday, unable to make or accept payments, with chaos reported in shops."

So what happened? This is the result of the Visa investigation.

"On Friday 1 June, our UK processing systems were disrupted such that many UK consumers were unable to complete transactions using their Visa debit and credit cards," Hogg told the Treasury Select Committee.

She added: "At its peak, the disruption affected people in the midst of returning home from work, socialising in restaurants and pubs, and doing end-of-day shopping."

This is exactly what happened:

14:35 Friday - UK data centre operations team became aware of "a partial degradation in our processing system" - it was not a cyber-attack.

The backup doesn't kick in - the entire Visa entire system has a full safety net that kicks in if the main system fails. In this case, however, the failure was only "partial" so the secondary system did not take over

Additionally, the main system kept trying to synchronise with the back up - leading to a backlog of messages and slower operations at the secondary system. The team at Visa start switching systems off and cleaning up messages at the main system to clear the backlog - both manually and with automated programs

14:45 Friday - Visa's technical response team holds an assessment meeting, they escalate the issue to all relevant European support teams

15:05-15:15 Friday - First period of peak disruption, 35 percent of transactions fail

16:00 Friday - The full crisis management team meets, led by Visa Europe chief executive Charlotte Hogg

16:05 Friday - A public statement is sent to the media reading: "Visa is currently experiencing a service disruption. This incident is preventing some Visa transactions in Europe from being processed. We are investigating the cause and working as quickly as possible to resolve the situation."

17:40 - 18:30 Friday - Second period of peak disruption, 35 percent of transactions fail

19:10 Friday - The failing system is fully deactivated. The secondary system is now operating and processing most transactions normally

20:15 Friday - Problem mostly fixed

00:45 Saturday - Normal service is fully resumed at both primary and secondary data centres.

"The service disruption lasted from 14:35 to 00:45. This length of time was clearly unacceptable," Hogg said.

Visa calculates 5.2 million transactions failed to process correctly.

When transactions failed, it happened in two ways, she added. "Either when Visa's system did not respond within required time limits (generally 10-15 seconds) so the transaction 'timed out', or the transaction was improperly declined due to a Visa system error."

Many cardholders tried again after initial failure and got through, 1.1million cards experienced failures.

Overall the whole period, one transaction in 11 failed (9 percent) - but Visa accepts this is far from good enough.

"We do absolutely hold ourselves accountable to the consumers and merchants who use our cards," Hogg told the Treasury Select Committee.

"In this instance, we failed to meet your and our own expectations, and we apologise again unreservedly to everyone who was affected by the incident."

What Visa is doing to stop it happening again

Visa said its priority was making sure people do not lose out, and working with its partners to stop anything like this happening again.

"Since 1 June, we have also been focused on identifying all necessary steps to prevent a reoccurrence and ensure that Visa's incident response is enhanced in the future by applying any lessons learned," Hogg said.

"We are undertaking both a rigorous internal review, and bringing in an independent third party to ensure we fully understand and embrace lessons to be learned from this incident."

Visa removed the parts of the switch that failed, and replaced them with new versions from the manufacturer.

It's also conducting a forensic examinations of the broken switch with the hardware manufacturer to find out exactly why it failed when it did.

A new monitoring program is being brought in that will check the switches, meaning if a similar problem happens again it will be spotted and shut down immediately, letting the back up kick in straight away.

Visa is also looking at how it can isolate failing components anywhere in the system faster in future.

Further out, Visa plans to move to a new system with multiple data centres, all working in tandem and with a lot more processing capacity. This move should be complete by the end of the year.

The new system can also isolate and remove a failing component with a single command - this plan has been in place since February.

There is also an external review of the incident form EY, looking at causes as well as Visa's response.

"We hold ourselves to extremely high standards and are very disappointed that this incident caused difficulty for consumers and businesses," Hogg said.