The Financial Services Authority says banks mis-sold specialist insurance known as interest rate swaps tied to thousands of small businesses.
The FSA said it had found "serious failings" in the sale of the products, designed to protect firms taking out loans against rising interest rates.
The FSA said it had reached agreement with Barclays, HSBC, Lloyds and RBS over providing "redress".
The mis-selling is the third case of serious malpractice at the UK's banks.
The FSA said it believed the swaps had had a "severe impact on a large number of these businesses".
It did not say how much money would be necessary to compensate the businesses involved.
Sandy Chen, from Cenkos Securities, estimated that the total cost to the banks collectively would be between £1.1bn to £1.4bn, with most of that falling on Barclays.
Around 28,000 interest rate protection products were sold to thousands of small businesses, starting in 2001.
The businesses affected should now be contacted by their bank to instruct them whether they are included in a review of these sales. Those which were found to have been the victims of mis-selling will eventually be offered compensation.
Assurances
The managing director of the FSA's conduct business unit, Martin Wheatley, said the practice had been costly for the victims: "For many small businesses this has been a difficult and distressing experience with many people's livelihoods affected," he said.
He added that the bosses of the banks, including Barclays chief executive Bob Diamond, had given a personal assurance they would sort out the problems caused.
