HMRC’s failure to clamp down on rich tax dodgers is undermining confidence in the whole system, the public accounts committee said.

The highly critical report released on Friday examined HMRC’s specialist unit, which collects tax from high net-worth individuals with more than £20m. It found that “the amount of tax paid by this very wealthy group of individuals has actually fallen by £1bn since the unit was set up” in 2009 – even as tax receipts rose to £23bn.

Meg Hillier, the Labour MP who chairs the committee, said HMRC’s claims about the success of its strategy to deal with the very wealthy did not add up.

“Cosy terms such as ‘customer relationship manager’ and HMRC’s reluctance to be open add to the picture of arrangements that, while beyond the reach of ordinary taxpayers, are also ill-suited to the increasingly sophisticated methods the super rich can use to reduce the tax they pay,” she said.

“If the public are to have faith in the tax system then it must be seen to have fairness at its heart. It also needs to work properly. In our view, HMRC is failing on both counts.”

Tax officials calculated that there were about 6,500 high net-worth people in 2015-16, about one in every 5,000 taxpayers. In 2009, a specialist unit was set up to bring in more money from them.

MPs questioned the role of the specialist unit and some of its practices.

“We were not convinced by [HMRC’s] assertion that there is a clear line between giving its view on potential transactions and giving tax advice and we do not think there is enough clarity about what customer relationship managers can and cannot do,” the report says.

The committee pointed out that advice from officials to wealthy taxpayers was not recorded. “While calls from most taxpayers to HMRC call centres are recorded routinely, meetings and phone calls with high net-worth individuals are not recorded,” the report says.

The committee also highlighted concerns about “potential abuse” of image rights by top footballers and entertainers to minimise their tax liabilities. It confirmed that HMRC had “open inquiries” relating to the use of image rights by 43 footballers, 12 clubs and eight agents.

Committee members said they were appalled to learn that not all clubs were providing HMRC with the data it required under the terms of a voluntary agreement with the Premier League.

However, they praised HMRC’s managers for trying take action against the clubs. “We were encouraged by the evidence HMRC’s senior management gave to the committee on image rights and we look forward to news of meaningful action in this area.”

HMRC said the pursuit of high net-worth individuals had resulted in the collection of an additional £2.5bn in revenues. But it was unable to explain why the income tax they paid fell by 20% – from £4.5bn in 2009-10 to £3.5bn in 2014-15 – when the overall income tax take rose to £23bn.

The committee said about a third of the individuals concerned were likely to be under inquiry by HMRC for unpaid tax – with cases with a potential value of £1.9bn currently under investigation.

However, the report found HMRC had a “dismal record” when it came to prosecuting the very wealthy for tax fraud in the criminal courts.

In the five years to 31 March 2016, it completed just 72 fraud investigations into high net-worth individuals, with all but two having been dealt with using its civil powers. Only one case resulted in a successful criminal prosecution.

Of the 850 penalties issued to the very wealthy since 2012, the average charge was £10,500 – a figure the committee said was likely to be too small to act as a deterrent.

The problem was likely to become more acute because wealthy people were moving from off-the-peg tax avoidance schemes – the “high street equivalent of Primark or Next” – to bespoke “made-to-measure Savile Row” arrangements, the report says.