Difficulties getting hold of HM Revenue and Customs has cost the taxpayer the equivalent of nearly £100m and may have put some people at risk of debt.
Phone call waiting times of more than an hour for some was down to a “collapsed” service as HMRC cut staff while at the same time trying to encourage people to do their tax returns online, said the National Audit Office (NAO).
The spending watchdog said HMRC’s view that its digital strategy would require fewer employees to answer the phones had badly misfired.
Its report said: “The quality of service provided by HMRC for personal taxpayers collapsed in 2014-15 and the first seven months of 2015-16 when average call waiting times tripled.
“Services have subsequently improved following the recruitment of additional staff but whether this performance is sustainable depends on HMRC achieving successful outcomes from its programme to make tax digital.”
The NAO said that self-assessment callers were left waiting for an average of 47 minutes for their calls to be answered during the deadline week for paper returns in October 2015.
HMRC said waiting times now averaged about six minutes.
Using HMRC’s own criteria, the NAO estimated call charges had cost people £10m, the value of customers’ time spent while waiting at £66m and the value of their time actually talking to someone at HMRC once they have got through was put at £21m.
The NAO also estimated that more than three million people may have been left paying the wrong amount of tax thanks to the problems.
Citizens Advice chief executive, Gillian Guy, said: “Long waiting times not only cause frustration and increase the cost of the call, but can also mean people miss important deadlines.
“For example if you don’t return your tax form on time you face a fine – which for some households can be an additional cost they can’t afford to pay.”
Ruth Owen, HMRC’s director general for customer services, told the BBC: “We recognise that early in 2015 we didn’t provide the standard of service that people are entitled to expect and we apologised at the time.
“We have since fully recovered and are now offering our best service levels in years.”