Black Box Blues – When Nobody Can Explain the Numbers
The black box problem in audit and financial reporting
There is a moment in this episode of Ethics in the Age of AI that captures the whole problem in one exchange. An audit engagement manager, asked by a client’s CFO why the firm’s new AI platform flagged certain transactions as high risk, can only offer: ‘The AI identified them as anomalous.’
‘That’s not an explanation,’ replies Alex, one of the podcast’s two AI hosts. ‘That’s just saying the same thing again.’
Explainability, the episode argues, is not a nice-to-have in the accounting profession. It is foundational to professional accountability. And AI is making it very complicated.
The audit that nobody could explain
The first scenario follows a partner at a mid-sized firm that has invested in an impressive AI audit platform. It analyses entire transaction populations rather than samples, flags anomalous patterns and generates risk assessments automatically. The dream, essentially. Until the cracks appear.
The team cannot explain the AI’s findings to the client. Junior auditors have started focusing entirely on what the AI flags, setting aside the broader professional scepticism that might catch what the AI misses. Documentation is becoming thinner because everyone assumes the automated workpapers cover it. And client data is flowing through the platform’s cloud servers without ever being properly addressed in the engagement letter.
The suggested way forward is not to abandon the technology. It is training, a blended approach of AI-directed and traditional risk-based testing, independent verification of data completeness, stronger documentation and updated engagement letters that explicitly address AI usage and data handling. The episode also highlights the Financial Reporting Council’s recently published Generative and Agentic AI Guidance as a genuinely helpful starting point for firms working through these questions.
The Audit Committee that could not get answers
The second scenario moves to the boardroom, where an experienced Chartered Accountant chairs the Audit Committee of a listed technology company. The company’s AI system automates complex revenue recognition calculations and intangible asset valuations, exactly the judgement-intensive areas where transparency matters most. Yet when the Chair probes how key figures were reached, the finance team can only point to the system’s complexity and its proprietary algorithms. Essentially: it is the AI, we trust it.
The external auditors cannot verify what happens inside the system either. And these figures underpin market communications and regulatory filings, information that investors and regulators rely on. The system, it turns out, was adopted without any documented board-level discussion or approval.
The Chair’s response sets the standard: do not approve the quarterly financials until satisfactory explanations arrive. That is not obstruction. That is the job.
A question worth asking
In our profession, the hosts conclude, ‘I don’t know, the algorithm said so’ has never been and should never be an acceptable professional answer. Transparency is not just an internal virtue. It is part of what the profession owes to the public.
Listen to this episode of Ethics in the Age of AI below or subscribe on Apple Podcasts, Spotify, Amazon Music or your favourite podcast library, and explore the full case studies and ethical frameworks behind the series at ccab.org.uk.