How ChatGPT may disrupt auditing
From writing audit file notes to giving expert advice and potentially identifying high risk areas, Imran Vanker, director of standards at IRBA, explains how ChatGPT could change auditing.
How many auditors does it take to change a light bulb? It’s not a matter of how many but whether the change is properly documented and supported by sufficient evidence.
ChatGPT wrote that joke for me. As Imran Vanker, director of standards at the Independent Regulatory Board for Auditors (IRBA), points out, ChatGPT is also good at creating sufficient evidence.
If you ask ChatGPT to describe itself, it will tell you, “I am an artificial intelligence language model trained on a large amount of text data to understand and generate human-like language. I can answer questions, have conversations, and provide information on a wide range of topics.”
Vanker has been experimenting with the technology. In January, he used it to write a communique on artificial intelligence and the audit profession. Neither his senior manager nor the language editor made major changes to the communique ChatGPT authored.
Vanker has found that ChatGPT’s auditing applications extend to areas.
Using ChatGPT to create audit file note
To illustrate, Vanker gave ChatGPT the following prompt.
“I am an auditor. The year-end has just passed. Write me a file note for my audit file on the year-end stock take. Focus on the existence and valuation of the stock. At the count, I found only one error, and it was not a big deal. The condition of the stock was good. Last year’s problems were not found again this year. The stock was in a brewery.”
To which ChatGPT replied:
“If we inspected the file with this note, it would be a perfect file note, says Vanker. “It would meet all the requirements of the standard.”
“Other than the fact that it’s a fake file note or if you looked for some contradictory evidence, you’d come to the conclusion that this meets the sufficiency and appropriateness test that the audit standards require from us.”
“It knows things about a stock count that I didn’t ask. For example, it gave the date of the count, but it said it was present throughout the count, which is a condition of a valid stock count. And then I told it that the stock was in reasonably good condition. It drew the conclusion that it did not observe any signs that would impact its value or usability.”
The danger? “My challenge is that an audit clerk would have spent the day writing this file note in the absence of the manager who’s reviewing this, not knowing the note has been generated by ChatGPT. What if actually the stocktake never happened? The clerk didn’t go to the stocktake, then I can tell a very convincing story,” says Vanker.
Can ChatGPT provide advice like a seasoned auditor?
It’s not only audit clerks who’d find ChatGPT useful.
“You could have a firm in a small town with one partner in the firm looking at this file saying, ‘I wish there was someone here who could help me’. The traditional audit software doesn’t give them the answer,” says Vanker.
To illustrate, Vanker asks ChatGPT the following.
“I am an experienced audit engagement partner. I am reviewing the reporting section of the audit file. I see that we have a number of audit differences. All with credits to the income statement. Please give some advice with references from ISA 700 (revised) and ISA 220 about my responsibilities.”
“My sense is that the advice that, without us giving it any more specifics, is roughly equivalent to what the experienced audit partner would get from the firm’s technical department,” says Vanker.
“Interestingly, I didn’t reference ISA 330 or ISA 450. I didn’t know that. The experienced practitioner is getting guided to exactly where the answer is. And, of course, this is a very positive response. If indeed the answer was right if it wasn’t making it up. In this case, I happen to know it’s not.”
The problem of AI hallucinations
Sometimes though, ChatGPT does make things up, with great eloquence.
During a demonstration, Vanker asked ChatGPT if it knew about auditing standards ISA 620, which deals with the use of experts. ChatGPT responded that the standard deals with the concept of materiality.
This flaw, referred to as AI hallucinations, is not only a ChatGPT problem. Self-driving cars will see objects that don’t exist. Google Cloud computing reportedly mistook a picture of a rifle for a helicopter.
However, Vanker says this can become less likely as technology develops. “If we could get GPT to ingest the (auditing) standards and say, look at the standards to come to a conclusion,” explains Vanker.
OpenAI, the company behind ChatGPT, recently started selling access to businesses looking to integrate the technology into their products.
It’s like accessing the internet in 1995
“We may be excited by what it is now, but its true potential is still to come.”
One application Vanker has reflected on when discussing the technology with colleagues is a model that works similarly to how banks grant loan applications. Instead of staff at the branch making decisions an algorithm collects and processes relevant data and produces a loan decision.
“What happens if that logic is applied to the entire back end of the audit industry? So a firm has 400 clients. It sustains the sources of information at the back, then ISA 315, which is the risk ID standard, may become a very rich experience.”
From a regulator’s viewpoint, AI could be similarly powerful. “If I can convert PDF financial statements into a version the AI can read. Then I can use it as the regulator for business intelligence purposes in the same way firms would. I would ingest this information, and it would give me indicators which I would then apply my expertise over and decide.”
“Our thinking up to now was let’s gather a lot of data in a spreadsheet and try to find trends in numbers. Now we may have the ability to find trends in narratives.”
Will AI make it tougher to catch bad actors
Once AI tools like ChatGPT are widespread among firms and regulators, will it be tough for auditors to flag the activity of malicious actors who are also using advanced tools?
“But (if this happens) we would be no worse off than where we are now,” says Vanker. “Audit today is premised on the fact that you only accept the engagements of clients you’ve done appropriate background checks on. The journey of audit evidence at the moment only seeks to identify evidence that corroborates assertions. The conventional financial statement audit does not seek out contradictory evidence. So we’re not trying to prove management wrong. We’re looking for evidence supporting what management says to us.”
AI is not going to bridge the expectation gap.
“If I go back to the inventory count procedure, management says we’re happy that our stock of beer in the brewery has not expired. When I walk around, I look at the bottles’ labels and confirm that it’s not expired. Therefore it’s valid. Of course, I don’t know that the night before the stocktake, they relabeled all the beer. They changed all the expired labels to valid labels. I wouldn’t have figured that out either way. But that’s not my burden as a conventional financial statement audit that’s within the realm of a forensic audit.”
As with any new technology, the ethics of ChatGPT’s use cases are likely to depend on the ethics of its user.
