New research has raised concerns about the reliability of AI in answering finance-related questions, after findings revealed that over one-third of AI-generated responses were inaccurate.
The study, conducted by broker analysis site Investing in the Web, tested ChatGPT—one of the most popular generative AI chatbots—by asking it 100 finance and investment-related questions.
These questions covered various financial topics, including savings strategies for children’s education, pension comparisons to average salaries, and the pros and cons of investing in gold. The results highlight the potential risks of relying on AI for financial guidance.
The study also asked both evergreen questions, where the right answer will always remain relatively consistent, and current questions, where the answers are more dependent on recent information.
The responses from ChatGPT were considered ‘correct’ by the industry experts in 65% of cases, which is a reasonably high result. However, in 29% of cases, the response was incomplete or misleading, and in 6% of cases, the answer was entirely wrong.
This means that over one third of the questions posed to the AI bot were answered incorrectly or unsuitably.
These findings have sparked concerns from investment experts that a high proportion of people who use ChatGPT may ask for financial advice and unknowingly receive a poor response.
Pedro Braz, CEO of Investing in the Web, reflected on the findings of the research, offering the following conclusion:
“As with any other tool, ChatGPT serves its purpose if used in moderation. A good rule of thumb is to use it early in the research process, to help clarify terms or shed light on a general topic. Using it to ask more personalised questions is where the risk of miscommunication comes into play.
“It is always advisable to ask for and check the listed sources from ChatGPT’s answers, and to cross-reference its main points with other trusted sites, especially when dealing with financial inquiries that may have a significant impact on our lives.
“For information and data that is subject to change such as interest rates, the current performance of specific stocks, or the average pension, it is usually best to go to the very source of the information, rather than asking AI chatbots for data, as ChatGPT has well-recognised issues with up-to-date information.
“Overall, ChatGPT search is a welcome application that can save us a lot of time, but it can also misdirect us if we’re not careful while using it. Where the subject matter is as serious as personal finances, overuse of AI can have a significant negative impact.”