Artificial intelligence has become an increasingly common part of the modern world, with the nascent technology beginning to be deployed across a range of professions and industries.
The trend has fueled a surge in enthusiasm around the potential applications of AI and powered a spike in the valutions of big-name firms like semiconductor company Nvidia (NASDAQ:NVDA) and software giant Microsoft (NASDAQ:MSFT) who are viewed as frontrunners in an AI arms race. These gains have, in turn, underpinned the broader stock market even during recent times of wider economic uncertainty.
But doubts remain over the effectiveness of AI and -- potentially -- its ability to replace tasks traditionally performed by humans.
In a note, analysts at Bernstein aimed to explore this question by challenging AI models to assess reams of data and deliver professional-level financial analysis.
Their test involved both horizontal -- or general-purpose -- large-language models like OpenAI’s ChatGPT or xAI’s Grok and more specific "vertical" LLMs. It was also split into two phases, one focused on foundational skills like chart-making and data extraction, and the other examining whether a model can formulate its own opinions and make reliable judgements of management decisions.
The first phase found that many AI platforms had issues with terminology changes, such as substituting "personnel costs" in for "employee benefits expense." Although they said the models’ chart-making prowess was "robust," data reliability remained a key issue, with only a couple able to fetch figures with reasonable accuracy.
In phase two, the models demonstrated "remarkable proficiency" in more cognitively complex tasks, like extracting key investor concerns from management calls or crafting a timeline of problems at a company. Some models were even able to effectively perform tone analysis on earnings calls, "pointing out where the management appeared less confident or evasive," they said.
"The results were remarkable, and given the turnaround time for this vast data, dare we say surpassing humans?" the analysts wrote.
However, they said the "euphoria was short-lived," as this high performance faltered "dramatically" once they introduced the "element of higher cognition in the questions."
"Given the vast amount of information we fed to the platforms, we’d expect a human to be fairly good at judging industry trends, but the AI platforms were not effective in connecting seemingly unlinked macro factors to forecast industry patterns, unless prompted explicitly," they said. AI, as a whole, is not a "natural skeptic," they added, and is a "cognitive tool at best."
"Its effectiveness? As good as the human using it," the Bernstein analysts said.
Source :
https://www.investing.com/news/stock-market-news/could-ai-models-do-the-same-tasks-as-financial-analysts-bernstein-investigates-4198013
