AI can’t write good analyst research yet, says analyst

{"text":[[{"start":null,"text":""}],[{"start":7.03,"text":"Maybe one day, sell-side research departments will look like the interior of the Discovery One in Stanley Kubrick’s 2001: A Space Odyssey: no desks, white and minimalist. In it will sit a HAL-like server offering, on command, reports on the three-year financial prospects for XYZ.com."}],[{"start":28.82,"text":"FTAV and others have hashed over the possibility that AI can replace hard-working City analysts (also this on coders). Mostly, the tone has been negative for their prospects, never mind for financial journalists."}],[{"start":44.26,"text":"But how about when you make the AI models analyse a company or sector, put together a predictive financial model and write a research note? Bernstein Société Générale tested this. The AI models started well, but then everything got a bit messy."}],[{"start":61.16,"text":"Bernstein’s team led by Venugopal Garre, head of India research, first had to choose which models to use:"}],[{"start":null,"text":"

We went through hordes of AI tools out there and picked up the most widely used ones, along with a few lesser known. Google’s Gemini, Grok and ChatGPT were the usual candidates, and we added Perplexity, Microsoft’s Copilot, Claude, Meta AI, DeepSeek and a few others (including vertical LLMs tailored for finance) to it.

"}],[{"start":68.75999999999999,"text":"Using various tests, Garre aimed to mimic the thought process of an equities analyst then grade them by their humanlike qualities. Could AI not only extract data from publicly available data, including earnings call transcripts, but also synthesise everything and make judgments? The team wanted to see if ChatGPT, Gemini or any of the others could build a financial model to help predict outcomes and then write an initiation report on a company."}],[{"start":99.99999999999999,"text":"Next, Garre created a number of tests, basic and advanced ones, beginning with some seek-and-return tasks without feeding the AIs any information."}],[{"start":111.82999999999998,"text":"At this stage, when the model extracted publicly available data for presentation, everything went pretty well. While there were some hiccups with miscategorisations, which created inconsistencies across the AI responses, in general he found that the AI models could do a good job generating graphs of financial data."}],[{"start":134.10999999999999,"text":"For instance, Grok created an attractive interactive graph with dual axes of an Indian company, Dixon Technologies."}],[{"start":null,"text":""}],[{"start":141.79,"text":"What large language models can do well is find useful stuff in copious amounts of text, and even divine a change in tone about any subject over time. After uploading three years of quarterly earnings transcripts for specific companies, the AI tool was tasked with listing out any investor concerns and, in a separate exercise, rating how well management had addressed these concerns. Mostly, they handled this well. When asked to assess the quality of management by how confidently they answered questions, Gemini “stood out”."}],[{"start":177.63,"text":"After this, everything went a bit sour. Making pretty pictures and assessing the tone of earnings calls only make up a small portion of an analyst’s job. Using lots of data, plus one’s experience, to create long-term industry forecasts helps the analyst produce vital financial models for forecasting purposes."}],[{"start":201.53,"text":"These types of prompts proved too much:"}],[{"start":null,"text":"

Initiate on stock xyz as a sell-side analyst stating your view (buy, hold, sell) and reasons for the same. Give EPS forecasts, your target price and the calculation behind the same. (earnings call transcripts and financials provided, information about the sector of company provided)

Given the financials and split for a company in the sector ABC, come up with a basic model listing out the drivers which can be changed to forecast earnings for the next two years. (company financials for last ten years provided)

"}],[{"start":205.71,"text":"Despite feeding in the relevant data plus repeated, refined prompts, the models returned false information and error-strewn spreadsheets. “On modelling, [AI] absolutely failed,” Garre told me. “There are too many accounting nuances and differences from country to country.” Humans understand all this but computers require lots of learning to understand these subtleties."}],[{"start":232.04000000000002,"text":"Most of the AI tools couldn’t create a model at all. With a lot of coaxing, Gemini offered some Python code to make a financial model, but it still didn’t work due to errors. For those that did manage the feat, Garre said that these lacked much if any predictive power."}],[{"start":null,"text":""}],[{"start":249.67000000000002,"text":"In the end, no matter how much data and prompting Garre provided, none of the ten-plus models could properly analyse the outlook for companies. The company initiation reports lacked sufficient depth."}],[{"start":264.51,"text":"Nor could the AIs properly assess the outcome of management actions, such as creating a joint venture with a Chinese company with all its geopolitical considerations."}],[{"start":275.95,"text":"The overall average score for the group was poor. AI optimists will intone their mantra that these models will only get better. Realists will say that AI, like Excel, can only boost productivity and that’s enough to make a difference."}],[{"start":293.09,"text":"Sell-side analysts — which to be fair Garre wants to stay in the room — will take some comfort."}],[{"start":307.2,"text":""}]],"url":"https://audio.ftmailbox.cn/album/a_1757685438_3700.mp3"}