Evaluating Local Language Models: An Application to Bank Earnings Calls

Cook, Thomas R.; Kazinnik, Sophia; Lundgaard Hansen, Anne; McAdam, Peter

Working Paper

Evaluating Local Language Models: An Application to Bank Earnings Calls

Abstract: This study evaluates the performance of local large language models (LLMs) in interpreting financial texts, compared with closed-source, cloud-based models. We first introduce new benchmarking tasks for assessing LLM performance in analyzing financial and economic texts and explore the refinements needed to improve its performance. Our benchmarking results suggest local LLMs are a viable tool for general natural language processing analysis of these texts. We then leverage local LLMs to analyze the tone and substance of bank earnings calls in the post-pandemic era, including calls conducted during the banking stress of early 2023. We analyze remarks in bank earnings calls in terms of topics discussed, overall sentiment, temporal orientation, and vagueness. We find that after the banking stress in early 2023, banks tended to converge to a similar set of topics for discussion and to espouse a distinctly less positive sentiment.

Keywords: data; large language models; quantitative methods; banking and finance;

JEL Classification: C45; G21;