Calibration of a 2026-generation large language model on short-horizon sports events with concurrent market prices

Engineering notes

Engineering notes will be added by the aipentium editorial team.

Chinese explanation / 中文解读

中文解读待补充：本站会优先为大语言模型、生成式AI、ChatGPT相关技术、计算机视觉、深度学习等高价值论文补充中文说明。

Original abstract

Large language models are widely reported to be poorly calibrated when asked to state probabilities. Nearly all published evidence comes from models of the 2024-2025 generation, evaluated in single-turn chat on long-horizon questions that resolve in months or years. This study measures calibration of a 2026-generation model (Claude Opus 5 and Claude Sonnet 5) on a question class that is the opposite in every relevant respect: short-horizon, high-volume, and priced by a live market. Tennis matches resolve within hours, several thousand occur per year, and each carries a set of derivative statements (set winner, total games, set handicap) that resolve from the same match. A prediction-market or exchange price exists alongside a subset of them, providing an external benchmark that is unavailable in most forecasting domains. The study is prospective and forward-only. No prediction is made about an event that has already occurred, and no historical backtest of the model arms is performed, because matches predating the model knowledge cutoff may be present in its training data. The design separates two questions that are routinely conflated: whether the system is calibrated, and whether it beats the market. These require different sample sizes and are analysed on different subsets, specified in the sampling and analysis sections. No money is staked on any prediction at any point in this study, and no forecast is published as advice.

Links and sources

Comments