Forecast-Dojo: Replayable Environments for Benchmarking and Training LLM Forecasting Agents

arXiv Computer Science AI published Forecast-Dojo: Replayable Environments for Benchmarking and Training LLM Forecasting Agents. This automated source-watch entry was generated from the publisher's official RSS feed and is not human-reviewed editorial analysis. Source excerpt: arXiv:2609.28876v1 Announce Type: new Abstract: We introduce Forecast-Dojo, a replayable environment for benchmarking and training LLM forecasting agents. It combines resolved prediction-market questions with dated news, allowing agents to research an event and revisit their predictions at successive historical dates. The same tasks and tools support repeated evaluation, collection of training interactions, and feedback from recorded outcomes without waiting for new events to resolve. Forecast-Dojo contains 1,568 Polymarket events, split by time into training and evaluation periods, and 18.8M dated news articles. In an evaluation of 12 models, research tools lower Brier score for all 12. Forecasts also improve as events unfold, with the largest gains at steps where more newly dated evidence is recorded. Every model still trails historical market forecasts in both Brier score and…

• Read the original arXiv Computer Science AI article before relying on this summary.
• Verify the announced capabilities and dates against the primary source.
• Assess whether the change affects your agent stack or evaluation plan.