(The Eurographics Association, 2025) Jobst, Adrian; Atzberger, Daniel; Scheibel, Willy; Döllner, Jürgen; Diehl, Alexandra; Kucher, Kostiantyn; Médoc, Nicolas
Large Language Models (LLMs) are increasingly integrated into Natural Language Interfaces (NLIs) for visualizations, enabling users to inquire about visualizations through natural language. This work introduces a software framework for evaluating LLMs' visualization literacy, i.e., their ability to interpret and answer questions about visualizations. Our framework generates a set of data points across different LLMs, prompts, and question types, allowing for in-depth analysis. We demonstrate its utility by two experiments, examining the impact of the temperature parameter and predefined answer choices.