The KIMERA Infrastructure: Shifting from Evaluation-as-a-Service to Evaluation-in-the-Cloud

Experimental evaluation plays a key role in Information Retrieval (IR), and Evaluation-as-a-Service (EaaS) was proposed as a viable approach for efficiently running experiments without distributing experimental collections. We now introduce Kubernetes Infrastructure for Managed Evaluation and Resource Access (KIMERA), a cloud-based platform implemented with Kubernetes that advances EaaS toward Evaluation-in-the-Cloud (EitC), enabling researchers to develop and run IR systems directly through a web interface. KIMERA ensures scalability, reproducibility, and fairness across experiments, and it can integrate easy access to external services such as Large Language Models and Quantum Computing via APIs. It supports detailed resource tracking for a comprehensive evaluation of effectiveness and efficiency.