Augmented Intelligence in Technology-Assisted Review Systems (ALTARS 2022): Evaluation Metrics and Protocols for eDiscovery and Systematic Review Systems

In this workshop, we aim to fathom the effectiveness of Technology-Assisted Review Systems from different viewpoints. In fact, despite the number of evaluation measures at our disposal to assess the effectiveness of a "traditional" retrieval approach, there are additional dimensions of evaluation for these systems. For example, it is true that an effective high-recall system should be able to find the majority of relevant documents using the least number of assessments. However, this kind of evaluation usually discards the resources used to achieve this goal, such as the total time spent on those assessments, or the amount of money spent for the experts judging the documents.