Datasheet for ASAP

A toolkit for actively pairing conditions in pairwise comparison preference aggregation.

Pairwise comparison data arise in many domains with subjective assessment experiments. In these experiments participants are asked to express a preference between two conditions. However, many pairwise comparison protocols require a large number of comparisons to infer accurate scores, which may be unfeasible when each comparison is time-consuming or expensive. To address this problem we propose ASAP, an active sampling algorithm, offering the highest accuracy of inferred scores compared to the existing methods. Unlike most existing methods, which rely on partial updates of the posterior distribution, we are able to perform full updates and, therefore, much improve the accuracy of the inferred scores.