Tally

Many labs around the world conduct research that relies heavily on perception experiments with video. Standards for operating procedures, such as the ITU recommendations, exist for these kinds of experiments. However, the current tools for data collection for subjective testing are outdated or insufficient, and efforts to improve collection methods are too sparse and fractured to be widely effective.

To address this need, we have developed Tally, a web- based data collection tool for subjective video experiments. Our tool has numerous advantages, most notably the decoupled voting and viewing interface, flexibility and robustness, and the ability to share and collaborate on projects. We believe our web-based design is not only superior to previous attempts at such a tool, but is also the proper way of addressing the data collection problem.