Critical reflections on user studies’ evaluation methods for group recommender systems

Social choice-based aggregation strategies are often used in group recommender systems to aggregate individual preferences or recommendations. However, previous works evaluating group recommenders with user studies found that the diversity of the group members’ preferences impacts the effectiveness of the strategies. In this paper, we highlight and address the methodological limitations of those previous works. Specifically, the methodologies we introduce demonstrate the following three novelties: 1) We evaluated the strategies from the viewpoint of an “internal evaluator”; 2) We introduced a novel methodology for modeling a fictional but realistic group with specific preference profiles for the group members, defining scenarios with concrete users and items, that are still mapped to specific group configurations; 3) We evaluated the understanding of the participants, by measuring how well they can successfully apply the aggregation strategy to a new scenario. To do this we performed a randomized controlled trial (n=444) using a mixed design with two between-subject factors (the used aggregation strategy and the presented explanation type), and a within-subject factor (the group configuration). Our results, with friend groups, showed significant differences in the effectiveness of the aggregation strategies depending on the specific group configuration (i.e., depending on the internal diversity of group members’ preferences), with noticeable differences between evaluations of what is good for the group – external evaluation – and what is good for the participant – internal evaluation. We conclude with methodological implications for group recommender systems.