Reality Check – Conducting Real World Studies: Frontiers of Information Access Experimentation fortextlessbr/textgreaterResearch and Education

Bruce Ferwerda, Allan Hanbury, Bart P. Knijnenburg, Birger Larsen, Lien Michiels, Andrea Papenmeier, Alan Said, Philipp Schaer, Martijn Willemsen. 2023, "Reality Check – Conducting Real World Studies: Frontiers of Information Access Experimentation fortextlessbr/textgreaterResearch and Education". Frontiers of Information Access Experimentation for Research and Education.

Abstract

Information retrieval and recommender systems are deployed in real world environments. Therefore, to get a real feeling for the system, we should study their characteristics in “real world studies”. This raises the question: What does it mean for a study to be realistic? Does it mean the user has to be a real user of the system or can anyone participate in a study of the system? Does it mean the system needs to be perceived as realistic by the user? Does it mean the manipulations need to be perceived as realistic by the user?