ResQue

Abstract

This research was motivated by our interest in understanding the criteria for measuring the success of a recommender system from users’ point view. Even though existing work has suggested a wide range of criteria, the consistency and validity of the combined criteria have not been tested. In this paper, we describe a unifying evaluation framework, called ResQue (Recommender systems’ Quality of user experience), which aimed at measuring the qualities of the recommended items, the system’s usability, usefulness, interface and interaction qualities, users’ satisfaction with the systems, and the influence of these qualities on users’ behavioral intentions, including their intention to purchase the products recommended to them and return to the system. We also show the results of applying psychometric methods to validate the combined criteria using data collected from a large user survey. The outcomes of the validation are able to 1) support the consistency, validity and reliability of the selected criteria; and 2) explain the quality of user experience and the key determinants motivating users to adopt the recommender technology. The final model consists of thirty two questions and fifteen constructs, defining the essential qualities of an effective and satisfying recommender system, as well as providing practitioners and scholars with a cost-effective way to evaluate the success of a recommender system and identify important areas in which to invest development resources.

Keywords

Recommender systems, quality of user experience, e-Commerce recommender, post-study questionnaire.

Year

2010 – 2011

Researchers

Rong Hu, Li Chen and Pearl Pu

Sponsor

Swiss National Science Foundation

Publications

  • Pearl Pu, Li Chen and Rong Hu. A User-Centric Evaluation Framework for Recommender Systems. In Proceedings of the 5th ACM Conference on Recommender Systems (RecSys’11), pages 157 – 164, Chicago, IL, USA, October 23 – 27, 2011. 
  • Pearl Pu and Li Chen. A User-Centric Evaluation Framework of Recommender Systems. In the 3rd ACM Conference on Recommender Systems (RecSys’10), Workshop on User-Centric Evaluation of Recommender Systems and Their Interfaces (UCERSTI’10), Barcelona, Spain, Sept. 26-30, 2010.

Introduction of ResQue questionnaire

Recommender systems’ Quality of User Experience (ResQue) questionnaire is an evaluation framework developed by a team of researchers in the Human-Computer Interaction Group at École Polytechnique Fédérale de Lausanne (EPFL). It is designed to assess the perceived qualities of recommenders such as their usability, usefulness, interface and interaction qualities, users’ satisfaction of the systems, and the influence of these qualities on users’ behavioral intentions, including their intention to purchase the products recommended to them, return to the system in the future, and tell their friend about the system. This model thus identifies the essential qualities of an effective and satisfying recommender system and the essential determinants that motivate users to adopt this technology. 

The ResQue Questionnaire has been validated as a well-balanced user-centric evaluation framework for recommender systems. It defines the essential qualities of an effective and satisfying recommender system. Furthermore, it provides practitioners and scholars with a cost-effective way to evaluate the success of a recommender system and identify important areas in which to invest development resources. The questionnaires can be applied to assess different types of recommenders, regardless of the backend engines used. 

The current version contains a demographic questionnaire, a full version and a short version of the ResQueQuestionnaire. Each ResQue question is answered on a 5-point Likert-scale from 1 (strongly disagree) to 5 (strongly agree). One option “Not Applicable” is designed for the case where that question cannot be applied for the evaluated system. 

In addition to English, the ResQue Questionnaire is currently available in French. 

The ResQue Questionnaire is developed by the HCI Group of EPFL. It is free for the purpose of academic research. Any commercial usage, please contact us for permission. 

Offers

  1. Questionnaire Lists. We provide both a full version and a short version of ResQue Questionnaire with demographic questions. Both of them can be downloaded in word and pdf format. All questions can be edited to meet users’ specific needs. 
  2. Inventory. We provides a list of questions initially developed for Resque. 
  3. References. A selection of relevant papers detailing the development and validation of the ResQue model. 
  4. Online Survey. It is a live survey for our studies. Users are also welcomed to use this survey system to evaluate their recommenders. If you need more supports and information (e.g., obtain your survey data, personalize the survey questions, analyse users’ responses etc.), please contact us.

Question Examples

1.1.1 The items recommended to me matched my interests.
__Strongly Agree    __Agree    __Neither agree nor disagree    __Disagree    __Strongly Disagree    __Not Applicable 

1.1.2 This recommender system gave me good suggestions.
__Strongly Agree    __Agree    __Neither agree nor disagree    __Disagree    __Strongly Disagree    __Not Applicable 

7.1 Overall, I am satisfied with this recommender system.
__Strongly Agree    __Agree    __Neither agree nor disagree    __Disagree    __Strongly Disagree    __Not Applicable 

8.2.1 I will use this recommender again.
__Strongly Agree    __Agree    __Neither agree nor disagree    __Disagree    __Strongly Disagree    __Not Applicable 

8.2.2 I will use this recommender frequently.
__Strongly Agree    __Agree    __Neither agree nor disagree    __Disagree    __Strongly Disagree    __Not Applicable 

Survey

Licensing

The ResQue Questionnaire is developed by the HCI Group of EPFL. It is free for the purpose of academic research. Any commercial usage, please contact us for permission.