Publication: Critic-Proofing: Robust Validation Through Data-Mining

Critic-proofing is a modified heuristic evaluation technique, specifically designed to provide a fine-grained, prioritized list of heuristic violations. The critic-proofing technique weights the severity of a problem based on the frequency that similar problems are found in similar games. The severity ratings are calculated using data collected from game reviews, and the severity assigned to a problem during the heuristic evaluation process. However, heuristic techniques have had limited adoption within the video game industry. One reason for this is the perceived lack of validity and robustness of game specific heuristic principles. In this paper, we introduce and outline a new data-mining project designed to validate game-specific heuristic techniques, especially the critic-proofing technique by using the popular game-review aggregation website Metacritic.




Ian Livingston
Electronic Arts
Lennart Nacke
University of Ontario Institute of Technology, University of Saskatchewan
Regan Mandryk
University of Saskatchewan


Game Heuristics
Video gaming is a common and popular entertainment. The primary goal of video games is to entertain and engage users.


Livingston, I.J., Nacke, L.E., Mandryk, R.L. 2010. Critic-Proofing: Robust Validation Through Data-Mining. In Playability and player experience: Proceedings of the Fun and Games 2010 Workshop, NHTV Expertise Series 10. Leuven, Belgium. 81-94. Workshop paper.


@inproceedings {200-Metacritic-FinalPaper,
author= {Ian Livingston and Lennart Nacke and Regan Mandryk},
title= {Critic-Proofing: Robust Validation Through Data-Mining},
booktitle= {Playability and player experience: Proceedings of the Fun and Games 2010 Workshop},
year= {2010},
series= {NHTV Expertise Series 10},
address= {Leuven, Belgium},
pages= {81-94},
note= {Workshop paper}