An Improved Scale for the Novice-Expert Ratio Method

Abstract

The Novice-Expert ratio Method (NEM) compares the time it takes a novice user to complete each step of a task with the time it takes an expert user to complete the same step. A high ratio indicates that novice users are performing a step far below expert performance. This allows researchers to identify aspects of a user interface that impede its usability, because they are counterintuitive. However, NEM has three major shortcomings. First, the merits of a ratio scale have never been appraised relative to other possible scales, such as log, median, reciprocal, D, and Cohen’s d. Second, user error rates are not included in calculating the NEM ratio. Third, NEM does not specify how much time to allow for the completion of a step or what completion time to use for a step that was not completed. This study tested the validity of the original ratio scale of NEM and other candidate scales to ascertain whether another scale based on the objective usability measures of completion time and steps required might better represent the concept of usability. Scale validity was tested by comparing how well each scale correlates with other kinds of usability measures, including the ability to answer questions accurately and self-reported ratings of usability. A dataset was constructed from the amount of time and number of clicks it took participants to enter words on a counterintuitive cellular phone interface as implemented on a webpage. Cohen’s d measure correlated more highly with the number of clicks and completion time for novice-expert performance than the original NEM ratio.