The effects of running fewer subjects on time‐on‐task measures

There is a desire to streamline the development design process by conducting usability evaluations with fewer subjects. This study examined the impact of using fewer subjects on the accuracy and stability of the mean and median. Monte Carlo simulations were conducted to model the skewed task‐completion times typically found in usability studies. The results showed that the mean was a more accurate estimate of its respective population parameter than the median and should be the preferred metric in usability evaluations. However, the mean cannot always be used because it will underestimate the population mean when there are missing data as a result of subjects exceeding time limits. Alternatively, it was found that the median tended to consistently overestimate the population median by as much as 10% in one examined case. The average 95% confidence limits of both estimates were quite broad, particularly for the smaller sample sizes. Based on these results, it was concluded that with reduced sample sizes (e.g., 5), we are not able to measure population parameters with either a high degree of confidence or, in the case of medians, with any accuracy. It was recommended that usability measurements take a back seat to usability testing when conducting tests with small sample sizes.