Consider the following from pp. 254-256 of Sauro, J., & Lewis, J. R. (2016). Quantifying the User Experience: Practical Statistics for User Research, 2nd Ed. Cambridge, MA: Morgan-Kaufmann (you can look inside at https://www.amazon.com/Quantifying-User-Experience-Second-Statistics/dp/0128023082/).
DO YOU NEED TO TEST AT LEAST 30 USERS?
ON ONE HAND
Probably most of us who have taken an introductory statistics class (or know someone who took such a class) have heard the rule of thumb that to estimate or compare means, your sample size should be at least 30. According to the central limit theorem, as the sample size increases, the distribution of the mean becomes more and more normal, regardless of the normality of the underlying distribution. Some simulation studies have shown that for a wide variety of distributions (but not all—see Bradley, 1978), the distribution of the mean becomes near normal when n = 30.
Another consideration is that it is slightly simpler to use z-scores rather than t-scores because z-scores do not require the use of degrees of freedom. As shown in Table 9.1 and Fig. 9.2, by the time you have about 30 degrees of freedom the value of t gets pretty close to the value of z. Consequently, there can be a feeling that you don’t have to deal with small samples that require small-sample statistics (Cohen, 1990). ...
ON THE OTHER HAND
When the cost of a sample is expensive, as it typically is in many types of user research (e.g., moderated usability testing), it is important to estimate the needed sample size as accurately as possible, with the understanding that it is an estimate. The likelihood that 30 is exactly the right sample for a given set of circumstances is very low. As shown in our chapters on sample size estimation, a more appropriate approach is to take the formulas for computing the significance levels of a statistical test and, using algebra to solve for n, convert them to sample size estimation formulas. Those formulas then provide specific guidance on what you have to know or estimate for a given situation to estimate the required sample size.
The idea that even with the t-distribution (as opposed to the z-distribution) you need to have a sample size of at least 30 is inconsistent with the history of the development of the distribution. In 1899, William S. Gossett, a recent graduate of New College in Oxford with degrees in chemistry and mathematics, became one of the first scientists to join the Guinness brewery. “Compared with the giants of his day, he published very little, but his contribution is of critical importance. … The nature of the process of brewing, with its variability in temperature and ingredients, means that it is not possible to take large samples over a long run” (Cowles, 1989, p. 108–109).
This meant that Gossett could not use z-scores in his work—they just don’t work well with small samples. After analyzing the deficiencies of the z-distribution for statistical tests with small samples, he worked out the necessary adjustments as a function of degrees of freedom to produce his t tables, published under the pseudonym “Student” due to the policies of Guinness prohibiting publication by employees (Salsburg, 2001). In the work that led to the publication of the tables, Gossett performed an early version of Monte Carlo simulations (Stigler, 1999). He prepared 3000 cards labeled with physical measurements taken on criminals, shuffled them, then dealt them out into 750 groups of size 4—a sample size much smaller than 30.
This controversy is similar to the “five is enough” versus “eight is not enough” argument covered in Chapter 6, but applied to summative rather than formative research. For any research, the number of users to test depends on the purpose of the test and the type of data you plan to collect. The “magic number” 30 has some empirical rationale, but in our opinion, it’s very weak. As you can see from the numerous examples in this book that have sample sizes not equal to 30 (sometimes less, sometimes more), we do not hold this rule of thumb in very high regard. As described in our sample size chapter for summative research, the appropriate sample size for a study depends on the type of distribution, the expected variability of the data, the desired levels of confidence and power, and the minimum size of the effect that you need to be able to reliably detect.
As illustrated in Fig. 9.2, when using the t-distribution with very small samples (e.g., with degrees of freedom less than 5), the very large values of t compensate for small sample sizes with regard to the control of Type I errors (claiming a difference is significant when it really is not). With sample sizes these small, your confidence intervals will be much wider than what you would get with larger samples. But once you’re dealing with more than 5 degrees of freedom, there is very little absolute difference between the value of z and the value of t. From the perspective of the approach of t to z, there is very little gain past 10 degrees of freedom.
It isn’t much more complicated to use the t-distribution than the z-distribution (you just need to be sure to use the right value for the degrees of freedom), and the reason for the development of the t-distribution was to enable the analysis of small samples. This is just one of the less obvious ways in which usability practitioners benefit from the science and practice of beer brewing. Historians of statistics widely regard Gossett’s publication of Student’s t-test as a landmark event (Box, 1984; Cowles, 1989; Stigler, 1999). In a letter to Ronald A. Fisher (one of the fathers of modern statistics) containing an early copy of the t tables, Gossett wrote, “You are probably the only man who will ever use them” (Box, 1978). Gossett got a lot of things right, but he certainly got that wrong.
Box, G. E. P. (1984). The importance of practice in the development of statistics. Technometrics, 26(1), 1-8.
Box, J. F. (1978). Fisher, the life of a scientist. New York, NY: John Wiley.
Bradley, J. V. (1978). Robustness? British Journal of Mathematical and Statistical Psychology, 31, 144-152.
Cohen, J. (1990). Things I have learned (so far). American Psychologist, 45(12), 1304-1312.
Cowles, M. (1989). Statistics in psychology: An historical perspective. Hillsdale, NJ: Lawrence Erlbaum.
Salsburg, D. (2001). The lady tasting tea: How statistics revolutionized science in the twentieth century. New York, NY: W. H. Freeman.
Stigler, S. M. (1999). Statistics on the table: The history of statistical concepts and methods. Cambridge, MA: Harvard University Press.