# Small n: non-parametric or parametric tests?

I am in a situation where I have sample with $n=19$, and the the data are not normally distributed (according to the results of a Kolmogorov-Smirnov test) for all the variables. The data were collected using Likert scales in a questionnaire. So now I am unsure whether or not I can use a parametric test or whether I need to use non-parametric tests.

I have found books stating that if you have a small $n$, you should always use non-parametric tests. However I have also found citations stating that the choice between parametric and non-parametric tests depends on the level of your data (Likert can be seen as nominal), so I should use parametric tests.

• It is worth keeping in mind that non-parametric tests also make some assumptions regarding the shape of the distribution. – chl Nov 2 '12 at 17:14

The data were collected using Likert scales in a questionnaire.

From the moment you decided to use Likert scales, the issue of whether or not the data were actually from a normal distribution was decided (in the negative). It's pointless to test (and answer with chance of error) a question to which you already know the answer with certainty (a large enough sample would always lead to rejection by a suitable test, but you can tell it is the case with no data at all). Your data are not from a normal distribution normal; that was already certain.

[However, it's also not a useful question to answer; a better question to answer is not 'are the data from a normal distribution?' (are they ever?) but 'how much impact does it have on my inference?', a question not answered by hypothesis tests.]

So now I am unsure whether or not I can use a parametric test

That depends on the parametric test. Parametric doesn't necessarily imply "normal"; you may be able to make some other distributional assumption that will be consistent enough with your situation that you would be content with the impact of whatever deviation from assumptions you have.

or whether I need to use non-parametric tests.

Beware - nonparametric tests also have assumptions, and in some cases may be somewhat sensitive to them.

Many nonparametric tests assume continuous data, for example, and if you don't account for heavy discreteness you may get tests with quite different properties from their nominal ones. Some assume symmetry. In addition, suitability of particular tests may depend on the precise hypothesis you're interested in - you may need some additional assumptions (or perhaps a somewhat different nonparametric procedure) to get a test of your actual hypothesis.

I have found books stating that if you have a small n, you should always use non-parametric tests.

For very small $n$, that's not necessarily useful advice, since you may have no useful significance levels available to you. At larger (but still small) $n$, in cases where the assumptions of a suitable nonparametric procedure are tenable, it sometimes makes sense to avoid making parametric assumptions to which your inferences may be sensitive (though there's sometimes the possibility of choosing different, less sensitive procedures).

However I have also found citations

Which citations? What did they actually say?

stating that the choice between parametric and non-parametric tests depends on the level of your data (Likert can be seen as nominal)

No. A single item intended as part of a Likert scale is at least ordinal. If you have constructed a Likert scale by adding a number of such questions you have already assumed it was interval - by assuming things like '5'+'2' = '4'+'3' (which must be the case if you're able to add the scores and treat every '7' as the same), every component item had to have been interval. If they're interval, their sum certainly is.

so I should use parametric tests.

I don't see how "use a parametric test" follows from that.

You say very little about what kind of hypotheses you have (what are you trying to find out?); more might be said in those circumstances.

The sample size is not an issue here. You can only use nonparametric procedures (depending on the particular question Wilcoxon test, rank correlation, Kruskal-Wallis test or others) with Likert scale data due to their ordinal scale.

Parametric procedures use the spaceing between different levels. But on a Likert scale, this spaceing is not informative: You could code five levels by "1,2,3,4,5" or you could code them without changeing information by "1,10,20,34,100" or by "10,Jack,Queen,King,Ace". So parametric tests that would include this arbitrary information do not yield reproducible results (and in the latter case none at all).

Sometimes however, the sample size is a criterion to decide between parametric and nonparametric tests. Namely, if you are unsure if the assumptions for one or another parametric test hold, but you have a large sample size, you can cross fingers that by the central limit theorem, the mean is approximately normally distributed. For small sample sizes, this approximation is rather not so good (e.g. if your data are exponentially distributed, you need really huge sample sizes for the t-test to be usable), so you can only rely on nonparametric tests. This way you would sacrifice the information about the spaceings between the observations and keep only their ranks. But this is better than a parametric test that is liberal because you unintentionally chose the wrong assumptions.

• This advice seems too narrow. A parametric two sample t test is an approximately valid test, even with sample sample sizes, of the null hypothesis that the data in the two groups come from identical distributions. The "spacing" issue does not invalidate the test as the nominal levels are approximately correct under the null model. Rank-based tests can also be used, but they have the same null hypothesis (equal distributions in the Likert scale case), they are not necessarily better in terms of achieved type I error rate, and they are more often less powerful for Likert scale data. – Peter Westfall Oct 27 '18 at 13:21
• You are roughly right about the error rates, but the "spacing issue" concerns the way how to interpret the single result. It is the equivariance criterion Casella and Berger (2002) mention. Likert scale data are ordinal. Changing the measurement from 1,2,3,4,5 to something else representing the ordinal relation "<" does not change the fact the data show, so meaningful inference must not rely on this arbitrary choice of symbolic representation. The 1 in the Likert scale is not a number, it's just a symbol. You can't calculate means out of if, let alone t-tests. – Horst Grünbusch Feb 6 '19 at 14:02
• Sure you can use the t-test. The t-test is just one way to test the hypothesis that the distributions are equal, and it is perfectly valid if you interpret it that way. Of course there are other tests. Which is more powerful just depends on the distribution; easy to find out via simulation. – Peter Westfall Feb 6 '19 at 19:05
• Oh no, the t-test doesn't test the "hypothesis that the distributions are equal". It tests if the population means are equal. Variances and their differences are just nuisance parameters. Please have a look at the book I referred to or at least any other book on applied statistics. – Horst Grünbusch Feb 7 '19 at 10:05
• Ok, here are the facts, found anywhere. It's just math; no arguments are possible. Assume iid data, with finite variance. Break the data into two groups, form the t-stat. The asymptotic distribution of the statistic is N(0,1), implying asymptotic level-$\alpha$ validity of the t-test. If the distributions have different means, then the power tends to 1.0. As far as the "spacing" issue goes, as long as there is a shift in the distribution, you get a different mean in the Likert responses, implying validity of the test. So the spacing issue is not relevant in that regard. – Peter Westfall Feb 8 '19 at 13:05