This question arose from reading this post: T-test for non normal when N>50?

In the response to this post, the author outlines really well that the assumption of normality with regard to the t-test is about the sampling distribution of the means and not the population distribution.

However, parametric statistics are using sample data to draw inference about population parameters like the mean, or the difference in means between two populations. If the population distribution is not normal, it means that the population mean is a poor representation of the central tendency of the population. So even though the test assumptions itself of say a t-test are not violated when data are not normal and the sample size is large enough, is the inference gained from the test informative when the population is not normal and is thus poorly described by the mean and variance?

  • $\begingroup$ I think the reference post answers your question. 50 is not large enough for some distributions. There are other choices. Frank Harrell points out that the Wilcoxon test is relatively efficient. So that is one option and bootstrap is also available. If the population distribution is close to normal is it close enough to make the distribution of the test statistic close to the t distribution? You are talking too generally in your question. In the case of a two sample t test you need to consider whether or not the variances are equal. $\endgroup$ Dec 24 '16 at 19:24
  • $\begingroup$ When the population variances exist they take longer to converge which also affect sample size considerations. Take the one sample case with a Cauchy distribution even though the you can create a t statistic it is pretty meaningless. Both the population mean and the population variance are nonexistent. $\endgroup$ Dec 24 '16 at 19:29
  • $\begingroup$ Also technically if the data are non-normal the assumptions of the t test are violated. Look at the comments and answers about assumption in the refrenced post. $\endgroup$ Dec 24 '16 at 19:31
  • $\begingroup$ I think you have misunderstood my question. I'm not talking about sample size at all really. I mean that if we are conducting inference about the population mean using sample data (which we are in parametric statistics) but the population is not normally distributed, then the mean which we are conducting inference on is uninformative as it does not represent the central tendency of the population. I am wondering whether this affects our conclusions. $\endgroup$
    – jimbo
    Dec 24 '16 at 22:19
  • $\begingroup$ It is not necessarily true that the sample mean is a poor measure of central tendency when the population density is non normal. It depends on what distribution you are talking about. For a uniform distribution on any interval [a,b] where a<b the mean equals the median and is a a reasonable measure of central tendency. A triangular distribution on [0,1] with density f(x)=2x for 0<=x<=1/2. and f(x)=2-2x for 1/2<x<=1. This density has an isosceles triangle shape and the mode =mean =median. $\endgroup$ Dec 25 '16 at 1:39

I need to correct a number of mistaken or partly misplaced ideas in the question first (as well as some that aren't in the question but are commonly seen and may be indirectly influencing the way you ask your question), but I will return to the main issue.

The answer you link to says:

The t-test assumes that the means of the different samples are normally distributed; it does not assume that the population is normally distributed.

As one of the comments below the answer points out, this claim is incorrect.

The derivation that the t-statistic has a t-distribution relies on the distribution from which the observations were drawn being normal, not just the means. Note that the t-statistic is not just a numerator, but a ratio of two things.

You need the denominator in the t-statistic to be (i) distributed as scaled chi and (ii) independent of the numerator.

There's no result I'm aware of that makes the t-statistic have a t-distribution in any other general circumstance of interest.

In addition, while you can use the CLT plus Slutsky's theorem to argue that a t-statistic will asymptotically approach a standard normal, (so that as sample sizes go to infinity you approach a level-$α$ test) there's nothing that makes it approach the $t$ (e.g. at least in some situations, it might always be better approximated by the normal than it is by the t, all along the way as $n$ increases).

There's also nothing that says that such a test will have good power (i.e. that would make us care to use it). In fact asymptotically the power may be quite poor compared to alternatives (indeed the ARE of the t-test can go to zero).

[One of the comments under that answer points this error out but it has not been addressed by the person who answered.]

That said, however, often the t-test does work at least moderately well; but that's casual observation and not in any sense a theorem. It would not do to be overly reliant on it without understanding something of the circumstances with which one was dealing.

parametric statistics are using sample data to draw inference about population parameters like the mean, or the difference in means between two populations.

I recognize that you're not saying that parametric statistics is only about means there, but I want to take a moment to deal with a not-uncommon idea that they do.

Some parametric methods do focus on the mean but parametric methods are not in general about means. (A lot of books that don't go into the theory properly manage to get this completely wrong. The term "parametric" is not about normality and it is not about means.)

If the population distribution is not normal, it means that the population mean is a poor representation of the central tendency of the population.

One can care about a population mean without remotely caring about whatever "central tendency" might be.

Consider the exponential distribution.

This distribution is commonly used to model a number of things. A typical example is inter-event times (indeed you can derive it as the inter-event time in a Poisson process). Let's imagine we're doing just that (using it as a model for inter-event times). It looks like this:

plot of exponential density

Here the labels on the x-axis represent multiples of the population mean (since it's in units of the mean, the population mean is at 1).

I'm not even sure we can say that the exponential distribution has a central tendency -- but it makes perfect sense to use its population mean to describe the process, it's still the average inter-event time. It's not the typical inter-event time for a single event, but if I am repeatedly experiencing this process that mean is very much of interest.

is the inference gained from the test informative when the population is not normal and is thus poorly described by the mean and variance?

Outside some very particular situations, this is not the best way to think about it. It doesn't really matter whether the particular population parameter of interest is typical of the distribution (in whatever sense that 'typical' is intended).

It's easily possible for a population mean to be of interest even when the mean is not a 'typical' value (e.g. when the mean and the mode are quite different). Whether a particular population parameter is of interest (might be suitable for some hypothesis) is not so much a question of what the distribution looks like, but of what the question is that you want to answer.

For example, if I have to wait for 100 inter-event times in the above example, the population mean on the individual times is an important thing to be thinking about, even though most of the time I'll wait much less than the mean.

We can also construct tests based on sample means without assuming normality (this is an unrelated issue to the above one, but you seem as if you might be partly conflating the two issues, so I better discuss it). In the above exponential model we have a different parametric assumption, and we can base a suitable test on that assumption (and it can sensibly be based on the sample mean, though it's not quite optimal choice when comparing two small samples). Or if we don't have a parametric model, we can nevertheless construct a test that doesn't make a parametric assumption but still deals with (say) a difference in population means (such as a permutation test of the difference in sample means).

However, in the other hand, just because I may want to test something in relation to a population mean, that does not necessarily imply that the sample mean is a good way to construct a test of it. If I were interested in the difference in population means for two samples from logistic distributions, I would be better to use something other than the sample mean as the basis of the test (indeed, a Wilcoxon-Mann-Whitney test -- used as a test of a shift in population mean -- would be an excellent choice in that case).

In summary: let us begin to summarize that by passing back to the title question.

Is the inference from a parametric test valid when the population distribution is not normal?

It certainly can be. The consideration of validity of a parametric test is not related to whether or not we're dealing with normality.

  • the mean can be a perfectly reasonable population quantity to make hypotheses about, even when the mean is not at all close to the median or the mode.

  • a parametric test needn't be about the mean. It might be about the population minimum, for example, or the population median, or the population interquartile range or any number of other things. Nor is a parametric test necessarily related to normality at all (like a test for the upper limit of a uniform distribution or the shift parameter in a shifted exponential)

  • we must be careful not to conflate considerations of what population quantities we're interested in with what sample quantities we use for making inferences about them. Sometimes it's good to use the sample equivalent to estimate the population quantity and sometimes it's not -- and it's not always intuitive. With a (right skewed) exponential distribution, using the sample mean in inference about the population mean works well. With a symmetric, bell-shaped, not very heavy-tailed distribution like a logistic (and one you're likely to think looks pretty normal-ish in a histogram), not so much. And if it's Laplace even less so.

When looking at what to test and how to test it, I think it boils down to answering two main questions --

  1. What question about the population are you interested in answering? (This consideration needn't necessarily relate much to distribution shapes.)

  2. Given what you know/assume (or don't know/won't assume) about the distribution, what's a good way to answer that question? (it's here that questions of efficiency and robustness of tests come in, and where possible distribution shapes may be a central consideration)

  • $\begingroup$ Nice and careful answer. Thanks for sharing. I'd like to hear a bit more about the definition of parametric statistics. I see you consider that "parametric statistics use sample data to draw inference about population parameters", but could you clarify and differentiate parametric inference from nonparametric inference? Papers and books are valuable. Thanks. $\endgroup$
    – Luis
    Mar 29 '20 at 3:34
  • 1
    $\begingroup$ Parametric statistics focus on getting the best use of the data given the specific parametric assumption (eg. for distributional models: most efficient parameter estimates, most powerful tests, shortest confidence intervals). Nonparametric statistics tend to focus on trying to maintain some of their properties irrespective of a specific parametric situation (e.g. in relation to distributional assumptions, maintaining significance level in a test, coverage in a confidence interval), avoiding the necessity of conditioning on the truth of a model which is sure not to be quite correct $\endgroup$
    – Glen_b
    Mar 29 '20 at 4:07
  • 1
    $\begingroup$ @Luis Note that a nonparametric test can still test a hypothesis about specific population parameter(s) -- such as testing equality of two population means, which might, for example, be done via a permutation test. $\endgroup$
    – Glen_b
    Mar 29 '20 at 4:31
  • 1
    $\begingroup$ Keeping in mind that I don't often use hypothesis tests in practice (estimation and prediction are more relevant): For me, such a choice depends on the circumstances, what is known about the variables, and the potential audience; if by 'math scores' you mean test scores or numeric grades, those are bounded and not usually very skewed. A t-test probably would probably be fine, but if I was less confident (didn't have previous information about the distribution, say) I might well base a nonparametric test off the t-statistic itself. $\endgroup$
    – Glen_b
    Mar 30 '20 at 2:51
  • 1
    $\begingroup$ I'd tend to argue that in many instances of experimental statistics, the usual hypothesis tests people use are often not answering a very useful question. $\endgroup$
    – Glen_b
    Mar 30 '20 at 14:18

It depends on the degree and type of non-normality. Even with relatively small sample sizes the test is conservative with skewed distributions but not strictly "valid" since the actual p is lower than the nominal p (if you use p=0.05 as a cutoff you will reject a true null hypothesis less than 0.05 of the time. If your browser lets you run unsigned applets you can explore this yourself here: http://onlinestatbook.com/stat_sim/robustness/index.html

Other deviations from normality (such as kurtosis) may have different effects.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.