Relevance of assumption of normality, ways to check and reading recommendations for non-statisticians

Question

I started reading around the topic of modern robust methods, consulted various statistics texts and did some research on the CV forum. I ended up being rather confused regarding the relevance of the assumption of normality.

While some authors state that even small deviations from normality can, under certain circumstances, cause major issues when using classical parametric tests, others argue the assumption of normality was not that important.

Recommendations on how to check (if at all) the assumption of normality vary broadly. For a non-statistician as myself, who uses textbooks as a basis to analyse research data, it is difficult to identify trustworthy resources and procedures that are well-accepted in the field.

Can anyone recommend a source with helpful and accessible information on current recommendations?

Edit

For some more background: My research is in the area of psychology, which - in my case - often requires t-tests, ANOVAS (often two-ways) or regressions. On a regular basis, I encounter issues with skewed distributions (e.g. in some groups, many people chose or obtained the/one of the highest values - which even makes sense content-wise, but still violates the assumption of normality). I found the skewed data to be difficult to "correct" in many cases (because the skew is so extreme or because groups differed in their skew, so the transformation didn't solve some of the issues or even caused new ones). This is why I started looking for alternatives and came across the modern robust methods (e.g. robust two-way ANOVA, WRS2 package for R).

Answer 1

You are right to be confused, as this is a confusing issue indeed.

I'm afraid it will be hard to find everything good to know in a single reference, but some may give that to you - I will not look around but rather tell you what I think.

(This answer is more about the background, how to think about model assumptions in general; I have written another answer with some more practical hints.)

1. Models are idealisations and they never hold precisely in practice, so we will routinely apply methods to data for which the methods' model assumptions are more or less obviously violated.

2. Having a model assumption for a certain method does not mean that the assumption has to be fulfilled for the method to make sense. It only means that if the model assumption is fulfilled, there is a theoretical guarantee that the method does what it's supposed to do (these guarantees are not always the same and can be stronger or weaker). The tricky issue here is that obviously there is then no theoretical guarantee in case that the model assumption is violated; and in fact it is both possible that our analysis is still fine, or that it is misleading, and it is hard to tell these two possibilities apart.

3. Many statistics are asymptotically normally distributed even if the underlying data are not normal due to the Central Limit Theorem (CLT; which itself has assumptions that may be violated, but see above). This means that if your sample size is large, many deviations from normality are not problematic as results from assuming normality will hold approximately.

4. There is a catch with item 3, which is that it depends on the unknown true underlying distribution how large a sample size is actually required so that results are satisfactorily normal (also it depends on what exactly you do because the CLT doesn't apply to everything; it does apply to the arithmetic mean though, on which many statistics are based).

5. The key issue is not whether data are normal or not, and even not whether data are approximately normal, but rather whether normality is violated in ways that will mislead conclusions. In fact, some tiny violations of normality (a single gross outlier) may be harmful, whereas a distribution such as the uniform that looks very obviously non-normal will usually not cause problems for inference based on a normality assumption.

6. Depending on what exactly you do, in most cases the following things are most problematic: Extreme outliers (either observations where something is wrong or essentially different from the others, or generally heavy distributional tails) and strong skewness. On the other hand, if there are no heavy tails, normality based inference is very often harmless, for example for discrete 5-point (or other) Likert scaled data (data that can only take values -2,-1,0,1,2 and are therefore pretty much guaranteed to not have outliers).

7. In many situations other violations of model assumptions such as dependence are more critical than non-normality. In particular, the CLT requires independently identically distributed data or some alternative assumptions that are not much weaker. I have read once (I think in the Hampel et al. book on Robust Statistics) that many high quality astronomical data sets are heavy tailed, and some of them look more normal than they should due to long range dependence - and for the sake of analysis one would be better off with less normality and less dependence.

Answer 2

I will write a second more practically oriented answer. The first one is rather on a more abstract level about understanding the relevance (or limited relevance) of the model assumptions. So here are some hints.

1. Although the thread opener is apparently well aware of this, for general reference I start by saying, make sure that you understand what exactly is assumed to be normally distributed in your problem. For example, if you run an ANOVA or regression, the normality assumption applies to the random variation ("error") term, estimated by the residuals, rather than to the raw observations of any variable, i.e., in one-way ANOVA it applies within groups seperately, and this is different from having "normal data", i.e., normality of all groups taken together.

2. Use some a priori knowledge about your data: Is the value range limited, and would you normally expect that data regularly occur at or close to both extremes? In this case methods backed up by normal assumption theory will usually be fine. (This would for example be the case for most Likert scale data or data that has a natural range between 0 and 1 unless there are reasons to expect that almost all data is on one side.)

3. What are potential reasons for outliers in the data? Can you somehow track outliers down and check whether they are in fact erroneous? In this case the advice would be to remove such observations. (I do not recommend to generally remove outliers as these may be relevant and meaningful, but if they are in fact erroneous, out with them!) Generally, improving data quality is always worthwhile where possible, even though robust methods exist that can deal with some issues.

4. Have a look at your data and see whether there are outliers or extreme skewness. Important: Do not be too picky! Technically, deciding what test (or other method) to use conditionally on the data themselves will invalidate the theory behind the methods. I had written in the first answer that model assumptions are never perfectly fulfilled anyway, and for this reason one can argue that making decisions conditionally on how the data look like can be tolerated to some extent (better to invalidate theory "a bit" than doing something grossly inappropriate), however there is a tendency to overdo this and I'd generally say the less of this you do, the better (I mean less data dependent decision making, not less looking at the data!). So by all means look at your data and get a proper idea of what is going on, however have an analysis plan before you do that and be determined to go through with it unless there is a really strong indication in the data that this will go wrong.

5. You can get a better feel for these things simulating, say, 100 normal datasets of the size of interest, and looking at them. This will show you how much variation there can be if the normal assumption really holds. If your data look quite non-normal, you can also (although this is more sophisticated) simulate many datasets from a skew distribution that looks to some extent like your data, or normal data with added outliers, or a uniform or whatever, compute the test you want to perform, and check whether its performance (type I and type II error) is still fine, or look at the distribution of the test statistic. Such things cause some work but give you a much better feel for what happens in such situations.

6. If you have outliers that can't be pinned down to be erroneous, and the data look otherwise fine, standard robust methods should do fine. There's a school of thought that says you should really always use robust methods as they do little harm even if model assumptions are fulfilled. I don't fully agree with this, as although they will not lose much in case the data are really normal, their quality loss may be more substantial in cases in which normality is not fulfilled but the CLT approximation works well such as uniform data or discrete data (particularly if there is a large percentage of observations on a single value, which can be detrimental for robust methods).

7. As you have already indicated yourself, transformation is often a good tool for skew data, however if you compare groups and different groups have different kinds of skewness, it may not help. One important thing in such situations is that you need to be clear about what exactly you want to compare. t-tests and standard ANOVA compare means. Their performance may be OKish is such situations despite skewness (if the skewness is not too extreme), but a more tricky issue is whether the group mean represents in a proper way the location of the group. In a normal distribution, mean, median and mode are the same, for a skew distribution this is not the case. The mean may be too dependent on extreme observations. Robust ANOVA may help but there are some subtleties. Particularly if you compare two groups that have opposite skew, a robust method may downweight large observations in one group and small observations in the other, which would not be fair (it depends on which exact method you use). Using rank sums, as the Kruskal-Wallis test does, may be more appropriate, however its theory also assumes the same distributional shape in all groups if not normal (it may still perform fairly well otherwise; as long as the rank sums are a good summary of the relative locations of the groups to be compared, I'd think this is fine). In case your distributions are differently distributed in a way that on some areas one group tends to score higher, and another one in other areas (the simplest case is if one group has a slightly smaller mean and a much smaller variance, meaning that the other group has the highest as well as the lowest observations), it may be more appropriate anyway to give a differentiated interpretation of the situation rather than breaking things down to a single statistic/p-value.

8. Whatever your result is, look at the data again afterwards and try to understand how the data led to the specific result, also understanding the statistics that were involved. If the result differs from your intuition what the result should be (from how the data look like - I don't mean subject matter knowledge here as hopefully you don't want to bias your results in favour of your subject matter expectations), either the result or your intuition are misleading, and you can learn something (and maybe use a different method in case your plot shows that the method has done something inappropriate).

9. A comment hinted at methods that explicitly require other assumptions than normality. Obviously this is fine where data are of that kind, however a similar discussion applies to their assumptions.

10. Don't neglect other model assumptions thinking about normality too much. Dependence is often a bigger problem; use knowledge about how the data were obtained to ask yourself whether there may be issues with dependence (and think about experimental design or conditions of data collection if this is a problem; one obvious issue could be several observations that stem from the same person). Plot residuals against observation order if meaningful; also against other conditions that might induce dependence (geographical location etc.). I'm not usually much concerned by moderately different variances, however if variances are strongly different, it may be worthwhile to not use a method that assumes them to be the same but rather visualise the data and give a more detailed description, see above.