When is a statistical test said to be robust? Also, is normal distribution applicable to economic variables as well?

Question

I was doing a student satisfaction survey and analysed the results using MANOVA. None of the variables thAt was under study followed a normal distribution and thus, violated the assumption of MANOVA. My professor said that we could still do this test since it was robust. On another project, in economics, I found that both investments and GDP growth rate which I had taken under study did not follow normal distribution.

Answer 1

Note that MANOVA (together with the whole family of linear regressions) does not assume normality of neither the dependent, nor the indepentend variables. The only normality assumption is that the residuals (i.e. the errors) are normally distributed. And that is a thing that you have to check AFTER you fit your model (or perform MANOVA).

In regard to your question, a test or procedure is defined "robust", when it remains valid even when some of the assumptions are violated by some amount. Generalized Linear Models (of which MANOVA is part) should be robust in regard to normality of residuals when the sample is big. (I was tought n>20 for each cell/condition, but I have no reference for this number)

Answer 2

When is a statistical test said to be robust?

That depends on who is saying it!

There are two main things with hypothesis tests -- how they perform under the null and how they perform under the alternative.

1. Many people only consider performance under the null -- impact on the significance level -- i.e. to see if it remains close to the chosen significance level (in some sense of 'close'). Often people check for level robustness against deviations from some assumption -- that is they relax the assumption to some degree, and in some manner, and see what impact it has on actual significance level (compared to the one you had specified). So for example they might see the impact of distributions other than the normal if there's a normal assumption. Some people check only a few cases or only small deviations and yet may conclude something fairly general. [If you want to get a better handle on things, it helps if you have some sense of how to generate "worst case scenarios".]

   [Note that there are usually multiple assumptions; I'd hesitate to refer to a test as "robust" without specifying which assumptions it was robust to.]

2. Performance when the null is false is also important. For example, I can give you a test that always has a significance level of 5%, no matter how badly you break the assumptions -- the ultimate in level-robustness. If level-robustness was all that mattered, you should certainly use it, since it can't be bettered on that score. The problem comes when you look at power -- this particular super-robust test also has its power always at 5%.

   So you need to take account of the effect on power as well. Typically, I'd tend to consider power under a sequence of deviations from the null (e.g. increasing skewness by considering a sequence of alternative distributions, or increasing dependence by making values more and more correlated, and so on). Some people might be mostly interested in power at some particular alternative, but I think it's also important to be aware of the possibility of impact on smaller effects -- especially to consider the possibility of test bias, for example.

So when someone says a test is robust, you need to consider what basis they have for the claim -- what is it robust against, exactly -- which assumptions is it not sensitive to? How much can you push them? How was that checked -- which cases were considered? What impact did they have? Did they look at power as well as significance level?