2
$\begingroup$

1) Is non-parametric test robust to Gaussian data? For example in a case where the post-processed data is both Gaussian and non-Gaussian (eg RSME of multiple variables). I have growing concern if the analysis would be accurate if I do a parametric test on non-Gaussian data or vice versa. In my case for example I always choose Mann Whitney $U$ test instead of $t$-test without checking the normality, as my initial assumption would be that my data would be non-Gaussian (although it might be Gaussian)

2) Is it correct to choose non-parametric test if I have a low sample size (e.g. n=18) although the data is Gaussian?. Since non-parametric test is more robust to outlier (as it used median instead of mean). If I prefer not to remove outlier, is it justified to just use non-parametric test?

$\endgroup$
5
$\begingroup$

Is non-parametric test robust to Gaussian data?

If you are asking about level robustness, yes; it doesn't matter what the distribution is.

If you're asking if nonparametric tests have good power at the Gaussian, then it depends entirely on the test. Many have excellent power at the Gaussian, some are fully efficient or very close to it. Others are not. If power exactly at the Gaussian matters a lot to you, you need to check what it is for that test against the alternatives that matter to you.

Is non-parametric test robust to parametric data?

Please note that Gaussian is not the same thing as parametric! There's no such thing as "parametric data". Data are just data. A model is (finite-)parametric (or it isn't). For example, see the opening paragraph here:

Note the definition doesn't relate to normality (though some of the examples do). There are parametric procedures (a huge host of them) that do no assume normality. I'd guess I use one or another of them more often than I use procedures that assume normality.

post-processed data is both parametric and non-parametric

It's not clear what this means. Do you mean "normal" and "non-normal"? If so, how can you know something is normal?

(Failure to reject in a goodness of fit test of normality doesn't tell you something is normal - it will almost never be the case that you have normality; what you have is a lack of power to detect whatever non-normality is present. Not every deviation from normality that's hard to detect is inconsequential!)

In my case for example I always choose Mann Whitney U test instead of t

The first considerations are

  • what are your null and alternative?

  • what assumptions are you prepared to make

Note that testing your assumptions is not generally a good idea.

In the case of testing normality, some of the issues are well explained here:

Is normality testing 'essentially useless'?

(though there are other issues)

Similar issues apply to other tests of assumptions. They don't answer the question you actually need answered, and choosing between tests on the basis of such a test is frequently worse than not making an assumption you don't have some reasonable basis to assume before you see the data.

test without checking the normality, as my initial assumption would be that my data would be non-Gaussian (although it might be Gaussian)

It sounds like you have no good reason to assume normality. Why then would power at the normal matter much to you?

Consider something like a two-sample t-test against a Wilcoxon-Mann-Whitney. The latter has about 96% power in large samples at the normal, but you only need very slightly heavier tails before that advantage completely disappears. There's no distribution for which the t does very strongly better that the W-M-W, but it can do much worse. This suggests that most of the location-difference information is already captured by ranks, when we're (in some sense) near the normal.

Is it correct to choose non-parametric test if I have a low sample size (e.g. n=18) although the data is parametric?

Again, it's not clear what you mean by parametric data; if you mean that the data are sampled from a normal distribution how do you know? Merely on the basis that you have little power to detect it isn't? What good is that?

There are two main issues with small samples

  • you won't have much basis to assess whether your assumptions were wrong

  • nonparametric tests don't offer good control of level at very small sample sizes; it's not so much a problem at say n=18 but at lower sample sizes you might be left with a choice between $\alpha = 0.058$ or $\alpha= 0.021$ when you want (say) a 5% test. You can do a randomized test but that has consequences many people find highly unsatisfactory (such as two different people coming to different conclusions using exactly the same test procedure on exactly the same data set using the same significance level).

    At very low sample sizes there might well be no test with a significance level below 10%.

Since non-parametric test is more robust to outlier (as it used median instead of mean).

Actually hardly any nonparametric tests are based on the median. None of the common ones are (e.g. the signed rank test and the Wilcoxon-Mann-Whitney are examples of tests which are not based on the median)

If I prefer not to remove outlier,

It's rarely a good idea to arbitrarily "remove outliers" unless you have some clear basis for concluding that they should not have been part of your sample. This is also an issue from the point of view of choosing your analysis on the basis of the appearance of your data.

It does make sense to use procedures that are not badly impacted by potential outliers. That doesn't necessarily restrict you to nonparametric tests; suitably robustified parametric tests may do quite well for example.

is it justified to just use non-parametric?

  1. Not all nonparametric tests are robust to outliers (if you mean to ask about their power rather than their significance level). Some are, but you can't just assume that because a test is nonparametric it retains good power in the presence of outliers. An example of a nonparametric test that can be badly impacted by outliers is the Tukey-Duckworth Quick test (though there are ways to improve that)

  2. You should not be choosing tests on the basis of the appearance of your sample. This impacts the behaviour of the distribution of the test statistic and so your p-values no longer carry their usual meaning.

    Instead, you should understand the behaviour of your variables as well as possible ahead of time and make reasonable assumptions where it makes sense to do so, while (where possible) avoiding highly consequential assumptions you don't have a good basis for.

$\endgroup$
  • $\begingroup$ I mean Gaussian and non-Gaussian data. Sorry I get mixed up between parametric and Gaussian. My post processed data can be both normal and non-normal, however, I don't usually prefer to test if my data is normal or not, instead I just skip to performing Mann Whitney U or Wilxocon Sign Rank instead of t-test. The only justification I have is because I have small sample size i.e. 18 and it can easily be changed from Gaussian to non Gaussian. I am just worried this might not be enough justification. I performed qqplot and Shapiro Wilk test. $\endgroup$ – Sharah Jul 12 '18 at 12:38
  • $\begingroup$ I edited my question to correct the term. Note that most of the time I just use t-test, U test and Wilcoxon test. $\endgroup$ – Sharah Jul 12 '18 at 12:39
  • 1
    $\begingroup$ Note that any data-based assessment of "Gaussian-ness" can be fooled. So it's safest just to almost always use non-parametric or semi-parametric methods. $\endgroup$ – Frank Harrell Jul 12 '18 at 12:48
  • 1
    $\begingroup$ @FrankHarrell yes, I read somewhere saying that it's not right to check if the data is Gaussian or not. It should be in the initial assumption. $\endgroup$ – Sharah Jul 13 '18 at 10:18
  • $\begingroup$ Even if you do some check, you can still not say that you have data drawn from a normal distribution. For example, failure to reject normality (having $p>\alpha$ in a test of normality) is not at all the same thing as actually having normality. It just means you lacked the power to detect the difference you had from normality. $\endgroup$ – Glen_b Jul 14 '18 at 1:33
4
$\begingroup$

In terms of the choice of non-parametric vs. parametric tests, if the assumptions of the parametric test are met then the parametric test will be more powerful than the non-parametric one. This doesn't mean that the non-parametric can't be used, just that there is another test out there that would perform better.

So in your case, if you are always using Mann Whitney without checking the assumptions, then you could be losing performance in the cases where the t tests assumptions are met.

$\endgroup$
  • 2
    $\begingroup$ may want to note that the asymptotic relative efficiency of (e.g.) the Wilcoxon test is about 96% (see here), which means that at least for large samples you're not actually losing very much power ... $\endgroup$ – Ben Bolker Jul 11 '18 at 15:15
  • $\begingroup$ @benBolker what is large sample, this is intriguing. is there's a threshold for large/small sample? $\endgroup$ – Sharah Jul 11 '18 at 21:19
  • $\begingroup$ @astel while I understand I might be losing performance, however, it's impossible to do multiple tests on same dataset, if the data is both normally and non normally distrbuted. $\endgroup$ – Sharah Jul 11 '18 at 21:21
  • 1
    $\begingroup$ The assertion in your first sentence is not always true - there are nonparametric tests that are fully efficient. $\endgroup$ – Glen_b Jul 11 '18 at 23:25
  • $\begingroup$ @Sharah There's no "threshold"... the relative efficiency in small samples at the normal is a little lower than the asymptotic value, but it's not by much and it smoothly progresses as you increase the sample size. $\endgroup$ – Glen_b Jul 11 '18 at 23:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.