3
$\begingroup$

I refer in particular to such choices: 1) t-test (or its generalizations: ANOVA or Hotelling's $t^2$) vs its non-parametric alternatives (e.g., U Mann-Whitney test and its generalizations); 2) Pearson's correlation vs its non-parametric alternatives (Spearman correlation, Kendall's Tau, Goodman's Gamma). I know that using first a test on Normality and then the final test (non-parametric if Normality was rejected, parametric otherwise) is heavily critized due to loss of control of type I Error, however I'd prefer to ignore this problem here. Also, I've found out a typical suggestion is to judge basing on graphical tests, that is something I can only consider for explorative aims. I'm also aware that the issue of whether using parametric or non-parametric tests is often debated (please notice I'm only interested in inference, not on the observed value of correlation or of the difference between means per se). However I find replies to such questions basically conflicting. See for example here: Independent samples t-test: Do data really need to be normally distributed for large sample sizes? where two replies supporting contrasting views have both been evaluated positively by many users.

Specifically, I am referring to cases where we assume observations are i.i.d. with finite mean and variance but traditional tests reject normality (both on the original variables and on its transformations), so that only the Central Limit Theorem (alongside the Slutsky's theorem, in case we are concerned with the distribution of variance) may be invoked to adopt the usual parametric tests. I read here:

What references should be cited to support using 30 as a large enough sample size?

that we cannot really rely on the rule of thumb of 30 observations to assume Normality of the distribution of sample mean, because the speed of convergence to normality obviously depends on the distribution of single observations (see points 3 and 4 here: Normality assumption and sample size. The only possible approach to solve this issue I've found among the discussions on such riddle is the use of the Berry-Esseen theorem. What I'd like to understand is: can we use it to understand whether parametric tests are sufficiently valid and powerful (i.e. if error I type and the distribution of error II type are acceptable) with the specific dataset at hand, so that one can have an idea on whether such tests are fit or not? In case such issue (parametric vs non-parametric) is still controversial, can I get a reference describing the pros and cons of each position?

++++++++++ EDIT 30 NOV 2018 ++++++++++++++++

Ok: one could check the performance of the t-test compared to the ideal test given a specific alternative distribution. Nevertheless, I'm referring to situations where traditional Normality tests are rejected also for the logarithmic (and squared root, etc.) transformation of original data. What I’d like to have is a comparison between t-test/Pearson correlation and a non-parametric alternative. In particular, while we typically ignore the population distribution of continuous variables, we may observe their sample moments (and any other characteristic of their sample distribution), so I wonder how we can effectively use such information to solve the question: parametric or not?

Improve this question
$\endgroup$
3
  • $\begingroup$ You might like to redo Antoni's lognormal simulation with a lognormal with $\sigma=4$ (rather than the medium-skewed $\sigma=1$ case), and a smaller effect size, and then compare its power with a more powerful test of a change in the $\mu$ parameter of the lognormal (the obvious one in that instance being to take logs and perform a t-test on those values). Good questions to answer would be "what's the achieved significance level?", "at some moderate sample size, how much power do we lose by using a t-test compared to the best test?" and "how large a sample would we need for good power?" $\endgroup$
    – Glen_b
    Nov 30, 2018 at 3:38
  • $\begingroup$ ... & perhaps "is there a test that would have power close to that best test without requiring us to know it's lognormal?" Both Antoni and Frank make some good points but they're mostly addressing different things. whuber's comment at the end under Antoni's answer is useful because it explains that the t-test can certainly fail to perform well in some circumstances, even though it frequently performs reasonably well in a variety of situations. It's possible to do something more robust than the t-test but the extent to which it makes sense to depends on the precise hypothesis of interest. $\endgroup$
    – Glen_b
    Nov 30, 2018 at 3:54
  • $\begingroup$ Thanks for the various edits. I made a small edit to include the title of the post with the linked answer and removed comments related to it. I've left my comments related to some of the issues raised in the question for now. $\endgroup$
    – Glen_b
    Nov 30, 2018 at 11:30

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.