6
$\begingroup$

I have been having a hard time deciding which statistical test to choose for a dataset. The more a read on the web, the more I get confused since frequently there are different opinions when it comes to chose the right test.

To this extent, when in doubt, I apply one parametric and one non-parametric tests, for example, an one-way ANOVA and a Kruskal-Wallis, or a two-sample t-test and a Mann-Whiteney, hoping that both tests give me the same output (generally $p < 0.05$). If they do, I am done; if not, then I need to work harder.

Is there some well recognized site out there that provides some kind of decision support tree for choosing statistical tests?

Is there some tool that checks as much as possible the assumptions of a statistical test on a given dataset before applying it? For example, for one-way ANOVA it could check for normality and variance homogeneity automatically!

I think such site or tool would help a lot, but probably I am asking too much ...

Thanks

Improve this question
$\endgroup$
10
  • 10
    $\begingroup$ Having some kind of decision support tree is always beneficial. But software designed to automatically check the assumptions, if it means "automatically choosing the most appropriate test" is a project for castrating a man mentally. Unfortunately, some otherwise respectible packages (e.g. SPSS) have already started deadly efforts in this direction. $\endgroup$
    – ttnphns
    Commented Jun 30, 2012 at 6:20
  • 1
    $\begingroup$ Evolution is to move forward into supporting higher-level abstractions to make human lives easier (e.g., computer languages and handheld scientific calculators with lots of high-level functions), so I think those "deadly efforts" are welcome. (For example, being a programmer, I don't imagine going back to use assembly language.) Anyway, I think there should be support for the statistian who wants to control every analysis parameter, and also for those who occasionally need some statistical assessment. $\endgroup$
    – mljrg
    Commented Jun 30, 2012 at 8:47
  • 1
    $\begingroup$ @ttnphns: I agree with "A statistician must reflect to himself about every decision made (why did I prefer this test not that)", but better practical assistance doing that would be extremelly helpful has the options can be very hard to decide. (See my comment below to Michael Chernick where I give a great example of that in software programming) $\endgroup$
    – mljrg
    Commented Jun 30, 2012 at 9:46
  • 6
    $\begingroup$ I too wholeheartedly disagree with @mljrg . Two things explain why. First think of all the silly tests of normality that people do to justify the use of the $t$-test, assuming the test has a power of 1.0. Secondly, there will be wide disagreement among statisticians on the exact "rules" for "satisfying assumptions". $\endgroup$
    – Frank Harrell
    Commented Jun 30, 2012 at 17:45
  • 1
    $\begingroup$ @Macro: that's a good topic. Clearly not everything can be automated (I never meant that when I wrote above "tool that checks as much as possible the assumptions of a statistical test") but once I decided for a statistical test based on knowledge of the sampled data, probably some other assumptions could be checked automatically by the chosen test function in the package being used. $\endgroup$
    – mljrg
    Commented Jul 1, 2012 at 12:49

3 Answers 3

Reset to default
6
$\begingroup$

I think you should look at applied statistics texts. An easy one to read the is one of my favorites was written by the late Rupert Miller (I took the applied statistics sequence that he taught when I was a graduate student at Stanford). At that time we had notes. His book was not finished but it is a marvel. He was a great teacher and writer. The book was published by Wiley and is titled Beyond ANOVA, Basics of Applied Statistics (on Amazon). It was originally published by Wiley but apparently is reprinted currently by Chapman & Hall/CRC. This goes through all the assumptions needed for parametric ANOVA and the methods to check them.

Improve this answer
$\endgroup$
12
  • $\begingroup$ Too much expensive, especially for me who will be using statistics occasionally, not as daily work ... but indeed a good (and less expensive) applied statistics text would be helpful (something to the point, like a kind of recipe). But which one? $\endgroup$
    – mljrg
    Commented Jun 30, 2012 at 0:58
  • $\begingroup$ A very useful reference in software programming is Design Patterns: Elements of Reusable Object-Oriented Software, which provides systematic and well-justified solutions for common problems and their variants. Is there something like this for statistics in practice? Is there a good reference that lists for several "data analysis problems" the corresponding "statistical tests"? This would be very helpful for the occasionally statistian, like me. $\endgroup$
    – mljrg
    Commented Jun 30, 2012 at 9:43
  • $\begingroup$ @mljrg, the price differences are mostly due to supply & demand; it costs a lot to get a book out & there just aren't that many sales over which to distribute those costs. As a result, statistics texts are pretty much always expensive :-(! OTOH, there is a much bigger market for programming books, making them cheaper. $\endgroup$
    – gung - Reinstate Monica
    Commented Jul 1, 2012 at 17:33
  • 2
    $\begingroup$ mljrg, following on @GregSnow's answer, you need to know about those formulas (etc) in order to know what's going on (or will be if you do different things) so that you can make intelligent choices about which test to use & how to interpret it. Re: the book that will help you best in your situation, I would put a lot of stock in what Michael Chernick has offered, b/c he is the master of the topic. Re: the 1 overall book for someone to read, you may want to read this CV question & my answer there. $\endgroup$
    – gung - Reinstate Monica
    Commented Jul 2, 2012 at 1:26
  • 1
    $\begingroup$ David Moore's The Basic Practice of Statistics is one good one to look at amazon.com/The-Basic-Practice-Statistics-Student/dp/1429224266/… I also think the books by DeVore and Peck and the ones by DeVeaux are also excellent. $\endgroup$
    – Michael R. Chernick
    Commented Jul 2, 2012 at 15:51
10
$\begingroup$

The information needed to decide if the assumptions about a statistical test are reasonable are often exterior to the data itself. This means that an automated program would not have the information needed. For example it is usually assumed that the data was collected independently (or conditionally independently), but looking at the data how can you tell the difference between a simple random sample (usually fine for many stats tests) and a snowball sample (not good for most quantitative tests)? Since a simple random sample has every possible sample equally likely then any non independent sample could have also resulted from a simple random sample. You need to know how the data was collected, not just the data itself.

Also note that if you do a normality test in order to decide which test to use then you are generally either getting a meaningless answer to a meaningful question (small sample size) or a meaningful answer to a meaningless question (large sample size). I expect that many of the other tests for assumptions (without outside knowledge) will have similar problems.

If you "test" for every assumption that could affect the results of your test (without outside knowledge suggesting which might be the most meaningful) then you are likely to either always reject at least one assumption (if you don't correct for multiple comparisons) or you will have so little power to detect assumption violations (when you do correct for multiple comparisons) that the results will be little better than generating a p-value from a uniform distribution. Knowledge of the science that lead to the data is needed to assess which assumptions to further investigate (and plots are probably as useful as formal tests).

Also note that the non-parametric tests mentioned above and the normal based tests above are testing different null hypotheses. If the results don't agree it could be that both are giving correct (or at least approximately, with a good approximation, correct) answers to different questions.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I completely agree with your points about the need of outside knowledge to decide which test, but once you use that knowledge to pick a test, I think the test could further automate some other assumptions. My dataset comprises a dependent variable "elapsed time" measured for three treatments, but since the size of the treatments are small (<= 21), from what you said I conclude that checking for normality is meaningless, so I should go for a non-parametric test (Kruskal-Wallis). Thanks. $\endgroup$
    – mljrg
    Commented Jul 1, 2012 at 12:41
  • 1
    $\begingroup$ @mljrg Wrong conclusion! Doing a normality test is meaningless, but that does not mean that normality-based tests cannot be used. However, time-to-event variables such as elapsed time are usually quite skewed and heteroskedastic (external information!) It is easy to misinterpret even non-parametric tests under these conditions. Other options such as a log-rank test might be a good alternative. But, to reiterate, you need to incorporate your knowledge about the outcome variable. $\endgroup$
    – Aniko
    Commented Jul 2, 2012 at 13:58
1
$\begingroup$

This is old but your library may have it:

"A guide for selecting statistical techniques for analyzing social science data" 2nd ed 1981; institute for social research,university of michigan Andrews, FM; Klem, L; Davidson, TN; O'Malley, PM; rodgers, WL

Laurence

Improve this answer
$\endgroup$
0

Not the answer you're looking for? Browse other questions tagged or ask your own question.