Is there a software package designed to automatically check the assumptions of various statistical tests?  I have been having a hard time deciding which statistical test to choose for a dataset. The more a read on the web, the more I get confused since frequently there are different opinions when it comes to chose the right test.
To this extent, when in doubt, I apply one parametric and one non-parametric tests, for example, an one-way ANOVA and a Kruskal-Wallis, or a two-sample t-test and a Mann-Whiteney, hoping that both tests give me the same output (generally $p < 0.05$). If they do, I am done; if not, then I need to work harder.
Is there some well recognized site out there that provides some kind of decision support tree for choosing statistical tests?
Is there some tool that checks as much as possible the assumptions of a statistical test on a given dataset before applying it? For example, for one-way ANOVA it could check for normality and variance homogeneity automatically!
I think such site or tool would help a lot, but probably I am asking too much ...
Thanks
 A: I think you should look at applied statistics texts.  An easy one to read the is one of my favorites was written by the late Rupert Miller (I took the applied statistics sequence that he taught when I was a graduate student at Stanford). At that time we had notes. His book was not finished but it is a marvel.  He was a great teacher and writer.  The book was published by Wiley and is titled Beyond ANOVA, Basics of Applied Statistics (on Amazon).  It was originally published by Wiley but apparently is reprinted currently by Chapman & Hall/CRC.   This goes through all the assumptions needed for parametric ANOVA and the methods to check them.
A: The information needed to decide if the assumptions about a statistical test are reasonable are often exterior to the data itself.  This means that an automated program would not have the information needed.  For example it is usually assumed that the data was collected independently (or conditionally independently), but looking at the data how can you tell the difference between a simple random sample (usually fine for many stats tests) and a snowball sample (not good for most quantitative tests)?  Since a simple random sample has every possible sample equally likely then any non independent sample could have also resulted from a simple random sample.  You need to know how the data was collected, not just the data itself.
Also note that if you do a normality test in order to decide which test to use then you are generally either getting a meaningless answer to a meaningful question (small sample size) or a meaningful answer to a meaningless question (large sample size).  I expect that many of the other tests for assumptions (without outside knowledge) will have similar problems.  
If you "test" for every assumption that could affect the results of your test (without outside knowledge suggesting which might be the most meaningful) then you are likely to either always reject at least one assumption (if you don't correct for multiple comparisons) or you will have so little power to detect assumption violations (when you do correct for multiple comparisons) that the results will be little better than generating a p-value from a uniform distribution.  Knowledge of the science that lead to the data is needed to assess which assumptions to further investigate (and plots are probably as useful as formal tests).
Also note that the non-parametric tests mentioned above and the normal based tests above are testing different null hypotheses.  If the results don't agree it could be that both are giving correct (or at least approximately, with a good approximation, correct) answers to different questions. 
A: This is old but your library may have it:
"A guide for selecting statistical techniques for analyzing social science data"
2nd ed 1981; institute for social research,university of michigan
Andrews, FM; Klem, L; Davidson, TN; O'Malley, PM; rodgers, WL
Laurence
