After the fact hypothesis testing

Question

I'm not really a statistician, but rather in need of statistical guidance, so I hope this is not too much of an off-topic question.

I'm writing a master's thesis (computational linguistics/NLP), and I've got several result sets I'm comparing. Now, I didn't really formulate a null and an alternative hypothesis before I ran the experiments, which I understand means that ideally I shouldn't really go about using the T-test on my datasets. But some of the different results are so close that I've given in to the temptation and T-tested them. How unclean is this? Is it bad enough that I should leave it out of my thesis entirely, or is it less severe?

In case it matters, the data are the error rates of different language models, by 10-fold cross-validation.

Answer 1

It's not necessarily a problem that you didn't formulate hypotheses before running the study, but you may be doing a post-hoc analysis, which would be relevant. Also consider whether what your tests mean.

The population

I feel the need to point out that the population is sort of bizarre in this situation. If I understand your study correctly, you have 10 error rates for each model. If you were just interested in the performance for this particular partitioning scheme, you would not need to use statistical tests; these 10 error rates would be the population.

The entire corpus would interest you more. There are (N choose N/10) ways you could partition the data into 90% training set and 10% test set, and you could run the models on a random sample of these partitioning schemes. It seems that this approach is called repeated random sub-sampling validation.

Differences between models

If I understand your dataset, the tests that I am about to describe may not really be valid because the 10 error rates are sort of dependent on each other; they are all taken from the same partitioning scheme.

But here we go anyway! I assume that you are trying to see whether any of the models perform significantly differently from than the others. This is a valid hypothesis, but you'll need to use something like ANOVA because you have more than two models.

On the other hand, if you are just trying to tell the difference between two models because they are the two best models, you have to account for how you decided to compare these two after the fact. Look at post-hoc tests and p-value adjustments.

Answer 2

I did a Google search and bumped into the following title:

"A study of cross-validation and bootstrap for accuracy estimation and model selection"

I would post the link but it's one of those citeseerx.ist.psu.edu links that comes across with an IP number for the URL. If you do the search, you should be able to find it. In that article, they give a calculation for a confidence interval.

Here's a similar article:

http://www.public.asu.edu/~ltang9/papers/ency-cross-validation.pdf

If those don't help, here are some links on "Multiple Comparison Tests":

http://www.itl.nist.gov/div898/handbook/prc/section4/prc47.htm

http://www.uvm.edu/~dhowell/StatPages/More_Stuff/RepMeasMultComp/RepMeasMultComp.html

http://www.utdallas.edu/~herve/Abdi-Bonferroni2007-pretty.pdf

http://www.csub.edu/~psmith3/Teaching/310-8.pdf

http://www.graphpad.com/faq/viewfaq.cfm?faq=1091

http://en.wikipedia.org/wiki/Multiple_comparisons

I'm no expert, but I'm in a similar situation.

In a recent problem (that's still going on), until I looked at the data, I had no idea what was a reasonable null, much less what was the "right" null. After some playing around with the data, I noticed some possible relationships. Am I data snooping (dredging)? Who knows? I do know that I can't use an uncorrected t-test, p-value, etc on any of my results/discoveries.

What I started looking for was a way to "put a fence around the problem". In other words, some way to define how many relationship combinations were available when I found "my" relationship. Do you have a way to do that? In other words, can you categorize/count the POSSIBLE solutions that you've tested/allowed-for into XX groups? If so, that might be a way to approach a Multiple Comparison Test.