Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 3 down vote favorite

... where the significance of the results has been checked with statistical tests.

If no, then why?

side note: The data is generated based on specified prior probabilities.

share|improve this question
what is exactly the question here ? If one wants to make a statement on the superiority of performance of one technique over another one should always use significance testing :). Do you mean: What is the value of such a statement if it has been made on artificial data only ? – steffen Dec 19 '11 at 9:00
Yes, I should have been clearer. But you've got it now. Is it a bit contradictory to test for statistical significance when the artificial data comes from some fixed model? – dubby Dec 19 '11 at 9:44
1  
no, since statistical tests "only" check whether the difference in techniques is true or has occurred by chance. See for an example the Kaggle competition "Dont Overfit" where the competiton data is purely artificial. Now imagine a statistical sig-test on a crossvalidation-result on the mentioned 250 cases: Real improvement over random model or fluke ? – steffen Dec 19 '11 at 12:30
I edited the question according to your response. Please check if I have accidentically changed the meaning. – steffen Dec 19 '11 at 12:37

1 Answer

up vote 5 down vote accepted

Your problem is much like any other machine learning problem: you have some data and you use some of it as training data, to tweak your algorithms, then you use some of of the data (that wasn't used for training) to choose a "winner" algorithm, and finally you use some of the data (that wasn't used for training or testing) to give yourself an idea of how your winner will work in the Real World.

In your case, the initial data is artificial. The question is: how well does the artificial data mimic the Real World where you intend to use your technique? Your technique reflects a model of the world that is simplified. The main fear would be that your data-generation method reflects the same simplifications, and thus provides data that fits your model well, as opposed to your model fitting the Real World well.

(Even if your data were not artificial, the issue still arises: when you say a technique is "superior", all that you can really say is that you were able to adjust it to work better on the data on which you tested it. If you did your work well and the technique is sound, it will probably be a good choice on similar data. You have to be careful with what you mean by "superior".)

share|improve this answer
1  
(+1). I want to add: Artificial data is generally used to emphasize (or validate) the properties of a classifier, not to prove that it'll work in the real world. – steffen Dec 19 '11 at 15:47

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.