Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

At this moment I have a dataset with 4000 samples (50% positive and 50% negative). Normally I would do cross validation for this approach, however besides normal data mining techniques I am also trying an alternate ILP approach.

Since I can't implement cross validation using the ILP system I am using, to maintain dataset coherence between the two types of techniques I decided to instead do a split validation. I keep hearing about split validation should only be used in large datasets, but how large should they be? Would implement it on this one produce unacceptable "bad" results? Should I really avoid this approach?

share|improve this question
2  
It's unclear which "ILP" you are using. Otherwise, you are asking largely for opinions about what you should do and about what is good or bad. This site is better suited to questions about how things work or about what the truth of a matter is. –  rolando2 Jul 16 at 10:14
    
I am using the Aleph system. And I am asking how split validation would work work in this situation, so I think it's on the scope of this site. –  user697110 Jul 16 at 10:22
    
Please explain in a short sentence your en.wikipedia.org/wiki/ILP. I don't see why you cannot do resampling validation. –  cbeleites Jul 16 at 11:16
1  
ILP here stands for Inductive Logic Programming. –  user697110 Jul 16 at 12:34

2 Answers 2

Can you explain why you cannot do resampling validation?:

  • All you really need for resampling validation is a method to get predictions for unknown cases.

  • And you can certainly also apply the very same splits to more than one classification algorithm in order to get paired data for a comparison.

share|improve this answer
    
Ah, the answer I was looking for :) –  Andy Blankertz Jul 16 at 13:27
    
Because I am using rapidminer, and I can't see how exactly he splitting the data for cross validatio, whereas I can recreate the split validation for the Aleph dataset. –  user697110 Jul 16 at 13:46
1  
But you could write a script that does the splitting on your desktop and then push(es) training data and unknown test cases to rapidminer, right? I mean, doing a single iteration/repetition of a 4-fold cross validation isn't that much hassle, is it? And it gives you so much more information (at least if you can get the model out of Rapidminer). –  cbeleites Jul 16 at 15:33

I think it's cromulent to create a test file by randomly sampling from the data (can't remember if it's with or without replacement).

Can anyone confirm this?

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.