Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

At this moment I have a dataset with 4000 samples (50% positive and 50% negative). Normally I would do cross validation for this approach, however besides normal data mining techniques I am also trying an alternate ILP approach.

Since I can't implement cross validation using the ILP system I am using, to maintain dataset coherence between the two types of techniques I decided to instead do a split validation. I keep hearing about split validation should only be used in large datasets, but how large should they be? Would implement it on this one produce unacceptable "bad" results? Should I really avoid this approach?

share|improve this question
It's unclear which "ILP" you are using. Otherwise, you are asking largely for opinions about what you should do and about what is good or bad. This site is better suited to questions about how things work or about what the truth of a matter is. – rolando2 Jul 16 '14 at 10:14
I am using the Aleph system. And I am asking how split validation would work work in this situation, so I think it's on the scope of this site. – user697110 Jul 16 '14 at 10:22
Please explain in a short sentence your I don't see why you cannot do resampling validation. – cbeleites Jul 16 '14 at 11:16
ILP here stands for Inductive Logic Programming. – user697110 Jul 16 '14 at 12:34

Can you explain why you cannot do resampling validation?:

  • All you really need for resampling validation is a method to get predictions for unknown cases.

  • And you can certainly also apply the very same splits to more than one classification algorithm in order to get paired data for a comparison.

share|improve this answer
Ah, the answer I was looking for :) – JenSCDC Jul 16 '14 at 13:27
Because I am using rapidminer, and I can't see how exactly he splitting the data for cross validatio, whereas I can recreate the split validation for the Aleph dataset. – user697110 Jul 16 '14 at 13:46
But you could write a script that does the splitting on your desktop and then push(es) training data and unknown test cases to rapidminer, right? I mean, doing a single iteration/repetition of a 4-fold cross validation isn't that much hassle, is it? And it gives you so much more information (at least if you can get the model out of Rapidminer). – cbeleites Jul 16 '14 at 15:33

I think it's cromulent to create a test file by randomly sampling from the data (can't remember if it's with or without replacement).

Can anyone confirm this?

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.