I am currently working on a kaggle competition; ignoring the details, each training & test set example, in this competition, essentially consists of product and search term pairs. Given each of these pairs, we are expected to be able to predict the relevance of the search term to the given product.

I want to be able to tune my model's parameters and get a out-of-sample root mean squared error in order to evaluate my model's performance. However, I am currently unsure of how to split the data. There are two schemes I have thought of so far:

1) Do a standard 70/30 or 80/20 split on my training set where the splitting is done such that none of the products or search terms present in the resulting validation set are present in the resulting training set.

2) Split the training data in such a way that the resulting validation set has a similar distribution to the actual test set. It turns out that the test set is negatively skewed while the training set is positively skewed. As a result, the largest split possible is a 98/2 split.

Note: The "distribution" I am referring to is characterized by several frequency based values I have chosen because I believe they represent the data well. For example:

value 1 = what percentage of test set examples (product, search term) pairs have a product which exists in the training set but a search term that does not

value 2 = what percentage of test set examples (product, search term) pairs have a product which does not exist in the training set but a search term which does


Are either of these approaches correct? If not, how should one go about splitting this data?

  • $\begingroup$ How are you splitting? $\endgroup$ – Glen_b Mar 24 '16 at 3:26
  • $\begingroup$ The training set? I currently have not split it because my plan to do a 70/30 split with the 30 split having a distribution similar to the test set is not possible; the test and train sets vary so much that I would only be able to do a 98/2 split $\endgroup$ – Jay Karimi Mar 24 '16 at 3:30
  • $\begingroup$ You should be explaining what exactly is not possible there - and why - in your question $\endgroup$ – Glen_b Mar 24 '16 at 3:32
  • $\begingroup$ I think I understand what you mean now; I reformulated my question as a result. $\endgroup$ – Jay Karimi Mar 24 '16 at 5:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.