I am currently working on a kaggle competition; ignoring the details, each training & test set example, in this competition, essentially consists of product and search term pairs. Given each of these pairs, we are expected to be able to predict the relevance of the search term to the given product.
I want to be able to tune my model's parameters and get a out-of-sample root mean squared error in order to evaluate my model's performance. However, I am currently unsure of how to split the data. There are two schemes I have thought of so far:
1) Do a standard 70/30 or 80/20 split on my training set where the splitting is done such that none of the products or search terms present in the resulting validation set are present in the resulting training set.
2) Split the training data in such a way that the resulting validation set has a similar distribution to the actual test set. It turns out that the test set is negatively skewed while the training set is positively skewed. As a result, the largest split possible is a 98/2 split.
Note: The "distribution" I am referring to is characterized by several frequency based values I have chosen because I believe they represent the data well. For example:
value 1 = what percentage of test set examples (product, search term) pairs have a product which exists in the training set but a search term that does not
value 2 = what percentage of test set examples (product, search term) pairs have a product which does not exist in the training set but a search term which does
Are either of these approaches correct? If not, how should one go about splitting this data?