0
$\begingroup$

I have a data-set where customers reported their numeric preference for a sized product, comprised of ~600 samples with 15 different products. The data-set also includes 10 other numeric features, used to predict the size for each product. Sample size per product isn't huge, and ranges between 25-50.

Each product behaves a bit differently, so each gets its own ML model.

To perform model selection, I'm testing several regression models using a leave one out cross validation methodology. The best performing model type is selected for each product, then trained over the entire data-set.

The weird bit is this: When I use the entire data-set as a test set, the models which performed better during cross validation sometimes perform less well than another model which lost during model selection.

Since I'm using a leave one out for cross validation, I assumed results should differ that much between cross validation and testing on the entire data-set, but apparently that is not true.

I aware of the problem with testing with the entire data-set you used to train your models and the risks of over-fitting. I've done this in this case simply as a sanity check step rather than normal procedure.

I'd be happy to hear suggestions as to why this might be the case, at least for some of the products.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

Whenever we get data, we divide it into three parts.

  1. Training set (60% of total examples)
  2. Cross validation set (20% of total examples)
  3. Test set (20% of total examples)

Training Set are the set of examples that are used for learning (i.e) to find the optimal weights or parameters.

Cross Validation set is used to tune the parameter of the classifier. As we already found optimal weights using training set, we cannot use the same examples to check whether our algorithm is working properly or not because these examples will be in likely work well. We need other examples to test them. So, we use cross validation set. Cross validation set can also be used to determine the best model that suits.

After finding the model, we need test cases. So, we use test set for this purpose.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Not really what I was asking, though I'll clarify the question. In this case feature selection and hyper-parameter optimization was already performed. The cross validation process is done for model selection, but it seems to provide unexpected results when testing with the entire dataset. I tested with the entire dataset for curiosity and sanity checking, after performing the training and cross validation testing with train-test splits. $\endgroup$
    – TLousky
    Commented Feb 12, 2018 at 12:11
  • $\begingroup$ The algorithm that perform well in cross validation set, it does not mean that it's truly the best one. So in your case it gives unexpected results in entire data which means you need to repeat the process starting from the training phase and choose better model. $\endgroup$
    – user195278
    Commented Feb 12, 2018 at 17:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.