0
$\begingroup$

All mainstream machine learning approaches I've seen depend on a test and usually a validation dataset to measure model accuracy during and after training.

This seems like it uses up quite a lot of data that could be used for training.

Are there any machine learning approaches that can estimate model accuracy without extra datasets? I've seen theoretical approaches like VC dimension and Occam learning, which can estimate model accuracy without the extra datasets, but I'm not aware of mainstream, practical applications of these theories.

Also, is there a reason why machine learning engineers prefer to split their datasets instead of using all the data for training and verifying model accuracy in another way?

$\endgroup$
  • 3
    $\begingroup$ What other way for verifying a model do you have in mind, other than collecting new data and testing the model on it? Whether the data is available from the beginning of the modeling exercise (so we can/need to split it into training/testing sets) or whether we first train the model on all the data we have and only subsequently collect more data for testing is a minor consideration. $\endgroup$ – S. Kolassa - Reinstate Monica May 8 '19 at 13:40
  • $\begingroup$ a) are you including crossvalidation? b) theoretical approaches like AIC, BIC bayesian model evidence $\endgroup$ – seanv507 May 8 '19 at 14:09
  • 1
    $\begingroup$ Check the "optimized approximation algorithm" mentioned in this answer $\endgroup$ – Jan Kukacka May 8 '19 at 15:07
  • $\begingroup$ @StephanKolassa the methods I mention: VC and Occam, use model size, number of training samples and accuracy to estimate out of sample error. In theory you can then use all your data for training and still know how it will perform on new data. However, I've not seen real world use of these theories, so I'm wondering why. $\endgroup$ – yters May 8 '19 at 18:06
  • $\begingroup$ @seanv507 yes, I refer to crossvalidation too. In regards to heuristics like regularization, the problem is they don't give precise numbers on what we can expect for out of sample error. They just allow us to say one model is better than another because it ranks better regarding the heuristic metric. So, they cannot in a practical setting take the place of using a separate dataset to estimate the error outside the training set. At least from what I understand, I could be wrong. $\endgroup$ – yters May 8 '19 at 18:08
2
$\begingroup$

Out-of-sample testing simulates the real way machine learning is used. Think about how Amazon designs Alexa. The goal is to analyze future speech, sentences and speakers she hasn't necessarily heard.

The goal of out-of-sample testing is to determine if your model has overfit the data on which it was trained. In other words, we don't want to memorize patterns in the training data or model mere coincidences. We want something that will generalize to new data, much like Alexa should.

I do agree that out-of-sample testing is an excellent way to assess performance, since it's kind of the ultimate test of the generality in which a model works. However, in the regression setting (think OLS), there is adjusted R^2 which gives a penalty for having many parameters, with the assumption being that throwing all kinds of parameters at a model will end up memorizing the data. Remember, if you have N points in the plane, you can hit all of them with a polynomial of order N-1 (parabola connects 3 points) and achieve perfect accuracy on your training data, but that's little more than playing connect the dots.

$\endgroup$
2
$\begingroup$

It's a simplistic description to say that the data is split into training and validation. In reality often you end up using all data for training. For example, you can split the dataset into two equal parts, and run two validations with each using one of the samples as training and the other as testing (aka leave-one-out cross validation). Or, you could validate the model on 80%/20% split, then for the production model you simply re-estimate the model on the whole dataset. These are just two examples where effectively entire dataset is used for training. Then you have cases where the data keeps coming. You don't re-estimate data every time new batch arrived, but you can use the new data to monitor ongoing performance.

Summarizing, the way you stated, surely, it would appear that training/validation approach is not optimal. However, the practice is not so straightforward as you describe. So, the purported inefficiency is not as significant, and maybe even negligible.

$\endgroup$
  • $\begingroup$ My understanding is the more you mix the testing and training data, the less you can depend on the out-of-sample error estimate you get from the testing data, even if this is what happens in the real world. $\endgroup$ – yters May 8 '19 at 18:12
  • 1
    $\begingroup$ @yters, you're absolutely right. That's why cross validation and test split exists to relieve the issue, though it doesn't solve it completely. You use cross-validation for model selection, then at the very end use the last bullet (test sample) to validate the choice, presumably, only once. $\endgroup$ – Aksakal May 8 '19 at 18:42
1
$\begingroup$

You already got two nice answers. Without repeating what was already said, let me add my three cents.

Notice that no matter what criteria for model selection you'll choose, in each case you make a number of assumptions that need to hold and if they are met, the methods give you some guarantees of optimality. Those assumptions may be more or less unrealistic.

The idea of having hold out test set, is to simulate scenario where you would make prediction on other dataset then was used for training. What follows, the only assumption that you make is that your test set is similar to the "real" data that your algorithm will see in the wild. This is a rather basic assumption, since you already gathered this sample assuming that it will be reasonably representative for the "real" data, otherwise you wouldn't use it, or at least trust it. Any other method makes much stronger (less realistic) assumptions.

$\endgroup$
  • $\begingroup$ Is there no approach that starts from the basic assumption of similarity, and then can avoid using a test dataset to rigorously estimate out of sample error? It seems we humans can do this quite frequently. Throughout most of life we develop our mental models for the most part without separating our perceptual data into cross validation and testing, and then later find out they match reality pretty well. I wonder what allows us to make such good inferences without using the standard machine learning approach. $\endgroup$ – yters May 8 '19 at 18:31
  • $\begingroup$ @yters human cognition has close to nothing in common with machine learning, so I don't think that such comparisons are fruitful. $\endgroup$ – Tim May 8 '19 at 19:06
  • $\begingroup$ Yes, the mechanism is very different, but there is the same mathematical question. Given a sample how can we construct a model that has good out of sample accuracy? Whether human cognition or extreme gradient boosting, all the approaches have some way of answering this fundamental question. So, since humans can do it without validation and testing data, seems there is a way of doing it with a computational approach. $\endgroup$ – yters May 8 '19 at 19:47
  • 1
    $\begingroup$ @yters sure. If we knew that, we wouldn't be using cross-validation. If you don't want to overfit, then you can simply choose algorithms that will have worse fit, but be less likely to overfit (i.e. forget about gradient boosting or neural networks), much of old-fashioned statistics is about that. $\endgroup$ – Tim May 8 '19 at 20:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.