8
$\begingroup$

For model building I typically apply the following process (which I am simplifying somewhat here for brevity):

  1. Split the data into test and training
  2. Use cross-validation on my training data to find the best model parameters relative to some performance metric (accuracy, ROC etc) and pick this as my best model
  3. Evaluate my model on the test data

So far so good.

One problem that arises is that my test data is often valuable to me. Suppose I am predicting the value of a stock. On the one hand, I want my test data to be the most recent data so that I can see how the model performs on recent stock prices. On the other hand, I want to fit my model on my latest stock price data because that is the most relevant data for predicting tomorrows stock price.

But After applying 3. above, I have a model that is not using the test data to fit the model.

My question is, is it legitimate (or standard practice) to refit the model with the final tuning parameters to the entire data set as a final fourth step. For example, suppose in 2. my best parameter for a random forest model is mtry = 5. So at the end of 3. I refit the entire model with mtry = 5. Note, using seeds in R I am able to ensure the best model I picked in 2. will be identical in form to the model I fit in this final step i.e. I do not want to refit the final model with 5 new random features - I want to use the same features as the model I picked in step 2.

Thoughts please. Is this good / bad? Does it violate some fundamental principal in machine learning approach?

$\endgroup$

2 Answers 2

5
$\begingroup$

Yes,

Of course, the accuracy you should report is from training with the test set held out. But now it's time to make predictions and you want them to be as good as they can be. This is certainly legitimate and important for exactly the reasons you mention. Your most recent data is what you want to test against and what you want final predictions to be made based on. There are some things worth being careful of.

In some machine learning algorithms, parameters are sensitive to the size of your training data for instance, in k-Nearest-Neighbour Regression, the optimal $k$ grows with $n$. So you may want to cross validate with several training sizes and extrapolate to find the best $k$ for your full training data.

Another example is if you are using a regularized GLM, with an objective function of the form $f(\theta) = L(\theta) + \lambda R(\theta)$ where $L$ is your loss and $R$ is your regularization. If you Total Logloss as $L$ this will be sensitive to training set size, whereas Average Logloss will not. So using average logloss makes the $\lambda$ you establish in CV usable with more data.

$\endgroup$
1
  • $\begingroup$ thanks for the clarification. Good point I had not thought about relationship between parameters and training size. I will have to give this a bit of thought. But I get the general idea. $\endgroup$
    – Ofir
    Sep 25, 2015 at 2:21
1
$\begingroup$

It depends on the model you're using. Unless you're limiting yourself to a simple class of convex models/loss functions, you're considerably better off keeping a final test split. Here's why:

Let's say you collect iid sample pairs from your data generating distribution, some set of (x, y). You then split this up into a training and test set, and train a model on the training set. Out of that training process you get a model instance, f(x; w). Where w denotes the model parameters.

Let's say you have N observations in the test set. When you validate this model on that test set you form the set of test predictions, {f(x_i, w) : i=1,2,...,N} and compare it to the set of test labels {y_i : i=1,2,...,N} using a performance metric.

What you're able to say using N independent observations is how you expect that model instance, i.e. the function given a specific w, will generalize to other iid data from the same distribution. Importantly, you only really have one observation (that w you found) to comment on your process for determining f(x, w), i.e. the training process. You can say a little more using something like k-fold cross validation, but unless your willing to do exhaustive cross-validation (which is not really feasible in many ML settings), you'll always have less data on the reliability of your training process.

Take a pathological example, where you draw the model parameters at random, and you don't train them at all. You obtain some model instance f(x, w_a). Despite the absurdity of your (lack of) training process, your test set performance is still indicative of how that model instance will generalize to unseen data. Those N observations are still perfectly valid to use. Maybe you'll have gotten lucky and have landed on a pretty good w_a. However, if you combine the test and training set, then "retrain" the model to obtaining a w_b, you're in trouble. The results of your previous test performance amounts to basically a point estimate of how well your next random parameter draw will fare.

There are statistical results that you can use to comment on the reliability of the entire training process. But they require some assumptions about your model class, loss function, and your ability to find the best f(x, w) from within that class for any given set of training observation. With all that, you can get some bounds on the probability that your performance on unseen data will deviate by more that a certain amount from what you measured on the training data. However, those results do not carry over (in a useful way) to overparameterized and non-convex models like modern neural networks.

The pathological example above is a little over the top. But as an ML researcher and consultant, I have seen neural network training pipelines that occasionally latch on to terrible local minima, but otherwise perform great. Without a final test split, you'd have no way of being sure that hadn't happened on your final retraining.

More generally, in a modern machine learning context, you cannot treat the models coming out of your training process as interchangeable. Even if they do perform similarly on a validation set. In fact, you may see considerable variation from one model to the next when using the full bag of stochastic optimization tricks. (For more details on that, check out this work on underspecification.)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.