Is hyperparameter tuning on sample of dataset a bad idea?

Question

I have a dataset of 140000 examples and 30 features for which I am training several classifiers for a binary classification (SVM, Logistic Regression, Random Forest etc)

In many cases hyperparameter tuning on the whole dataset using either Grid or Random search is too costly time-wise.

I started using the following technique

• Sub sample my dataset
• Use the obtained fraction to tune the hyperparameters on
• Use the obtained parameters to train a model using the whole dataset

To evaluate each set of parameters on the second step I use sklearn's GridSearchCV with cv=10. To evaluate the final model that I create in the third step I use sklearn's cross_val_predict. In that sense I evaluate my models leaving a 10% percent of data out, I train on the rest and measure the predictive accuracy on the 10%, iteratively 10 times, then taking the average of the scores.

What made me worry is that the prediction accuracy I get from training on my whole dataset, is really close to the evaluation I get when tuning the parameters for the best set of parameters (each tested set of parameters outputs a score obtained from averaging 10-fold-cross validation results).

Most of the times the accuracy that cross_val_predict measured using all the training examples (whole dataset) is a little bit above what the evaluation of the best parameters returned.

To illustrate this here is the evaluation of a set of parameters (on a smaller dataset than what I described above but the effect is the same)

Best parameters set found on development set:
{'kernel': 'rbf', 'C': 9, 'gamma': 0.1}
Scores for all sets of parameters
0.851 (+/-0.006) for {'kernel': 'rbf', 'C': 3, 'gamma': 0.5}
0.852 (+/-0.006) for {'kernel': 'rbf', 'C': 3, 'gamma': 0.1}
0.829 (+/-0.006) for {'kernel': 'rbf', 'C': 3, 'gamma': 0.001}
0.853 (+/-0.006) for {'kernel': 'rbf', 'C': 9, 'gamma': 0.1}
...

And here are the averaged scores (from cross_val_predict) I got from training on my whole dataset using the best parameters

precision    recall  f1-score   support

      0       0.86      0.85      0.86     15417
      1       0.86      0.87      0.87     16561

avg / total       0.86      0.86      0.86     31978

acc score: 0.863750078179
roc au score: 0.863370490059
[[13147  2270]
 [ 2087 14474]]

As you can see training on the whole dataset improves the results. I have also validated that badly tuned model (e.g. using the default values or random values for C and gamma) leads to much worse prediction accuracy.

Overall I think that tuning the hyperparameters on a subset is not ideal but can potentially lead to relatively good results without having to wait too long. I for example before using that approach used optunity package for tuning the hyperparameter on the whole dataset. This procedure would take 3-5 days to complete and would produce results that either had really good precision or really good recall but not both, so although for each class either the precision or the recall was really high (higher than what any of my other classifiers had achieved) the f1 meassure was really low. In the contrary using the later approach leads to some hours of training and a better f1 meassure.

My concerns are:

Do I limit my classification accuracy? Do I avoid using all the prediction power that my dataset can offer by tuning only on a subset? If such a harm of performance is happening is it somehow limited by some factor?

Answer 1

In addition to Jim's (+1) answer: For some classifiers, the hyper-parameter values are dependent on the number of training examples, for instance for a linear SVM, the primal optimization problem is

$\mathrm{min} \frac12\|w\|^2 + C\sum_{i=1}^\ell \xi_i$

subject to

$y_i(x_i\cdot w _ b) \geq 1 - \xi_i, \quad \mathrm{and} \quad \xi_i \geq 0 \quad \forall i$

Note that the optimisation problem is basically a measure of the data mis-fit term (the summation over $\xi_i$) and a regularisation term, but the usual regrularisation parameter is placed with the data misfit term. Obviously the greater the number of training patterns we have, the larger the summation will be and the smaller $C$ ought to be to maintain the same balance with the magnitude of the weights.

Some implementations of the SVM reparameterise as

$\mathrm{min} \frac12\|w\|^2 + \frac{C}{\ell}\sum_{i=1}^\ell \xi_i$

in order to compensate, but some don't. So an additional point to consider is whether the optimal hyper-parameters depend on the number of training examples or not.

I agree with Jim that overfitting the model selection criterion is likely to be more of an issue, but if you have enough data even in the subsample then this may not be a substantial issue.

Answer 2

Is hyperparameter tuning on sample of dataset a bad idea?

A: Yes, because you risk overfitting (the hyperparameters) on that specific test set resulting from your chosen train-test split.

Do I limit my classification accuracy?

A: Yes, but common machine learning wisdom is: with your optimal hyperparameters, say $\lambda^*$, refit your model(s) on the whole dataset and make that model your final model for new, unseen, future cases.

Do I avoid using all the prediction power that my dataset can offer by tuning only on a subset?

A: see previous answer.

If such a harm of performance is happening is it somehow limited by some factor?

A: idem.

I measure my accuracy using 10-fold cross as I use to also evaluate the parameters

A: Note that this is different from what is asked in the title. 10-fold CV iterates over 10 test-train splits to arrive at an "unbiased" (less-biased) estimate of generalizability (measured in this case by accuracy). 10-fold CV exactly addresses the issue I talk about in the first answer.

the prediction accuracy I get from training on my whole dataset

A: this is an "in-sample" measure that could be optimistically biased. But don't forget that you have many cases and relatively few features, so that this optimism bias may not be an issue. Machine learning nugget: "the best regularizer is more data."

[cont'd], is always really close to the evaluation I get when tuning the parameters for the best set of parameters.

A: see previous answer. Look at the hyperparameter plots: does tuning decrease error and by how much? From what you are saying, the tuning is not doing much.

You could test this as follows. Take a 70%-30% train-test split. Compare predictive performance of:

1. an untuned model trained on the train set,
2. a 10-fold-CV tuned model trained on the train set.

Let both models predict the test set. If performance is very close, then tuning is not doing much. If performance is different in favor of the tuned model, then continue with the tuning approach.

Answer 3

I'll answer for artificial neural networks (ANNs).

The hyperparameters of ANNs may define either its learning process (e.g., learning rate or mini-batch size) or its architecture (e.g., number of hidden units or layers).

Tuning architectural hyperparameters on a subset of your training set is probably not a good idea (unless your training set really lacks diversity, i.e. increasing the training set size doesn't increase the ANN performance), since architectural hyperparameters change the capacity of the ANN.

I would be less concerned tuning the hyperparameters that define the learning process on a subset of your training set, but I guess one should validate it empirically.

Answer 4

This paper is about the topic of taking other/smaller datasets for the tuning of bigger datasets: https://papers.nips.cc/paper/5086-multi-task-bayesian-optimization.pdf

I think it is not a bad idea in contrast to what Jim said.

Answer 5

You can use hyperparameter optimization algorithms which support multifidelity evaluations, i.e., evaluations on sub-sets of your data in order to get a rough but useful estimate about optimal hyperparameter values for the entire dataset. Such approaches typically allow to the reduce the total computational cost needed to run hyperparameter optimization.

Answer 6

You can take a look at https://link.springer.com/chapter/10.1007/978-3-319-53480-0_27 in which we've investigated the effects of random sampling on SVM hyper-parameter tuning using 100 real-world datasets...