1
$\begingroup$

I'm dealing with a fully connected NN, and I'm wondering if there are any rules of thumb for adjusting hyperparameters for changes to dataset size. For example, if I increase number of obs by 20%, then I should reduce epochs by 20%, or increase batch size by 20%, or decrease learning rate by X%, or whatever...

For context: After hyperparam tuning on a validation or test set, I'm taking my final model, and retraining on all available training data to maximize performance. Since the training data has now slightly increased in size, I want to know if I should make final fine-tune adjustments (which can't be validated) to any part of the model.

If using 10-fold cv or something, then this increase is only 10%, so not a big deal. But two situations come to mind where the increase could be more substantial. 1) Feature space is so big that 10-fold, or even 5-fold could be computationally cost-prohibitive. 2) With time series data out-of-time validation is preferred, which means the validation data always must come after training. So it is not possible to get 10 folds trained on 90% of the data. If you want many "folds", you are likely using 50% or less of training data in each fold.

$\endgroup$
1
  • $\begingroup$ To elaborate on the motivation, I'm primarily interested in the second scenario, using multiple past-to-future splits. This notebook shows a good visualization of how this split would work, and why the scale-up could easily be 2x or more. $\endgroup$ Jan 28, 2021 at 1:08

1 Answer 1

1
$\begingroup$

Most of the time you will be fine if your dataset size increases and you don't change your hyperparameters. However, if your dataset size is increasing, then the risk you overfit your model will decrease. Therefore, if you are less worried about overfitting then:

  • You can reduce the amount of regularization in your model (e.g. tune an L2 regularization hyperparameter lambda).
  • Dropout is less important.
  • Data augmentation is less important.
  • You can consider increasing the capacity of your network by adding layers or adding more nodes per layer.

You mentioned that an increase in dataset size might make the training cost expensive. I would suggest using mini-batch gradient descent instead of batch gradient descent. Choose a mini-bath size of 32. You should converge faster this way.

$\endgroup$
4
  • $\begingroup$ Thanks Brian, thats helpful. But ideally I'm looking for specifics. So for instance if my training data doubles, should I half my dropout rate? What are the optimal number of layers to add to get the most out of the extra signal? The key thing here is that once I am using my full training data set, I no longer have a validation set to tune these parameters. $\endgroup$ Jan 28, 2021 at 0:55
  • $\begingroup$ @PaulFornia, no need to alter your dropout rate, it's independent of the dataset size. As for the number of new layers to add, that is hard to say. I would training multiple models of different sizes and see how big you can go before you start to overfit. Finally, remember that the hyperparameter tuning process helps you choose the hyperparameters for you final model. So even though the hyperparameters were selected on a smaller dataset, no need to change them when you train the final model (with your selected hyperparameters) on the entire training dataset. $\endgroup$
    – Brian_E
    Jan 28, 2021 at 13:54
  • $\begingroup$ you say "Dropout is independent of data size" and "no need to change [hyperparams]". Do you have any evidence to support this? This seems at odds with your original answer that says I should reduce regularization and add layers, and that dropout is less important. "I would run multiple models..." I can't measure overfitting when training on all available data. That is the main point I am trying to get at, please let me know if I can clarify the original question at all. $\endgroup$ Jan 28, 2021 at 15:30
  • $\begingroup$ @PaulFornia You might find section 7.4 Effect on Data Set Size useful. (jmlr.org/papers/volume15/srivastava14a/srivastava14a.pdf) "As the size of the data set is increased, the gain from doing dropout increases up to a point and then declines." Note how they kept the dropout rate constant as they continued to increase the dataset size. In general, I'm not saying you should reduce regularization. Rather that it is something to consider and to experiment with. $\endgroup$
    – Brian_E
    Jan 29, 2021 at 13:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.