3
$\begingroup$

I have a conceptual problem choosing the best strategy for training a neural network. So, let me explain my situation. I have trained an autoencoder on a huge (unlabeled) dataset using train and validation split. Now I would like to extend the architure by fully connected layers to predict a small labelled dataset similar to the approach described here. As input features to the fully connected layers, the encoding part of the autoencoder will be used.

The parameters of the fully connected layers must be optimized. For SVM and Random Forest I would use nested cross-validation to optimize the hyparameters using random or grid search (or just one cross-validation with default parameters). With fully connected layers this is not a good options I think. Usually, I would split the data into train, test and validation set but my dataset consists only of 800 points in class 1 and 500 points in class 2. So train, test, validation split is probably also not a good option.

Using nested-crossvalidation and random search is very time consuming for fully connected layers. Moreover, using cross-validation I cannot observe the training and validation loss anymore and adjust the parameters acoordingly (as I did for tuning the autonecoder). So cross-validation does not work for manual tuning the network.

What is the best strategy to optimize the fully connected layers in my case?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

tl;dr

Since you have a small dataset, I'd go with cross-validation

Reasoning

Usually, small datasets don't come hand in hand with slow computation times. If your dataset is small you can typically use cross-validation to optimize hyperparameters. If it is very large and training times are very slow then you can safely use a train/val/test split.

In particular

Using nested-crossvalidation and random search is very time consuming for fully connected layers.

I disagree with this point. It shouldn't be slow in such a small dataset. I don't think that it is slower let's say than training a SVM.

One reason might be that your model is larger than it has to be (i.e. lots of parameters). If your dataset is small then you shouldn't use very large models because they will be harder to train and will probably be more prone to overfitting.

Another issue might be that your choice of hyperparameters is slowing down the process. Possibly a low learning rate, or maybe too many epochs.

Moreover, using cross-validation I cannot observe the training and validation loss anymore and adjust the parameters acoordingly (as I did for tuning the autonecoder)

I don't get this point either. I'm assuming you refer to things you do during training (e.g. learning rate adjustments). During $k$-fold CV you train your model $k$ times, each is a regular training process. Whatever you do with a train and a validation set you can do in each of the CV folds.

Improve this answer
$\endgroup$
7
  • $\begingroup$ I will go with cross-validation. Yes, I can observe the training and validation loss for each split in the CV but for 10-fold CV this would be 10 splits... not only one split as for one train, validation, test split... And for nested CV this would be even more splits, so for 10 fold nested CV this would be 10 x 10 = 100 splits in total. And to observe the training and validation loss 100 times is a bit too much... $\endgroup$
    – machinery
    Commented Aug 1, 2019 at 12:40
  • $\begingroup$ By nested CV you mean you crossvalidate on both the training and the validation sets? 100 trainings for a single set of hyperparameters is a bit too much, I just hold out the test set and run it once. Furthermore, with so few training data you need 1 or 2 batches per epoch... It shouldn't have a too long training time. $\endgroup$
    – Djib2011
    Commented Aug 1, 2019 at 12:49
  • $\begingroup$ With nested CV I mean doing a 10-fold CV and for each split doing again a CV to estimate the parameters (otherwise you will have a bias). If the inner CV is also 10-fold this will make a total of 100 iterations. $\endgroup$
    – machinery
    Commented Aug 1, 2019 at 17:23
  • $\begingroup$ What do you mean by "I just hold out the test set and run it once"? With 10-fold CV you have 10 different test and validation sets. $\endgroup$
    – machinery
    Commented Aug 1, 2019 at 17:26
  • $\begingroup$ No I mean having a single test set that you use once in the end. Have only one 10-fold CV to select hyperparameters. It should save you a lot of time $\endgroup$
    – Djib2011
    Commented Aug 1, 2019 at 22:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.