I have a conceptual problem choosing the best strategy for training a neural network. So, let me explain my situation. I have trained an autoencoder on a huge (unlabeled) dataset using train and validation split. Now I would like to extend the architure by fully connected layers to predict a small labelled dataset similar to the approach described here. As input features to the fully connected layers, the encoding part of the autoencoder will be used.
The parameters of the fully connected layers must be optimized. For SVM and Random Forest I would use nested cross-validation to optimize the hyparameters using random or grid search (or just one cross-validation with default parameters). With fully connected layers this is not a good options I think. Usually, I would split the data into train, test and validation set but my dataset consists only of 800 points in class 1 and 500 points in class 2. So train, test, validation split is probably also not a good option.
Using nested-crossvalidation and random search is very time consuming for fully connected layers. Moreover, using cross-validation I cannot observe the training and validation loss anymore and adjust the parameters acoordingly (as I did for tuning the autonecoder). So cross-validation does not work for manual tuning the network.
What is the best strategy to optimize the fully connected layers in my case?