I am training a deep net with greedy layer-wise pre-training (as in the algorithm in Bengio's "Practical Recommendations"). At some point, I have to keep only a few top candidate networks according to their performance in the classification task. Bengio suggests to rank the networks using a simple linear classifier (in order to evaluate the network quickly).
I've concerns about how to train this simple model. I.e., if I train an extra supervised layer for a few epochs with SGD, accuracy will depend on the arbitrary initialization of the weights and not only on the network itself. Also, it takes considerable time to train even simple models using SGD. Ideally, I'd like to use a more deterministic approach (SVM + convex optimization), but this would require loading all training data into memory, which is unfeasible in my case. As a compromise, I could train minibatches with deterministic approach and then average evaluations of each sub-model, but this might have 2 problems: 1) it depends on the arbitrary sampling of minibatches and 2) the train minibatches could not be representative enough. Any suggestion on how to quickly and un-biasedly evaluate network performance for classification task?