1
$\begingroup$

With techniques such as batch normalization and Xavier normal initialization, is it still necessary to pretrain the neural network to get a better normalization for unsupervised task?

In here, the pretraining procedure refers to training the model in an unsupervised way to force the neurons to have a good representation of the data without the labels.

My model overfits heavily, does pretraining help with reduce the overfitting here?

$\endgroup$
1
$\begingroup$

No, pretraining was mostly used in the days when sigmoid was the most used activation function. Since it suffered from vanishing gradient problem, unsupervised pretraining helped a lot. With ReLUs you don't have such problem, so pretraining is just an artifact of the past.

Also most of the time weights learned through pretraining were not optimal for the task. For example: lets say you want to build acats vs dogs classifier. Probably you're going to have a lot of different backgrounds in your dataset, but the background doesn't tell you anything about the kind of animal that is represented. However when you're pretraining using a reconstructing autoencoder some of your neurons HAVE to represent background for the reconstruction to be faithful. So you're wasting some of your network architecture for things that do not help you in any way with your task.

As for overfitting: just apply batch norm/dropout and you should be good.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.