How to pretrain Convolution filter

Question

I was implementing convolutional neural network, For classification of natural images like face, car, flower etc of about 10 categories. I read(from Andrew NG notes) that pre trained convolutional filter are much efficient and less resource consuming way to implement cnn. We know that one way to pretrain conv filter is to use sparse autoencoder.

I trained the first layer by input of 28x28x3 image patches with 100 hidden units.

Then I convoluted the image with those feature map.

similar as above I used 28*28*100 (new images of 100 depth or hidden units) image patches as input and performance sparse auto encoder, but it did not generate any useful feature map.

Is the approach what I applied correct. Is there any other way I can pretrain feature map or conv filter. please help, looking for some solution.

Answer 1

As far as I know, pre-training is not so popular nowadays.

• Try to use proper initialisation like this or this.
• Use Batch Normalisation (it will make the previous point less important, though). You will get more in terms of accuracy, but with some computational overhead.

P.S.

Now when I wrote this, I found a great answer on Cross Validated.