2
$\begingroup$

Re-posting this from scicomp based on recommendation in the comments.

(I'm a total ML noob, apologies if I've worded things badly, used the wrong terms or posted in the wrong SE site!)

I've been watching Magnus Erik Hvass Pederson's Tensorflow tutorials on YouTube.

One of the things I was previously confused about was where the set of features a convnet detects comes from, however after watching these videos I now believe these are learned; starting off by being initialised randomly.

In Tutorial #13 he shows a way of visualising what features are being detected by each layer (eg.: 9:36):

Detected features

My question is; if these things are all initialised randomly and fed the same input data, what's to stop these features ending up the same? (Or another way, what ensures they end up reasonably distributed to detect all the different things required?)

$\endgroup$
1
$\begingroup$

however after watching these videos I now believe these are learned; starting off by being initialised randomly.

correct

If these things are all initialised randomly and fed the same input data, what's to stop these features ending up the same?

Since the weights are initialized randomly, they'll receive different weight updates during backpropagation. If the weights were initialized to 0, then they'll all stay the same during the training phase, which we don't want.

So if the only difference between them is the initial random input, does this mean training with same params and the same training data twice might result in two completely different results (and that one might pick up great features and work really well and another pick up poor/duplicated features and perform terribly)?

In theory, yes. In practice, for image classification, the learnt features tend to be quite similar. But feature 1 of your neural network 1 could (loosely) correspond to feature 29 of your neural network 2. Since ensembling a set of trained neural networks often yield higher performances than a single trained neural network, this means that the learnt features aren't exactly the same.

Wouldn't we be better initialising with some known good starting points (shapes, patterns, etc.) instead of random noise?

I guess they must have been some papers published about it in computer vision, but I don't have any off the top of my head. In natural language processing, it is common to initialize word vectors with pre-trained word vectors. Sometimes it helps, sometimes not.

$\endgroup$
  • $\begingroup$ So if the only difference between them is the initial random input, does this mean training with same params and the same training data twice might result in two completely different results (and that one might pick up great features and work really well and another pick up poor/duplicated features and perform terribly)? Wouldn't we be better initialising with some known good starting points (shapes, patterns, etc.) instead of random noise? $\endgroup$ – Danny Tuppeny Nov 20 '16 at 17:04
  • 1
    $\begingroup$ I managed to find this which has a section on "Weight Initialization" which talks about this a little too $\endgroup$ – Danny Tuppeny Nov 20 '16 at 17:34
  • 1
    $\begingroup$ If ensembling yields higher performances then doesn't that mean that the nets are not converging to similar learned features? ensembling doesn't help that much if two models work in similar ways $\endgroup$ – Hugh Nov 20 '16 at 18:25
  • 2
    $\begingroup$ @Hugh this is correct, I just wanted to point out that learnt features aren't exactly the same. $\endgroup$ – Franck Dernoncourt Nov 20 '16 at 18:34
  • 1
    $\begingroup$ @Danny The whole "pre-training" idea using e.g. RBM/DBN or autoencoders is nothing else then initializing a network with "known" good starting points. You train a DBN, and use these weights as starting points for training the feedforward neural network. $\endgroup$ – hbaderts Nov 20 '16 at 20:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.