Timeline for Random initialization/order in neural network — bias or variance?
Current License: CC BY-SA 3.0
6 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jul 14, 2016 at 6:34 | comment | added | user20160 | I agree w/ all of that. If I understand correctly, variance isn't synonymous w/ overfitting, but variability across training sets drawn from the same distribution, which overfitting contributes to. Seems randomization would contribute too. If randomization contributes to bias, I think this would mean that randomization causes the learned function to differ from the true function in a way that's preserved across training sets drawn from the same distribution. Not seeing why that would be true | |
| Jul 14, 2016 at 4:43 | history | edited | horaceT | CC BY-SA 3.0 |
New thought in light of a comment
|
| Jul 14, 2016 at 4:37 | comment | added | horaceT | @user20160 This' a question of the topology of the objective function. Although the weights are randomly picked, some may lead to much better local optimum than others. There is no guarantee that the expectation with respect to the random starts would come close to the true function. | |
| Jul 14, 2016 at 4:31 | comment | added | horaceT | @user20160 Thanks for making me rethink the question. Let's address the variance part first. Suppose you increase hidden units/hidden layers to infinity. Take a sample data and train an NN, i think you'd agree that the model you get would fit the data very well, ie. overfit. Overfit occurs regardless of where you init your weights. So variance is a function of the hidden units/layers. Now, fix hidden units/layers, draw a sample from the true distribution, pick a random start, train a model, and repeat. Does the expectation of this procedure convert to the true distrib? I think it depends. | |
| Jul 14, 2016 at 2:00 | comment | added | user20160 | I originally upvoted but, after thinking about it some more, I'm not so sure that randomizing contributes to bias. Although randomization is independent of the data, it doesn't seem like it would lead to systematic error in a particular direction. Is there a more detailed argument to be made? It's an interesting question, would be glad to hear your thoughts. | |
| Jul 13, 2016 at 23:21 | history | answered | horaceT | CC BY-SA 3.0 |