For most ML models we say they suffer from high bias or high variance, then we correct for it. However, in DL do neural networks suffer from the same concept in the sense that they initially have high bias or high variance and then you correct through regularization and/or dropout? I would argue they initially suffer from high variance and they overfit the data. Then you correct through regularization, add dropout, image pre-processing in the case of CNNs, etc. Is this train of thought correct?
$\begingroup$
$\endgroup$
1
asked Sep 18, 2020 at 0:09
-
$\begingroup$ Neural nets are initialised with weights close to zero, so you can say they start with high bias/low variance. stopped training again is a form of regularisation (bias decreases as you increase the number of training iterations. $\endgroup$– seanv507Commented Mar 19, 2021 at 16:47
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
1
In general NNs are prone to overfitting the training set, which is case of a high variance. Your train of thought is generally correct in the sense that the proposed solutions (regularization, dropout layers, etc.) are tools that control the bias-variance trade-off.
answered Sep 18, 2020 at 16:03
-
1$\begingroup$ I always enjoy a bit of confirmation bias haha. $\endgroup$ Commented Sep 18, 2020 at 16:47
$\begingroup$
$\endgroup$
Neural networks, including DNNs, don't by themselves suffer from high variance any more than other machine learning algorithms. It is just that we find it easier to start the training with more complex networks and control for variance by the techniques you mentioned, than to start with simpler (less expressive) networks and gradually increase their complexity. By observing the network behaviour during training you can already get hints regarding simplification. If you'd start with a simpler network, you'd be tapping in the dark how to augment it.
