2
$\begingroup$

This question already has an answer here:

From what I understand, a one layer network (no hidden layers) is exactly the same as logistic regression.

Now suppose I have a function $f(x)$ that is exactly a logistic regression function with say d features. If I insist on running a neural network with say 3 layers with d/3 neurons each, is this a bad idea? That is, will it:

  1. Take much longer to train such a network?
  2. The final optimum weights produce a function that is not a great approximation to the original function? (Note the regression model reproduces f(x) exactly).

This would be a hypothetical example of a situation where naively using deep learning is worse than shallow learning so to say.

$\endgroup$

marked as duplicate by kjetil b halvorsen, Reinstate Monica, Michael Chernick, usεr11852 says Reinstate Monic, jbowman Mar 31 at 0:52

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • $\begingroup$ With small sample sizes use logistic regression, neural network needs huge sample size! $\endgroup$ – kjetil b halvorsen Apr 12 '17 at 12:06
  • $\begingroup$ So if I have less amount of data, you are saying it is better to use logistic regression (i.e. a shallow neural network) with few features than deep neural networks with the same number of features ? Is this to avoid overfitting perhaps ? $\endgroup$ – Femto Apr 12 '17 at 12:10
  • $\begingroup$ a) It won't unless $d$ is large ($10^3$ and more) b) It is a much more complex model so it may overfit. (But only if you add noise to your data, otherwise it will give same result) Also i think you meant to say weights, or parameters (of neural network) rather than features - features are your inputs $\endgroup$ – Łukasz Grad Apr 12 '17 at 12:25
  • $\begingroup$ Related: stats.stackexchange.com/questions/43538/… $\endgroup$ – Ferdi Apr 12 '17 at 13:08