From what I understand, a one layer network (no hidden layers) is exactly the same as logistic regression.

Now suppose I have a function $f(x)$ that is exactly a logistic regression function with say d features. If I insist on running a neural network with say 3 layers with d/3 neurons each, is this a bad idea? That is, will it:

  1. Take much longer to train such a network?
  2. The final optimum weights produce a function that is not a great approximation to the original function? (Note the regression model reproduces f(x) exactly).

This would be a hypothetical example of a situation where naively using deep learning is worse than shallow learning so to say.

  • $\begingroup$ With small sample sizes use logistic regression, neural network needs huge sample size! $\endgroup$ Commented Apr 12, 2017 at 12:06
  • $\begingroup$ So if I have less amount of data, you are saying it is better to use logistic regression (i.e. a shallow neural network) with few features than deep neural networks with the same number of features ? Is this to avoid overfitting perhaps ? $\endgroup$
    – Femto
    Commented Apr 12, 2017 at 12:10
  • $\begingroup$ a) It won't unless $d$ is large ($10^3$ and more) b) It is a much more complex model so it may overfit. (But only if you add noise to your data, otherwise it will give same result) Also i think you meant to say weights, or parameters (of neural network) rather than features - features are your inputs $\endgroup$ Commented Apr 12, 2017 at 12:25
  • $\begingroup$ Related: stats.stackexchange.com/questions/43538/… $\endgroup$
    – Ferdi
    Commented Apr 12, 2017 at 13:08