From what I understand, a one layer network (no hidden layers) is exactly the same as logistic regression.
Now suppose I have a function $f(x)$ that is exactly a logistic regression function with say d features. If I insist on running a neural network with say 3 layers with d/3 neurons each, is this a bad idea? That is, will it:
- Take much longer to train such a network?
- The final optimum weights produce a function that is not a great approximation to the original function? (Note the regression model reproduces f(x) exactly).
This would be a hypothetical example of a situation where naively using deep learning is worse than shallow learning so to say.