If computing power is not an issue, then are there any downsides to having tons of hidden layers in the neural network? Does this lead to overfitting? Can a really good neural network be created for classification with like a hundred hidden layers?
 A: Using 'old school' techniques for building neural networks, there are many problems with making deep networks. Nowadays, there are many modern techniques designed to address these.
In no particular order


*

*Vanishing gradients: As the gradient is fed through layer after layer, and some neurons have saturated activations, the gradients get smaller and smaller. This make the pace of learning for early layers very small. Sometimes the opposite can happen and the gradients explode. Residual and/or skip connections have largely overcome this problem.

*Overfitting: As you add more parameters to your model, you may end up training something that works really well in sample, but does not generalize out of sample. This is because you have 'fit the noise'. Techniques such as L1 and L2 regularization and drop out help address this problem.

*Internal Covariate Shift: Each layer can be thought of (loosely) as trying to learn from the layer before it. But as you train your model, the distribution of activations of earlier layers keep changing, and then later layers are trying to learn from a moving target.: Techniques such as batch normalization help to reduce this problem.


There are plenty of others, but those are three big ones.
A: In addition to the good points covered by the answer above, it is worth going back to basics and considering the needed complexity of the predictive signal.  It is amazing how often an additive model competes well with regard to predictive discrimination.  In other words, non-additive effects (interactions) are often weaker than main effects.  In many problems, lack of fit is caused more by nonlinear effects of predictors than by non-additive effects.  In the neural network framework this relates to how many layers are really needed for the problem at hand.
The use of prior knowledge results in better predictions.  For example, one often should  not "waste parameters" by allowing interactions to be as important as main additive effects.  A Bayesian modeler would put a prior on interaction effects that reflects this.  Putting more emphasis on effects likely to be stronger results in less overfitting and better predictions.
