Disadvantages of Neural network method I want to list the advantages and disadvantages of Neural network methods  for classification or estimation purposes. I have already found the advantages of NN method in many papers. But they didn't write anything special about disadvantages of NN method as it was their own proposed method ;) . I guess they have some disadvantages too. For example as we know the performance of NN method is based on some settings, like number of hidden layers nodes for MLP neural network, Also there is no analytical method to fine-tune these settings except trial and error. Am I right? If yes, Do you guys have some references about this disadvantage? Please give some strong references(Books or papers).  
Please also do tell if there are other disadvantages exist and provide some references for them.
 A: Neural nets have many advantages, but here are some disadvantages:
Large number of hyperparameters. This includes network architecture (how many layers, layer size, layer type), activation function for each layer, optimization algorithm, regularization methods, initialization method, and many associated hyperparameters for each of these choices. Hyperparameters can strongly interact with each other to affect performance. Good settings are highly problem dependent, and there typically isn't a clear, obvious choice. Hyperparameters are typically chosen using some combination of intuition/experience, copying past work, manual tweaking, and black box hyperparameter optimization algorithms. Training large neural nets on large problems takes a long time, so tuning the hyperparameters can be very laborious and/or computationally expensive.
Non-convex loss functions. Optimization for neural nets is trickier than for methods with convex loss functions. We don't get the nice guarantees that come with convex optimization. Additionally, we have to deal with issues like local minima and saddle points, which can trap the optimization algorithm at bad solutions. Optimization for neural nets is an active research topic, and much work has gone into trying to mitigate these issues.
Black box model. Neural nets implement complicated, nonlinear functions that are not straightforward to interpret. This is fine if we only care about the output (e.g. prediction in classification and regression problems). It's a disadvantage if we want to understand something about the mapping between input and output. Some methods for peering into the black box do exist.
Little theoretical understanding. Neural nets are less well understood than other methods, from a theory standpoint. Naturally, this isn't an inherent disadvantage of neural nets, but of our current understanding. Hopefully, this will improve with time.
