Why not just dump the neural networks and deep learning? [closed]

Question

Fundamental problem with deep learning and neural networks in general.

1. The solutions that fit training data are infinite. We don't have precise mathematical equation that is satisfied by only a single one and that we can say generalizes best. Simply speaking we don't know which generalizes best.

2. Optimizing weights is not a convex problem, so we never know we end up with a global or a local minimum.

So why not just dump the neural networks and instead search for a better ML model? Something that we understand, and something that is consistent with a set of mathematical equations? Linear and SVM do not have this mathematical drawbacks and are fully consistent with a a set of mathematical equations. Why not just think on same lines (need not be linear though) and come up with a new ML model better than Linear and SVM and neural networks and deep learning?

Answer 1

1. Not being able to know what solution generalizes best is an issue, but it shouldn't deter us from otherwise using a good solution. Humans themselves often do not known what generalizes best (consider, for example, competing unifying theories of physics), but that doesn't cause us too many problems.

2. It has been shown that it is extremely rare for training to fail because of local minimums. Most of the local minimums in a deep neural network are close in value to the global minimum, so this is not an issue. source

But the broader answer is that you can talk all day about nonconvexity and model selection, and people will still use neural networks simply because they work better than anything else (at least on things like image classification).

Of course there are also people arguing that we shouldn't get too focused on CNNs like the community was focused on SVMs a few decades ago, and instead keep looking for the next big thing. In particular, I think I remember Hinton regretting the effectiveness of CNNs as something which might hinder research. related post

Answer 2

As the comments to your question point out, there are a lot of people working on finding something better. I would though like to answer this question by expanding the comment left by @josh

---

All models are wrong but some are useful (Wiki)

The above statement is a general truth used to describe the nature of statistical models. Using data that we have available, we can create models that let us do useful things such as approximate a predicted value.

Take for example Linear Regression

Using a number of observations, we can fit a model to give us an approximate value for a dependent variable given any value(s) for the independent variable(s).

Burnham, K. P.; Anderson, D. R. (2002), Model Selection and Multimodel > Inference: A Practical Information-Theoretic Approach (2nd ed.):

"A model is a simplification or approximation of reality and hence will not reflect all of reality. ... Box noted that “all models are wrong, but some are useful.” While a model can never be “truth,” a model might be ranked from very useful, to useful, to somewhat useful to, finally, essentially useless."

Deviations from our model (as can be seen in the image above) appear random, some observations are below the line and some are above, but our regression line shows a general correlation. Whilst deviations in our model appear random, in realistic scenarios there will be other factors at play which cause this deviation. For example, imagine watching cars as they drove through a junction where they must turn either left or right to continue, the cars turn in no particular pattern. Whilst we could say that the direction the cars turn is completely random, does every driver reach the junction and at that point make a random decision of which way to turn? In reality they are probably heading somewhere specific for a specific reason, and without attempting to stop each car to ask them about their reasoning, we can only describe their actions as random.

Where we are able to fit a model with minimal deviation, how certain can we be that an unknown, unnoticed or immeasurable variable wont at some point throw our model? Does the flap of a butterfly’s wings in Brazil set off a tornado in Texas?

The problem with using the Linear and SVN models you mention alone is that we are somewhat required to manually observe our variables and how they each affect each other. We then need to decide what variables are important and write a task-specific algorithm. This can be straight forward if we only have a few variables, but what if we had thousands? What if we wanted to create a generalised image recognition model, could this realistically be achieved with this approach?

Deep Learning and Artificial Neural Networks (ANNs) can help us create useful models for huge data sets containing huge amounts of variables (e.g. image libraries). As you mention, there's an incomprehensible number of solutions which could fit the data using ANNs, but is this number really any different to the amount of solutions we would need to develop ourselves through trial and error?

The application of ANNs do much of the work for us, we can specify our inputs and our desired outputs (and tweak them later to make improvements) and leave it up to the ANN to figure out the solution. This is why ANNs are often described as "black boxes". From a given input they output an approximation, however (in general terms) these approximations don't include details on how they were approximated.

And so it really comes down to what problem you are trying to solve, as the problem will dictate what model approach is more useful. Models are not absolutely accurate and so there is always an element of being 'wrong', however the more accurate your results the more useful they are. Having more detail in the results on how the approximation was made may also be useful, depending on the problem it may even be more useful than increased accuracy.

If for example you are calculating a persons credit score, using regression and SVMs provides calculations that can be better explored. Being able to both tweak the model directly and explain to customers the effect separate independent variables have on their overall score is very useful. An ANN may aid in processing larger amounts of variables to achieve a more accurate score, but would this accuracy be more useful?

Answer 3

The global minimum may as well as be useless, so we don't really care if we find it or not. The reason is that, for deep networks, not only the time to find it becomes exponentially longer as the network size increases, but also the global minimum often corresponds to overfitting the training set. Thus the generalization ability of the DNN (which is what we really care about) would suffer. Also, often we prefer flatter minima corresponding to a higher value of the loss function, than sharper minima corresponding to a lower value of the loss function, because the second one will deal very badly with uncertainty in the inputs. This is becoming increasingly clear with the development of Bayesian Deep Learning. Robust Optimization beats Determinist Optimization very often, when applied to real world problems where uncertainty is important.

Finally, it's a fact that DNNs just kick the ass of methods such as XGBoost at image classification and NLP. A company which must make a profit out of image classification will correctly select them as models to be deployed in production (and invest a significant amount of money on feature engineering, data pipeline, etc. but I digress). This doesn't mean that they dominate all the ML environment: for example, they do worse than XGBoost on structured data (see the last winners of Kaggle competitions) and they seem to not still do as well as particle filters on time series modelling. However, some very recent innovations on RNNs may modify this situation.

Answer 4

I think the best way to think about this question is through the competitive market place. If you dump deep learning, and your competitors use it, AND it happens to work better than what you used, then you'll be beaten on the market place.

I think that's what's happening, in part, today, i.e. deep learning seems to work better than anything for the whole lot of problems on market place. For instance, online language translators using deep learning are better than the purely linguistic approaches that were used before. Just a few years ago this was not the case, but advances in deep learning brought those who used to the leadership positions on the market.

I keep repeating "the market" because that's what's driving the current surge in deep learning. The moment business finds something useful, that something will become wide spread. It's not that we, the committee, that decided that deep learning should be popular. It's business and competition.

The second part, is that in addition to actual success of ML, there's also fear to miss the boat. A lot of businesses are paranoid that if they miss out on AI, they'll fail as businesses. This fear is being fed by all these consulting houses, Gartners etc., whispering to CEOs that they must do AI or die tomorrow.

Nobody's forcing businesses to use deep learning. IT and R&D are excited with a new toy. Academia's cheering, so this party's going to last until the music stops, i.e. until deep learning stops delivering. In the meantime you can dump it and come up up with a better solution.

Answer 5

There are excellent answers, mostly weighing in with the usefulness of DL and ANNs. But I would like to object the OP in a more fundamental way, since the question already takes for granted the mathematical inconsistency of neural networks.

First of all, there is a mathematical theory behind (most models of) Neural Networks. You could likewise argue that linear regression does not generalize, unless the underlying model is... well, linear. In neural algorithms, a model is assumed (even if not explicitly) and the fitting error is computed. The fact that algorithms are modified with various heuristics does not void the original mathematical support. BTW, local optimization is also a mathematically consistent, let alone useful, theory.

Along this line, if Neural Networks just constitute one class of methods within the whole toolbox of scientists, which is the line that separates Neural Networks from the rest of techniques? In fact, SVMs were once considered a class of NNs and they still appear in the same books. On the other hand, NNs could be regarded as a (nonlinear) regression technique, maybe with some simplification. I agree with the OP that we must search better, well founded, efficient algorithms, regardless you label them as NNs or not.

Answer 6

I guess for some problem we care less for the mathematical rigor and simplicity but more for its utility, current status is neural network is better in performing certain task like pattern recognition in image processing.

Answer 7

There is a lot in this question. Lets go over what you've wrote one by one.

The solutions that fit training data are infinite. We don't have precise mathematical equation that is satisfied by only a single one and that we can say generalizes best.

The fact that there are infinite many solutions comes from learning problem being an ill-posed problem so there cannot be a single one that generalizes best. Also, by no free lunch theorem whichever method we use cannot guarantee that it is the best across all learning problems.

Simply speaking we don't know which generalizes best.

This statement is not really true. There are theorems on empirical risk minimization by Vapnik & Chervonenkis that connect the number of samples, VC dimension of the learning method and the generalization error. Note, that this only applies for a given dataset. So given a dataset and a learning procedure we know the bounds on generalization. Note that, for different datasets there are no and cannot be single best learning procedure due to no free lunch theorem.

Optimizing weights is not a convex problem, so we never know we end up with a global or a local minimum. So why not just dump the neural networks and instead search for a better ML model?

Here there are few things that you need to keep in mind. Optimizing non-convex problem is not as easy as convex one; that is true. However, the class of learning methods that are convex is limited (linear regression, SVMs) and in practice, they perform worse than the class of non-convex (boosting, CNNs) on a variety of problems. So the crucial part is that in practice neural nets work best. Although there are a number of very important elements that make neural nets work well:

1. They can be applied on very large datasets due to stochastic gradient descent.
2. Unlike SVMs, inference with deep nets does not depend on the dataset. This makes neural nets efficient at test time.
3. With neural nets it is possible to directly control their learning capacity (think of number of parameters) simply by adding more layers or making them bigger. This is crucial since for different datasets you might want bigger or smaller models.

Something that we understand, and something that is consistent with a set of mathematical equations? Linear and SVM do not have this mathematical drawbacks and are fully consistent with a a set of mathematical equations. Why not just think on same lines (need not be linear though) and come up with a new ML model better than Linear and SVM and neural networks and deep learning?

Dumping things that work because of not understanding them is not a great research direction. Making an effort in understanding them is, on the other hand, great research direction. Also, I disagree that neural networks are inconsistent with mathematical equations. They are quite consistent. We know how to optimize them and perform inference.

Answer 8

How about viewing neural networks from an experimental point of view? Just because we created them doesn't mean that we're obligued to understand them intuitively. Or that we're not allowed to play with them in order to have a better grasp of what they're doing.

Here's a couple of thoughts I have on them:

• Structure: they are hierarchies. They are like trees that share inputs. The roots are the inputs and the leafs are the output layer. The closer the layer is to the outputs, the more relevant it is to them, the greater level of abstraction It contains (it's more about the picture than the pixels).
• Functionality: they "play" with data, the modus operandi is to experiment with relationships in neurons (weights) until things "click" (the error margin is acceptable).

This is consistent with how we think. It's even consistent with how the scientific method operates. So by cracking neural networks we may also be solving the general question of what knowledge represents.

Answer 9

Don't forget, there is a vast field of research that use LMs, GLM, multilevel modelling. Lately Bayesian techniques and Hamiltonian Monte Carlo(the STAN community is really at the forefront of this) have come of age and a number of problems that are solved by STAN really easily and don't really need NNs or deep nets. Social Science research, Microeconomics are two(large) examples of such fields adopting Stan rapidly.

Stan models are very "readable". The coefficients actually have a posterior distributional interpretation and so do the predictions. The priors are part of the data generating process and don't need to be conjugate to be performant(like gibbs). The model fitting in stan is a delight, it actually tunes the pesky MCMC params automatically pretty darn well and warns you when the exploration is stuck with really nice visualizations.

If you haven't tried it already see awesome stan demos here).

At the end of the day I think people don't talk about this stuff so much because the research in this field and the problems are not so "sexy"/"cool" as with NNs.

Answer 10

What typically happens when there is no mathematical consistency (atleast in this case of neural networks)...when its not giving results as desired, on the test set, your boss will come back and say...Hey why don't you try Drop out (which weights,which layer, how many is your headache as there isn't mathematical way to determine), so after you try and hopefully got a marginal improvement but not the desired, your boss will come back and say, why not try weight decay(what factor?)? and later, why don't you try ReLU or some other activation on some layers, and still not, why not try 'max pooling'? still not, why not try batch normalization, still not, or atleast convergence, but not desired result, Oh you are in a local minimum, try different learning rate schedule, just change the network architecture? and repeat all above in different combinations! Keep it in a loop until you succeed!

On the other hand, when you try a consistent SVM, after convergence, if the result is not good, then okay, the linear kernel we are using is not good enough as the data may not be linear, use a different shaped kernel, try a different shaped kernel if you have any hunch, if still not, just leave it, its a limitation of SVM.

What I am saying is ,the neural networks being so inconsistent, that it is not even wrong! It never accepts its defeat! The engineer/designer takes the burden, in case it does not work as desired.