# Neural networks vs everything else

Of course if the data I have is of the order of millions then deep learning is the way.

And I have read that when I do not have big data then maybe it is better to use other methods in machine learning. The reason given is over-fitting. Machine learning: i.e. looking at data, feature extractions, crafting new features from what is collected etc. things such as removing heavily correlated variables etc. the whole machine learning 9 yards.

And I have been wondering: why is it that the neural networks with one hidden layer are not panacea to machine learning problems? They are universal estimators, over-fitting can be managed with dropout, l2 regularization, l1 regularization, batch-normalization. Training speed is not generally an issue if we have just 50,000 training examples. They are better at test time than, let us say, random forests.

So why not - clean the data, impute missing values as you would generally do, center the data, standardize the data, throw it to an ensemble of neural networks with one hidden layer and apply regularization till you see no over-fitting and then train them to the end. No issues with gradient explosion or gradient vanishing since it is just a 2 layered network. If deep layers were needed, that means hierarchical features are to be learned and then other machine learning algorithms are no good as well. For example SVM is a neural network with hinge loss only.

An example where some other machine learning algorithm would outperform a carefully regularized 2 layered (maybe 3?) neural network would be appreciated. You can give me the link to the problem and I would train the best neural network that I can and we can see if 2 layered or 3 layered neural networks falls short of any other benchmark machine learning algorithm.

• Neural Networks ARE a machine learning algorithm... – Matthew Drury May 3 '18 at 16:43
• There are certainly some domains where deep learning is king, such as image recognition, but in most other areas they tend to be dominated by gradient boosting, as is clear from looking at the results of Kaggle competitions. – Jake Westfall May 3 '18 at 16:45
• @MatthewDrury - Indeed it is! apologies for being lousy in terminologies. I hope the message was conveyed though. changing the question nonetheless, to be more robust. Thank you for pointing out – MiloMinderbinder May 3 '18 at 17:36
• Regarding network depth, definitely check this out: stats.stackexchange.com/questions/182734 – jld May 3 '18 at 18:06
• en.wikipedia.org/wiki/No_free_lunch_in_search_and_optimization – Tim Seguine May 3 '18 at 20:03

Each machine learning algorithm has a different inductive bias, so it's not always appropriate to use neural networks. A linear trend will always be learned best by simple linear regression rather than a ensemble of nonlinear networks.

If you take a look at the winners of past Kaggle competitions, excepting any challenges with image/video data, you will quickly find that neural networks are not the solution to everything. Some past solutions here.

apply regularization till you see no over-fitting and then train them to the end

There is no guarantee that you can apply enough regularization to prevent overfitting without completely destroying the capacity of the network to learn anything. In real life, it is rarely feasible to eliminate the train-test gap, and that's why papers still report train and test performance.

they are universal estimators

This is only true in the limit of having an unbounded number of units, which isn't realistic.

you can give me the link to the problem and i would train the best neural network that i can and we can see if 2 layered or 3 layered neural networks falls short of any other benchmark machine learning algorithm

An example problem which I expect a neural network would never be able to solve: Given an integer, classify as prime or not-prime.

I believe this could be solved perfectly with a simple algorithm that iterates over all valid programs in ascending length and finds the shortest program which correctly identifies the prime numbers. Indeed, this 13 character regex string can match prime numbers, which wouldn't be computationally intractable to search.

Can regularization take a model from one that overfits to the one that has its representational power severely hamstrung by regularization? Won't there always be that sweet spot in between?

Yes, there is a sweet spot, but it is usually way before you stop overfitting. See this figure:

If you flip the horizontal axis and relabel it as "amount of regularization", it's pretty accurate -- if you regularize until there is no overfitting at all, your error will be huge. The "sweet spot" occurs when there is a bit of overfitting, but not too much.

How is a 'simple algorithm that iterates over all valid programs in ascending length and finds the shortest program which correctly identifies the prime numbers.' an algorithm that learns?

It finds the parameters $\theta$ such that we have a hypothesis $H(\theta)$ which explains the data, just like backpropagation finds the parameters $\theta$ which minimize the loss (and by proxy, explains the data). Only in this case, the parameter is a string instead of many floating point values.

so if i get you correctly you are making the argument that if the data is not substantial the deep network will never hit the validation accuracy of the best shallow network given the best hyperparameters for both?

Yes. Here is an ugly but hopefully effective figure to illustrate my point.

but that doesnt make sense. a deep network can just learn a 1-1 mapping above the shallow

The question is not "can it", but "will it", and if you are training backpropagation, the answer is probably not.

We discussed the fact that larger networks will always work better than smaller networks

Without further qualification, that claim is just wrong.

• Thank you for the answer! Can regularization take a model from one that overfits to the one that has its representational power severely hamstrung by regularization? Won't there always be that sweet spot in between? – MiloMinderbinder May 4 '18 at 15:19
• Regarding your toy problem. How is a 'simple algorithm that iterates over all valid programs in ascending length and finds the shortest program which correctly identifies the prime numbers.' an algorithm that learns? – MiloMinderbinder May 4 '18 at 15:26
• Please note cs231n lecture notes - 'We discussed the fact that larger networks will always work better than smaller networks, but their higher model capacity must be appropriately addressed with stronger regularization (such as higher weight decay), or they might overfit. We will see more forms of regularization (especially dropout) in later sections.' cs231n.github.io/neural-networks-1 – MiloMinderbinder May 4 '18 at 15:32
• @user46478 i addressed questions your above by editing the answer – shimao May 4 '18 at 17:31
• 'There is no guarantee that you can apply enough regularization to prevent overfitting without completely destroying the capacity of the network to learn anything.' - I am sorry but i still don't understand this statement. I take the training data, {train, val} split, make biggest network my machine permits, increase dropout probability till i see validation error minimized among the validation errors that i am getting at various dropout probabilities. Can this process lead to a useless from an over-fitted model without going through a good model? – MiloMinderbinder May 4 '18 at 17:42

I would add that there is not such thing as a machine learning panacea:

By the no free lunch theorem:

If an algorithm performs well on a certain class of problems then it necessarily pays for that with degraded performance on the set of all remaining problems

• This is very depressing (but don't worry, i wont argue from consequences) 1. Reading this i understand this is about algorithms with same distribution over functions they can learn (which would mean same co-domain as well). So neural networks can only be pitied against universal estimators. 2. This does not talk about how fast the function space is searched through. Can we say that a neural network with 2 hidden layers is necessarily equivalent or superior to a neural network with one hidden layer on every problem? i think the assumption of continuity has to do a lot here – MiloMinderbinder May 4 '18 at 0:21
• with proper hyperparameter tuning a two hidden layered neural network jumps through functions that are have similar co-domains faster than a single hidden layered neural network. your inputs? – MiloMinderbinder May 4 '18 at 0:24
• 'Can we say that a neural network with 2 hidden layers is necessarily equivalent or superior to a neural network with one hidden layer on every problem'. No unfortunately not. As a counter example, imagine a logistic regression (a nn with zero hidden layers) vs a 1 layer neural network with the number of hidden units equal to the number of datapoints for a fully separable dataset. There is no information about the decision boundary that the l.r. can't capture, and the high complexity n.n. is likely to overfit – user3684792 May 4 '18 at 7:24
• Agree with the statement here, but I wanted to add the standard caveat to the NFL, also taken from the wikipedia page: "note that NFL only applies if the target function is chosen from a uniform distribution of all possible functions". – mkt - Reinstate Monica May 4 '18 at 8:38
• @user46478 I agree with the relevance of the NFL theorem in response to your question "why is it that the neural networks with one hidden layer are not panacea to machine learning problems?" – mkt - Reinstate Monica May 4 '18 at 15:37