I am using neural networks for most problem. The point is that it's in most cases more about the experience of the user than about the model. Here are some reasons why I like NNs.
- They are flexible. I can throw whatever loss I want at them: hinge loss, squared, cross entropy, you name it. As long as it is differentiable, I can even design a loss which fits my needs exactly.
- They can be treated probabilistically: Bayesian neural networks, variational Bayes, MLE/MAP, everything is there. (But in some cases more difficult.)
- They are fast. Most MLPs will be two matrix multiplications and one nonlinearity applied component wise in between. Beat that with an SVM.
I will go through your other points step by step.
Have a strong founding theory
I'd say, NNs are equally strong in that case: since you train them in a probabilistic framework. That makes the use of priors and a Bayesian treatment (e.g. with variational techniques or approximations) possible.
Reach the global optimum due to quadratic programming
For one set of hyperparameters. However, the search for good hps is non-convex, and you won't know whether you found the global optimum as well.
Have no issue for choosing a proper number of parameters
With SVMs, you have to select hyper parameters as well.
Needs less memory to store the predictive model
You need to store the support vectors. SVMs will not in general be cheaper to store MLPs, it depends on the case.
Yield more readable results and a geometrical interpretation
The top layer of an MLP is a logistic regression in the case of classification. Thus, there is a geometrical interpretation (separating hyper plane) and a probabilistic interpretation as well.