I'm currently teaching myself how to do classification, and specifically I'm looking at three methods: support vector machines, neural networks, and logistic regression. What I am trying to understand is why logistic regression would ever perform better than the other two.
From my understanding of logistic regression, the idea is to fit a logistic function to the entire data. So if my data is binary, all my data with label 0 should be mapped to the value 0 (or close to it), and all my data with value 1 should be mapped to value 1 (or close to it). Now, because the logistic function is continuous and smooth, performing this regression requires all my data to fit the curve; there is no greater importance applied to data points near the decision boundary, and all data points contribute to the loss by different amounts.
However, with support vector machines and neural networks, only those data points near the decision boundary are important; as long as a data point remains on the same side of the decision boundary, it will contribute the same loss.
Therefore, why would logistic regression ever outperform support vector machines or neural networks, given that it "wastes resources" on trying to fit a curve to lots of unimportant (easily-classifiable) data, rather than focussing only on the difficult data around the decision boundary?