Can we say SVM and logistic regression are the same model under different loss functions?

Question

Could we say that linear SVM and logistic regression are the exact same models optimized using different losses, respectively hinge loss and logistic loss?

Answer 1

Could we say that linear SVM and logistic regression are the exact same models optimized using different losses, respectively hinge loss and logistic loss?

Yes and no at the same time.

Both can be seen under an empirical risk minimization light, where one is interested in minimizing the following function with regards to the coefficients $w$:

$$L(x,y,w)=\sum_i^n f(x_i,y_i,w)+R(w)$$

When $f \equiv \text{hinge loss}$ and $R(w) =w^Tw$ we have the SVM, and when $f \equiv \text{logistic loss}$ and $R(w) =0$ we have logistic regression.

So you can treat the loss and regularization as parameters of a model, which is empirical risk minimization.

Most, if not all, linear (kernelized/regularized) models can be put under this umbrella as well, it's a really broad definition.

Of course, the SVM and logistic regression can be seen as different models, because they have different functional forms, having completely different specialized techniques for optimization.

Answer 2

It probably depends on your point of view. Many methods can be studied from different angles! and that can be a useful exercise. But, mostly logistic regression is a probability model used for risk estimation. Support vector machines, on the other hand, is just searching for a separating hyperplane, there is no underlying probability model, and it is not clear (at least not for me) how they can be used for risk estimation.

So I would say the answer is mostly no, they are not different representations of the same model. See also Why isn't Logistic Regression called Logistic Classification? of which this is maybe a duplicate.