What are the pros and cons of logistic regression and SVM (support vector machines)?
Logistic Regression as its name suggests is a regression technique: it estimates class membership probability whereas SVM on its own is only a classifier.
Such a probability estimate is more informativte than the SVM's distance to the class boundary.
libsvmcan calculate such probabilities by actually fitting a logistic regression to the distance from class boundary.
They use different loss functions: binomial loss for logistic regression vs. hinge loss for SVM.
In consequence, SVM puts even more emphasis on cases at the class boundaries than logistic regression (which in turn puts more emphasis on cases close to the class boundary than LDA). In fact, it ignores all cases that are not directly adjacent to the class boundaries. Cases somewhat further from the class boundary have more influence on logistic regression (but diminishing with distance from boundary).
This makes SVM (but not LR) a sparse model.
In high dimensional spaces (lots of input features), points tend to be "at the outside", i.e. many data points will be adjacent to some class boundary in some direction. In consequence, many (possibly all) points may become support vectors, and the SVM isn't sparse any more. This is typically a cumbersome situation from a computational point of view (and is often a bad sign terms of overfitting).
SVM maximize an existing (clear) margin between the classes, dealing with not perfectly separable classes is a "standard add-on". For logistic regression it's the other way round: while it naturally deals with not perfectly separable situations, perfect separation needs some "add-on" (regularization)
Kernels are not really a difference, since they can be used not only with SVM but also with logistic regression (and many other models)
The question is too broad to be answered well. Different models work differently, we need to pick the right model for the given data.
In general, Here are major differences:
SVM is a more complex model (non-linear model) than logistic regression (linear model). It may provide more accuracy, but may suffer from overfitting.
SVM will not work well for large amount of data (say a million data points). See Can support vector machine be used in large data? But logistic regression is fine (logistic regression will suffer more on number of features/columns instead of number of rows).
The main thing you need to understand here is,
- SVM always tries to maximize the margin between support vectors to differentiate classes.
- Logistic Regression uses the Logit Function (using log-likelihoond function) and Odds ratio that obtain the probabilistic values to find [0,1] output.
- SVM has kernel methods which can classify features by mapping data in higher dimensions using orthogonal projections and RBF kernels.
- Since SVM can handle complex data, there would be less room for errors compared to Logistic Regression.
- Logistic regression is more sensitive to outliers, hence SVM performs better in presence of outliers.
SVM is preferred when there are higher dimensions and higher classes in data and Logistic regression is great when there are less dimensions and less dependent features.