Timeline for What is the statistical model behind the SVM algorithm?
Current License: CC BY-SA 3.0
27 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| May 12, 2019 at 7:03 | vote | accept | i_love_thu_ha | ||
| Nov 15, 2017 at 7:26 | comment | added | Glen_b | @Lyndon Sure looks like it. If nobody writes that up I will put something from it into my answer when time permits. | |
| Nov 15, 2017 at 6:52 | comment | added | Frames Catherine White | Maybe icml-2011.org/papers/386_icmlpaper.pdf is a reference for this?(I've only skimmed it) | |
| Nov 14, 2017 at 23:37 | comment | added | user541686 | @DeltaIV: Haha, well do note that I said repeating in your comments, not entirely comprising them. =P I can't rule out that we've talked to the same people (though it seems unlikely) but I can tell you pretty much definitely that we haven't been to the same conferences. :-) And I haven't had read/taken/listened/any-other-verb'd anything from Andrew Ng or Bishop haha. | |
| Nov 14, 2017 at 22:52 | comment | added | DeltaIV | @Mehrdad I don't see you mentioning hinge loss, or empirical risk minimization, or the Carin's team paper. But the part on generative, discriminative and "less than discriminative" models is actually identical! I didn't read your question before writing my comments. Could it be that we were at the same conference and/or talked to the same people? Anyway, it's a pretty common interpretation - I think either Bishop's book or Andrew Ng's course mention that SVM are neither generative or discriminative, but I cannot check right now. | |
| Nov 14, 2017 at 14:05 | comment | added | Cagdas Ozgenc | @DeltaIV There is a reason for that. As I explained here stats.stackexchange.com/questions/208529/… under misspecification (which is almost always) these approaches are more fail safe. | |
| Nov 14, 2017 at 12:39 | comment | added | DeltaIV | It's kind of funny - in Machine Learning, we have generative models for classification, i.e., methods which try to estimate the joint distribution of predictors and labels (e.g., LDA). Then we have discriminative models, i.e., models which estimate the conditional distribution of labels given predictors (e.g., logistic regression). SVMs, in a way, are "less than discriminative" models - they don't even care to estimate the conditional probabilities (at least in the optimization interpretation), they just estimate the position of the maximum margin hyperplane (or manifold). | |
| Nov 14, 2017 at 12:27 | comment | added | DeltaIV | @Glen_b sure - I'm not saying you can't cast SVM for classification in a probabilistic framework, I'm just saying that researchers in the field told me they were introduced precisely to make do w/o probability theory (see also Firebug's comment on SVMs not predicting class probabilities, confirmed also by ESL). Bayesian decision theory is exactly what I was getting at - do you have ESP powers? :) Here's the relevant NIPS paper: people.ee.duke.edu/~lcarin/svm_nips2014.pdf I won't talk about SVMs for regression (which I know as SVR) because I know too little about them. | |
| Nov 14, 2017 at 12:21 | comment | added | Firebug | SVM is instrinsically linked to quantile regression, to the point regularized quantile regression is often performed using adapted SVM solvers, and sometimes distributed within the same frameworks. | |
| Nov 14, 2017 at 12:16 | comment | added | Firebug | @CagdasOzgenc It's not that it doesn't "emit correct probabilities", it does not output probabilities at all. The only output is the (signed) distance to the margin. | |
| Nov 14, 2017 at 11:09 | comment | added | Glen_b | It may be possible to do something with a decision theory framework and back out a distributional assumption from a Bayesian approach, but that sounds pretty close to what DeltaIV is getting at above. | |
| Nov 14, 2017 at 11:06 | history | edited | Glen_b | CC BY-SA 3.0 |
added 159 characters in body
|
| Nov 14, 2017 at 11:00 | history | edited | Glen_b | CC BY-SA 3.0 |
added 159 characters in body
|
| Nov 14, 2017 at 10:36 | comment | added | Glen_b | @Delta on the hinge loss, in a regression framework I think that's quantile regression stats.stackexchange.com/questions/251600/… ... | |
| Nov 14, 2017 at 10:34 | comment | added | Glen_b | Just because you don't have to have a probabilistic framework... doesn't mean what you're doing doesn't correspond to one. One can do least squares without assuming normality, but it's useful to understand that's what it's doing well at ... and when you're nowhere near it that's it may be doing much less well. | |
| Nov 14, 2017 at 10:33 | comment | added | DeltaIV | ...that someone has cooked up a probabilistic model. I will look ELS up later - maybe they have something. | |
| Nov 14, 2017 at 10:32 | comment | added | DeltaIV | If the OP is asking about SVM, s/he is probably interested in classification (which is the most common application of SVMs). In that case the loss is hinge loss which is a bit different (you don't have the increasing part). Concerning the model, I heard academics saying at conference that SVMs were introduced to perform classification without having to use a probabilistic framework. Probably that's why you can't find references. On the other hand, you can and you do recast hinge loss minimization as empirical risk minimization - which means... | |
| Nov 14, 2017 at 10:23 | history | edited | Glen_b | CC BY-SA 3.0 |
added 26 characters in body
|
| Nov 14, 2017 at 10:15 | comment | added | Firebug | +1, plus the vanilla SVM also has a Gaussian prior on its parameters through the $\ell_2$-norm. | |
| Nov 14, 2017 at 10:02 | comment | added | Tim | +1 actually this is very strange that it is that hard to find this in print. Is seems that everyone is discussing SVM in model-free terms. | |
| Nov 14, 2017 at 9:42 | history | edited | Glen_b | CC BY-SA 3.0 |
added 12 characters in body
|
| Nov 14, 2017 at 8:37 | history | edited | Glen_b | CC BY-SA 3.0 |
added 23 characters in body
|
| Nov 14, 2017 at 8:32 | history | edited | Glen_b | CC BY-SA 3.0 |
added 23 characters in body
|
| Nov 14, 2017 at 8:20 | history | edited | Glen_b | CC BY-SA 3.0 |
added 367 characters in body
|
| Nov 14, 2017 at 7:49 | history | edited | Glen_b | CC BY-SA 3.0 |
added 143 characters in body
|
| Nov 14, 2017 at 7:44 | comment | added | Glen_b | (I answered this earlier elsewhere but I deleted that and moved it here when I saw you also asked here; the ability to write mathematics and include pictures is much better here -- and the search function is better too, so it's easier to find in a few months) | |
| Nov 14, 2017 at 7:43 | history | answered | Glen_b | CC BY-SA 3.0 |