What is the statistical model behind the SVM algorithm?

Question

I have learned that, when dealing with data using model-based approach, the first step is modeling data procedure as a statistical model. Then the next step is developing efficient/fast inference/learning algorithm based on this statistical model. So I want to ask which statistical model is behind the support vector machine (SVM) algorithm?

Answer 1

You can often write a model that corresponds to a loss function (here I'm going to talk about SVM regression rather than SVM-classification; it's particularly simple)

For example, in a linear model, if your loss function is $\sum_i g(\varepsilon_i) = \sum_i g(y_i-x_i'\beta)$ then minimizing that will correspond to maximum likelihood for $f\propto \exp(-a\,g(\varepsilon))$ $= \exp(-a\,g(y-x'\beta))$. (Here I have a linear kernel)

If I recall correctly SVM-regression has a loss function like this:

That corresponds to a density that is uniform in the middle with exponential tails (as we see by exponentiating its negative, or some multiple of its negative).

There's a 3 parameter family of these: corner-location (relative insensitivity threshold) plus location and scale.

It's an interesting density; if I recall rightly from looking at that particular distribution a few decades ago, a good estimator for location for it is the average of two symmetrically-placed quantiles corresponding to where the corners are (e.g. midhinge would give a good approximation to MLE for one particular choice of the constant in the SVM loss); a similar estimator for the scale parameter would be based on their difference, while the third parameter corresponds basically to working out which percentile the corners are at (this might be chosen rather than estimated as it often is for SVM).

So at least for SVM regression it seems pretty straightforward, at least if we're choosing to get our estimators by maximum likelihood.

(In case you're about to ask ... I have no reference for this particular connection to SVM: I just worked that out now. It's so simple, however, that dozens of people will have worked it out before me so no doubt there are references for it -- I've just never seen any.)

Answer 2

I think someone already answered your literal question, but let me clear up a potential confusion.

Your question is somewhat similar to the following:

I have this function $f(x) = \ldots$ and I'm wondering what differential equation it is a solution to?

In other words, it certainly has a valid answer (perhaps even a unique one if you impose regularity constraints), but it's a rather strange question to ask, since it was not a differential equation that gave rise to that function in the first place.
(On the other hand, given the differential equation, it is natural to ask for its solution, since that's usually why you write the equation!)

Here's why: I think you're thinking of probabilistic/statistical models—specifically, generative and discriminative models, based on estimating joint and conditional probabilities from data.

The SVM is neither. It's an entirely different kind of model—one that bypasses those and attempts to directly model the final decision boundary, the probabilities be damned.

Since it's about finding the shape of the decision boundary, the intuition behind it is geometric (or perhaps we should say optimization-based) rather than probabilistic or statistical.

Given that probabilities aren't really considered anywhere along the way, then, it's rather unusual to ask what a corresponding probabilistic model could be, and especially since the entire goal was to avoid having to worry about probabilities. Hence why you don't see people talking about them.