Does a linear SVM behave in the same way as correlation except with the imposition of a large margin?

Question

I want to understand the relationship between correlation and SVMs. My question is based on initial studies that used correlation as a way to examine distributed processing in the cortex with fMRI. This approach involved showing that within class correlations of evoked activity were greater than between class correlations (apparently this is similar to nearest neighbour methods).

Oddly, many researchers still use simple similarity measures, like correlations, even though there are many more sophisticated techniques available. It seems to me that correlations continue to be used since correlations only assess differences in patterns of response, not changes in magnitude. This is an attractive property since it provides a measure of whether different categories of stimuli evoke distinct patterns of activity within a region without measuring whether the two categories evoke different levels of activation.

I've only recently completed an introduction to kernel methods with SVMs and, to my understanding, the classifier forms the decision boundary based on a 'correlation-like' similarity measure between the examples. So my questions are;

Does a linear SVM behave in the same way as correlation except with the imposition of a large margin?

AND if so, does a linear SVM retain the so called 'independence' to classes that only differ in magnitude?

ELSE if no, can a linear SVM use a correlation matrix instead of the standard similarity measures? (or is this a horrible franken-algorithm)?

Answer 1

In a great simplification, the point of SVM is to make function of predictors $f(X_i)$ so that $f(X_i)<0$ for class A and $>0$ for B; the core of SVM is that this $f$ is done so that the distance between maximum value of $f$ for A and minimal for B is as large as possible, not counting outliers.

So the linear SVM (for which $f=A_iX_i+B$) is a kind of weird linear regression, thus in fact can be just described in means of inter-class correlation minimization (plus minus usual attribute normalisation step) and claimed nothing interesting.

However, this is not really the point -- you can swap linear kernel with other, arbitrarily sophisticated method of comparing observations; this way your $f$ can become arbitrary interesting while the margin should protect you from overfitting.

Answer 2

My understanding of this question is a little different from the previous answers, so let me put my views too.

From your example of using correlation for classification, the correlation is over all the samples within the class w.r.t to a test sample, i.e., trying to find which class has the largest number of "templates" that match your test sample the most. Such is not the case with the SVM. The key to the SVM is, it tries to find the subset of samples that belong to two classes that separate the two classes. In that regard, what SVM tries to do is a clever "template matching" to choose a class. To me that is the plus (a big one) in SVM.

Also, I would like to add that I do not agree with the comment that swapping linear kernel with more "sophisticated" kernel is any different. Again, it is a clever way for template matching in an RKHS (and you can find similar analogy to finding correlations in that RKHS)