4
$\begingroup$

Let's say we have a standard classification problem where we want to classify samples into two groups based on some number of predictors.

Is it possible to do this with above-chance accuracy, if there is absolutely no difference between the two target groups in means of each individual predictors using linear SVM?

My reasoning is, that means is quite sensitive to outliers or just long tails and differences in distributions, but hinge loss of SVM should be more robust. On the other hand, SVM cares about the margin, which should be different than just looking at means.

I was trying to classify some distributions with the same means, but outliers, however, I could never produce an example where linear SVM would be able to do this. Can such an example be created? If not, why not?

Edit: Here is an example of a situation where means of orange and blue cases for both x1 and x2 variables are exactly 0 (due to blue outliers), although it is easy to imagine quite a good decision boundary. However, SVM seems not to be able to reliably classify sample in this example.enter image description here

$\endgroup$
0
$\begingroup$

SVM with linear kernel can do that because mean is sensitive to outliers as you said. Consider a simple 1D case, with two classes. One class is distributed around -1, and the other around +1 with very low variance, i.e. doesn't overlap. But, there is an outlier for positive class and negative class, which are around $-N$ and $N$, where $N$ is the number of data points. such that both means are at $0$. But, this is a thought experiment and there will be cases where SVM cannot succeed, just as in your example, where there seems to be large number of so called outliers.

Edit: Removing my hinge-loss argument since it's not completely correct but also not needed, I could generate the following experiment for the above case.

import numpy as np
from sklearn.svm import LinearSVC

N = 100
x1 = np.concatenate([np.ones(N), [-N]])
y1 = np.ones(N+1)
x2 = np.concatenate([-np.ones(N),[N]])

print("Means: {}, {}".format(x1.mean(),x2.mean()))

y2 = -np.ones(N+1)
x  = np.concatenate([x1,x2])
y  = np.concatenate([y1,y2])

svc = LinearSVC()
svc.fit(x.reshape(-1,1),y)

print("Threshold: {:4f}".format((-svc.intercept_ / svc.coef_[0])[0]))

yp = svc.predict(x.reshape(-1,1))
print("Accuracy: {:4f}".format(np.mean(y==yp)))

Having the following output:

Means: 0.0, 0.0
Threshold: -0.014788
Accuracy: 0.990099
$\endgroup$
  • $\begingroup$ Hinge loss is not a zero-one loss, it stills increase as the distance from the boundary increases. I can convince myself with thought experiments that this should happen, but then it never actually worked when I fit the model. $\endgroup$ – rep_ho Jan 24 '19 at 12:58
  • 1
    $\begingroup$ Thanks! This is exactly what I would expect and it works. It doesn't work for me in R, but maybe I am just doing something wrong and that is also why my other examples were failing. $\endgroup$ – rep_ho Jan 24 '19 at 14:07
  • 1
    $\begingroup$ it works! even if I add noise and run it many times. It seems like it depends on an implementation: it works in liblinear but not in kernlab, also it does not work if I scale the variables on top of just centering them. Probably this 0 means setup creates some numerical problems for some algos. $\endgroup$ – rep_ho Jan 24 '19 at 15:27
0
$\begingroup$

The problem doesn't come from the same means.

If there is only two classes,the linear SVM will help you to separate the data in classes thanks to a straight line (2 dimensions/variables) / a plane (in 3D)

enter image description here

On your example, an SVM can't classify your data because it can't be separated by a straight line. BUT you can create more data with simple math that will allow you to use the SVM. On this example, addidng a third variable : x3 = x1*x2 will create a third dimension, where a straight plane can divide your data into two classes.

enter image description here

A whole part of machine learning is the creation of features, it can be more important than the actual model.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.