Hot answers tagged

195

In a SVM you are searching for two things: a hyperplane with the largest minimum margin, and a hyperplane that correctly separates as many instances as possible. The problem is that you will not always be able to get both things. The c parameter determines how great your desire is for the latter. I have drawn a small example below to illustrate this. To the ...


23

This link should help: http://www.csie.ntu.edu.tw/~cjlin/libsvm/faq.html#/Q3:_Data_preparation It's mentioned that the data is stored in a sparse array/matrix form. Essentially, it means only the non-zero data are stored, and any missing data is taken as holding value zero. For your questions: a) Index merely serves as a way to distinguish between the ...


23

I'm trying to give a simple and easy-to-understand answer. A complete answer would likely need to cover everything from the purpose behind SVMs to the finer details of loss and support vectors. If you want to dig deeper into those details you might need to look into e.g. the chapters about SVMs in the machine learning books out there. SVMs are large margin ...


17

Many SVM implementations address this by assigning different weights to positive and negative instances. Essentially you weigh the samples so that the sum of the weights for the positives will be equal to that of the negatives. Of course, in your evaluation of the SVM you have to remember that if 95% of the data is negative, it is trivial to get 95% accuracy ...


13

Most of the online/incremental SVM utilities are for linear kernels and I suppose its not as difficult as it is for non-linear kernels. Some of the notable Online/incremental SVM tools currently available: + Leon Bottous's LaSVM: It supports both linear and non-linear kernels. C++ code + Bordes's LaRank: It supports both linear and non-linear kernels. C++...


13

A likely cause is the fact you are not tuning your model. You need to find good values for $C$ and $\gamma$. In your case, the defaults turn out to be bad, which leads to trivial models that always yield a certain class. This is particularly common if one class has much more instances than the others. What is your class distribution? scikit-learn has ...


11

The problem does turn out to be parameter testing. I did not try when gamma is between 0.0 (which is 1/n_feature) and 1. On my data gamma should be turn to something around 1e-8


10

C is a regularization parameter that controls the trade off between the achieving a low training error and a low testing error that is the ability to generalize your classifier to unseen data. Consider the objective function of a linear SVM : min |w|^2+C∑ξ. If your C is too large the optimization algorithm will try to reduce |w| as much as possible leading ...


10

You've actually hit on something of an open question in the literature. As you say, there are a variety of kernels (e.g., linear, radial basis function, sigmoid, polynomial), and will perform your classification task in a space defined by their respective equations. To my knowledge, no one has definitively shown that one kernel always performs best on one ...


10

A simple Matlab code using adaBoost+SVM, probably you can start from here... N = length(X); % X training labels W = 1/N * ones(N,1); %Weights initialization M = 10; % Number of boosting iterations for m=1:M C = 10; %The cost parameters of the linear SVM, you can... perform a grid search for the optimal value as ...


9

The no "no free lunch" theorems suggest that there is no a-priori superiority of one classifier over another, and which works best depends on the nature of the learning task. So for some datasets the SVM will give better performance, and for others it will be regularised logistic regression. The real distinction between the two classifiers is that the SVM ...


9

This means, that optimization algorithm detected that with high probability (not in the strict, mathematical sense) you can speed up your training by turning the -h 0 flag in your options. Basically, -h is the shrinking heuristics, implemented in the libsvm package which for some data significantly reduces number of required computations, while in others - ...


9

It seems like you are mixing a couple of things up. First of all, cross-validation is used to get an accurate idea of the generalization error when certain tuning parameters are used. You can use svm-train in k-fold cross-validation mode using the -v k flag. In this mode, svm-train does not output a model -- just a cross-validated estimate of the ...


8

SVMs work fine on sparse and unbalanced data. Class-weighted SVM is designed to deal with unbalanced data by assigning higher misclassification penalties to training instances of the minority class.


8

Setup Recall that an SVM can be viewed as a weight vector $w$ and an intercept $b$, and that the output function for a test input $x$ is is $\langle w, x \rangle + b$. To get a binary prediction, we take $f(x) = \mathrm{sign}(\langle w, x \rangle + b)$. (I'm going to use some primal notations here, but use $\langle \cdot, \cdot \rangle$ to denote that ...


7

Support vector machine classifiers use the following decision function to determine the label for a test instance $\mathbf{z}$: $f(\mathbf{z})=\mathtt{sign}\big(\sum_{i=1}^{totalSV} y_i \alpha_i \kappa(\mathbf{x}_i,\mathbf{z})-\rho\big)=\mathtt{sign}\big(\langle\mathbf{w},\Phi(\mathbf{z})\rangle-\rho\big)$, where $\kappa(\cdot,\cdot)$ is the kernel ...


7

Try the Gaussian kernel. The Gaussian kernel is often tried first and turns out to be the best kernel in many applications (with your bag-of-words features, too). You should try the linear kernel, too. Don't expect it to give good results, text-classification problems tend to be non-linear. But it gives you a feeling for your data and you can be happy about ...


7

No one can definitively tell you what a low C value means without playing and looking at your data as well. A low C value could be caused by: A low amount of data relative to the dimension Inherent noise in the data set Improper weight scaling Inherent difficult of the problem and more This doesn't even get into issues about how big the change in accuracy ...


6

In your clf, coef_ are the weights assigned to the features; (Note it only works for linear SVM) support_vectors_ and support_ are the support vectors and the corresponding index; dual_coef_ is the coefficients of the support vector in the decision function; and intercept_ is the bias in decision function. In linear SVM, $w^Tx+b=0$ is the decision ...


6

Proportion classified correctly is a discontinuous improper scoring rule that is optimized by a bogus model. I would not believe anything that you learn from it.


6

I figured out what is needed to be done. Actually, it's something simple, but its seems I had a matlaboid bug... Here is the code and the resulting figure for the "XOR" binary classification problem. gamma = getGamma(); b = getB(); points_x1 = linspace(xLimits(1), xLimits(2), 100); points_x2 = linspace(yLimits(1), yLimits(2), 100); [X1, X2] = ...


6

Those two formulae are different things: $\frac{1}{2} w^T w + C \sum \xi_i$ is one form of the objective function, the function which is minimized over $w$, $b$, and $\xi_i$ (subject to certain constraints, which are where $b$ comes in) to find the best SVM solution. Once you've found the model (defined by $w$ and $b$), predictions on new data $x$ are done ...


6

In short, C is the penalty on the slack variables, which measure the degree to which the margin constraints are violated. A training pattern violates the margin constraint if the kernel expansion (i.e. the output of the SVM) has a value between -1 and +1, and all patterns violating this constraint will be support vectors. If you increase C, a greater ...


6

The goal of $\nu$ is similar to that of $C$. The parameter $\nu$ is there to finetune the trade-off between overfitting and generalization. The problem with $C$ is that it is positive and unbounded which makes it difficult to choose optimally during cross-validation. Thus, the range $[0,1]$ makes the regularization more interpretable in terms of $\nu$ but ...


6

The gamma parameter in the RBF kernel determines the reach of a single training instance. If the value of Gamma is low, then every training instance will have a far reach. Conversely, high values of gamma mean that training instances will have a close reach. So, with a high value of gamma, the SVM decision boundary will simply be dependent on just the points ...


5

The answers above are excellent. After carefully reading your questions, I found there are 2 important facts we might overlooked. You are using linear kernel Your training data is linearly separable, since "There is no error on the training set". Given the 2 facts, if C values changes within a reasonable range, the optimal hyperplane will just randomly ...


5

Another possibility is alpha-seeding. I am not aware whether libSVM supports it. The idea is to divide a huge amount of training data into chunks. Then you train a SVM on the first chunk. As the resulting support vectors are nothing but some of the samples of your data, you take those and use them to train your SVM with the next chunk. Also, you use that SVM ...


5

Should the training samples all be positive examples or not? Yes, in one class SVM (and any other outlier detection algorithm) you need just one class. If it is positive or negative depends on your naming convention, but it it more probable, that you will seek for positive examples which are underrepresented. Which kernel function can get better result, ...


5

In the case of sparse data like that SVM will work well. As stated by @Bitwise you should not use accuracy to measure the performance of the algorithm. Instead you should calculate the precision, recall and F-Score of the algorithm.


5

In one-class SVM the notion of accuracy is out of place. One-class SVM is designed to estimate the support of a distribution. Basically, it's output for a given instance is a measure of confidence of that instance belonging to the data that was used in training the model. When constructing a one-class SVM model, you have to decide how much of your data can ...


Only top voted, non community-wiki answers of a minimum length are eligible