Questions tagged [svm]

Support Vector Machine refers to "a set of related supervised learning methods that analyze data and recognize patterns, used for classification and regression analysis."

636 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
23
votes
0answers
1k views

How does a Relevance Vector Machine (RVM) work?

Relevance Vector Machines (RVMs) are really interesting models when contrasted with the highly geometrical (and popular) SVMs. In the light of a question like How does a Support Vector Machine (SVM) ...
7
votes
0answers
796 views

What is the deep(er) math that makes the 'kernel trick' in SVMs work?

The kernel trick gets used very heavily in SVMs. And it is impressive: not only can you get the inner product in a larger-dimensional space (including an infinite-dimensional one) that comes from a ...
7
votes
0answers
658 views

Machine learning with ordered labels

The usual method for adapting binary classifiers like various SVMs to multilabel data is one-vs-all, which assumes that labels are independent and in case of a prediction error we don't care what ...
6
votes
0answers
222 views

Optimising dual SVM: why do some authors drop constraints?

In Hastie's Elements of Statistical Learning the dual problem is put as $$ \begin{align} \text{arg min}_\alpha \quad &\ \frac{1}{2}\alpha^\top Q\, \alpha_i- \sum_i \alpha_i\\ \text{subject to}\...
6
votes
0answers
447 views

Rank of kernel Gram matrix and classifier performance

In kernel machines we have some kernel function $k$ and we compute the $n \times n$ Gram matrix $K$ where $K_{ij} = k(x_i, x_j)$ for observations $x_i, x_j \in \mathbb R^p$. I'm letting $n$ denote the ...
6
votes
2answers
993 views

SVM with quadratic loss

I've seen some statement where I got the impression that SVM with a quadratic loss is no more than having a kernel matrix where a multiple of the unit matrix is subtracted from the kernel. It was ...
5
votes
0answers
270 views

Is linear regression equivalent to the support vector regression with a linear kernel?

I notice the objective function of the linear regression and the support vector regression (SVR) with a linear kernel could be the same, except for the SVR has two error constraints for each data ...
5
votes
0answers
137 views

Equivalent Gradients in Kernelized SVM

Let $\varphi: \mathcal{X} \to \mathcal{H}$ a mapping with corresponding kernel $K:\mathcal{X}\times\mathcal{X}\to \mathbb{R}$ (that is, $K\left(x,x'\right) = \left<\varphi\left(x\right), \varphi\...
5
votes
1answer
3k views

How probabilities are calculated for SVM model?

I would like to know, how probabilities are calculated in support vector machine. I have used Iris data set and here is my decision values for three "SupportVectorMachine" (please find the PMML below ...
5
votes
1answer
130 views

Regarding the size of training data for building classifier

When we build a classifier, like SVM or Naive Bayesian, are there any generic rules or theoretical derivations on the size of training data set? For example, to train a SVM-based classifier, what ...
4
votes
0answers
45 views

Figuring out the margin for the soft margin SVM (exam question)

This is an exam question and I am not sure whether it is solveable with the given information. We were given a graphic that displayed binary labelled points $x^{(i)}\in \mathbb{R}^2$ with $y^{(i)} \...
4
votes
0answers
648 views

Why do we need the gamma parameter in the polynomial kernel of SVMs?

The polynomial kernel is sometimes defined as just: $$ K(x,y):=(\left<x,y\right>+c)^d $$ with two parameters: the degree $d$ and constant coefficient $c$. But others (e.g., libsvm, and sklearn ...
4
votes
0answers
149 views

Are null space of matrix and kernel function same?

I have recently started learning about machine learning and have come across kernels and null spaces. I understand that null space is the set of all vectors that satisfy the equation A.v = 0 (Where A ...
4
votes
0answers
74 views

soft SVM - degenerate case

According to "A Note on Support Vector Machine Degeneracy", Theorem 4, if the dual problem for soft-SVM has a solution with $\alpha_i \in \{0,C\}, \forall i$, then $w=0$ for the primal problem. In "...
4
votes
0answers
928 views

SVM Primal Optimization and Dual Optimization. Under what cases one better than another?

Primal $$argmin_{w} \frac{1}{2}||w||^2 + \frac{1}{N}\sum max(1-t^{(i)}(w^Tx+b),0)$$. The dual form $$max_{\alpha} \sum\alpha_i - \frac{1}{2}\sum t^{(i)} t^{(j)}\alpha_i\alpha_j K(x^{(i)},x^{(j)})$$...
4
votes
0answers
45 views

Can SVM leak training data?

Is it possible to have access to trained model, e.g. through some API, and reverse engineer the model by asking for predictions for some arbitrary data, therefore recover the support vectors of the ...
4
votes
0answers
145 views

How to mitigate the hierarchical error propagation in tree-structured classification

Suppose we have a multi-class classification problem, where the number of classes $K \geq 3$ We use a tree structure of multiple SVMs to divide and conquer the problem, with one example in the figure ...
4
votes
0answers
2k views

How to know when to use Kernel SVM and not Linear SVM?

If I have more than 3 features in a dataset, then I can't visualize them to see if my classes are scattered in a non linear fashion. So how do I know when is the right way to use linear model with non-...
4
votes
0answers
1k views

How to draw plot of the values of decision function of multi class svm versus another arbitrary values?

I am trying to draw a plot of the decision function ($f(x)=sign(wx+b)$ which can be obtain by fit$decision.values in R using the svm function of e1071 package) versus another arbitrary values. From ...
4
votes
0answers
590 views

How to understand kernel functions and how to choose a suitable kernel?

I am trying to describe my understand of kernels in the Support Vector Machine(SVM) and why some of them are more popular, but I am not sure if I misunderstand these concepts: 1) There are a large ...
4
votes
0answers
120 views

Vapniks proof of the basic lemma

In his book Statistical Learning Theory (1998), Vladimir Vapnik proves an inequality needed to prove a bound on the risk for indicator loss functions. Theorem 4.1 on page 133 he derives the following ...
4
votes
0answers
3k views

How to train SVM correctly on a 1D dataset

I am trying to use svmtrain (Statistic Toolbox) to train a linear (2 class) SVM on a 1D feature vectors. The features are not fully separable and the classes intersect. The naive approach would be ...
4
votes
0answers
2k views

SVM classifier (with soft-margin) implementation in R, gamma value and quadprog

I'm trying to implement a Support Vector Machine classifier in R and I have to solve the optimization problem using the quadprog R package which solves problems of the form : $$min_b \frac{1}{2} b^...
4
votes
0answers
959 views

Calibrating multiple binary SVM classifiers for one-vs-all multi-class classification

I'm classifying text using the one-vs-all approach. There are three classes. I've trained 3 different binary SVM classifiers using 10-fold cross-validation. The accuracy of the binary classifiers ...
4
votes
0answers
671 views

Kernel in PenalizedSVM R package

There is not option to select kernel in penalizedSVM R package. What kernel do they use? Is there some other R package with penalized SVM methods where I can choose various kernels?
4
votes
0answers
125 views

Classifiers with post-training constraints on the prediction space

I'm familiar with using tools like SVMs and decision trees for discrete classification problems. But one detail that I have not encountered in that domain is: what do you do if your classifier must ...
3
votes
0answers
45 views

Does distance from the decision boundary suggest higher confidence that the class prediction is correct using SVM?

Does further distance from the decision boundary threshold suggest higher confidence that the class prediction is correct when using SVM with probability estimates enabled? This is not a question ...
3
votes
0answers
47 views

SVM Scaling problem with One-Class SVM

I'm trying to mess around with a one-class SVM implementation I hacked together from ArduinoSVM. I'm using an RBF kernel and training the model with just "in" datapoints with sklearn. First, as is ...
3
votes
0answers
67 views

One-Class SVM - Decision function

The following is based on the paper: Schölkopf et.al - SVM for Novelty Detection First let us consider the (classical) Soft Margin SVM optimization problem: ${\displaystyle {\text{minimize }}{\frac {...
3
votes
1answer
545 views

k-means clustered data: how to label newly incoming data

I have a data set with labels that were produced by a k-means clustering algorithm. Now there is some data (with the same data structure) from another source and I wonder what is the most sensible way ...
3
votes
0answers
200 views

Support Vector Machines: a beginner's question about the underlying math

I'm new to Support Vector Machines and I've been trying to get into the underlying math (instead of just using Scikit Learn or something like that). I understand the math behind it up to the point ...
3
votes
0answers
481 views

Using confusion matrix to improve my SVM

I ran an SVM classifier on the CIFAR_10 classification workbench. I got about 2/3 accuracy (which I think is Ok, but I want to improve...) Here is my confusion matrix: ...
3
votes
0answers
65 views

What are some machine learning problems that can be attacked with continuous multiobjective optimization?

I am working on continuous vector optimization, and hence continuous multiobjective optimization is a particular case. I am interested in finding applications in machine learning for this problems. Is ...
3
votes
0answers
136 views

Reducing the multiclass SVM problem with 2 classes to the standard SVM problem

Given the multiclass SVM problem with a hinge loss, and no bias (e.g. $b=0$): $$ f= \sum_{j\in[K]}\frac12||\mathbf w_j||^2+\frac{C}m \sum_{i=1}^m \ell(\mathbf w_1,...,\mathbf w_K,\mathbf x_i,y_i)$$ ...
3
votes
0answers
142 views

External Validation for SVM

As important as I have found external model validation to be, there is certainly a lack of material out there. The closest thing I have found is a paper that is focused on external validation for a ...
3
votes
0answers
105 views

Kernel function between time series of different lengths

I'm studying a data set composed of time series of different lengths; some are up to an order of magnitude longer than others. (If it matters, the data aren't actually temporally related; it's just ...
3
votes
0answers
666 views

How can I use gradient descent on the dual form of the linear SVM problem?

I understand that this is the dual form of the linear SVM problem (with a hard margin): $J(\mathbf{\alpha}) = \dfrac{1}{2}\sum\limits_{i=1}^{m}{ \sum\limits_{j=1}^{m}{ \alpha_i \alpha_j y_i y_j {\...
3
votes
0answers
449 views

Counter intuitive behavior from scikit-learn's SGDClassifier

I am working with SGDClassifier from Python library scikit-learn, a function which implements linear classification with a ...
3
votes
0answers
126 views

Centering for the regression. How to do it properly?

I have read a paper described an analysis of using support vector regression. In the paper it mentioned: It is worth mentioning that, in our implementation, we subtract the mean value of the ...
3
votes
0answers
227 views

Hyperparameter optimization in 6-dimensional continuous space

I am using Random Forest and Stochastic Gradient Boosting to predict a categorical target variable exhibiting severe between-class imbalance. I am using oversampling to make sure that the models do ...
3
votes
1answer
163 views

nonseparable case of classification problem (SVM)

I am learning Soft Margin Classification (SVM) right now. In cases when the classes are non-separable by the usual hyperplane with a margin $M >0$, we modify the constraints and say that it is ok ...
3
votes
1answer
47 views

Is there some theory of SVMs with infinitely many data?

I am trying to understand what does it means to have a (linear) SVM classifier (with soft margins) given the generative model of the data. And I realize I have not seen any paper on it, nor can I ...
3
votes
0answers
372 views

Wide swings in SVM performance with different training/test sets

I'm trying to train a classifier on 10 classes, using 249 samples and a (currently) 16-dimensional feature vector. I'm using an SVM with RBF kernel, through Python's scikit-learn module. The ...
3
votes
0answers
967 views

Feature importance scores of SVM multiclass one-vs-one design

Info about dataset: 5 classes, 200 trials, 100 features. (I know about the trial to feature ratio being very low, but can not avoid this here and still got well enough classification results.) ...
3
votes
0answers
698 views

Differences between SVR with a linear kernel and linear least squares

I've been working on a toy problem of predicting reviews a product will get in the future. I found that SVR with a linear kernel worked better than doing a linear least squares regression on the data (...
3
votes
0answers
7k views

Probability output from support vector machine (svm) with soft margin

Based on my very simple understanding of SVMs, it seems like a probabilistic output would be a very useful thing to have. Soft margin seems to part of the way toward accounting for noisy data, but ...
3
votes
0answers
499 views

SVM hyperplane equation

I am trying to understand the hyperplane of the SVM algorithm. My problem is that different sources state different equations. Which equation is the right one and if both are right, why? Wikipedia: ...
3
votes
1answer
915 views

Meaning of alpha = C in SVM

I have been studying SVM lately, following Andrew Ng's CS229 lecture notes. I can understand most of the notes. But for the case where the KKT condition is satisfied at ...
3
votes
1answer
370 views

Finding optimal hyperplane

I have a set of vectors $\{V_i\}$ in $n$-dimensional space. There is a number corresponded to each vector $\alpha_i = f(V_i)$ ($\alpha_i$ can be negative). I want to find a hyperplane which would ...
2
votes
1answer
28 views

Logistic PCA and the train/test split

I did a lot of readings about how to do PCA with train/test split. see PCA and the train/test split I understand that we should apply the PCA on train set and then apply the same transformation to the ...

1
2 3 4 5
13