Questions tagged [svm]

Support Vector Machine refers to "a set of related supervised learning methods that analyze data and recognize patterns, used for classification and regression analysis."

617 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
22
votes
0answers
986 views

How does a Relevance Vector Machine (RVM) work?

Relevance Vector Machines (RVMs) are really interesting models when contrasted with the highly geometrical (and popular) SVMs. In the light of a question like How does a Support Vector Machine (SVM) ...
7
votes
0answers
626 views

Machine learning with ordered labels

The usual method for adapting binary classifiers like various SVMs to multilabel data is one-vs-all, which assumes that labels are independent and in case of a prediction error we don't care what ...
6
votes
0answers
218 views

Optimising dual SVM: why do some authors drop constraints?

In Hastie's Elements of Statistical Learning the dual problem is put as $$ \begin{align} \text{arg min}_\alpha \quad &\ \frac{1}{2}\alpha^\top Q\, \alpha_i- \sum_i \alpha_i\\ \text{subject to}\...
6
votes
0answers
741 views

What is the deep(er) math that makes the 'kernel trick' in SVMs work?

The kernel trick gets used very heavily in SVMs. And it is impressive: not only can you get the inner product in a larger-dimensional space (including an infinite-dimensional one) that comes from a ...
6
votes
0answers
437 views

Rank of kernel Gram matrix and classifier performance

In kernel machines we have some kernel function $k$ and we compute the $n \times n$ Gram matrix $K$ where $K_{ij} = k(x_i, x_j)$ for observations $x_i, x_j \in \mathbb R^p$. I'm letting $n$ denote the ...
6
votes
2answers
914 views

SVM with quadratic loss

I've seen some statement where I got the impression that SVM with a quadratic loss is no more than having a kernel matrix where a multiple of the unit matrix is subtracted from the kernel. It was ...
5
votes
0answers
241 views

Is linear regression equivalent to the support vector regression with a linear kernel?

I notice the objective function of the linear regression and the support vector regression (SVR) with a linear kernel could be the same, except for the SVR has two error constraints for each data ...
5
votes
1answer
2k views

How probabilities are calculated for SVM model?

I would like to know, how probabilities are calculated in support vector machine. I have used Iris data set and here is my decision values for three "SupportVectorMachine" (please find the PMML below ...
5
votes
2answers
126 views

Regarding the size of training data for building classifier

When we build a classifier, like SVM or Naive Bayesian, are there any generic rules or theoretical derivations on the size of training data set? For example, to train a SVM-based classifier, what ...
4
votes
0answers
123 views

Are null space of matrix and kernel function same?

I have recently started learning about machine learning and have come across kernels and null spaces. I understand that null space is the set of all vectors that satisfy the equation A.v = 0 (Where A ...
4
votes
0answers
61 views

soft SVM - degenerate case

According to "A Note on Support Vector Machine Degeneracy", Theorem 4, if the dual problem for soft-SVM has a solution with $\alpha_i \in \{0,C\}, \forall i$, then $w=0$ for the primal problem. In "...
4
votes
0answers
111 views

Equivalent Gradients in Kernelized SVM

Let $\varphi: \mathcal{X} \to \mathcal{H}$ a mapping with corresponding kernel $K:\mathcal{X}\times\mathcal{X}\to \mathbb{R}$ (that is, $K\left(x,x'\right) = \left<\varphi\left(x\right), \varphi\...
4
votes
0answers
43 views

Can SVM leak training data?

Is it possible to have access to trained model, e.g. through some API, and reverse engineer the model by asking for predictions for some arbitrary data, therefore recover the support vectors of the ...
4
votes
0answers
142 views

How to mitigate the hierarchical error propagation in tree-structured classification

Suppose we have a multi-class classification problem, where the number of classes $K \geq 3$ We use a tree structure of multiple SVMs to divide and conquer the problem, with one example in the figure ...
4
votes
0answers
1k views

How to draw plot of the values of decision function of multi class svm versus another arbitrary values?

I am trying to draw a plot of the decision function ($f(x)=sign(wx+b)$ which can be obtain by fit$decision.values in R using the svm function of e1071 package) versus another arbitrary values. From ...
4
votes
0answers
118 views

Vapniks proof of the basic lemma

In his book Statistical Learning Theory (1998), Vladimir Vapnik proves an inequality needed to prove a bound on the risk for indicator loss functions. Theorem 4.1 on page 133 he derives the following ...
4
votes
0answers
3k views

How to train SVM correctly on a 1D dataset

I am trying to use svmtrain (Statistic Toolbox) to train a linear (2 class) SVM on a 1D feature vectors. The features are not fully separable and the classes intersect. The naive approach would be ...
4
votes
0answers
2k views

SVM classifier (with soft-margin) implementation in R, gamma value and quadprog

I'm trying to implement a Support Vector Machine classifier in R and I have to solve the optimization problem using the quadprog R package which solves problems of the form : $$min_b \frac{1}{2} b^...
4
votes
0answers
947 views

Calibrating multiple binary SVM classifiers for one-vs-all multi-class classification

I'm classifying text using the one-vs-all approach. There are three classes. I've trained 3 different binary SVM classifiers using 10-fold cross-validation. The accuracy of the binary classifiers ...
4
votes
0answers
669 views

Kernel in PenalizedSVM R package

There is not option to select kernel in penalizedSVM R package. What kernel do they use? Is there some other R package with penalized SVM methods where I can choose various kernels?
4
votes
0answers
123 views

Classifiers with post-training constraints on the prediction space

I'm familiar with using tools like SVMs and decision trees for discrete classification problems. But one detail that I have not encountered in that domain is: what do you do if your classifier must ...
3
votes
0answers
42 views

Figuring out the margin for the soft margin SVM (exam question)

This is an exam question and I am not sure whether it is solveable with the given information. We were given a graphic that displayed binary labelled points $x^{(i)}\in \mathbb{R}^2$ with $y^{(i)} \...
3
votes
1answer
173 views

k-means clustered data: how to label newly incoming data

I have a data set with labels that were produced by a k-means clustering algorithm. Now there is some data (with the same data structure) from another source and I wonder what is the most sensible way ...
3
votes
0answers
482 views

Why do we need the gamma parameter in the polynomial kernel of SVMs?

The polynomial kernel is sometimes defined as just: $$ K(x,y):=(\left<x,y\right>+c)^d $$ with two parameters: the degree $d$ and constant coefficient $c$. But others (e.g., libsvm, and sklearn ...
3
votes
1answer
192 views

Schölkopf's One-class SVM: Role of $\rho$ in the cost function

I have read and re-read the original paper and articles on Schölkopf's One-class SVM but elements of it still baffles me. The paper defines the cost function as: $$ L = \frac{1}{2}||w^{2}|| + \frac{1}{...
3
votes
0answers
424 views

Using confusion matrix to improve my SVM

I ran an SVM classifier on the CIFAR_10 classification workbench. I got about 2/3 accuracy (which I think is Ok, but I want to improve...) Here is my confusion matrix: ...
3
votes
0answers
64 views

What are some machine learning problems that can be attacked with continuous multiobjective optimization?

I am working on continuous vector optimization, and hence continuous multiobjective optimization is a particular case. I am interested in finding applications in machine learning for this problems. Is ...
3
votes
0answers
132 views

Reducing the multiclass SVM problem with 2 classes to the standard SVM problem

Given the multiclass SVM problem with a hinge loss, and no bias (e.g. $b=0$): $$ f= \sum_{j\in[K]}\frac12||\mathbf w_j||^2+\frac{C}m \sum_{i=1}^m \ell(\mathbf w_1,...,\mathbf w_K,\mathbf x_i,y_i)$$ ...
3
votes
0answers
140 views

External Validation for SVM

As important as I have found external model validation to be, there is certainly a lack of material out there. The closest thing I have found is a paper that is focused on external validation for a ...
3
votes
0answers
102 views

Kernel function between time series of different lengths

I'm studying a data set composed of time series of different lengths; some are up to an order of magnitude longer than others. (If it matters, the data aren't actually temporally related; it's just ...
3
votes
0answers
602 views

How can I use gradient descent on the dual form of the linear SVM problem?

I understand that this is the dual form of the linear SVM problem (with a hard margin): $J(\mathbf{\alpha}) = \dfrac{1}{2}\sum\limits_{i=1}^{m}{ \sum\limits_{j=1}^{m}{ \alpha_i \alpha_j y_i y_j {\...
3
votes
0answers
399 views

Counter intuitive behavior from scikit-learn's SGDClassifier

I am working with SGDClassifier from Python library scikit-learn, a function which implements linear classification with a ...
3
votes
0answers
1k views

How to know when to use Kernel SVM and not Linear SVM?

If I have more than 3 features in a dataset, then I can't visualize them to see if my classes are scattered in a non linear fashion. So how do I know when is the right way to use linear model with non-...
3
votes
0answers
124 views

Centering for the regression. How to do it properly?

I have read a paper described an analysis of using support vector regression. In the paper it mentioned: It is worth mentioning that, in our implementation, we subtract the mean value of the ...
3
votes
0answers
222 views

Hyperparameter optimization in 6-dimensional continuous space

I am using Random Forest and Stochastic Gradient Boosting to predict a categorical target variable exhibiting severe between-class imbalance. I am using oversampling to make sure that the models do ...
3
votes
1answer
45 views

Is there some theory of SVMs with infinitely many data?

I am trying to understand what does it means to have a (linear) SVM classifier (with soft margins) given the generative model of the data. And I realize I have not seen any paper on it, nor can I ...
3
votes
0answers
553 views

How to understand kernel functions and how to choose a suitable kernel?

I am trying to describe my understand of kernels in the Support Vector Machine(SVM) and why some of them are more popular, but I am not sure if I misunderstand these concepts: 1) There are a large ...
3
votes
0answers
372 views

Wide swings in SVM performance with different training/test sets

I'm trying to train a classifier on 10 classes, using 249 samples and a (currently) 16-dimensional feature vector. I'm using an SVM with RBF kernel, through Python's scikit-learn module. The ...
3
votes
0answers
937 views

Feature importance scores of SVM multiclass one-vs-one design

Info about dataset: 5 classes, 200 trials, 100 features. (I know about the trial to feature ratio being very low, but can not avoid this here and still got well enough classification results.) ...
3
votes
0answers
688 views

Differences between SVR with a linear kernel and linear least squares

I've been working on a toy problem of predicting reviews a product will get in the future. I found that SVR with a linear kernel worked better than doing a linear least squares regression on the data (...
3
votes
0answers
7k views

Probability output from support vector machine (svm) with soft margin

Based on my very simple understanding of SVMs, it seems like a probabilistic output would be a very useful thing to have. Soft margin seems to part of the way toward accounting for noisy data, but ...
3
votes
0answers
494 views

SVM hyperplane equation

I am trying to understand the hyperplane of the SVM algorithm. My problem is that different sources state different equations. Which equation is the right one and if both are right, why? Wikipedia: ...
3
votes
1answer
836 views

Meaning of alpha = C in SVM

I have been studying SVM lately, following Andrew Ng's CS229 lecture notes. I can understand most of the notes. But for the case where the KKT condition is satisfied at ...
3
votes
1answer
363 views

Finding optimal hyperplane

I have a set of vectors $\{V_i\}$ in $n$-dimensional space. There is a number corresponded to each vector $\alpha_i = f(V_i)$ ($\alpha_i$ can be negative). I want to find a hyperplane which would ...
2
votes
0answers
34 views

In semi-supervies learning, is “low density separation” the same thing as “pseudo-labelling”?

I'm looking into the different methods of semi-supervised learning. In the wikipedia page, one of the methods described is called "low-density separation", where we attempt to minimize this loss ...
2
votes
0answers
41 views

One-Class SVM - Decision function

The following is based on the paper: Schölkopf et.al - SVM for Novelty Detection First let us consider the (classical) Soft Margin SVM optimization problem: ${\displaystyle {\text{minimize }}{\frac {...
2
votes
0answers
52 views

Is kernalized linear regression parametric or nonparametric?

We know that for linear regression, we can predict: $$\hat{y} = w^Tx +b$$ Where $w$ is the parameter that minimizes the square loss. It is easy to prove that for the final solution using gradient ...
2
votes
0answers
27 views

SVM classifier: strange location of support vector

I am playing around with Matlab's example which involves classifying whether data lie inside a circle of radius 1 (label: -1) or out of it (label: 1). I decided to experiment with things and flipped ...
2
votes
0answers
47 views

SVM optimization problem with constraint

I am studying SVM from Andrew ng machine learning notes. I don't fully understand the optimization problem for svm that is stated in the notes. So we have optimization problem $$\max_{\gamma, w, b}\...
2
votes
0answers
252 views

SVM/Linear Regression after PCA and making up numbers

I recently saw an answer on Stackoverflow where a person tried to run a linear regression after doing PCA but the answerer suggested making up the dependent variable. Is this something that's ...

1 2 3 4 5 13