Questions tagged [svm]
Support Vector Machine refers to "a set of related supervised learning methods that analyze data and recognize patterns, used for classification and regression analysis."
2,288
questions
0
votes
0
answers
23
views
About the hinge loss and slack variables
I'll be denoting the $ith$ training example, target label and slack variable as $\mathbf{\vec x}^{(i)}$, $y^{(i)}$ and $\xi_i$ respectively.
Hinge Loss :
The hinge loss function in the context of ...
- 115
0
votes
0
answers
39
views
How is the SVM optimization objective derived from the hinge loss function?
The hinge loss function, in the context of SVMs, is given as:
$$
\mathcal{L}(\mathbf{\vec w}, b\,; \mathbf{\vec x}^{(i)}, y ^{(i)}) = \max(0, 1-y ^{(i)}(\mathbf{\vec w}\cdot \mathbf{\vec x}^{(i)} + b))...
- 115
3
votes
1
answer
36
views
How to determine one-class SVM's $r$ parameter after obtaining $\alpha$ from QP programming solver?
I'm reading about one-class SVM in wiki here: One-class SVM. One-class SVM attempts to learn $r$ and $c$ to fit a hypersphere to the dataset. The formula for assigning labels is:
$$sign(r^2 - ||\phi(x)...
0
votes
0
answers
22
views
Scholkopf single class linear SVM equation: why ρ substracted to 1/2 ||w||² is the same as maximizing the distance
In the one class linear SVM, the equation is :
$\min_{w, \rho} \frac{1}{2} \|w\|^2- \rho + C\sum_{i=1}^{n} \xi_i$
subject to:
$\begin{align*}
& w \cdot x_i \geq \rho - \xi_i, \\
& \xi_i \geq 0,...
- 116
0
votes
1
answer
32
views
Learning Curve to Know Underfitting or Overfitting
I want to know if the model I am using tends to be overfitting or underfitting. I am using SVM and Random Forest algorithms. How to figure it out?
- 3
0
votes
0
answers
8
views
Introducing bias via combining probability outputs from multiple models
I am working on a classification task, where I am trying to estimate the probability that a patient may not die.
I did use a Survival Analysis approach at first, but the results seemed unintuitive and ...
- 23
0
votes
0
answers
14
views
Can I find the explicit feature map that generates exponent of a kernel?
Let's say I have a kernel $K$, and another kernel of the form :
$$
K' = e^K
$$
now I know how to prove K' is a kernel, I can do it using taylor expansion of $e^x$ around $0$,
but let's say if I want ...
- 123
1
vote
0
answers
35
views
Support Vector Classifiers for Overlapping Classes
I am currently studying support vector classifiers (SVC), more specifically, the solution to the Lagrangian (Wolfe) dual function with the help of the book "The Elements of Statistical Learning&...
- 11
0
votes
0
answers
11
views
Does L2-SVM involve always using the squared hinge loss?
Im trying to understand the math behind the L1 and L2 SVM but Im kind of confused at this point : in the following picture we see that the regularization term is squared for the L2-SVM. Does that mean ...
2
votes
2
answers
61
views
How is the Representer theorem used in the derivation of the SVM dual form?
This is the primal form of the SVM hypothesis :
$$
h _{\mathbf{\vec w}, b}(\mathbf{\vec x}^{(i)}) = \mathbf{\vec w}\cdot \mathbf{\vec x}^{(i)} + b
$$
The Representer theorem as formulated here ...
- 115
3
votes
1
answer
94
views
Why is the regularization term multiplied by the error term in the cost function of SVM?
The cost function of the Optimal Margin Classifier(non-kernelized SVM) is given as :
$$
J(\mathbf{\vec w}, b) = \frac{1}{2}\|\mathbf{\vec w}\|_{2}^{2} + C \sum_{i=1}^{n}\max(0, 1-y ^{(i)}(\mathbf{\vec ...
- 115
3
votes
1
answer
152
views
Scenario where minimizing 0-1 loss is different than minimizing hinge loss
Suppose we're using linear predictors. I'm trying to conceptually understand how minimizing hinge loss and 0-1 loss aren't necessarily the same. For instance I was told that one can choose a set of ...
0
votes
0
answers
41
views
How to use random kitchen sinks for $\sigma \neq 1$?
The RBF kernel is given by
$$
k(x,y) = \exp\left(-\frac{\| x - y \|_2^2}{2 \sigma^2}\right)
$$
where $\sigma$ is the length-scale parameter. I want to use the random kitchen sinks method to create a ...
- 11
0
votes
0
answers
11
views
Linear SVM vs Decision Stumps for AdaBoost
I have heard that AdaBoost can use a linear SVM as a weak classificer. I wonder why Decision Stumps is often used with AdaBoost? Booth are binary classifiers.
In my opinion, linear SVM seems to be a ...
- 415
3
votes
2
answers
155
views
Support Vector machine - Hingeloss
What does it mean that 'The SVM hinge loss
estimates the mode of the posterior class probabilities'(Elements of statistical Learning p.427).
The decision function f(x) assigns to the positive class(+...
- 359
1
vote
1
answer
35
views
Availability of Linear Grouping Algorithms to Linearly Cluster Datasets
I have been trying to cluster a scatter plot that has a triangular graph, ideally the proper clustering plot should have a linear form, as shown below:
I tried using Spectral Clustering:
and ...
- 39
0
votes
0
answers
15
views
How to use RFE for RF and SVM
Considering I have a big data (lots of OTUs and clinical), which I will be using to input into RF and SVM for prediction (classification), will it make sense to perform RFE as a feature selection step?...
- 11
1
vote
0
answers
26
views
Feature selection before ML (RF and SVM)
I am new to machine learning and have to work with big data (lots of OTUs along with clinical) which I will input into 2 different machine learning models (RF and SVM) that will be used for prediction ...
- 11
2
votes
1
answer
29
views
Interpreting the formula for Riemannian metric tensor
In Improving support vector machine classifiers by modifying kernel functions, the authors defined Riemannian metric tensor for a kernel as follows:
$$
\begin{align}
g(\vec{x}) &= \text{det}|g_{ij}...
- 95
1
vote
2
answers
554
views
Support Vector Regression vs. Linear Regression
I am new to ML and I am learning the different algorithms one can use to perform regression. Keep in mind that I have a strong mathematical background, but I am new in the ML field.
So I understand ...
- 145
0
votes
1
answer
31
views
An extremely simple classification problem leads to intractable SVM program
In the popular textbook Mathematics for Machine Learning, creating a SVM requires solving:
$\text{min}_{w,b} \dfrac{1}{2}\|w\|^2$
subject to $y_n (w^T x_n + b) \geq 1$, for all $n = 1, \ldots, N$
Ok, ...
- 1,550
2
votes
0
answers
111
views
Textbook Recommendation other than ESL [duplicate]
My current background is as follows: (core subjects only)
Math : Linear Algebra, Analysis, (half of) Measure TheoryStats : Mathematical Statistics, Regression Analysis, Multivariate Analysis
"...
- 311
1
vote
0
answers
29
views
Convexitiy of multi-class hinge loss
The empirical risk of a multi-class hinge-loss is given by
$$L(\Theta,(x,y) = \max_{j \neq y} \Big[1+ \sum_{i=1}^{d} x_i(\Theta_{ij} - \Theta_{iy}) \Big]_{+} $$
where $x \in \mathbb{R}^{d}$ is a ...
- 245
2
votes
0
answers
25
views
Implement Nesterov's acceleration for SVM
I am trying to implement Nestrov's acceleration gradient descent for SVM. The objective function I need to minimize is $$\frac{1}{2}\lVert Au-Bv\rVert_2^2$$ with constraints $\sum_{i}u_i=\sum_{j}v_j=1$...
0
votes
0
answers
12
views
Classify text by topics using SVM: Derive the upper bound for the norm of the weight vector
In the section on SVM in one book I'm reading, the authors wrote:
Consider the problem of learning to classify a short text document according to its topic, say, whether the document is about sports ...
- 710
0
votes
0
answers
25
views
Sklearn feature selection performs strangely with 2 groups (and with SVC)
Previously I've successfully performed support vector classification with recursive feature elimination in R using the e1071 package, but I'm now hoping to move over to SciKit Learn given that Python ...
1
vote
0
answers
26
views
How to forecast changepoints from Gas Concentration Data?
So I'm trying to predict when gas concentrations change from sensor conductivity readings over a day. The gases randomly change concentrations around every 80-120 seconds and are kept constant between ...
- 11
0
votes
0
answers
43
views
Is $\ell_1$ regularization not compatible with SVM?
In the notes of Andrew Ng's CS229 Machine Learning course, it is mentioned:
The $\ell_2$ norm regularization is
much more commonly used with kernel methods because $\ell_1$ regularization is
...
2
votes
2
answers
268
views
What method should be used if the clusters contains different classes?
Assume that you having $N$ clusters. Each cluster have multiple classes. So we know the class ID for every major clusters, but not the class ID for the data points inside the major clusters.
Each ...
- 415
1
vote
1
answer
82
views
Lasso for feature selection in classification models
I want to perform classification of breast cancer cases by using models like SVM or Random Forest. When I was browsing the web I saw that one could use Lasso for feature selection, and then applied ...
- 621
0
votes
0
answers
10
views
Which method should be used if I want to find relations between two variables
Assume that you have a matrix $X \in \Re^{M x N}$ that have $M$ rows and $N$ columns. The rows $M$ can vary in length, but the $N$ columns remains constant.
Each row is labeled with a class ID. The ...
- 415
0
votes
0
answers
20
views
What is the value that the "b" term should have before the optimization problem starts?
I have been beating my head against this wall: There is this optimization problem for SVM (photo taken from Andrew Ng's lecture notes)
What I did not get is how, by using a quadratic solver for ...
0
votes
1
answer
44
views
Maxima in the dual of hard margin and soft margin SVMs?
The dual problem for hard margin SVM is:
\begin{align*}
&\max_{\alpha} \left( \sum_{i=1}^{N} \alpha_i - \frac{1}{2} \sum_{i=1}^{N} \sum_{j=1}^{N} \alpha_i \alpha_j y^{(i)} y^{(j)} \langle x^{(i)}, ...
1
vote
0
answers
39
views
SVM kernels corresponding to different types of distance measures
This answer to Data normalization for RBF kernel points out that RBF kernel implies Eucledean distance. Are there kernels corresponding to other popular distance/dissimilarity measures, such as Bray-...
- 3,933
2
votes
0
answers
10
views
How to prove that 2d support vectors are enough for Hard Margin Linear SVM?
As the question states, how can I prove mathematically that 2d support vectors are enough to always be able to formulate the Maximum Margin Hyperplane in d dimensions?
- 21
2
votes
1
answer
46
views
In case of no correlation, can a model make predictions above the expected values?
For simplicity's sake, let's suppose a binary classification problem, with a perfect 50% of probability for each of the classes, and a SkLearn's SVC model.
Let's ...
0
votes
1
answer
32
views
The distance from the hyperplane to the points
In Support Vector Machines, the distance from the hyperplane to each class of nearest points should have the same length.
Is this correct?
0
votes
1
answer
74
views
Performing a classification if having categorial labels and a distance matrix
I encountered a multi-class classification problem and I wonder which model would work the best in my scenario. I have around 50,000 vectors (each of size 200) with corresponding categorical labels ...
0
votes
0
answers
11
views
Kernel + Mutliple SVM's + Platt Scaling = 1 layer neural network?
I have built my own Support Vector Machine by using quadratic programming and I'm using Kernel PCA with SVM. The output is tanh e.g Platt scaling.
When I combinde ...
- 415
0
votes
0
answers
5
views
Which method should be used to determine the class ID of multiple SVM models?
I'm using Support Vector Machine(SVM) with image classification.
Each SVM model results a linear model
$$y = wx + b$$
Where $w$ and $b$ is the SVM parameters.
If I have multiple SVM models, I will get ...
- 415
1
vote
0
answers
20
views
Can multiple Support Vector Machine models achieve the same accuracy as one deep neural network?
Assume that we have one deep neural network with an input for images. That deep neural network can classify images.
On the other side, we have multiple Support Vector Machines that classify parts of ...
- 415
4
votes
2
answers
135
views
Scikitlearn: Why are hyperplane coefficients not available if kernel is not linear
I am interested in learning the math behind support vector machines. So far, I understand that SVMs attempt to find hyperplanes that maximize the margin distance between support vectors associated ...
- 41
1
vote
0
answers
103
views
Geometric intuition of kernel trick
I would like to understand better the geometry underlying the Kernel trick with the Gaussian Kernel. In particular my question is:
How the Kernel trick can be interpreted geometrically, in particular ...
- 880
1
vote
1
answer
488
views
Determine 'w' and 'b' in hard margin SVM
I have been asked the following question related to SVM (Hard Margin) in the exam, and I failed to answer it. Can anyone help me find the solution?
Consider the dataset M:
\begin{align*}
& \left(\...
- 11
0
votes
1
answer
27
views
Result after applying kernel trick
I understand when the data is not linearly separable, it has to transformed into higher dimensional space, to make it linearly separable. Applying kernel trick can perform it without even computing ...
2
votes
1
answer
194
views
How to solve alphas (or the dual equation) after getting Lagrangian dual of SVM
I'm trying to learn SVM by myself, and I'm stuck after getting the dual of SVM. I understand getting the dual after the primal. But, I am stuck here. Please help.
We assume that the hard margin case ...
- 21
3
votes
1
answer
209
views
Can SVM and Decision Trees be seen as instances of neural networks?
We already know that neural networks with specific choices of activation function as well as connections can generalize large amount of ML models.
My question is: neural network also generalize SVM ...
- 1,550
1
vote
0
answers
64
views
Achieve 99% accuracy with svm
I was going through this paper: https://www.mdpi.com/2227-7080/9/3/52, it compares different data scaling methods on different algorithms to see how performance is impacted.
using this database: https:...
- 23
0
votes
0
answers
33
views
SVM with too many observations, any library or solution?
In R Studio I have a database with 77,000 rows and 50 columns, I divide the observations into train and test, and the train table is left with 59,000 observations, I am making the SVM model, I am now ...
- 13
1
vote
0
answers
13
views
Is it necessary to choose predictors for svm or I use all my variables?
I have transformed my categorical variables to dummies and I have used the lasso method to decide which variables I choose to do the logistic regression, my question is: for the svm model do I need to ...
- 13