Hot answers tagged

220 votes

What is the influence of C in SVMs with linear kernel?

In a SVM you are searching for two things: a hyperplane with the largest minimum margin, and a hyperplane that correctly separates as many instances as possible. The problem is that you will not ...
198 votes

How to intuitively explain what a kernel is?

Kernel is a way of computing the dot product of two vectors $\mathbf x$ and $\mathbf y$ in some (possibly very high dimensional) feature space, which is why kernel functions are sometimes called "...
86 votes

How does a Support Vector Machine (SVM) work?

Ryan Zotti's answer explains the motivation behind the maximization of the decision boundaries, carlosdc's answer gives some similarities and differences with respect to other classifiers. I'll give ...
77 votes

How to intuitively explain what a kernel is?

A visual example to help intuition Consider the following dataset where the yellow and blue points are clearly not linearly separable in two dimensions. If we could find a higher dimensional space ...
73 votes
Accepted

Why do Convolutional Neural Networks not use a Support Vector Machine to classify?

What is an SVM, anyway? I think the answer for most purposes is “the solution to the following optimization problem”: $$ \begin{split} \operatorname*{arg\,min}_{f \in \mathcal H} \frac{1}{n} \sum_{i=1}...
  • 22.5k
63 votes

What makes the Gaussian kernel so magical for PCA, and also in general?

I think the key to the magic is smoothness. My long answer which follows is simply to explain about this smoothness. It may or may not be an answer you expect. Short answer: Given a positive ...
  • 2,043
49 votes

Why not just dump the neural networks and deep learning?

Not being able to know what solution generalizes best is an issue, but it shouldn't deter us from otherwise using a good solution. Humans themselves often do not known what generalizes best (consider, ...
48 votes

How to intuitively explain what a kernel is?

A very simple and intuitive way of thinking about kernels (at least for SVMs) is a similarity function. Given two objects, the kernel outputs some similarity score. The objects can be anything ...
45 votes

How can SVM 'find' an infinite feature space where linear separation is always possible?

This answer explains the following: Why perfect separation is always possible with distinct points and a Gaussian kernel (of sufficiently small bandwidth) How this separation may be interpreted as ...
  • 10.2k
38 votes
Accepted

What algorithms need feature scaling, beside from SVM?

In general, algorithms that exploit distances or similarities (e.g. in the form of scalar product) between data samples, such as k-NN and SVM, are sensitive to feature transformations. Graphical-model ...
  • 546
37 votes

What algorithms need feature scaling, beside from SVM?

Here is a list I found on http://www.dataschool.io/comparing-supervised-learning-algorithms/ indicating which classifier needs feature scaling: Full table: In k-means clustering you also need to ...
34 votes
Accepted

Is there any supervised-learning problem that (deep) neural networks obviously couldn't outperform any other methods?

Here is one theoretical and two practical reasons why someone might rationally prefer a non-DNN approach. The No Free Lunch Theorem from Wolpert and Macready says We have dubbed the associated ...
  • 19.5k
33 votes
Accepted

How to know whether the data is linearly separable?

There are several methods to find whether the data is linearly separable, some of them are highlighted in this paper (1). With assumption of two classes in the dataset, following are few methods to ...
32 votes

Are neural networks better than SVMs?

Short answer: On small data sets, SVM might be preferred. Long answer: Historically, neural networks are older than SVMs and SVMs were initially developed as a method of efficiently training the ...
  • 6,725
31 votes
Accepted

hinge loss vs logistic loss advantages and disadvantages/limitations

Logarithmic loss minimization leads to well-behaved probabilistic outputs. Hinge loss leads to some (not guaranteed) sparsity on the dual, but it doesn't help at probability estimation. Instead, it ...
  • 16.1k
30 votes

How to prove that the radial basis function is a kernel?

I'll add a third method, just for variety: building up the kernel from a sequence of general steps known to create pd kernels. Let $\mathcal X$ denote the domain of the kernels below and $\varphi$ the ...
  • 22.5k
29 votes

Comparing SVM and logistic regression

Image signifies the difference between SVM and Logistic Regression and where to use which method this picture comes from the coursera course : "machine learning" by Andrew NG. It can be ...
28 votes
Accepted

What is the statistical model behind the SVM algorithm?

You can often write a model that corresponds to a loss function (here I'm going to talk about SVM regression rather than SVM-classification; it's particularly simple) For example, in a linear model, ...
  • 264k
28 votes

Is it a good idea to use CNN to classify 1D signal?

I guess that by 1D signal you mean time-series data, where you assume temporal dependence between the values. In such cases convolutional neural networks (CNN) are one of the possible approaches. The ...
  • 120k
28 votes

Are neural networks better than SVMs?

You may have heard of the "no free lunch theorem" in machine learning. For each model, there are pros and cons for specific data and use case. So. NN is not better than SVM and I can give ...
  • 33.8k
27 votes
Accepted

Why are optimization algorithms defined in terms of other optimization problems?

You are looking at top level algorithm flow charts. Some of the individual steps in the flow chart may merit their own detailed flow charts. However, in published papers having an emphasis on ...
27 votes

Is there any supervised-learning problem that (deep) neural networks obviously couldn't outperform any other methods?

Somewhere on this playlist of lectures by Geoff Hinton (from his Coursera course on neural networks), there's a segment where he talks about two classes of problems: Problems where noise is the key ...
  • 4,737
26 votes
Accepted

Does Dimensionality curse effect some models more than others?

In general, the curse of dimensionality makes the problem of searching through a space much more difficult, and effects the majority of algorithms that "learn" through partitioning their vector space. ...
26 votes
Accepted

What does the cost (C) parameter mean in SVM?

I'm trying to give a simple and easy-to-understand answer. A complete answer would likely need to cover everything from the purpose behind SVMs to the finer details of loss and support vectors. If you ...
  • 3,731
25 votes

How does one interpret SVM feature weights?

I am trying to interpret the variable weights given by fitting a linear SVM. A good way to understand how the weights are calculated and how to interpret them in the case of linear SVM is to perform ...
23 votes
Accepted

How to determine the optimal threshold for a classifier and generate ROC curve?

Use the SVM classifier to classify a set of annotated examples, and "one point" on the ROC space based on one prediction of the examples can be identified. Suppose the number of examples is ...
  • 818
23 votes

PCA on train and test datasets: should I run one PCA on train+test or two separate on train and on test?

(1) is incorrect, because if you run PCA on the two sets separately, you will end up with two different spaces. You cannot train a classifier in one space, and apply it to a different space. (2) is ...
  • 421
21 votes

How does a Support Vector Machine (SVM) work?

The technique is predicated upon drawing a decision boundary line leaving as ample a margin to the first positive and negative examples as possible: As in the illustration above, if we select an ...
21 votes

What does average of word2vec vector mean?

You can think of it in terms of physical analogy. You can take a flat surface, like a table, and arrange 30 balls on it. Then you can cut legs from the table and replace it with a single leg. In order ...
  • 7,314
20 votes
Accepted

Difference between ep-SVR and nu-SVR (and least squares SVR)

In $\nu$-SVR, the parameter $\nu$ is used to determine the proportion of the number of support vectors you desire to keep in your solution with respect to the total number of samples in the dataset. ...

Only top scored, non community-wiki answers of a minimum length are eligible