Questions tagged [kernel-trick]

Kernel methods are used in machine learning to generalize linear techniques to nonlinear situations, especially SVMs, PCA, and GPs. Not to be confused with [kernel-smoothing], for kernel density estimation (KDE) and kernel regression.

Filter by
Sorted by
Tagged with
33
votes
2answers
44k views

Which search range for determining SVM optimal C and gamma parameters?

I am using SVM for classification and I am trying to determine the optimal parameters for linear and RBF kernels. For the linear kernel I use cross-validated parameter selection to determine C and for ...
8
votes
1answer
16k views

Non-linear SVM classification with RBF kernel

I'm implementing a non-linear SVM classifier with RBF kernel. I was told that the only difference from a normal SVM was that I had to simply replace the dot product with a kernel function: $$ K(x_i,...
2
votes
1answer
1k views

How do I access or compute the posterior covariance matrix returned by kernlab::gausspr R function?

I am looking to compute the covariance matrix of an inferred Gaussian process in R. Below I outline how I would do this manually, but I realize that the kernlab ...
2
votes
4answers
2k views

comparing predictive model with hold out set

In Rapid Miner, I created a predictive model (SVM) with Kernel type = polynomial, c= 10, and obtained 80.77% accuracy using cross validation. When compared to hold out set my accuracy on the test ...
2
votes
0answers
70 views

Confusion related to a derivation in a paper

I was reading this paper: Ristovski, K., Das, D., Ouzienko, V., Guo, Y., Obradovic, Z. Regression Learning with Multiple Noisy Oracles. This paper is basically about regression. The idea is ...
0
votes
1answer
3k views

Free data set for comparing kernel logistic regression and regular logistic regression [closed]

I'm looking for a data set that is easily accessible for comparing Kernel Logistic Regression (KLR) and regular logistic regression. All the paper that I find using KLR use synthetic data sets. I'm ...
2
votes
1answer
1k views

Given a kernel, how to find mapping phi?

I'm not clear about kernel. How could I construct my own kernel that is valid? Is the only method the Mercer Theorem (positive semi-definite)? I mean if I know $K$ is a valid kernel, do I know that $...
2
votes
0answers
368 views

Polynomial kernel in logistic regression?

So I have put together a nice logistic regression program that works quite well. Now, I have used two dimensions to test it and see how it works, and guided by some online tutorials, have increased ...
2
votes
0answers
394 views

How can I find the number of support vectors in an SVM, depending on kernel?

Here on slide 3 I see the claim that for linearly separable cases the number of support vectors will be d+1, where d is the dimension we are working on. What if my data is not linearly separable and ...
9
votes
2answers
10k views

Use of the Gamma parameter with support vector machines

When using libsvm, the parameter $\gamma$ is a parameter for the kernel function. Its default value is setup as $$\gamma = \frac{1}{\text{number of features.}}$$ ...
14
votes
2answers
2k views

How to prove there is no finite-dimensional feature space for Gaussian RBF kernel?

How to prove that for the radial basis function $k(x, y) = \exp(-\frac{||x-y||^2)}{2\sigma^2})$ there is no finite-dimensional feature space $H$ such that for some $\Phi: \text{R}^n \to H$ we have $k(...
3
votes
0answers
148 views

Why we solve the dual problem in SVM? [duplicate]

Possible Duplicate: Why bother with the dual problem when fitting SVM? What advantages do we get from solving in the dual?
35
votes
3answers
18k views

How to prove that the radial basis function is a kernel?

How to prove that the radial basis function $k(x, y) = \exp(-\frac{||x-y||^2)}{2\sigma^2})$ is a kernel? As far as I understand, in order to prove this we have to prove either of the following: For ...
3
votes
1answer
245 views

Representation within a RKHS framework

Given a p.s.d kernel $Q$, can minimization/maximization of $Tr(X^TQX)$ over X be represented within a reproducing kernel Hilbert space (RKHS) framework? If there is a primary concern with the trace ...
1
vote
1answer
538 views

Possible reason for failing to build a support vector machine

I was trying to build a classifier for a set of documents using a support vector machine. I choose to build the feature space using term occurrence. While experimenting, I found the following scenario:...
12
votes
1answer
2k views

The relationship between the number of support vectors and the number of features

I ran an SVM against a given data set, and made the following observation: If I change the number of features for building the classifier, the number of resulting support vectors will also be changed. ...
0
votes
1answer
2k views

MATLAB implementaion of kernel ridge regression [closed]

I am trying to implement the Kernel Ridge Regression algorithm but I am getting some quite strange results. I am afraid that I have made some silly mistakes, so I need your help to find out how to fix ...
2
votes
1answer
768 views

Does the product of two p.s.d kernel matrices result in a kernel matrix?

In a ML setting, where $a_1,..., a_n$ are a set of training points. A kernel function is a function $κ$ that gives the inner product between two vectors in the feature space: $κ(a_i, a_j ) = ψ(a_i) · ...
7
votes
4answers
4k views

Support vector machine for text classification

I am currently having a data set, class 1 with about 8000 short text files and class 2 with about 3000 short text files. I applied LibSVM and tried a couple of parameter combinations in the cross-...
4
votes
2answers
665 views

Does a linear SVM behave in the same way as correlation except with the imposition of a large margin?

I want to understand the relationship between correlation and SVMs. My question is based on initial studies that used correlation as a way to examine distributed processing in the cortex with fMRI. ...
5
votes
3answers
12k views

How to select best parameter for polynomial kernel?

I am using LibSVM library for classification. For my problem I am using polynomial kernel and I need to select best parameters (d = degree of polynomial kernel, and ...
5
votes
0answers
193 views

How to learn similarity of typed/attributed graphs?

I have a question for graph machine learning gurus :). For this project I'm working on, I need to be able to learn similarity between typed graphs. By typed I mean that every vertex and every edge of ...
0
votes
4answers
494 views

Kernel Selection [closed]

First, I want to know how to analyze a dataset to discover its pattern. And second, how can I select the best kernel function for classifying a dataset?
4
votes
2answers
4k views

SVM using RBF and nearest neighbor classification method

SVM using RBF kernel is claimed to be similar (equivalent) to the K nearest neighbor classification method. I am not very clear about the analysis process of building this kind of relationship. Thanks ...
7
votes
2answers
3k views

Kernel logistic regression

I heard Kernel Logistic Regression is a classical combination of kernel methods and Logistic regression, but I cannot find any major reference (book, or paper) on this topic. Can you give me any ...
2
votes
2answers
2k views

Polynomial kernel function

Consider SMV with the polynomial kernel $k(x_1,x_2)=(\langle x_1, x_2\rangle + 1)^d,$ where $d > 1.$ Is it true that if the dataset is separated with a hyperplane then the SVM (with the kernel $k$) ...
4
votes
1answer
4k views

Linear separability for a sum of kernel functions

Suppose we have 2 kernel functions $K_1(x,y)$ and $K_2(x,y)$. We know, that the dataset ($(x_1,y_1),\ldots,(x_l,y_l),$ $y_i \in \{-1,1\}$ ) is separated with the first one (that is, there are $w,$ $...
4
votes
1answer
2k views

How to run K-means clustering on data points of varying dimensionality?

I'm trying to aggregate $T$ local image descriptors (i.e. histograms) into a vector, namely, the Fisher Vector as described in this paper by H. Jégou et al., Aggregating local image descriptors into ...
2
votes
1answer
1k views

Large data set for testing kernel logistic regression [closed]

Does anyone know of a large data set (upwards of $10^7$ rows, but I'll take $10^5$ as well) that would be appropriate for testing kernel logistic regression? Continuous variables, 2 to 50 independent ...
2
votes
4answers
209 views

Multi task learning

I have a dataset where all observations are measured several times and reported outcomes correspond to those measurements. In other words, my set of data points looks like $\{x_i, y_{i_1}, y_{i_2}, .....
1
vote
1answer
7k views

Parameters to change for different kernels for SVM

I am carrying out SVM and was interested in knowing the parameters that could be varied for each kernel. I am using 3 kernels: RBF, linear and polynominal. These are the parameters that i think can ...
16
votes
6answers
21k views

Fastest SVM implementation

More of a general question. I'm running an rbf SVM for predictive modeling. I think my current program definitely needs a bit of a speed up. I use scikit learn with a coarse to fine grid search + ...
10
votes
1answer
219 views

Beyond Fisher kernels

For a while, it seemed like Fisher Kernels might become popular, as they seemed to be a way to construct kernels from probabilistic models. However, I've rarely seen them used in practice, and I have ...
13
votes
3answers
4k views

What is a kernel and what sets it apart from other functions

There seem to be many machine learning algorithms that rely on kernel functions. SVMs and NNs to name but two. So what is the definition of a kernel function and what are the requirements for it to be ...
6
votes
2answers
905 views

CCA/KCCA for more than two views

Canonical Correlation Analysis (CCA) (and its kernel equivalent (KCCA)) can be used to find linear (nonlinear) relationships between two aligned multivariate datasets (or views). Is there a way to ...
6
votes
1answer
2k views

Kernel matrix normalisation

Originally posted in stats.SE but never got an answer so reposting here. Is it ever a bad idea to normalise the kernel matrix? By this I mean the method described on page 113 of Shawe-Taylor & ...
6
votes
4answers
8k views

Train a SVM-based classifier while taking into account the weight information

Currently I have a data set which are known to belong to two classes, and would like to build a classifier using SVM. However, there exist different confidence levels for this data set. For example, ...
6
votes
1answer
412 views

Can simple vector distance work as a SVM kernel?

I'm considering using the following simple function as an SVM kernel. It basically computes the distance between the 2 input vectors (norm): $K(\vec{x}_1, \vec{x}_2) = \left\| \vec{x}_1- \vec{x}_2 \...
3
votes
1answer
581 views

Weights of radial basis function networks

If I use radial basis function networks (RBFNs) for probability estimation by plugging the output of the RBFNs into the Logistic function are weights between 0 and 1 sufficient?
100
votes
4answers
75k views

How to select kernel for SVM?

When using SVM, we need to select a kernel. I wonder how to select a kernel. Any criteria on kernel selection?
4
votes
1answer
206 views

Constrain decision boundary to fall on grid lines in multiple class logistic regression

I would like to use multiple class logistic regression to learn the decision boundaries separating the different classes (denoted by color) in the image below. Kernel logistic regression with a RBF ...
12
votes
5answers
57k views

How to calculate a Gaussian kernel effectively in numpy [closed]

I have a numpy array with m columns and n rows, the columns being dimensions and the rows datapoints. I now need to calculate kernel values for each combination of data points. For a linear kernel $...
2
votes
2answers
600 views

What is an analogue of PCA in the regression context?

I'm writing code to approximate a function $y=f(\vec{x})$ where $y\in\mathbb{R}$ and $\vec{x}\in\mathbb{R}^N$ for medium-sized $N$ ($N$ between 20 and 50, approx.). I have a ton of examples, however, ...
1
vote
4answers
366 views

Viewing kernel regression in a Bayesian framework

If one wanted to use Kernel Regression in a Bayesian Framework, any ideas on how one would go about it? Kernel Regression
7
votes
1answer
8k views

How do I choose what SVM kernels to use?

I am having trouble determining what kernel I should use in a non-linear SVM without testing in advance. I want to know if there are any other ways to determine the best kernel without tests? How does ...
7
votes
2answers
394 views

Number of eigenfunctions for kernel

While studying machine learning, I've read the following statement: The kernel $K(x,y)=(x\cdot y+1)^d$ , for $x, y \in \mathbb{R}^p$, has $M={p+d \choose d}$ eigenfunctions that span the space of ...
6
votes
2answers
476 views

Learning a univariate transform (kernel?) for novelty detection

I have 150 observations, 500 features, and I am interested in novelty detection (outlier detection): given a new observation (let's say 'patient') I want to know if it is different from the previous ...
6
votes
2answers
2k views

Implementing the 'kernel trick' for a support vector machine in R

I've heard a bit about the 'kernel trick' for support vector machines, and I was wondering: How do you identify problems that might benefit from the kernel trick? How to implement it in R? Thank you
9
votes
2answers
6k views

Is it possible to use kernel PCA for feature selection?

Is it possible to use kernel principal component analysis (kPCA) for Latent Semantic Indexing (LSI) in the same way as PCA is used? I perform LSI in R using the ...
2
votes
1answer
698 views

Linear discriminant analysis and the “kernel trick”?

This is problem 12.10 in "The Elements of Statistical Learning": Suppose you wish to carry out a linear discriminant analysis (two classes) using a vector of transformations of the input ...

1
9 10 11
12
13