Kernel trick refers to kernel methods in machine learning, such as kernel support vector machine (SVM) or kernel principal components analysis (PCA). It allows to generalize linear techniques to non-linear situations. DO NOT USE this tag for [kernel] which is reserved for non-parametric estimation ...

learn more… | top users | synonyms

0
votes
0answers
30 views

What are the potential disadvantages of doing kernel PCA?

I was trying to learn more of the motivation around kernel PCA. Its clear to me that one might need to change the representation of the data if it lies in a non-linear space, hence, the projection ...
4
votes
1answer
24 views

Does a polynomial kernel with degree less than 1 satsify mercers condition

Consider the polynomial kernel: $$K(\boldsymbol{x}, \boldsymbol{x}') = (\boldsymbol{x}^{T} \boldsymbol{x}'+c)^{d}$$ This kernel satisfies the mercers theorem/condition. Since I never saw any ...
0
votes
0answers
18 views

What is a kernel parameter in Extreme Learning Machines?

I am using the MATLAB function elm_kernel but I don't know what the Kernel_Para variable do. Where can I learn about this? Also Regularization_coefficient. It looks similar to ...
4
votes
1answer
74 views

How to choose a kernel for kernel PCA?

What are the ways to choose what kernel would result in good data separation in the final data output by kernel PCA (principal component analysis), and what are the ways to optimize parameters of the ...
3
votes
1answer
56 views

What exactly is the procedure to compute principal components in kernel PCA?

In kernel PCA (principal component analysis) you first choose a desired kernel, use it to find your $K$ matrix, center the feature space via the $K$ matrix, find its eigenvalues and eigenvectors, then ...
1
vote
0answers
22 views

Integrating length for input-space feature PC projections in kernel PCA

I read a paper detailing the algebraic process of kernel PCA. I have question though: the paper details the projection of new points onto the new eigenvectors in the feature space, but what if I want ...
17
votes
4answers
329 views

What makes the Gaussian kernel so magical for PCA, and also in general?

I was reading about kernel PCA (1, 2, 3) with Gaussian and polynomial kernels. How does the Gaussian kernel separate seemingly any sort of nonlinear data exceptionally well? Please give an intuitive ...
0
votes
1answer
31 views

What are the various basic kernels available?

I am currently following the book Gaussian Processes for Machine Learning by C.E. Rasmussen and C.K.I. Williams and I have come across various kernels in their Chapter 4 I have also gone through the ...
1
vote
0answers
24 views

How to fit a single quadratic term to a regression

I have a high dimensional multivariate model and am fitting linear weights to each of the $N$ free variables using a classic stable SVD matrix solver. This works. I want to improve the fit by using a ...
0
votes
0answers
9 views

U statistics and RKHS

Is there a relation between the kernel function in U-statistics and RKHS theory? Namely, can the kernel trick be seen as an order-2 U-statistic?
1
vote
1answer
52 views

Is it possible to project a new vector onto the PC space using kernel PCA?

Let $X_{N \times d}$ be the data matrix, where $N$ is the number of samples and $d$ the size of the features space. Using kernel PCA (kPCA), one first computes a kernel matrix $K_{N \times N}$, and ...
1
vote
1answer
37 views

Intuition behind RKHS

Why has RKHS become such an important concept in machine learning in recent times. Is it because it allows us to represent a function of combination of linear functions? What areas of mathematic does ...
1
vote
0answers
37 views

Non-decaying eigenvalues in Kernel PCA with small kernel width

I noticed that when I use a small width kernel (RBF) with PCA, I get my desired result (clustering in this case), but I do not get a decay in the eigenvalues (they stay about the same value). Is that ...
0
votes
0answers
10 views

Kernel Methods for Binary Vectors

I am currently involved in a project which requires a minor point in choosing a proper similarity metric for a set of binary vectors, i.e. all components are either 1 or 0 . Currently, the go-to ...
2
votes
2answers
36 views

Kernel methods on Categorical Data

I have a basic understanding of kernel methods and the kernel-trick and the advantages of it, why it is preferred over conventional machine learning algorithms etc. However, I have some trouble using ...
0
votes
0answers
16 views

Pyramid Match Kernel: How to fit histogram grid around data points?

This question is directly related with Kernel methods and SVM, so I think this is a good place to ask it. I am planning to use Pyramid Match Kernel method for object recognition from depth images: I ...
2
votes
1answer
48 views

How does the $\phi(x_i)$ function look for Gaussian RBF kernel?

I am trying to write programs for simple SVM cases. And what I am stuck at is that I am unable to find $\phi(x_i)$ functions for given kernels. For example there is Gaussian Radial Basis Function ...
0
votes
0answers
28 views

Support Vector Machine with zero bias term

I'm looking for an algorithm to solve SVM with zero bias term. So dual form of such SVM is $max_\alpha \sum_i^n \alpha_i -1/2\sum_i^n \sum_j^ny_iy_jK(x_ix_j)\alpha_i\alpha_j$ subject to: $0 \leq ...
0
votes
0answers
39 views

The Shogun Machine Learning toolbox for SVM with precomputed kernel and zero bias

Can I use the Shogun Machine Learning toolbox for SVM with precomputed kernel and zero bias. I should be able to input pre-computed kernel and I also should be able to set bias zero.
0
votes
0answers
33 views

SVM with pre-computed kernel and zero bias

I have an optimization function, where I need to give my own kernel matrix and bias value is zero. The kernel matrix is calculated using the data but there is no specific formula for it. If I have a ...
0
votes
0answers
40 views

LIBSVM for pre-computed kernel and zero bias (b values is zero)

I want to do binary classification and I'm using LIBSVM library for that. I have a precomputed Kernel and my bias value (b) is zero. Can I do this in LIBSVM or do I have to use some other library? ...
0
votes
0answers
22 views

Usage of libsvm with RBF kernel and no Offset

I'm using libsvm for the binary classification and using a precomputed Kernel. In my particular problem there is no bias term (it's zero). Is there anyway to adjust the bias term in libsvm (and not ...
0
votes
1answer
49 views

What is the toolkit that implements Cost sensitive Support Vector Machine?

I need implementation of cost sensitive support vector machine. The cost is different for each training example (unlike each class). So problem is to solve $max_\alpha$ $-1/2 \sum_{i,j} ...
1
vote
1answer
97 views

Which PCA (or kernel PCA) basis better describes a single test sample?

I have two PCA bases obtained by decomposition of two groups of training data. I also have some samples of test data. How can I decide which PCA basis fits better each test sample? I tried to ...
0
votes
0answers
15 views

definig distances using radial basis functions

in svm the kernels are supposed to measure the distance between two vectors in the feature space. however, rbf is largest at 0 meaning that in that new space the distance between a feature and itself ...
2
votes
1answer
74 views

Support vector regression versus kernel ridge regression

I have a question concerning the difference between support vector regression and kernel regression. I will try to write down all the math so no misunderstandings arise (hopefully). Let's begin with ...
5
votes
1answer
316 views

How to apply a Gaussian radial basis function kernel PCA to nonlinear data?

I have an assignment to implement a Gaussian radial basis function-kernel principal component analysis (RBF-kernel PCA) and have some challenges here. It would be great if someone could point me to ...
2
votes
1answer
46 views

Are eigenvectors obtained in Kernel PCA orthogonal?

As Kernel PCA is the same as PCA in higher dimension space, shouldn't the eigenvectors obtained be orthogonal? Suppose, I have $n$ data points and let $a$ and $b$ be two eigenvectors of covariance ...
1
vote
1answer
96 views

Support Vector Machine Question

I need help with the following problem. I provided my current (partial) solution, and I hope someone can correct me and/or give me suggestions as to how I should solve the parts that I've left out. ...
1
vote
1answer
76 views

Where can I use kernels other than Gaussian (like Cauchy, laplacian) in kernel methods in machine learning? Or maybe in kernel density estimation?

In few papers I read that - kernel used doesn't really matter for kernel density estimation but bandwidth of the kernel is the most important factor. But I did not see any mathematical explanation to ...
0
votes
0answers
8 views

How accurate sum of kernel function needs to be, so that we can use it in Mean shift algorithm (may be for image segmentation)?

Mean shift is a procedure for locating the maxima of a density function given discrete data sampled from that function. It is useful for detecting the modes of this density. This is an iterative ...
1
vote
2answers
670 views

Plotting the decision boundary of a kernel SVM (RBF)

Suppose we are given a training set of 2D points that are linearly non-separable. I train a binary SVM with an RBF kernel in order to classify them. What I want to do is to draw the desicion boundary. ...
4
votes
3answers
114 views

Do kernel methods “scale” with the amount of data?

I've been reading about kernel methods, where you map original $N$ data points to a feature spaces, compute the kernel or gram matrix and plug that matrix into a standard, linear algorithm. This all ...
5
votes
2answers
427 views

Is Kernel PCA with linear kernel equivalent to standard PCA?

If in kernel PCA I choose a linear kernel $K(\mathbf{x},\mathbf{y}) = \mathbf x^\top \mathbf y$, is the result going to be different from the ordinary linear PCA? Are the solutions fundamentally ...
3
votes
1answer
125 views

Understand the reasons of using Kernel method in SVM

I understand that one can use kernel functions (i.e. radial kernel) to create non-linear decision boundary. However, there is something with my logic and I am sure there is something that I clearly ...
0
votes
0answers
162 views

What's wrong with my Kernel algorithm (Kernel SVD)?

I have a user-item matrix $A$ as data input, which is a sparse matrix containing a large number of missing values (as zeros). Each row is a user, and each column is an item. Generally, I am conducting ...
4
votes
1answer
306 views

What are the advantages of kernel PCA over standard PCA?

I want to implement an algorithm in a paper which uses kernel SVD to decompose a data matrix. So I have been reading materials about kernel methods and kernel PCA etc. But it still is very obscure to ...
1
vote
1answer
67 views

Linear Kernel in Baysian Linear Regression

I came up with http://mlg.eng.cam.ac.uk/duvenaud/cookbook/index.html and it is actually very useful. At some point it says If you use just a linear kernel in a GP, you're simply doing Bayesian ...
1
vote
0answers
95 views

Derive squared exponential covariance function

In Gaussian Processes, SVMs, kernels are used (as to my understanding) as similarity measure. However, they have the constraint that any kernel has to be represented as a dot product. i.e. ...
0
votes
1answer
43 views

What's wrong with the intuition that kernel measures similarity between observations?

Near the middle of page 16 of Andrew Ng's notes on SVM, he explained an intuitive view of kernel as measuring similarity between observations, but then added the caveat that there are things wrong ...
2
votes
2answers
66 views

How to select a number of components to retain in kernel PCA?

I'm using kpca function from kernlab and try to get the proportion of variance explained by each component as in standard PCA. I ...
2
votes
2answers
630 views

The difference of kernels in SVM?

Can someone please tell me the difference between the kernels in SVM: Linear Polynomial Gaussian (RBF) Sigmoid Because as we know that kernel is used to mapped our input space into high ...
1
vote
1answer
56 views

Can one use eigenvalues to choose a number of components to retain in kernel PCA?

When using Kernel PCA for dimensionality reduction, is there any simple criterion which can be used to determine the number of components to use? I am using Kernel PCA with linear kernel, which would ...
2
votes
2answers
82 views

Is this a decent summary of the kernel trick?

Here's my understanding of the kernel trick. The motivation is to find a linear separator in a higher dimensional space than what you have (because the data are not currently linearly separable.) ...
9
votes
3answers
160 views

Should I use the Kernel Trick whenever possible for non-linear data?

I recently learned about the use of the Kernel trick, which maps data into higher dimensional spaces in an attempt to linearize the data in those dimensions. Are there any cases where I should avoid ...
0
votes
0answers
324 views

Explicit mapping of input space to a high dimensional feature space

I am now studying on Kernel Methods, all the theory behind it to understand Support Vector Machines. Of course, I understood some very well but there is something I could not completely comprehend. ...
1
vote
1answer
153 views

Kernel PCA with an SVD algo

Suppose that I have a great algo for calculating the SVD and I want to do Kernel PCA. It is possible to first apply the Kernel function to my data and then run the SVD algo on the transformed data?
3
votes
0answers
86 views

Kernel interpretation of Manhattan distance / Lp norms

Euclidean distance corresponds to the linear kernel similarity via: $$d_\text{Euclidean}(x,y)^2=\sum_i (x_i-y_i)^2=\sum_i x_i^2 + y_i^2 - 2 x_i y_i = ...
1
vote
2answers
169 views

Applying kernel function to input data before giving it to algorithm

I have gene expression data, I do dimensionality reduction and clustering with self organizing maps, but self organizing maps do not perform well with my data. I want to map my data to feature space ...
2
votes
1answer
57 views

How to transform this dataset to make classes linearly separable?

I have this data set: And I want to transform the data (with a RBF kernel?) in order to be able to do a simple linear ridge-classifier. I know I can do more or less the same thing using a kernel ...