Kernel trick refers to kernel methods in machine learning, such as kernel support vector machine (SVM) or kernel principal components analysis (PCA). It allows to generalize linear techniques to non-linear situations. DO NOT USE this tag for [kernel] which is reserved for non-parametric estimation ...

learn more… | top users | synonyms

0
votes
0answers
20 views

Kernel Function Probability Distribution Connection

What is the connection between a kernel function in machine learning to the underlying distribution of the data set, if there is any such relation of course? For example can we say something like ...
0
votes
1answer
32 views

Locally weighted regression VS kernel linear regression?

I am trying to make it clear the relationship of the listed three methods. According to my understanding kernel regression means : the weight vector W lies in the space spanned by training data. $$ ...
1
vote
1answer
33 views

Kernel methods in machine learning?

I am beginning to tackle geostatistics problems where I tried to apply kriging(gaussian processes) to interpolate demographical water drop. According to my understanding, kernel methods are something ...
3
votes
1answer
57 views

How to use the kernel trick on data that can't be visualized

I'm reviewing the kernel trick and there are a lot of toy examples of how a 2D classification which can't usually be separated by a linear SVM can be separated in 3 space. This is fine, but how is ...
1
vote
2answers
48 views

Kernel PCA and classification

I need to perform kernel PCA on the colon-­‐cancer dataset and then I need to plot number of principal components vs classification accuracy with PCA data. For the first part I am ...
0
votes
2answers
22 views

Data normalization for RBF kernel

I have a matrix of values where rows are individuals and columns are attributes. I want to extract a similarity value for every pair of individuals, and I use an rbf kernel: $$k(x_i,x_j) = ...
3
votes
1answer
61 views

How do you transform a decision boundary in the angle kernel to the original space?

Say I have training data $S_n$ and each point is of the form $x = \langle x_1 , x_2 \rangle$ in the original space (i.e. $x^{(i)} \in \mathbb{R}^2$). I was considering the following kernel: $$ ...
0
votes
0answers
29 views

SVM for an unbalanced textual dataset?

I have a text classification task, currently I can classify the data with very poor precision. This are the scores: ...
1
vote
1answer
32 views

Regularizing the dual variables in SVM

Consider the the optimization program of the kernelized SVM: $$\text{maximize}_{\alpha} ~~ \alpha^T1-\alpha^TQ\alpha$$ $$\text{subject to:} \sum_{i=1}^N \alpha_iy_i=0,~0\leq \alpha_i\leq c$$ where ...
0
votes
0answers
41 views

What are the potential disadvantages of doing kernel PCA?

I was trying to learn more of the motivation around kernel PCA. Its clear to me that one might need to change the representation of the data if it lies in a non-linear space, hence, the projection ...
4
votes
1answer
31 views

Does a polynomial kernel with degree less than 1 satsify mercers condition

Consider the polynomial kernel: $$K(\boldsymbol{x}, \boldsymbol{x}') = (\boldsymbol{x}^{T} \boldsymbol{x}'+c)^{d}$$ This kernel satisfies the mercers theorem/condition. Since I never saw any ...
0
votes
0answers
32 views

What is a kernel parameter in Extreme Learning Machines?

I am using the MATLAB function elm_kernel but I don't know what the Kernel_Para variable do. Where can I learn about this? Also Regularization_coefficient. It looks similar to ...
4
votes
1answer
124 views

How to choose a kernel for kernel PCA?

What are the ways to choose what kernel would result in good data separation in the final data output by kernel PCA (principal component analysis), and what are the ways to optimize parameters of the ...
3
votes
1answer
73 views

What exactly is the procedure to compute principal components in kernel PCA?

In kernel PCA (principal component analysis) you first choose a desired kernel, use it to find your $K$ matrix, center the feature space via the $K$ matrix, find its eigenvalues and eigenvectors, then ...
1
vote
1answer
45 views

Integrating length for input-space feature PC projections in kernel PCA

I read a paper detailing the algebraic process of kernel PCA. I have question though: the paper details the projection of new points onto the new eigenvectors in the feature space, but what if I want ...
18
votes
4answers
437 views

What makes the Gaussian kernel so magical for PCA, and also in general?

I was reading about kernel PCA (1, 2, 3) with Gaussian and polynomial kernels. How does the Gaussian kernel separate seemingly any sort of nonlinear data exceptionally well? Please give an intuitive ...
0
votes
1answer
41 views

What are the various basic kernels available?

I am currently following the book Gaussian Processes for Machine Learning by C.E. Rasmussen and C.K.I. Williams and I have come across various kernels in their Chapter 4 I have also gone through the ...
1
vote
0answers
31 views

How to fit a single quadratic term to a regression

I have a high dimensional multivariate model and am fitting linear weights to each of the $N$ free variables using a classic stable SVD matrix solver. This works. I want to improve the fit by using a ...
1
vote
0answers
12 views

U statistics and RKHS

Is there a relation between the kernel function in U-statistics and RKHS theory? Namely, can the kernel trick be seen as an order-2 U-statistic?
1
vote
1answer
63 views

Is it possible to project a new vector onto the PC space using kernel PCA?

Let $X_{N \times d}$ be the data matrix, where $N$ is the number of samples and $d$ the size of the features space. Using kernel PCA (kPCA), one first computes a kernel matrix $K_{N \times N}$, and ...
1
vote
1answer
40 views

Intuition behind RKHS

Why has RKHS become such an important concept in machine learning in recent times. Is it because it allows us to represent a function of combination of linear functions? What areas of mathematic does ...
1
vote
0answers
44 views

Non-decaying eigenvalues in Kernel PCA with small kernel width

I noticed that when I use a small width kernel (RBF) with PCA, I get my desired result (clustering in this case), but I do not get a decay in the eigenvalues (they stay about the same value). Is that ...
0
votes
0answers
14 views

Kernel Methods for Binary Vectors

I am currently involved in a project which requires a minor point in choosing a proper similarity metric for a set of binary vectors, i.e. all components are either 1 or 0 . Currently, the go-to ...
2
votes
2answers
47 views

Kernel methods on Categorical Data

I have a basic understanding of kernel methods and the kernel-trick and the advantages of it, why it is preferred over conventional machine learning algorithms etc. However, I have some trouble using ...
0
votes
0answers
18 views

Pyramid Match Kernel: How to fit histogram grid around data points?

This question is directly related with Kernel methods and SVM, so I think this is a good place to ask it. I am planning to use Pyramid Match Kernel method for object recognition from depth images: I ...
2
votes
1answer
71 views

How does the $\phi(x_i)$ function look for Gaussian RBF kernel?

I am trying to write programs for simple SVM cases. And what I am stuck at is that I am unable to find $\phi(x_i)$ functions for given kernels. For example there is Gaussian Radial Basis Function ...
0
votes
0answers
33 views

Support Vector Machine with zero bias term

I'm looking for an algorithm to solve SVM with zero bias term. So dual form of such SVM is $max_\alpha \sum_i^n \alpha_i -1/2\sum_i^n \sum_j^ny_iy_jK(x_ix_j)\alpha_i\alpha_j$ subject to: $0 \leq ...
0
votes
0answers
52 views

The Shogun Machine Learning toolbox for SVM with precomputed kernel and zero bias

Can I use the Shogun Machine Learning toolbox for SVM with precomputed kernel and zero bias. I should be able to input pre-computed kernel and I also should be able to set bias zero.
0
votes
0answers
60 views

SVM with pre-computed kernel and zero bias

I have an optimization function, where I need to give my own kernel matrix and bias value is zero. The kernel matrix is calculated using the data but there is no specific formula for it. If I have a ...
0
votes
0answers
56 views

LIBSVM for pre-computed kernel and zero bias (b values is zero)

I want to do binary classification and I'm using LIBSVM library for that. I have a precomputed Kernel and my bias value (b) is zero. Can I do this in LIBSVM or do I have to use some other library? ...
0
votes
0answers
29 views

Usage of libsvm with RBF kernel and no Offset

I'm using libsvm for the binary classification and using a precomputed Kernel. In my particular problem there is no bias term (it's zero). Is there anyway to adjust the bias term in libsvm (and not ...
0
votes
1answer
65 views

What is the toolkit that implements Cost sensitive Support Vector Machine?

I need implementation of cost sensitive support vector machine. The cost is different for each training example (unlike each class). So problem is to solve $max_\alpha$ $-1/2 \sum_{i,j} ...
1
vote
1answer
118 views

Which PCA (or kernel PCA) basis better describes a single test sample?

I have two PCA bases obtained by decomposition of two groups of training data. I also have some samples of test data. How can I decide which PCA basis fits better each test sample? I tried to ...
0
votes
0answers
16 views

definig distances using radial basis functions

in svm the kernels are supposed to measure the distance between two vectors in the feature space. however, rbf is largest at 0 meaning that in that new space the distance between a feature and itself ...
2
votes
1answer
117 views

Support vector regression versus kernel ridge regression

I have a question concerning the difference between support vector regression and kernel regression. I will try to write down all the math so no misunderstandings arise (hopefully). Let's begin with ...
5
votes
1answer
515 views

How to apply a Gaussian radial basis function kernel PCA to nonlinear data?

I have an assignment to implement a Gaussian radial basis function-kernel principal component analysis (RBF-kernel PCA) and have some challenges here. It would be great if someone could point me to ...
2
votes
1answer
55 views

Are eigenvectors obtained in Kernel PCA orthogonal?

As Kernel PCA is the same as PCA in higher dimension space, shouldn't the eigenvectors obtained be orthogonal? Suppose, I have $n$ data points and let $a$ and $b$ be two eigenvectors of covariance ...
1
vote
1answer
112 views

Support Vector Machine Question

I need help with the following problem. I provided my current (partial) solution, and I hope someone can correct me and/or give me suggestions as to how I should solve the parts that I've left out. ...
1
vote
1answer
92 views

Where can I use kernels other than Gaussian (like Cauchy, laplacian) in kernel methods in machine learning? Or maybe in kernel density estimation?

In few papers I read that - kernel used doesn't really matter for kernel density estimation but bandwidth of the kernel is the most important factor. But I did not see any mathematical explanation to ...
0
votes
0answers
8 views

How accurate sum of kernel function needs to be, so that we can use it in Mean shift algorithm (may be for image segmentation)?

Mean shift is a procedure for locating the maxima of a density function given discrete data sampled from that function. It is useful for detecting the modes of this density. This is an iterative ...
1
vote
2answers
954 views

Plotting the decision boundary of a kernel SVM (RBF)

Suppose we are given a training set of 2D points that are linearly non-separable. I train a binary SVM with an RBF kernel in order to classify them. What I want to do is to draw the desicion boundary. ...
4
votes
3answers
126 views

Do kernel methods “scale” with the amount of data?

I've been reading about kernel methods, where you map original $N$ data points to a feature spaces, compute the kernel or gram matrix and plug that matrix into a standard, linear algorithm. This all ...
5
votes
2answers
582 views

Is Kernel PCA with linear kernel equivalent to standard PCA?

If in kernel PCA I choose a linear kernel $K(\mathbf{x},\mathbf{y}) = \mathbf x^\top \mathbf y$, is the result going to be different from the ordinary linear PCA? Are the solutions fundamentally ...
4
votes
1answer
137 views

Understand the reasons of using Kernel method in SVM

I understand that one can use kernel functions (i.e. radial kernel) to create non-linear decision boundary. However, there is something with my logic and I am sure there is something that I clearly ...
0
votes
0answers
191 views

What's wrong with my Kernel algorithm (Kernel SVD)?

I have a user-item matrix $A$ as data input, which is a sparse matrix containing a large number of missing values (as zeros). Each row is a user, and each column is an item. Generally, I am conducting ...
4
votes
1answer
380 views

What are the advantages of kernel PCA over standard PCA?

I want to implement an algorithm in a paper which uses kernel SVD to decompose a data matrix. So I have been reading materials about kernel methods and kernel PCA etc. But it still is very obscure to ...
1
vote
1answer
76 views

Linear Kernel in Baysian Linear Regression

I came up with http://mlg.eng.cam.ac.uk/duvenaud/cookbook/index.html and it is actually very useful. At some point it says If you use just a linear kernel in a GP, you're simply doing Bayesian ...
1
vote
0answers
123 views

Derive squared exponential covariance function

In Gaussian Processes, SVMs, kernels are used (as to my understanding) as similarity measure. However, they have the constraint that any kernel has to be represented as a dot product. i.e. ...
0
votes
1answer
45 views

What's wrong with the intuition that kernel measures similarity between observations?

Near the middle of page 16 of Andrew Ng's notes on SVM, he explained an intuitive view of kernel as measuring similarity between observations, but then added the caveat that there are things wrong ...
2
votes
2answers
74 views

How to select a number of components to retain in kernel PCA?

I'm using kpca function from kernlab and try to get the proportion of variance explained by each component as in standard PCA. I ...