Tagged Questions

Kernel trick refers to kernel methods in machine learning, such as kernel support vector machine (SVM) or kernel principal components analysis (PCA). It allows to generalize linear techniques to non-linear situations. DO NOT USE this tag for [kernel] which is reserved for non-parametric estimation ...

learn more… | top users | synonyms

1
vote
0answers
32 views

Non-decaying eigenvalues in Kernel PCA with small kernel width

I noticed that when I use a small width kernel (RBF) with PCA, I get my desired result (clustering in this case), but I do not get a decay in the eigenvalues (they stay about the same value). Is that ...
0
votes
0answers
8 views

Kernel Methods for Binary Vectors

I am currently involved in a project which requires a minor point in choosing a proper similarity metric for a set of binary vectors, i.e. all components are either 1 or 0 . Currently, the go-to ...
2
votes
2answers
26 views

Kernel methods on Categorical Data

I have a basic understanding of kernel methods and the kernel-trick and the advantages of it, why it is preferred over conventional machine learning algorithms etc. However, I have some trouble using ...
0
votes
0answers
5 views

Pyramid Match Kernel: How to fit histogram grid around data points?

This question is directly related with Kernel methods and SVM, so I think this is a good place to ask it. I am planning to use Pyramid Match Kernel method for object recognition from depth images: I ...
2
votes
1answer
23 views

How does the $\phi(x_i)$ function look for Gaussian RBF kernel?

I am trying to write programs for simple SVM cases. And what I am stuck at is that I am unable to find $\phi(x_i)$ functions for given kernels. For example there is Gaussian Radial Basis Function ...
0
votes
0answers
35 views

Understanding kernel PCA

Kernel SVMs are explained as follows: Apply kernel method to original data Check if we have a linear separator in the kernelized space. Map linear separator back to original space Is it fair to ...
0
votes
0answers
18 views

Support Vector Machine with zero bias term

I'm looking for an algorithm to solve SVM with zero bias term. So dual form of such SVM is $max_\alpha \sum_i^n \alpha_i -1/2\sum_i^n \sum_j^ny_iy_jK(x_ix_j)\alpha_i\alpha_j$ subject to: $0 \leq ...
0
votes
0answers
17 views

The Shogun Machine Learning toolbox for SVM with precomputed kernel and zero bias

Can I use the Shogun Machine Learning toolbox for SVM with precomputed kernel and zero bias. I should be able to input pre-computed kernel and I also should be able to set bias zero.
0
votes
0answers
18 views

SVM with pre-computed kernel and zero bias

I have an optimization function, where I need to give my own kernel matrix and bias value is zero. The kernel matrix is calculated using the data but there is no specific formula for it. If I have a ...
0
votes
0answers
19 views

LIBSVM for pre-computed kernel and zero bias (b values is zero)

I want to do binary classification and I'm using LIBSVM library for that. I have a precomputed Kernel and my bias value (b) is zero. Can I do this in LIBSVM or do I have to use some other library? ...
0
votes
0answers
19 views

Usage of libsvm with RBF kernel and no Offset

I'm using libsvm for the binary classification and using a precomputed Kernel. In my particular problem there is no bias term (it's zero). Is there anyway to adjust the bias term in libsvm (and not ...
0
votes
1answer
22 views

What is the toolkit that implements Cost sensitive Support Vector Machine?

I need implementation of cost sensitive support vector machine. The cost is different for each training example (unlike each class). So problem is to solve $max_\alpha$ $-1/2 \sum_{i,j} ...
1
vote
1answer
76 views

Which PCA (or kernel PCA) basis better describes a single test sample?

I have two PCA bases obtained by decomposition of two groups of training data. I also have some samples of test data. How can I decide which PCA basis fits better each test sample? I tried to ...
0
votes
0answers
13 views

definig distances using radial basis functions

in svm the kernels are supposed to measure the distance between two vectors in the feature space. however, rbf is largest at 0 meaning that in that new space the distance between a feature and itself ...
2
votes
1answer
54 views

Support vector regression versus kernel ridge regression

I have a question concerning the difference between support vector regression and kernel regression. I will try to write down all the math so no misunderstandings arise (hopefully). Let's begin with ...
4
votes
1answer
200 views

How to apply a Gaussian radial basis function kernel PCA to nonlinear data?

I have an assignment to implement a Gaussian radial basis function-kernel principal component analysis (RBF-kernel PCA) and have some challenges here. It would be great if someone could point me to ...
2
votes
1answer
35 views

Are eigenvectors obtained in Kernel PCA orthogonal?

As Kernel PCA is the same as PCA in higher dimension space, shouldn't the eigenvectors obtained be orthogonal? Suppose, I have $n$ data points and let $a$ and $b$ be two eigenvectors of covariance ...
1
vote
1answer
83 views

Support Vector Machine Question

I need help with the following problem. I provided my current (partial) solution, and I hope someone can correct me and/or give me suggestions as to how I should solve the parts that I've left out. ...
1
vote
1answer
68 views

Where can I use kernels other than Gaussian (like Cauchy, laplacian) in kernel methods in machine learning? Or maybe in kernel density estimation?

In few papers I read that - kernel used doesn't really matter for kernel density estimation but bandwidth of the kernel is the most important factor. But I did not see any mathematical explanation to ...
0
votes
0answers
5 views

How accurate sum of kernel function needs to be, so that we can use it in Mean shift algorithm (may be for image segmentation)?

Mean shift is a procedure for locating the maxima of a density function given discrete data sampled from that function. It is useful for detecting the modes of this density. This is an iterative ...
1
vote
2answers
396 views

Plotting the decision boundary of a kernel SVM (RBF)

Suppose we are given a training set of 2D points that are linearly non-separable. I train a binary SVM with an RBF kernel in order to classify them. What I want to do is to draw the desicion boundary. ...
4
votes
3answers
111 views

Do kernel methods “scale” with the amount of data?

I've been reading about kernel methods, where you map original $N$ data points to a feature spaces, compute the kernel or gram matrix and plug that matrix into a standard, linear algorithm. This all ...
4
votes
2answers
270 views

Is Kernel PCA with linear kernel equivalent to standard PCA?

If in kernel PCA I choose a linear kernel $K(\mathbf{x},\mathbf{y}) = \mathbf x^\top \mathbf y$, is the result going to be different from the ordinary linear PCA? Are the solutions fundamentally ...
3
votes
1answer
105 views

Understand the reasons of using Kernel method in SVM

I understand that one can use kernel functions (i.e. radial kernel) to create non-linear decision boundary. However, there is something with my logic and I am sure there is something that I clearly ...
0
votes
0answers
122 views

What's wrong with my Kernel algorithm (Kernel SVD)?

I have a user-item matrix $A$ as data input, which is a sparse matrix containing a large number of missing values (as zeros). Each row is a user, and each column is an item. Generally, I am conducting ...
3
votes
1answer
222 views

What are the advantages of kernel PCA over standard PCA?

I want to implement an algorithm in a paper which uses kernel SVD to decompose a data matrix. So I have been reading materials about kernel methods and kernel PCA etc. But it still is very obscure to ...
1
vote
1answer
64 views

Linear Kernel in Baysian Linear Regression

I came up with http://mlg.eng.cam.ac.uk/duvenaud/cookbook/index.html and it is actually very useful. At some point it says If you use just a linear kernel in a GP, you're simply doing Bayesian ...
1
vote
0answers
77 views

Derive squared exponential covariance function

In Gaussian Processes, SVMs, kernels are used (as to my understanding) as similarity measure. However, they have the constraint that any kernel has to be represented as a dot product. i.e. ...
0
votes
1answer
42 views

What's wrong with the intuition that kernel measures similarity between observations?

Near the middle of page 16 of Andrew Ng's notes on SVM, he explained an intuitive view of kernel as measuring similarity between observations, but then added the caveat that there are things wrong ...
0
votes
0answers
46 views

KPCA in R proportion of variability explained

I'm using kpca function from kernlab and try to get the proportion of variance explained by each component as in standard pca. I ...
2
votes
2answers
430 views

The difference of kernels in SVM?

Can someone please tell me the difference between the kernels in SVM: Linear Polynomial Gaussian (RBF) Sigmoid Because as we know that kernel is used to mapped our input space into high ...
2
votes
2answers
76 views

Is this a decent summary of the kernel trick?

Here's my understanding of the kernel trick. The motivation is to find a linear separator in a higher dimensional space than what you have (because the data are not currently linearly separable.) ...
9
votes
3answers
120 views

Should I use the Kernel Trick whenever possible for non-linear data?

I recently learned about the use of the Kernel trick, which maps data into higher dimensional spaces in an attempt to linearize the data in those dimensions. Are there any cases where I should avoid ...
0
votes
0answers
256 views

Explicit mapping of input space to a high dimensional feature space

I am now studying on Kernel Methods, all the theory behind it to understand Support Vector Machines. Of course, I understood some very well but there is something I could not completely comprehend. ...
1
vote
1answer
137 views

Kernel PCA with an SVD algo

Suppose that I have a great algo for calculating the SVD and I want to do Kernel PCA. It is possible to first apply the Kernel function to my data and then run the SVD algo on the transformed data?
3
votes
0answers
79 views

Kernel interpretation of Manhattan distance / Lp norms

Euclidean distance corresponds to the linear kernel similarity via: $$d_\text{Euclidean}(x,y)^2=\sum_i (x_i-y_i)^2=\sum_i x_i^2 + y_i^2 - 2 x_i y_i = ...
1
vote
2answers
162 views

Applying kernel function to input data before giving it to algorithm

I have gene expression data, I do dimensionality reduction and clustering with self organizing maps, but self organizing maps do not perform well with my data. I want to map my data to feature space ...
1
vote
1answer
145 views

Non-Orthogonality in PCA?

i) What is the main role of "only" trying to find orthogonal components in PCA? I can understand, that we would not want a zero-solution as well as find directions that are orthogonal in order to ...
7
votes
1answer
150 views

Kendall-tau and RKHS spaces

Given two random variables $X_1$ and $X_2$, the Kendall-tau correlation coefficient could be defined as $$ ...
0
votes
1answer
59 views

is it possible to use kernel function to calculate each instance of covariance matrix? If Yes why?

I saw a paper that uses Gaussian kernel for calculation at covariance matrix of given variables. Is it mathematically Correct or not? if it is okay,what is the intuition behind ? What about using any ...
1
vote
0answers
27 views

Confusion related to convexity of a function

I was reading this paper http://www.umiacs.umd.edu/~jags/pdfs/KSforNLP.pdf where they say that $trace(K \pi^{T} L \pi)$ is a convex function of $\pi$ where $\pi$ is a permutation matrix and K and L ...
2
votes
0answers
153 views

Generalized RBF Kernels

There is the notion of Generalized RBF Kernels, for example in "Towards Optimal Bag-of-Features for Object Categorization and Semantic Video Retrieval" from Jiang (1) or in formula (2.72) in ...
1
vote
1answer
2k views

Use Gaussian RBF kernel for mapping of 2D data to 3D

I am working on SVMs and try to get all the concepts involved. For instance, the kernel mapping. I would like to construct some parts of the algorithm by myself, to understand what is happening. My ...
1
vote
1answer
244 views

Prove (or disprove) that this function is a kernel

I devised a distance function similar to this form $d(x,y) = \sum_{i = 1}^{n-1} b(x_i, y_i,x_{i+1}, y_{i+1}) $ with $b(x_i, y_i,x_{i+1}, y_{i+1}) = 0 \mbox{ if } x_i \leq 0 \vee y_i \leq 0 \vee ...
2
votes
1answer
626 views

Kernel PCA (in R)

I am attempting to use the kernel PCA features in kernlab but am having trouble understanding the output. In particular, it's unclear what scale the results are in ...
1
vote
1answer
231 views

Number of kernel evaluations in SVM training

What is the typical number of kernel evaluations (between two training vectors) performed during a (kernelized) Support Vector Machine (SVM) training? I am asking this question because I need to ...
1
vote
0answers
144 views

How to choose kernel functions for support vector regression

Are there any good resources regarding how to design kernels for regression problems, specifically time-series regression type of problem. I am finding the choice of a kernel for regression extremely ...
0
votes
0answers
129 views

Does kernel trick correspond to real higher dimensions?

I've read that the SVM kernel tricks corresponds to viewing the data in a higher dimension. I can see that the expansion of the kernel looks like a cross-product of many different terms. However, it ...
7
votes
1answer
497 views

Kernel Ridge Regression Efficiency

Ridge Regression can be expressed as $$\hat{y} = (\mathbf{X'X} + a\mathbf{I}_d)^{-1}\mathbf{X}x$$ where $\hat{y}$ is the predicted label, $\mathbf{I}_d$ the $d \times d$ identify matrix, $\mathbf{x}$ ...
1
vote
1answer
484 views

Proof of closeness of kernel functions under pointwise product

How can I prove that pointwise product of two kernel functions is a kernel function?