# Questions tagged [kernel-trick]

Kernel methods are used in machine learning to generalize linear techniques to nonlinear situations, especially SVMs, PCA, and GPs. Not to be confused with [kernel-smoothing], for kernel density estimation (KDE) and kernel regression.

605 questions
Filter by
Sorted by
Tagged with
343 views

### Prove sum of slacks in SVM is an upper bound on the number of misclassified examples

How can we Prove that the sum of slacks $\sum \xi_n$ from the objective function of the SVM formulation with soft margin is an upper bound on the number of misclassified training examples? Some ...
751 views

### What is the difference between explicit and implicit mapping in SVM?

1) What is the difference between explicit and implicit mapping 2) What is the difference between mapping and kernel trick?
2k views

### How can we prove that a normalized kernel is also a kernel?

How can we prove that the normalized kernel is a kernel? That is how can we show $\frac{K(x,y)}{ \sqrt{( K(x,x) K(y,y) )}}$ is a valid kernel. Question: Also in real world, why do we normalize the ...
50 views

### Conceptual Understanding of Kernels

I'm currently reading up on Kernels specifically related to SVM. My understanding of Kernels is that the data points are projected to another dimension and an ...
48 views

### How to decide an SVM kernel is linear or not?

I found the definition of SVM polynomial kernel is $$K (x, x′) = (1 + (x · x′))^p$$ I want to know why it add $1$? Does the kernel $K (x, x′ ) = (x · x′ )^2$ a linear kernel or a polynomial kernel? ...
831 views

### SVM kernels and decision boundaries

I trying to figure out intuitively how the kernel trick gives rise to a decision boundary. I've always thought of SVMs as hyperdimensional spaces with a decision plane dividing them up and I'm trying ...
103 views

### Distribution Regression with Kernel Mean Embedding

I'm interested in doing distribution regression with Kernel Mean Embedding with a variety of kernel methods (SVM regression, Ridge Regression, Gaussian Process). Typically in distribution regression, ...
232 views

### Relationship between PCA and KPCA

The relation between PCA and KPCA seems somewhat confusing. Basically, the Kernel variant of PCA can be described as constructing the normalized kernel matrix of the data $n \times n$ (n is the number ...
2k views

### (SVM) Difference between linear kernel and polynomial kernel of degree 1?

I am new to machine learning. Could anyone tell me the difference between linear kernel vs. polynomial kernel of degree 1 wrt SVM (if there is any difference)? The reason I asked, I am getting ...
233 views

### Influence kernel function for Kernel Ridge Regression and Support Vector Regression

I am currently exploring two regression methods using kernels, namely Kernel Ridge Regression (KRR) and Support Vector Regression (SVR). I tune their parameters using a randomized grid search. Using ...
144 views

### Kernel Regression

In Kernel Regression with a linear kernel, we have $$\beta = X\alpha = X^T(XX^T+\lambda I)^{-1}Y$$ and the normal Ridge Regression solution is $$\beta = (X^TX+\lambda I)^{-1}X^TY.$$ How can you prove ...
125 views

### Number of coefficients in SVM solutions

We know that the SVM optimization problem with n data points has a solution in the form of $$f(x)=\sum_{i=1}^n a_i k(x_i,x)+b$$ which has n+1 coefficients. But as I understands it we don't ...
794 views

### What is the deep(er) math that makes the 'kernel trick' in SVMs work?

The kernel trick gets used very heavily in SVMs. And it is impressive: not only can you get the inner product in a larger-dimensional space (including an infinite-dimensional one) that comes from a ...
814 views

### Python - (SVM) Why is my Gram matrix not positive definite?

I am writing a support vector machine with 1-norm soft margins in Python, using the quadprog quadratic programming package. As far as I can tell, by using the Gaussian kernel I should be guaranteed a ...
1k views

2k views

4k views

### SVM: How to get predicted output from SVM with Gaussian / RBF Kernel? Andrew Ng's course svmPredict

I've been learning machine learning through Andrew Ng's Coursera. I've completed Andrew's homework on SVM, but it felt wishy washed and I'm having hard time taking it from 0 to finish. Say you ...
3k views

### Kernels in Gaussian Processes

I am trying to understand intuitively how a kernel works in a Gaussian process. I know that GP are distributions over functions, in short you have the model $y = f(x)+\epsilon$ and the $f(x)$ follows ...
28 views

### Is there a relationship between RKHS and mixtures?

Reproducing kernel Hilbert spaces (RKHS) (see here or here) seem to involve conic combinations of "kernel functions" (my understanding is very crude). (See pp. 34-35 of Behrends, unfortunately the ...
110 views

### Prediciting an output with (kernelized) ridge regression

The optimal weight w* for ridge regression is ($\lambda$ is a positive scalar): $$w^* = (XX^T+\lambda I_n)^{-1}Xy$$ I want to predict the output for a new datapoint $x_i$, whereby $w^*$ is already ...
1k views

463 views

### Validating Kernel Functions

I'd appreciate help in clarifying my understanding of how to valid kernel functions, using the following two examples: $K(x, t) = x^Tt - (x^Tt)^2$ $K(x, t) = e^{(x_1t_1)}$ where $x_1\ and\ t_1$ are ...
225 views

### General form of Epanechnikov kernel

This Wikipedia source states the Epanechnikov kernel to be of the form: k_{n,d}=\left\{ \begin{array}{ll} \frac{1}{2d} \cdot \left(1 + \frac{1}{2n}\right) \cdot \left(1 - \left(\...
32 views

### Approximate arbitrary covariance matrix using kernel

I would like to approximate a arbitrary covariance matrix using a kernel. In hear kernel can only be summation or multiplication of each kernel (rbf, linear ,periodic, constant). I don't want to use ...
For a Gaussian kernel, what is the sigma value, and how is it calculated? $K(\mathbf{x}_i,\mathbf{x}_j) = \exp{-\frac{\|\mathbf{x}_i-\mathbf{x}_j\|^2}{\sigma^2}}$ ? Is it the covariance of the ...