Questions tagged [kernel-trick]

Kernel methods are used in machine learning to generalize linear techniques to nonlinear situations, especially SVMs, PCA, and GPs. Not to be confused with [kernel-smoothing], for kernel density estimation (KDE) and kernel regression.

538 questions
Filter by
Sorted by
Tagged with
10 views

Determine dimension of mapping function Phi from form of Kernel

If I have a kernel of the form $k(x, y) = (x^Ty)^n$ where $x$ and $y$ are d-dimensional, how can I determine the dimension of the mapping function $\phi(x)$, in terms of n and d, without explicitly ...
15 views

Kernel Confusion

Consider the function $K(\vec{x},\vec{y})$ where $\vec{x},\vec{y} \in \mathbb{R}^n$. I have been asked to check that this is a valid kernel. Question 1 My understanding is that I can prove this in ...
6 views

16 views

General approach to prove if the given function is NOT a valid kernel function

In general, for proving that the given kernel function is valid, I try one of the following two approaches: Check if the gram matrix is Symmetric Postive Semi Definite. Check if the kernel function ...
26 views

Kernel:Why is the dot product a “measure of similarity” of instances? [duplicate]

Not a duplicate since the linked question does not answer this question: A measure of similarity should be maximal for instances which are the same (e.g. similarity between (1,1) and (1,1) should be ...
55 views

For what values of $\beta \in \mathbb{R}$ is $t(x-x')=-||x-x'||^\beta$ a kernel?

For what values of $\beta \in \mathbb{R}$ is $t(x-x')=-||x-x'||^\beta$ a kernel? I know that kernels of type $t(x-x')$ where $t$ is function that inverts the dissimilarity $x-x'$ into a similarity ...
120 views

Invertibility of Random Fourier Features

Is it possible to approximately reconstruct a point $\mathbf{x}$ in a vector space (say $\mathbb{R}^n$) given it's randomized feature map $z(\cdot)$ and respective projection $z(\mathbf{x})$ (in ...
42 views

I want to know the relationship between Discriminant functions and the kernel in SVM

The following articles are reprinte of #3338212 of math.stackexchange.com. It was recommended to ask this community at math.stackexchange.com. The following 【Quiz】 and 【Official Answer】are the ...
29 views

Support Vector Machine: identifying support vectors and kernel linear separability

I went through the MIT Artificial Intelligence lecture on Support Vector Machines by Professor Patrick Winston: https://www.youtube.com/watch?v=_PwhiWxHK8o I've got a couple of questions. Would be ...
53 views

Understanding the reproducing property of RKHS

I am currently trying to learn about Reproducing Kernel Hilbert spaces (RKHS) and would like to gain some intuition about its reproducing property. The RKHS is defined with kernel $k(x,x')$ which maps ...
21 views

why the result of primal sim is not the same as linear kernel?

I have a data set with 36000 rows and 9 columns. so n<< m. this is multi classification SVM. I solved this by primal model in OSQP. then I used R package e710 and svm with linear kernel. the ...
94 views

What are kernels in support vector machine? [duplicate]

What are kernels in support vector machines? I have tried many contents but i am not familiar with Lagrange and Laplace concept in mathematics. So anyone can please elaborate concepts of kernels in ...
33 views

Random fourier features and Bochner's Theorem

The paper, Random Fourier Features for Large-Scale Kernel Machines by Ali Rahimi and Ben Recht , makes use of Bochner's theorem which says that the Fourier transform $p(w)$ of shift-invariant ...
20 views

How would phi of the gaussian rbf kernel map a 100-by-3 dimensional feature matrix?

Would a 100-by-3 dimensional feature matrix be mapped into a 100 dimensional or into a infinite dimensional feature space, if the mapping would not be bypassed by the Gaussian RBF Kernel? Following ...
34 views

Kernel PCA: Find most important variables for each PC

To find the most important variable for each Principal Component is easy with PCA: With data->X and variables->variable_names ...
36 views

How to Interpret output Coefficients of Linear Support Vector Regression?

I'm looking to interpret the output from my SVR model. I know that with SVM you can't directly interpret the coefficients of the model but that you first have to take a dot product With that said, ...
28 views

How to show or prove a dataset is not linearly separable

I am looking to be pointed in the right direction. I am learning about kernels and I have a homework assignment to use the dual perceptron algorithm to classify datapoints from a spiral dataset, with ...
55 views

Are there examples of covariance functions used in Gaussian processes with negative non-diagonal elements?

I've been searching through numerous kernels used in Gaussian processes, and one common feature is that the covariance matrices always have only positive elements. Yet the only requirement on the ...
32 views

Maximum Mean Discrepancy Implementation

I am just beginning to learn about MMD as a way to measure the difference between two probability distributions using this tutorial. I want to implement it code-wise but I don't understand it ...
55 views

Why isn't a gaussian kernel subject to the curse of dimensionality?

This has been bugging me for a while now. I understand from this answer why gaussian kernels are effective. But I can't wrap my head around the intuition of why the infinite dimensional feature map 𝜙(...
41 views

Is kernalized linear regression parametric or nonparametric?

We know that for linear regression, we can predict: $$\hat{y} = w^Tx +b$$ Where $w$ is the parameter that minimizes the square loss. It is easy to prove that for the final solution using gradient ...
17 views

A derivation regarding kernel regression for the support vector machine

THis is from the Elements of Statistical Learning book page 437 in the section of support vector machine. Can anyone give me some hint for the missing derivation steps for why 12.49 is true (as seen ...
23 views

Expansion of inner product for polynomial kernel for SVMs

On page 424 in "The Elements of Statistical Learning" by Hastie et al (2013) (https://web.stanford.edu/~hastie/Papers/ESLII.pdf), we see the following expansion of a polynomial kernel with degree 2: ...
29 views

Relevant Dimension Estimation: Showing that $\sum_{i=1}^N s_i^2 = ||y||^2$

In relevant dimension estimation, we are given a Kernel Matrix $K \in \mathbb{R}^{n \times n}$, where $K_{ij} = k(x_i, x_j)$. We then compute the kernel eigenvector from the multiple solutions of the ...
11 views

The KL-divergence kernel based method

I have read some papers about applying KL-divergence on kernel method, but they don't give any details about why this KL-divergence kernel is positive definite, which confuses me a lot. So does anyone ...
70 views

Can someone provide a brief explanation as to why reproducing kernel Hilbert space is so popular in machine learning?

I thought functional analysis was long thought to be old fashioned and generally a dead research area. It seems that all of a sudden there is a huge fascination with so-called reproducing kernel ...
27 views

153 views

What kind of kernel is used by statsmodels.nonparametric.kernel_regression.KernelReg?

I am doing multivariate nonparametric kernel regression using the Python function as mentioned in the title. The documentation can be found here: https://www.statsmodels.org/stable/generated/...
25 views

Reproducing kernels: how do I numerically compute the decomposition?

Suppose I'm given a kernel, $$K(x,y) : \mathbb{R} \times \mathbb{R} \rightarrow \mathbb{R}$$ In order to describe/understand the (unique) associated RKHS, I seek its eigenfunctions, as per Mercer'...