Questions tagged [kernel-trick]

Kernel methods are used in machine learning to generalize linear techniques to nonlinear situations, especially SVMs, PCA, and GPs. Not to be confused with [kernel-smoothing], for kernel density estimation (KDE) and kernel regression.

Filter by
Sorted by
Tagged with
0
votes
1answer
223 views

How to Interpret output Coefficients of Linear Support Vector Regression?

I'm looking to interpret the output from my SVR model. I know that with SVM you can't directly interpret the coefficients of the model but that you first have to take a dot product With that said, ...
3
votes
1answer
283 views

Are there examples of covariance functions used in Gaussian processes with negative non-diagonal elements?

I've been searching through numerous kernels used in Gaussian processes, and one common feature is that the covariance matrices always have only positive elements. Yet the only requirement on the ...
1
vote
0answers
197 views

Maximum Mean Discrepancy Implementation

I am just beginning to learn about MMD as a way to measure the difference between two probability distributions using this tutorial. I want to implement it code-wise but I don't understand it ...
4
votes
1answer
251 views

Why isn't a gaussian kernel subject to the curse of dimensionality?

This has been bugging me for a while now. I understand from this answer why gaussian kernels are effective. But I can't wrap my head around the intuition of why the infinite dimensional feature map 𝜙(...
2
votes
0answers
107 views

Is kernalized linear regression parametric or nonparametric?

We know that for linear regression, we can predict: $$\hat{y} = w^Tx +b$$ Where $w$ is the parameter that minimizes the square loss. It is easy to prove that for the final solution using gradient ...
1
vote
1answer
30 views

A derivation regarding kernel regression for the support vector machine

THis is from the Elements of Statistical Learning book page 437 in the section of support vector machine. Can anyone give me some hint for the missing derivation steps for why 12.49 is true (as seen ...
0
votes
1answer
58 views

Expansion of inner product for polynomial kernel for SVMs

On page 424 in "The Elements of Statistical Learning" by Hastie et al (2013) (https://web.stanford.edu/~hastie/Papers/ESLII.pdf), we see the following expansion of a polynomial kernel with degree 2: ...
0
votes
1answer
29 views

Relevant Dimension Estimation: Showing that $\sum_{i=1}^N s_i^2 = ||y||^2$

In relevant dimension estimation, we are given a Kernel Matrix $K \in \mathbb{R}^{n \times n}$, where $K_{ij} = k(x_i, x_j)$. We then compute the kernel eigenvector from the multiple solutions of the ...
1
vote
1answer
365 views

Can someone provide a brief explanation as to why reproducing kernel Hilbert space is so popular in machine learning?

I thought functional analysis was long thought to be old fashioned and generally a dead research area. It seems that all of a sudden there is a huge fascination with so-called reproducing kernel ...
1
vote
1answer
35 views

Kernels and Features

I am studying about kernel ridge regression and please bear with me, I have a couple points that left me very startled. Here is the equation for Kernel ridge regression: $f^{ridge}(x) = \phi(x)^T\...
2
votes
1answer
34 views

Computational analysis kernel trick

I was reading through "kernel trick" since I wasn't familiar with it. It is my understanding that apart from a better classification boundary (literally the geometric boundary) there should be some ...
0
votes
0answers
68 views

Sum of two kernel is also a kernel, how can we prove that? [duplicate]

Suppose that k(·, ·) and k 0 (·, ·) are kernels. Prove that l(·, ·) where l(x, y) = k(x, y) + k0 (x, y) is also a kernel. I am having trouble proving this, can you help?
0
votes
0answers
34 views

Linear separation in higher dimension [duplicate]

I am having a problem comprehending with the relation of kernel, weight and linear separation. I have a case where I am given a kernel $k_1$. that has a corresponding mapping $\phi_1$. And we ...
1
vote
2answers
730 views

What kind of kernel is used by statsmodels.nonparametric.kernel_regression.KernelReg?

I am doing multivariate nonparametric kernel regression using the Python function as mentioned in the title. The documentation can be found here: https://www.statsmodels.org/stable/generated/...
0
votes
2answers
293 views

Intuition behind the length-scale of the Rational Quadratic Kernel

What is the meaning of the length-scale in a rational quadratic? \begin{equation} k_{\textrm{RQ}}(t, t') = \sigma^2 \left( 1 + \frac{(t - t')^2}{2 \alpha \ell^2} \right)^{-\alpha} \label{eq:...
0
votes
1answer
33 views

When kernels are not useful in SVM?

In SVM using kernels we map the original features to the higher, transformer space (feature mapping) and then perform linear SVM in this higher space. But when kernels are not useful? I could not find ...
0
votes
2answers
216 views

RBF kernel mapping

I was reading that the Gaussian/RBF kernel maps its input onto the surface of normalized hypersphere. Our RBF kernel given by: $k(x,z) = exp(\frac{- ||x-z||^2}{2\sigma^2})$ Can anyone explain why ...
1
vote
0answers
17 views

Is there a representer theorem for unsupervised learning (to justify kernel density estimation)?

In supervised learning, we get a representer theorem by considering regularized losses of the following form: In Kernel Density Estimation, we simply directly assume densities of the form Could this ...
3
votes
1answer
267 views

Can SVM with Gaussian RBF kernel separate all kinds of data theoretically?

Gaussian is well known because its corresponding feature mapping is to infinite dimension. So with finite number of training data, is that the case that we can achieve zero training error with some ...
1
vote
0answers
68 views

Kernlab, user-defined kernel on chosen variables [closed]

I want to make a user-defined kernel with different variables in the kernel and combine them. Does anyone know if this is possible with kernlab or what is wrong with my code? I use these packages: <...
1
vote
0answers
660 views

Gaussian process vs. Bayesian linear regression / computational cost in weight space

Gaussian process (GP) regression with the linear covariance function $$k(x_i, x_j) = \sigma_0^2 + \sigma_1^2 x_i x_j + \delta(i=j)$$ can be seen as a Bayesian linear regression (BLR) model $$ y_i = ...
3
votes
1answer
474 views

Earth Movers Distance and Maximum Mean Discrepency

By Kantorovich-Rubinstein duality the Earth Movers Distance (EMD)/Wasserstein Metric is equivalent to Maximum Mean Discrepancy (MMD) correct? See here for a more thorough explanation. Why then does ...
1
vote
0answers
34 views

How to account for experimental errors when computing the derivative of a Gaussian process?

When applying Gaussian process regression upon training data, the covariance function can be generally given in the form: $\Sigma_{i,j} = k(x_i, x_j) + \sigma(x_i) \delta_{i,j}$, where $k$ is a ...
2
votes
0answers
186 views

Kernel function with a feature space equipped with an inner product that is not the dot product

Premise: A function $K: \mathbb R^d \times \mathbb R^d \to \mathbb R$ is called a kernel function on $\mathbb{R}^d$ if there exists a Hilbert space $\mathcal{H}$ and a map $\phi: \mathbb R^d \...
1
vote
1answer
212 views

What is the intuition behind changing the dot product for another inner product in SVM?

I understand that, when classifying with a SVM using a non-linear kernel, we are basically changing the dot product for a "custom" inner product. Is there some reason for working with a different ...
2
votes
1answer
81 views

Is there a Gaussian Process Kernel that limits functions to sigmoids?

I am modeling a large number of Dose-response curves. I have strong reason to believe that the generating function will be sigmoidal against the concentration of the assay (Michaelis-Menten kinetics). ...
1
vote
1answer
95 views

Is this a valid Gaussian Process kernel?

$\mathcal{K}\Big( \; (x,y), (x',y') \; \Big) = \sigma_f^2 \exp{ \frac{(x-x')^2}{2l^2 \cdot (y+y')^2} } $, where $l > 0$ The variance associated with each training point (given by a vector) is a ...
2
votes
1answer
982 views

Is a polynomial kernel ridge regression really equivalent to performing a linear regression on those expanded features?

Say we have a dataset, X, which is Nx2 where N is the number of examples and 2 is the number of dimensions "features". If we were to run a kernel ridge regression (or SVM or whatever) on these ...
2
votes
0answers
223 views

What is the most intuitive proof that Gaussian kernel is positive definite? [duplicate]

I have general form of Gaussian kernel $K(x,x')=\exp(-\|x-x'\|^{2})$ (just not considering $\sigma$). I tried to prove its positive definiteness via Gram matrix properties, but couldn't. Is there any ...
1
vote
0answers
64 views

How to prove 1-norm radial function is kernel?

How should I prove that is a valid kernel: $K(x,y)=exp(-\alpha||x-y||_1) $ As I understand, there are three ways to prove that prove $K(x,y)=<\phi(x) ,\phi(y) >$ prove $\sum_{j,k=1}^n a_j\,...
1
vote
0answers
53 views

Kernel trick: computationally inexpensive

I am reading this post about the kernel trick. The author claims that a calculation of a dot product of two vectors will need 100.000.000 multiplications given that both vectors are of a dimension (...
1
vote
1answer
67 views

Kernelize Linear Regression

We can kernelize Ridge regression as shown in these notes: https://www.ics.uci.edu/~welling/classnotes/papers_class/Kernel-Ridge.pdf. However would it be possible to find a vector $\boldsymbol\alpha$...
2
votes
1answer
411 views

Is the absolute value of the difference a kernel?

In particular is $$ k(x_i,x_j)=|x_i-x_j|, \quad x_i,x_j\in \mathbb{R}$$ a valid kernel?
3
votes
1answer
382 views

Gaussian Processes, Prior function of constant, Linear and Polynomial Kernel

In my university course slide, regarding the exponential and Gaussian kernel there are these two relevant pictures. They show the prior function of each kernel: You can see that the resulting prior ...
1
vote
1answer
226 views

Probabilistic Interpretation of Radial Basis Function

I was wondering if someone could flesh out the probabilistic interpretation of using the Radial Basis Function to compute the probability between an observation and some reference value. My question ...
1
vote
1answer
428 views

Using gradient descent to train dual formulation of Kernel SVM

I've seen other posts about using gradient descent for the primal form, but not the dual form. In this book, the author discusses using (projected) gradient descent for the dual form: http://ciml....
0
votes
0answers
106 views

If I center my kernel does it no longer remain positive semidefinite?? If so why is it being used in algorithms like kernel pca?

If I center my kernel then can it still be used in operations where a positive semi-definite kernel is required such as SVM and ridge regression? I am centering my kernel as follows: $$K_c(\mathbf{t}...
1
vote
1answer
21 views

Does Kernel Function Only Apply To Support Vector In SVM?

We know that the if $α=0$ in below equation it is not a support vector; support vectors have $α \ne 0$. $$ L(w, b, α) = \sum_{i=1}^m α_i − \frac12 \sum_{i,j=1}^m y^{(i)} y^{(j)} α_i α_j (x^{(i)})^T x^...
2
votes
2answers
966 views

Is a kernel function basically just a mapping?

I'm currently studying machine learning (support vector machines to be more specific), and I was wondering how exactly I should understand what a kernel function is. I've read other questions on this ...
1
vote
1answer
765 views

Choice of Gaussian process in non-parametric regression

I have been trying to understand non-parametric regression using Gaussian processes (GP), which are used to represent prior distributions over the space of functions. The linear model considered is $$ ...
3
votes
0answers
100 views

random kitchen sinks as approximation to kernel machine

In the paper Rahimi, Ali, and Benjamin Recht. "Random features for large-scale kernel machines." Advances in neural information processing systems. 2008. the author introduces a way to ...
5
votes
1answer
233 views

Uniqueness of Reproducing Kernel Hilbert Spaces

Digging in the definition of Reproducing Kernel Hilbert Spaces (RKHS) I came across the following example taken from pages 49-51 of [1]: Given the kernel $k(x,y) = \langle x,y\rangle^2$, with $x,y\in ...
1
vote
0answers
180 views

How to choose hyper-parameter for Gaussian Process kernels?

I'm trying to fit Gaussian Process in scikit-learn, and start with using kernel = RBF_1 + RBF_2 + whitekernel(sum of two RBF kernels with different length_scale and ...
4
votes
0answers
646 views

Why do we need the gamma parameter in the polynomial kernel of SVMs?

The polynomial kernel is sometimes defined as just: $$ K(x,y):=(\left<x,y\right>+c)^d $$ with two parameters: the degree $d$ and constant coefficient $c$. But others (e.g., libsvm, and sklearn ...
1
vote
0answers
189 views

Precomputed Kernels for Support Vector Machines (SVM)

To calculate the linear kernel matrix for some training matrix X with dimensions n x d where d is the number of features and n is the number of data points, we can simply do: $X * X^T$. The result is ...
1
vote
1answer
972 views

Finding the feature map corresponding to a specific Kernel? (Polynomial Kernels)

I am just getting into machine learning and I am kind of confused about how to show the corresponding feature map for a kernel. For example, how would I show the following feature map for this kernel? ...
1
vote
0answers
26 views

Prove that given kernel is valid and find the relevant mapping

Understanding Machine Learning: From Theory to Algorithms, Section 16.6, Question 4 is For $x>z$, I formulate my kernel matrix as $K = [x \quad z;z \quad z]$ which gives the cofactors as $x, z(x-...
1
vote
1answer
177 views

Checking if a kernel is valid

The kernel is $K(x,z) = \sum_{i=1}^D (x_i+z_i)$ My approach was trying to express $K = \phi(x)^T\phi(z) = (x_1 x_2 ... x_D \quad 1 1 ...1)(1 1 ...1\quad z_1 z_2 ... z_D )^T$ where $\phi$ is 2Dx1 ...
0
votes
1answer
186 views

Kernel ridge regression with matrix-vector data set $S := \{ X_i, y_i \}_{i=1}^{N}$?

Please notice that this question was asked in MO, but it seems that it doesn't interest MO community. So, I have got a comment to post in this community in the hope that I may get some attention to ...
1
vote
0answers
74 views

Use the features selected with RFE SVM linear for prediction of SVM rbf

I was wondering if the features selected with RFE with SVM linear kernel are still "good" features when we use a non linear model, like SVM rbf kernel. This comes in practice when you want to use SVM ...

1 2
3
4 5
13