Questions tagged [kernel-trick]

Kernel methods are used in machine learning to generalize linear techniques to nonlinear situations, especially SVMs, PCA, and GPs. Not to be confused with [kernel-smoothing], for kernel density estimation (KDE) and kernel regression.

Filter by
Sorted by
Tagged with
2
votes
1answer
24 views

intuition behind Cover's theorem?

I was going over https://en.wikipedia.org/wiki/Cover%27s_theorem And I am a bit lost with the intuition. I do understand that if it's not linearly separable, then projecting it into a higher-...
0
votes
0answers
19 views

Problem with understanding this particular RBF kernel approximation

I can't figure out where this approximation comes from. I tried to read the bibliographical references proposed by the author but did not come to a conclusion. The paper I'm studying is the following:...
5
votes
2answers
125 views

Meaning of “Distribution over functions” in Gaussian Processes?

I'm reviewing PyMC3's Gaussian Process documents and it's illuminated that I might have a flawed understanding of what "distribution over functions" actually means. Consider the below code: <...
0
votes
0answers
9 views

Ambiguous kernel graph for Gaussian processes in Pattern Recognition and Machine Learning

In the book Pattern Recognition and Machine Learning, the author shows these graphs in the context of Gaussian processes for regression (section 6.4.2) and states, "One widely used kernel ...
0
votes
0answers
21 views

more comprehensive list of kernels than crsouza’s blog of 25 kernels?

What’s the most comprehensive list of positive semidefinite or kernels with feature maps that work well in practice that’s not covered in http://crsouza.com/2010/03/17/kernel-functions-for-machine-...
2
votes
0answers
19 views

What is the difference between pairwise kernels and pairwise distances?

What is the difference between pairwise kernels and pairwise distances? I frequently came across terms like pairwise kernels and pairwise distances while learning about Pairwise metrics, Affinities, ...
1
vote
0answers
41 views

Prove that $K(x_{1},x_{2})=f(x_{1})K_{1}(x_{1},x_{2})f(x_{2})$ is symmetric positive definite if $K_{1}(x_{1},x_{2})$ is symmetric positive definite

I have been trying to prove this 5th proposition on the 25th slide We can prove that $ K(x_{1},x_{2}) $ will be a valid kernel as: $$ K(x_{1},x_{2}) = f(x_{1})K_{1}(x_{1},x_{2})f(x_{2}) \\ = f(x_{1})\...
1
vote
0answers
34 views

Convolution kernels

Below are two different convolution kernel formulas, h and H, written in Python which I think are both symmetric. What is the ...
2
votes
0answers
18 views

Interpretation of the alpha parameter in the Rational Quadratic Kernel

I've been working on Gaussian processes and a problem that keeps bugging me is the alpha parameter on the rational quadratic kernel I know that the rational quadratic is an infinite sum of squared ...
1
vote
1answer
55 views

Is it possible to find cluster centroids in kernel K means?

Suppose ${x_1, \ldots, x_N}$ are the data points and we have to find $K$ clusters using Kernel K Means. Let the kernel be $Ker$ (not to confuse with $K$ number of clusters) Let $\phi$ be the implicit ...
0
votes
0answers
12 views

In classical kernel regression, is there a specific task which responds almost exclusively to a single kernel choice?

I'm curious if there is any well-known kernel regression/classification task which can only be "solved" using a specific choice of kernel?
0
votes
0answers
14 views

Why minimize radius in support vector clustering

I have recently started studying machine learning on my own. I am reading support vector machines and then support vector clustering. https://papers.nips.cc/paper/2000/file/...
3
votes
1answer
33 views

difference between the “Kernel Convolution” and “Kernel PCA”

Can anybody explain the difference between the "Kernel Convolution" and "Kernel PCA" to me, please?
0
votes
0answers
20 views

Nonlinear regression with multiple outputs

I'm trying to learn a smooth, nonlinear mapping between regions of $\mathbb{R}^2$ by doing a regression. In the past, I've used Gaussian process regression to learn similar mappings where the output ...
6
votes
1answer
29 views

two times square in distance calculation on one example?

I read a book on Kernels, See the following example. Why the authors take square two times here? what is the logic?
1
vote
0answers
30 views

how to find a feature map

I have a dataset and it's 2 concentric circles centered at 0 of radius 1 and 2 corresponding to the two different classes. It's easy to seperate the data to get a classifier with 100% accuracy (I'm ...
0
votes
0answers
46 views

How to use the kernel trick on a XOR-like dataset

Let's say that I have the following data: I want to find a transformation of this dataset that will make it linearly separable. My thought was to bring the data around the origin and then multiply $...
1
vote
1answer
32 views

How do we come up with the SVM Kernel giving $n+d\choose d$ feature space?

I was going through the CS229 notes on SVM and Kernel tricks and I came across the following line. More generally the kernel $K(x,z)=(xTz+c)^d$ corresponds to a feature mapping to an $n+d\choose d$ ...
5
votes
1answer
105 views

Probability density from Hilbert-Schmidt integral operator

The Hilbert-Schmidt integral operator determines the underlying measure, if a universal kernel is used. Now, do eigenvalues of the Hilbert-Schmidt integral operator determine the underlying measure up ...
1
vote
2answers
75 views

Why is Dirac kernel positive semi-definite?

I read a paper Weisfeiler-Lehman Graph Kernel. In this paper, it says: Let the base kernel $k$ be a function counting pairs of matching node labels in two graphs: $k\left(G, G^{\prime}\right)=\sum_{v ...
0
votes
1answer
34 views

Why is the Dual Formulation a valid reparametrization of a regression model

In polynomial regression problems, in which an input vector $\underline{\phi}(\underline{x})$ is used to map a feature vector to a higher dimensional space (an example of this being $(x_{1}, x_{2}) \...
0
votes
2answers
45 views

Relationship between structural or statistical properties and hardness of classification

I am trying to understand the relationship between structural or statistical properties of training dataset and hardness of classification in the context of binary classification with SVM using RBF ...
1
vote
0answers
17 views

Proving if a function is a kernal [closed]

Given a function $k(x,x') = s^T.x_i.x_j^T.s$ where $s \in \mathbf{R^d}$. I want to prove using the definition of positive definitness that for any vector $v \in \mathbf{R^n}$ $v^T.K.v \geq 0$.
1
vote
0answers
27 views

Kernel ridge regression and Gaussian Process Regression

One knows that through the both methods mentioned in the title, in regression setting, with the same kernel $K$, the result is the same. It may be a very naive question but why? To me, they are quite ...
1
vote
1answer
47 views

Proving that a function is not a kernel function

The function is defined as $k(x,x')=||x||$ Norm in Hilbert Spaces can be defined as $||x||= \sqrt{x^Tx} $. I am not sure about the feature map of this function that how will it be and I am positive ...
3
votes
1answer
33 views

Prove that a function is not a kernel

$k(x,x') = \alpha k_1(x,x) + \beta k_2(x',x')$ is a kernel if $k_1$ and $k_2$ are kernels Prove that this statement is false for all $\alpha,\beta \in \mathbb{R}$ How to check for symmetricity of $k(x,...
0
votes
0answers
36 views

Feature map of Polynomial Kernel

The polynomial kernel is defined as $k(x,x') = (\langle x,x' \rangle +c)^m $ The feature map for polynomial kernel as introduced by my lecturer is given as $\phi:x \mapsto c_i(x_1^{i_1}+x_2^{i_2}+x_3^{...
2
votes
0answers
29 views

How can I model interactions between two groups of features using kernel methods?

Basically I am looking to fit a linear regression model using two groups of features that are completely different in nature (genomic data and say... weather data). I'm looking to extract main effects ...
0
votes
0answers
10 views

Posterior Gaussian process covariance operator

In Gaussian processes, we often see updates for the posterior covariance matrix at a set of points. However, the posterior covariance is actually an infinite dimensional operator. We often see the ...
3
votes
0answers
126 views

Does training loss go to zero in kernel regression?

Edit Have left the original post in tact, scroll to bottom for updated thinking High Level Problem Statement While studying kernel regression, after playing around with some linear algebra, I appear ...
1
vote
1answer
29 views

What is a natural way to define RKHS over mixed spaces (discrete and continuous)?

It is well known that given a kernel $k$ over any space $\mathcal{X}$, there is a corresponding RKHS (Reproducing Kernel Hilbert Space) associated with the kernel $k$. For example, Radial basis ...
0
votes
0answers
20 views

Dimensionality problem in dual SVM regression formulation

Consider the Boston Housing dataset. If we denote the house price with $y$ and the vector of predicting variables with $x$, then the Kernel SVMs are solved by considering the following dual convex ...
2
votes
1answer
149 views

why is rbf kernel svm a non-parametric algorithm?

I was reading up the difference between parametric and non-parametric models on this site: https://sebastianraschka.com/faq/docs/parametric_vs_nonparametric.html It says that linear SVM is parametric ...
1
vote
0answers
42 views

How is a polynomial kernel with infinite degree different from RBF Kernel?

I was reading about polynomial and RBF Kernels. According to my understanding: Polynomial kernels with degree >1 map the non-linear data into a higher dimensional feature space. Data that aren't ...
0
votes
0answers
9 views

Is there a relationship between the reproducing property of RKHS and eigenpair integral equations?

When we solve for the eigenpairs of a kernel we have the following equation: \begin{align} \lambda\phi(x)&=\int k(t,x)\phi(t)dt \end{align} where the right hand side can be interpreted as an inner ...
0
votes
0answers
16 views

Prove that the mixed partial derivative of a valid kernel is still a valid kernel

I have a vague memory of reading somewhere that the mixed partial derivative of a valid kernel is still a valid kernel but I cannot seem to find the original source. Does anyone have anything on it?
2
votes
1answer
104 views

Random Fourier Features vs Eigenfunctions for Gaussian Process Kernel Approximations?

Say we define kernels in Gaussian processes. There are two approaches to approximating them: random fourier features and eigenfunctions of the kernel. What are the tradeoffs to using each? If we ...
1
vote
0answers
15 views

Are haar bases eigenfunctions for any kernel?

Are haar wavelet bases eigenfunctions for any kernel? If so, what Kernel is it, and how would we find the eigenvalues?
1
vote
0answers
30 views

Is it necessary that an explicit feature map exists with all kernels? [duplicate]

Consider the Radial Basis Kernel $$K(x,z) = \exp\left(-\frac{\|x−z\|^2}{2\sigma^2}\right)$$ Is it possible to find a feature map in this case? Is it necessary that an explicit feature map exists with ...
0
votes
0answers
164 views

Compare distributions using Maximum Mean Discrepancy (MMD)

I use MMD distance to run a permutation test and decide whether two sample distributions come from the same distribution or not. For the MMD, I use a gaussian kernel, the bandwidth of which I select ...
1
vote
1answer
70 views

What is the difference between kernel function and kernel trick?

My question is regarding the SVM topic. What is the difference between kernel function and kernel trick? Are they same and refer to the same thing?
0
votes
0answers
18 views

Can I customize the kernel function?

I want to know whether I can customize the kernel function? For example, the polynomial kernel is defined as: $$ K(x,y) = (x^Ty+c)^d $$ Could I modify it to the following: $$ K(x,y) = (||x-y||_2)^d $$ ...
0
votes
0answers
37 views

Pros and cons of different MKL algorithms

I have been using multiple kernel learning (MKL) to train a classifier and got some exposure to the field. However, I am quite new to machine learning and I have only an intuitive understanding of the ...
1
vote
1answer
283 views

Kernel approximation with Nystroem method and usage in scikit-learn

I am planning to use the Nystroem method to approximate a Gram matrix induced by any kernel function. I found the Nystroem implementation in scikit-learn. As far as I understood, the full Gram Matrix ...
1
vote
1answer
121 views

How to understand mapping function of kernel?

For a kernel function, we have two conditions one is that it should be symmetric which is easy to understand intuitively because dot products are symmetric as well and our kernel should also follow ...
1
vote
1answer
46 views

Proving that a kernel function is kernel

Let's suppose we have a kernel function $k(x,x')=10 $ In order to prove that this a valid kernel function there are generally two conditions It is symmetric There exists a map $\varphi:R^d \...
1
vote
1answer
54 views

Why people prefer neural network to kernel methods?

I am learning Kernel methods. Kernel methods are less a "black box" than neural networks. Nowadays, it seems neural networks gain more popularity and show more powerful in various ...
1
vote
2answers
58 views

Why is it called $\chi^2$ distance / kernel?

The $\chi^2$ distance function is defined as $$ \chi(u,v) = \sum_{i=1}^n \frac{(u_i-v_i)^2}{u_i+v_i} $$ and the $\chi^2$ kernel function, used in support vector machines, is $$ K(u,v) = \exp(-c \...
0
votes
1answer
32 views

Finding optimal kernel parameters

I want to perform multiple kernel learning on my dataset and apply each (rbf) kernel to a different subset of features to then combine them. I do not want to have the same kernel with a range of ...
0
votes
0answers
16 views

Applying different kernels to parts of a dataset and merging the output [duplicate]

I am trying to create a classifier using SVM on a dataset that is composed of 6 sets of data for each of my observations. When I train the SVM (rbf kernel), I get a better performance of the ...

1
2 3 4 5
13