Questions tagged [kernel-trick]

Kernel methods are used in machine learning to generalize linear techniques to nonlinear situations, especially SVMs, PCA, and GPs. Not to be confused with [kernel-smoothing], for kernel density estimation (KDE) and kernel regression.

Filter by
Sorted by
Tagged with
3
votes
0answers
65 views

Does training loss go to zero in kernel regression?

High Level Problem Statement While studying kernel regression, after playing around with some linear algebra, I appear to have managed to "prove" that training loss always goes to zero. I'll ...
1
vote
0answers
18 views

What is a natural way to define RKHS over mixed spaces (discrete and continuous)?

It is well known that given a kernel $k$ over any space $\mathcal{X}$, there is a corresponding RKHS (Reproducing Kernel Hilbert Space) associated with the kernel $k$. For example, Radial basis ...
2
votes
1answer
31 views

Random Fourier Features vs Eigenfunctions for Gaussian Process Kernel Approximations?

Say we define kernels in Gaussian processes. There are two approaches to approximating them: random fourier features and eigenfunctions of the kernel. What are the tradeoffs to using each? If we ...
2
votes
1answer
33 views

why is rbf kernel svm a non-parametric algorithm?

I was reading up the difference between parametric and non-parametric models on this site: https://sebastianraschka.com/faq/docs/parametric_vs_nonparametric.html It says that linear SVM is parametric ...
0
votes
0answers
16 views

Dimensionality problem in dual SVM regression formulation

Consider the Boston Housing dataset. If we denote the house price with $y$ and the vector of predicting variables with $x$, then the Kernel SVMs are solved by considering the following dual convex ...
71
votes
4answers
39k views

What makes the Gaussian kernel so magical for PCA, and also in general?

I was reading about kernel PCA (1, 2, 3) with Gaussian and polynomial kernels. How does the Gaussian kernel separate seemingly any sort of nonlinear data exceptionally well? Please give an intuitive ...
1
vote
0answers
13 views

How is a polynomial kernel with infinite degree different from RBF Kernel?

I was reading about polynomial and RBF Kernels. According to my understanding: Polynomial kernels with degree >1 map the non-linear data into a higher dimensional feature space. Data that aren't ...
0
votes
0answers
4 views

Is there a relationship between the reproducing property of RKHS and eigenpair integral equations?

When we solve for the eigenpairs of a kernel we have the following equation: \begin{align} \lambda\phi(x)&=\int k(t,x)\phi(t)dt \end{align} where the right hand side can be interpreted as an inner ...
0
votes
0answers
9 views

Prove that the mixed partial derivative of a valid kernel is still a valid kernel

I have a vague memory of reading somewhere that the mixed partial derivative of a valid kernel is still a valid kernel but I cannot seem to find the original source. Does anyone have anything on it?
0
votes
1answer
25 views

What is the difference between kernel function and kernel trick?

My question is regarding the SVM topic. What is the difference between kernel function and kernel trick? Are they same and refer to the same thing?
1
vote
0answers
15 views

Are haar bases eigenfunctions for any kernel?

Are haar wavelet bases eigenfunctions for any kernel? If so, what Kernel is it, and how would we find the eigenvalues?
28
votes
4answers
17k views

Feature map for the Gaussian kernel

In SVM, the Gaussian kernel is defined as: $$K(x,y)=\exp\left({-\frac{\|x-y\|_2^2}{2\sigma^2}}\right)=\phi(x)^T\phi(y)$$ where $x, y\in \mathbb{R^n}$. I do not know the explicit equation of $\phi$. I ...
1
vote
1answer
78 views

How to understand mapping function of kernel?

For a kernel function, we have two conditions one is that it should be symmetric which is easy to understand intuitively because dot products are symmetric as well and our kernel should also follow ...
1
vote
0answers
29 views

Is it necessary that an explicit feature map exists with all kernels? [duplicate]

Consider the Radial Basis Kernel $$K(x,z) = \exp\left(-\frac{\|x−z\|^2}{2\sigma^2}\right)$$ Is it possible to find a feature map in this case? Is it necessary that an explicit feature map exists with ...
0
votes
1answer
223 views

How to Interpret output Coefficients of Linear Support Vector Regression?

I'm looking to interpret the output from my SVR model. I know that with SVM you can't directly interpret the coefficients of the model but that you first have to take a dot product With that said, ...
0
votes
0answers
29 views

Compare distributions using Maximum Mean Discrepancy (MMD)

I use MMD distance to run a permutation test and decide whether two sample distributions come from the same distribution or not. For the MMD, I use a gaussian kernel, the bandwidth of which I select ...
1
vote
1answer
196 views

Better classification performance when using an RBF kernel function in high dimensional space?

I'm learning about SVM's and understand that boosting something into a higher dimension can sometimes help separate the data better. However, if I were to perform 1 nearest neighbor with the RBF ...
0
votes
1answer
37 views

Kernel approximation with Nystroem method and usage in scikit-learn

I am planning to use the Nystroem method to approximate a Gram matrix induced by any kernel function. I found the Nystroem implementation in scikit-learn. As far as I understood, the full Gram Matrix ...
0
votes
0answers
28 views

Pros and cons of different MKL algorithms

I have been using multiple kernel learning (MKL) to train a classifier and got some exposure to the field. However, I am quite new to machine learning and I have only an intuitive understanding of the ...
0
votes
0answers
16 views

Can I customize the kernel function?

I want to know whether I can customize the kernel function? For example, the polynomial kernel is defined as: $$ K(x,y) = (x^Ty+c)^d $$ Could I modify it to the following: $$ K(x,y) = (||x-y||_2)^d $$ ...
1
vote
1answer
34 views

Proving that a kernel function is kernel

Let's suppose we have a kernel function $k(x,x')=10 $ In order to prove that this a valid kernel function there are generally two conditions It is symmetric There exists a map $\varphi:R^d \...
1
vote
1answer
45 views

Why people prefer neural network to kernel methods?

I am learning Kernel methods. Kernel methods are less a "black box" than neural networks. Nowadays, it seems neural networks gain more popularity and show more powerful in various ...
0
votes
1answer
712 views

How does the shape of a decision boundary in relate between the original and kernel feature space?

I'm trying to get my head around the mathematics and implementation of SVM and hopefully gain some intuition into how kernels work and perhaps being able to, with a bit more confidence, define my own ...
0
votes
1answer
25 views

Finding optimal kernel parameters

I want to perform multiple kernel learning on my dataset and apply each (rbf) kernel to a different subset of features to then combine them. I do not want to have the same kernel with a range of ...
1
vote
2answers
43 views

Why is it called $\chi^2$ distance / kernel?

The $\chi^2$ distance function is defined as $$ \chi(u,v) = \sum_{i=1}^n \frac{(u_i-v_i)^2}{u_i+v_i} $$ and the $\chi^2$ kernel function, used in support vector machines, is $$ K(u,v) = \exp(-c \...
0
votes
0answers
16 views

Applying different kernels to parts of a dataset and merging the output [duplicate]

I am trying to create a classifier using SVM on a dataset that is composed of 6 sets of data for each of my observations. When I train the SVM (rbf kernel), I get a better performance of the ...
1
vote
1answer
1k views

Formulation for kernel k-means

Given $n$ points ${\boldsymbol{x_1},...,\boldsymbol{x_n}}$ and a nonlinear feature map $\Phi$, the kernel k-means problem is formulated as follows, $\underset{\boldsymbol{t}, \; \boldsymbol{\mu}_k \; ...
0
votes
0answers
18 views

What dimensionality reduction technique to use for high dimensional data?

I have a dataset which contains a lot of features (>>3). For computational reasons, I would like to apply a dimensionality reduction. At this point I could use different techniques: standard PCA ...
1
vote
1answer
30 views

Re-building a cross-validated SVM

Suppose we are cross-validating parameters of a Gaussian (radial) SVM on $k$ training observations. The parameters are the cost parameter $C$, and the deviation parameter $\gamma$. Then, $4k$ more ...
0
votes
0answers
19 views

Assumptions of SVM

I have a dataset that has been analysed using logistic regression. In this, several variables were non-linearly associated with the log odds of my outcome, so were transformed prior to their inclusion ...
0
votes
1answer
17 views

Kernel on two functions?

I am curious whether there are any literatures considering kernel functions whose inputs are two functions. For example I would like consider two 1-Liptchiz mappings $\pi, \sigma:\mathbb{R}^M\...
0
votes
0answers
15 views

Is there any possibilty to apply Kernel PCA to coupled data fields?

Thanks to a wealth of answers from the community here (with a special thank you to amoeba), kernel PCA became much clearer to me. So far I understand that the magic of kernel PCA lies in the kernel ...
1
vote
0answers
22 views

Meaning of support vectors in support vector regression

I know that the support vectors in a soft margin SVM classifier model essentially means the vectors on the margin or less than the margin(the ones within the tube containing the decision boundary), ...
5
votes
1answer
168 views

Kernels in SVM primal form

For a soft margin SVM in primal form, we have a cost function that is: $$J(\mathbf{w}, b) = C {\displaystyle \sum\limits_{i=1}^{m} max\left(0, 1 - y^{(i)} (\mathbf{w}^t \cdot \mathbf{x}^{(i)} + b)\...
2
votes
1answer
192 views

Convolution equality

If $k$ is an squared or absolutely integrable kernel are the belo equalities true ? $$z(s)=\int_{R}^{} \! k(u-d) x(u).du \ \ =\int_{R}^{} \! k(u+d) x(u).du \ \ $$ and $$\int_{R}^{} \! k(u-d) k(u-d^{...
2
votes
1answer
60 views

How to choose a kernel function for Gaussian process regression in a multivaritate settings?

Could you please suggest some lectures, books, or videos on how to choose a kernel function for Gaussian process regression in a multivaritate setting?
2
votes
1answer
101 views

Combination of two kernel functions [duplicate]

Could you help me with this kernel function? \begin{equation} K(x,y) = (x \cdot y)^{2} + (x \cdot y), \text{ where } x = (x_1, x_2)', y = (y_1, y_2)' \end{equation} I want to know if the ...
0
votes
0answers
15 views

1D representation for 2D toy data (about linear separability)

suppose there is a dataset with 2 features x1, and x2. the points (-1;-1); (1; 1); (-3;-3); (4; 4) belong to class 1 and (-1; 1); (1;-1); (-5; 2); (4;-8) belongs to class 2. I am confused in terms of ...
2
votes
1answer
1k views

Use RBF kernel with logistic regression?

There are some resources online (e.g. this one) on logistic regression with polynomial kernels, such as $$h_\theta(x)=logistic(\theta_0 + \theta_1x1+ \theta_3x_1^2 + \theta_4x_2^2)$$ I'm wondering ...
2
votes
0answers
25 views

tradeoff function for exploration/exploitation in the context of gaussian process bayesian optimisation

First, let's give some context to my issue: consider an objective function $f(x)$ defined over a compact set $D$, such that there exists $x^*, f(x^*) > f(x) \quad \forall x \in D$. My goal is to ...
8
votes
1answer
1k views

Eigenfunctions and eigenvalues of the exponential kernel

What are the eigenfunctions and the eigenvalues of the exponential kernel? The exponential kernel is defined as $$k(x,x')=\sigma^2\exp\left(\frac{||x-x'||}{l}\right)$$ where both $\sigma>0$ and $l&...
2
votes
1answer
477 views

Kernel exercise

I'm looking for any good reference that can help me to understand the following exercises about kernels and online learning. A training set $(x_1, y_1), ...,(x_m, y_m)$ is generic iff $\mathbf{x}_i = ...
0
votes
1answer
16 views

Don't understand one particular normal distribution notation from opaper Stochastic Variational Deep Kernel Learning, help needed

I'm in progress working with paper "Stochastic Variational Deep Kernel Learning" NIPS 2016 and I have the problem with understanding the meaning of this normal distribution notation from part 2 ...
2
votes
1answer
182 views

How to fit a single quadratic term to a regression

I have a high dimensional multivariate model and am fitting linear weights to each of the $N$ free variables using a classic stable SVD matrix solver. This works. I want to improve the fit by using a ...
0
votes
0answers
41 views

Feature map corresponding to diffusion kernel defined over graphs/discrete structures?

It is well known that there exists an explicit feature map representation for every kernel ($K$) i.e. for any two points $x_1$ and $x_2$ \begin{align} K(x_1, x_2) = \Phi(x_1)^T\Phi(x_2) \end{align} ...
0
votes
0answers
9 views

Kernel function in SVM influence the decision boundary?

Does the kernel function used in SVM training influence the 'shape' of its decision boundary? I seem to have an impression that the RBF kernel function used in training seem to 'appear' also in the ...
4
votes
1answer
936 views

How to plot ROC for knn (and potentially kernel spectral regression)

I understand how to plot ROC for logistic classifier (like varies the probability cutoff). For KNN, how can I find the ROC? Also, what about kernel spectral regression?
0
votes
0answers
23 views

Hilbert Schmidt Independence Criterion For Large Data Set

So I'm trying the implement the Backward Feature Selection algorithm using the Hilbert Schmidt Independence Criterion (a.k.a BAHSIC) but I'm getting an out of memory error when calculating the kernel'...
2
votes
1answer
374 views

How to calculate RKHS norm of a function under given kernel transformation

This was a question asked before in mathoverflow but not yet got answered. I have the same problem when reading Srinivas et al (2010) [appendix B]'s paper. Here are my problems: Definitions: ...
1
vote
0answers
59 views

Is $cos^n(x^2-y^2)$ a valid mercer kernel function?

How to show if $cos^n(x^2-y^2)$ is a valid mercer kernel function if $n$ is positive? For $cos(x^2-y^2)$ I would assume that: $cos(x^2-y^2) = sin(x^2)sin(y^2)+cos(x^2)cos(y^2)$ Is a valid mercer ...

1
2 3 4 5
13