Questions tagged [kernel-trick]

Kernel methods are used in machine learning to generalize linear techniques to nonlinear situations, especially SVMs, PCA, and GPs. Not to be confused with [kernel-smoothing], for kernel density estimation (KDE) and kernel regression.

Filter by
Sorted by
Tagged with
1
vote
0answers
18 views

how to find a feature map

I have a dataset and it's 2 concentric circles centered at 0 of radius 1 and 2 corresponding to the two different classes. It's easy to seperate the data to get a classifier with 100% accuracy (I'm ...
0
votes
0answers
12 views

How kernalised correlation filter-tracker works opencv [closed]

I'm using KCF-Tracker to track small colored cars (see image below) As you know implementing KCF in OpenCV is straight forward: ...
0
votes
0answers
30 views

How to use the kernel trick on a XOR-like dataset

Let's say that I have the following data: I want to find a transformation of this dataset that will make it linearly separable. My thought was to bring the data around the origin and then multiply $...
1
vote
1answer
23 views

How do we come up with the SVM Kernel giving $n+d\choose d$ feature space?

I was going through the CS229 notes on SVM and Kernel tricks and I came across the following line. More generally the kernel $K(x,z)=(xTz+c)^d$ corresponds to a feature mapping to an $n+d\choose d$ ...
5
votes
1answer
92 views

Probability density from Hilbert-Schmidt integral operator

The Hilbert-Schmidt integral operator determines the underlying measure, if a universal kernel is used. Now, do eigenvalues of the Hilbert-Schmidt integral operator determine the underlying measure up ...
0
votes
2answers
56 views

Why is Dirac kernel positive semi-definite?

I read a paper Weisfeiler-Lehman Graph Kernel. In this paper, it says: Let the base kernel $k$ be a function counting pairs of matching node labels in two graphs: $k\left(G, G^{\prime}\right)=\sum_{v ...
0
votes
1answer
32 views

Why is the Dual Formulation a valid reparametrization of a regression model

In polynomial regression problems, in which an input vector $\underline{\phi}(\underline{x})$ is used to map a feature vector to a higher dimensional space (an example of this being $(x_{1}, x_{2}) \...
0
votes
2answers
43 views

Relationship between structural or statistical properties and hardness of classification

I am trying to understand the relationship between structural or statistical properties of training dataset and hardness of classification in the context of binary classification with SVM using RBF ...
1
vote
0answers
16 views

Proving if a function is a kernal [closed]

Given a function $k(x,x') = s^T.x_i.x_j^T.s$ where $s \in \mathbf{R^d}$. I want to prove using the definition of positive definitness that for any vector $v \in \mathbf{R^n}$ $v^T.K.v \geq 0$.
1
vote
0answers
26 views

Kernel ridge regression and Gaussian Process Regression

One knows that through the both methods mentioned in the title, in regression setting, with the same kernel $K$, the result is the same. It may be a very naive question but why? To me, they are quite ...
1
vote
1answer
46 views

Proving that a function is not a kernel function

The function is defined as $k(x,x')=||x||$ Norm in Hilbert Spaces can be defined as $||x||= \sqrt{x^Tx} $. I am not sure about the feature map of this function that how will it be and I am positive ...
3
votes
1answer
28 views

Prove that a function is not a kernel

$k(x,x') = \alpha k_1(x,x) + \beta k_2(x',x')$ is a kernel if $k_1$ and $k_2$ are kernels Prove that this statement is false for all $\alpha,\beta \in \mathbb{R}$ How to check for symmetricity of $k(x,...
0
votes
0answers
24 views

Feature map of Polynomial Kernel

The polynomial kernel is defined as $k(x,x') = (\langle x,x' \rangle +c)^m $ The feature map for polynomial kernel as introduced by my lecturer is given as $\phi:x \mapsto c_i(x_1^{i_1}+x_2^{i_2}+x_3^{...
2
votes
0answers
26 views

How can I model interactions between two groups of features using kernel methods?

Basically I am looking to fit a linear regression model using two groups of features that are completely different in nature (genomic data and say... weather data). I'm looking to extract main effects ...
0
votes
0answers
9 views

Posterior Gaussian process covariance operator

In Gaussian processes, we often see updates for the posterior covariance matrix at a set of points. However, the posterior covariance is actually an infinite dimensional operator. We often see the ...
3
votes
0answers
119 views

Does training loss go to zero in kernel regression?

Edit Have left the original post in tact, scroll to bottom for updated thinking High Level Problem Statement While studying kernel regression, after playing around with some linear algebra, I appear ...
1
vote
1answer
26 views

What is a natural way to define RKHS over mixed spaces (discrete and continuous)?

It is well known that given a kernel $k$ over any space $\mathcal{X}$, there is a corresponding RKHS (Reproducing Kernel Hilbert Space) associated with the kernel $k$. For example, Radial basis ...
0
votes
0answers
18 views

Dimensionality problem in dual SVM regression formulation

Consider the Boston Housing dataset. If we denote the house price with $y$ and the vector of predicting variables with $x$, then the Kernel SVMs are solved by considering the following dual convex ...
2
votes
1answer
75 views

why is rbf kernel svm a non-parametric algorithm?

I was reading up the difference between parametric and non-parametric models on this site: https://sebastianraschka.com/faq/docs/parametric_vs_nonparametric.html It says that linear SVM is parametric ...
1
vote
0answers
27 views

How is a polynomial kernel with infinite degree different from RBF Kernel?

I was reading about polynomial and RBF Kernels. According to my understanding: Polynomial kernels with degree >1 map the non-linear data into a higher dimensional feature space. Data that aren't ...
0
votes
0answers
7 views

Is there a relationship between the reproducing property of RKHS and eigenpair integral equations?

When we solve for the eigenpairs of a kernel we have the following equation: \begin{align} \lambda\phi(x)&=\int k(t,x)\phi(t)dt \end{align} where the right hand side can be interpreted as an inner ...
0
votes
0answers
11 views

Prove that the mixed partial derivative of a valid kernel is still a valid kernel

I have a vague memory of reading somewhere that the mixed partial derivative of a valid kernel is still a valid kernel but I cannot seem to find the original source. Does anyone have anything on it?
2
votes
1answer
65 views

Random Fourier Features vs Eigenfunctions for Gaussian Process Kernel Approximations?

Say we define kernels in Gaussian processes. There are two approaches to approximating them: random fourier features and eigenfunctions of the kernel. What are the tradeoffs to using each? If we ...
1
vote
0answers
15 views

Are haar bases eigenfunctions for any kernel?

Are haar wavelet bases eigenfunctions for any kernel? If so, what Kernel is it, and how would we find the eigenvalues?
1
vote
0answers
30 views

Is it necessary that an explicit feature map exists with all kernels? [duplicate]

Consider the Radial Basis Kernel $$K(x,z) = \exp\left(-\frac{\|x−z\|^2}{2\sigma^2}\right)$$ Is it possible to find a feature map in this case? Is it necessary that an explicit feature map exists with ...
0
votes
0answers
86 views

Compare distributions using Maximum Mean Discrepancy (MMD)

I use MMD distance to run a permutation test and decide whether two sample distributions come from the same distribution or not. For the MMD, I use a gaussian kernel, the bandwidth of which I select ...
1
vote
1answer
34 views

What is the difference between kernel function and kernel trick?

My question is regarding the SVM topic. What is the difference between kernel function and kernel trick? Are they same and refer to the same thing?
0
votes
0answers
17 views

Can I customize the kernel function?

I want to know whether I can customize the kernel function? For example, the polynomial kernel is defined as: $$ K(x,y) = (x^Ty+c)^d $$ Could I modify it to the following: $$ K(x,y) = (||x-y||_2)^d $$ ...
0
votes
0answers
31 views

Pros and cons of different MKL algorithms

I have been using multiple kernel learning (MKL) to train a classifier and got some exposure to the field. However, I am quite new to machine learning and I have only an intuitive understanding of the ...
1
vote
1answer
209 views

Kernel approximation with Nystroem method and usage in scikit-learn

I am planning to use the Nystroem method to approximate a Gram matrix induced by any kernel function. I found the Nystroem implementation in scikit-learn. As far as I understood, the full Gram Matrix ...
1
vote
1answer
100 views

How to understand mapping function of kernel?

For a kernel function, we have two conditions one is that it should be symmetric which is easy to understand intuitively because dot products are symmetric as well and our kernel should also follow ...
1
vote
1answer
38 views

Proving that a kernel function is kernel

Let's suppose we have a kernel function $k(x,x')=10 $ In order to prove that this a valid kernel function there are generally two conditions It is symmetric There exists a map $\varphi:R^d \...
1
vote
1answer
49 views

Why people prefer neural network to kernel methods?

I am learning Kernel methods. Kernel methods are less a "black box" than neural networks. Nowadays, it seems neural networks gain more popularity and show more powerful in various ...
1
vote
2answers
47 views

Why is it called $\chi^2$ distance / kernel?

The $\chi^2$ distance function is defined as $$ \chi(u,v) = \sum_{i=1}^n \frac{(u_i-v_i)^2}{u_i+v_i} $$ and the $\chi^2$ kernel function, used in support vector machines, is $$ K(u,v) = \exp(-c \...
0
votes
1answer
31 views

Finding optimal kernel parameters

I want to perform multiple kernel learning on my dataset and apply each (rbf) kernel to a different subset of features to then combine them. I do not want to have the same kernel with a range of ...
0
votes
0answers
16 views

Applying different kernels to parts of a dataset and merging the output [duplicate]

I am trying to create a classifier using SVM on a dataset that is composed of 6 sets of data for each of my observations. When I train the SVM (rbf kernel), I get a better performance of the ...
0
votes
0answers
21 views

What dimensionality reduction technique to use for high dimensional data?

I have a dataset which contains a lot of features (>>3). For computational reasons, I would like to apply a dimensionality reduction. At this point I could use different techniques: standard PCA ...
0
votes
0answers
43 views

Assumptions of SVM

I have a dataset that has been analysed using logistic regression. In this, several variables were non-linearly associated with the log odds of my outcome, so were transformed prior to their inclusion ...
1
vote
1answer
30 views

Re-building a cross-validated SVM

Suppose we are cross-validating parameters of a Gaussian (radial) SVM on $k$ training observations. The parameters are the cost parameter $C$, and the deviation parameter $\gamma$. Then, $4k$ more ...
0
votes
1answer
18 views

Kernel on two functions?

I am curious whether there are any literatures considering kernel functions whose inputs are two functions. For example I would like consider two 1-Liptchiz mappings $\pi, \sigma:\mathbb{R}^M\...
0
votes
0answers
15 views

Is there any possibilty to apply Kernel PCA to coupled data fields?

Thanks to a wealth of answers from the community here (with a special thank you to amoeba), kernel PCA became much clearer to me. So far I understand that the magic of kernel PCA lies in the kernel ...
1
vote
0answers
30 views

Meaning of support vectors in support vector regression

I know that the support vectors in a soft margin SVM classifier model essentially means the vectors on the margin or less than the margin(the ones within the tube containing the decision boundary), ...
2
votes
1answer
74 views

How to choose a kernel function for Gaussian process regression in a multivaritate settings?

Could you please suggest some lectures, books, or videos on how to choose a kernel function for Gaussian process regression in a multivaritate setting?
0
votes
0answers
34 views

1D representation for 2D toy data (about linear separability)

suppose there is a dataset with 2 features x1, and x2. the points (-1;-1); (1; 1); (-3;-3); (4; 4) belong to class 1 and (-1; 1); (1;-1); (-5; 2); (4;-8) belongs to class 2. I am confused in terms of ...
2
votes
1answer
116 views

Combination of two kernel functions [duplicate]

Could you help me with this kernel function? \begin{equation} K(x,y) = (x \cdot y)^{2} + (x \cdot y), \text{ where } x = (x_1, x_2)', y = (y_1, y_2)' \end{equation} I want to know if the ...
6
votes
1answer
245 views

Kernels in SVM primal form

For a soft margin SVM in primal form, we have a cost function that is: $$J(\mathbf{w}, b) = C {\displaystyle \sum\limits_{i=1}^{m} max\left(0, 1 - y^{(i)} (\mathbf{w}^t \cdot \mathbf{x}^{(i)} + b)\...
2
votes
0answers
30 views

tradeoff function for exploration/exploitation in the context of gaussian process bayesian optimisation

First, let's give some context to my issue: consider an objective function $f(x)$ defined over a compact set $D$, such that there exists $x^*, f(x^*) > f(x) \quad \forall x \in D$. My goal is to ...
0
votes
1answer
17 views

Don't understand one particular normal distribution notation from opaper Stochastic Variational Deep Kernel Learning, help needed

I'm in progress working with paper "Stochastic Variational Deep Kernel Learning" NIPS 2016 and I have the problem with understanding the meaning of this normal distribution notation from part 2 ...
0
votes
0answers
12 views

Kernel function in SVM influence the decision boundary?

Does the kernel function used in SVM training influence the 'shape' of its decision boundary? I seem to have an impression that the RBF kernel function used in training seem to 'appear' also in the ...
0
votes
0answers
25 views

Hilbert Schmidt Independence Criterion For Large Data Set

So I'm trying the implement the Backward Feature Selection algorithm using the Hilbert Schmidt Independence Criterion (a.k.a BAHSIC) but I'm getting an out of memory error when calculating the kernel'...

1
2 3 4 5
13