Questions tagged [kernel-trick]

Kernel methods are used in machine learning to generalize linear techniques to nonlinear situations, especially SVMs, PCA, and GPs. Not to be confused with [kernel-smoothing], for kernel density estimation (KDE) and kernel regression.

Filter by
Sorted by
Tagged with
0
votes
0answers
14 views

Can I customize the kernel function?

I want to know whether I can customize the kernel function? For example, the polynomial kernel is defined as: $$ K(x,y) = (x^Ty+c)^d $$ Could I modify it to the following: $$ K(x,y) = (||x-y||_2)^d $$ ...
0
votes
0answers
24 views

Pros and cons of different MKL algorithms

I have been using multiple kernel learning (MKL) to train a classifier and got some exposure to the field. However, I am quite new to machine learning and I have only an intuitive understanding of the ...
0
votes
0answers
28 views

Whether cosine similarity kernel is characteristic kernel? [closed]

I want to know whether cosine similarity kernel is characteristic kernel. Is there someone that would tell me? If it is, please give mathematical proofs.
0
votes
1answer
20 views

Kernel approximation with Nystroem method and usage in scikit-learn

I am planning to use the Nystroem method to approximate a Gram matrix induced by any kernel function. I found the Nystroem implementation in scikit-learn. As far as I understood, the full Gram Matrix ...
1
vote
1answer
55 views

How to understand mapping function of kernel?

For a kernel function, we have two conditions one is that it should be symmetric which is easy to understand intuitively because dot products are symmetric as well and our kernel should also follow ...
1
vote
1answer
34 views

Proving that a kernel function is kernel

Let's suppose we have a kernel function $k(x,x')=10 $ In order to prove that this a valid kernel function there are generally two conditions It is symmetric There exists a map $\varphi:R^d \...
1
vote
1answer
44 views

Why people prefer neural network to kernel methods?

I am learning Kernel methods. Kernel methods are less a "black box" than neural networks. Nowadays, it seems neural networks gain more popularity and show more powerful in various ...
1
vote
2answers
42 views

Why is it called $\chi^2$ distance / kernel?

The $\chi^2$ distance function is defined as $$ \chi(u,v) = \sum_{i=1}^n \frac{(u_i-v_i)^2}{u_i+v_i} $$ and the $\chi^2$ kernel function, used in support vector machines, is $$ K(u,v) = \exp(-c \...
0
votes
1answer
24 views

Finding optimal kernel parameters

I want to perform multiple kernel learning on my dataset and apply each (rbf) kernel to a different subset of features to then combine them. I do not want to have the same kernel with a range of ...
0
votes
0answers
16 views

Applying different kernels to parts of a dataset and merging the output [duplicate]

I am trying to create a classifier using SVM on a dataset that is composed of 6 sets of data for each of my observations. When I train the SVM (rbf kernel), I get a better performance of the ...
0
votes
0answers
18 views

What dimensionality reduction technique to use for high dimensional data?

I have a dataset which contains a lot of features (>>3). For computational reasons, I would like to apply a dimensionality reduction. At this point I could use different techniques: standard PCA ...
0
votes
0answers
15 views

Assumptions of SVM

I have a dataset that has been analysed using logistic regression. In this, several variables were non-linearly associated with the log odds of my outcome, so were transformed prior to their inclusion ...
1
vote
1answer
30 views

Re-building a cross-validated SVM

Suppose we are cross-validating parameters of a Gaussian (radial) SVM on $k$ training observations. The parameters are the cost parameter $C$, and the deviation parameter $\gamma$. Then, $4k$ more ...
0
votes
1answer
17 views

Kernel on two functions?

I am curious whether there are any literatures considering kernel functions whose inputs are two functions. For example I would like consider two 1-Liptchiz mappings $\pi, \sigma:\mathbb{R}^M\...
0
votes
0answers
14 views

Is there any possibilty to apply Kernel PCA to coupled data fields?

Thanks to a wealth of answers from the community here (with a special thank you to amoeba), kernel PCA became much clearer to me. So far I understand that the magic of kernel PCA lies in the kernel ...
0
votes
0answers
14 views

Variance and Generalization Error in Kernel Methods

I'm working on analyzing a paticular methodology I'm developing, and wanted to ask a fast question as a sanity check, as I've been out of the "math" game for awhile. I've found a couple papers: https:...
1
vote
0answers
20 views

Meaning of support vectors in support vector regression

I know that the support vectors in a soft margin SVM classifier model essentially means the vectors on the margin or less than the margin(the ones within the tube containing the decision boundary), ...
2
votes
1answer
57 views

How to choose a kernel function for Gaussian process regression in a multivaritate settings?

Could you please suggest some lectures, books, or videos on how to choose a kernel function for Gaussian process regression in a multivaritate setting?
0
votes
0answers
15 views

1D representation for 2D toy data (about linear separability)

suppose there is a dataset with 2 features x1, and x2. the points (-1;-1); (1; 1); (-3;-3); (4; 4) belong to class 1 and (-1; 1); (1;-1); (-5; 2); (4;-8) belongs to class 2. I am confused in terms of ...
2
votes
1answer
98 views

Combination of two kernel functions [duplicate]

Could you help me with this kernel function? \begin{equation} K(x,y) = (x \cdot y)^{2} + (x \cdot y), \text{ where } x = (x_1, x_2)', y = (y_1, y_2)' \end{equation} I want to know if the ...
4
votes
1answer
153 views

Kernels in SVM primal form

For a soft margin SVM in primal form, we have a cost function that is: $$J(\mathbf{w}, b) = C {\displaystyle \sum\limits_{i=1}^{m} max\left(0, 1 - y^{(i)} (\mathbf{w}^t \cdot \mathbf{x}^{(i)} + b)\...
2
votes
0answers
23 views

tradeoff function for exploration/exploitation in the context of gaussian process bayesian optimisation

First, let's give some context to my issue: consider an objective function $f(x)$ defined over a compact set $D$, such that there exists $x^*, f(x^*) > f(x) \quad \forall x \in D$. My goal is to ...
0
votes
1answer
16 views

Don't understand one particular normal distribution notation from opaper Stochastic Variational Deep Kernel Learning, help needed

I'm in progress working with paper "Stochastic Variational Deep Kernel Learning" NIPS 2016 and I have the problem with understanding the meaning of this normal distribution notation from part 2 ...
0
votes
0answers
8 views

Kernel function in SVM influence the decision boundary?

Does the kernel function used in SVM training influence the 'shape' of its decision boundary? I seem to have an impression that the RBF kernel function used in training seem to 'appear' also in the ...
0
votes
0answers
23 views

Hilbert Schmidt Independence Criterion For Large Data Set

So I'm trying the implement the Backward Feature Selection algorithm using the Hilbert Schmidt Independence Criterion (a.k.a BAHSIC) but I'm getting an out of memory error when calculating the kernel'...
0
votes
0answers
40 views

Feature map corresponding to diffusion kernel defined over graphs/discrete structures?

It is well known that there exists an explicit feature map representation for every kernel ($K$) i.e. for any two points $x_1$ and $x_2$ \begin{align} K(x_1, x_2) = \Phi(x_1)^T\Phi(x_2) \end{align} ...
1
vote
0answers
58 views

Is $cos^n(x^2-y^2)$ a valid mercer kernel function?

How to show if $cos^n(x^2-y^2)$ is a valid mercer kernel function if $n$ is positive? For $cos(x^2-y^2)$ I would assume that: $cos(x^2-y^2) = sin(x^2)sin(y^2)+cos(x^2)cos(y^2)$ Is a valid mercer ...
0
votes
1answer
14 views

Bayesian regression kernel

I am reading Bishops book "pattern recognition and machine learning". In chapter 3 "linear models for regression" section 3.3.3 "equivalent kernel" equation 3.63 on page 160 is given as follows: $$...
0
votes
0answers
89 views

Calculate Gamma for RBF Kernel to get Gaussian Kernel

In order to measure the information density like proposed in section 3.2 of this paper I need a symmetric positive definite Kernel function. For this purpose I want to use the Gaussian Kernel like ...
0
votes
0answers
34 views

Implementation of a Gauss Kernel in Python possibly using RBF Client

I want to implement the following Gauss kernel in Python: I could implement the structure in Python up to this point. However, the last piece missing is the calculation of the parameter tau squared. ...
3
votes
1answer
175 views

Kernel Mean Embedding relationship to regular kernel functions

I am struggling to understand kernel mean embeddings and how it relates to typical kernel functions. Review of Kernel Basics: Basically, a kernel function maps points (or vectors) from one feature ...
1
vote
0answers
17 views

Can someone explain how a kernel trick makes things computationally easier?

This author states that when we use the kernel trick, we map original features to the distance between two values. I have two questions regarding this: I'm not seeing how this is computationally ...
0
votes
0answers
17 views

Resource for understanding kernel trick, kernel method, kernel functions?

So far my understanding about kernel methods is that they are ways to map our features to a higher dimension space - allowing us to fit non-linear data using linear models. I don't understand much ...
0
votes
0answers
7 views

How to assign weights using Kernel function based on a vector of pairwise Euclidean distance?

I want to quantify the dissimilarity between two group. Each group has 5 observations, so there are 25 combinations. For each combination, I have calculated their pairwise Euclidean distance (based ...
1
vote
2answers
28 views

hyperparameters optimisation with linear kernel

I want to conduct an SVM model-regression (i.e., support vector regression), using a linear kernel function. Does it make sense to perform a cross-validation hyperparameter optimization when the ...
1
vote
0answers
50 views

Gaussian Process - The variance term in Kernel matrix is not reducing to 0 at the Location of the Data Point

I am running a variance reduction experiment using GPy library and sampling using Bayesian optimization suggested location. I noticed that once the data at certain location is added to GP's data set, ...
0
votes
0answers
10 views

Why do Support Vector Data Description and One Class Support Vector Machine produce the same results?

Quoating from Chapter 5 of Kernel Methods in Computer Vision by Christoph H. Lampert 'A quick geometric check shows that if all data points have the same feature space norm and can be separated ...
0
votes
1answer
122 views

Intuitive Understanding of the Effect of Correlation on Random Variables in Gaussian Process

So In general covariance matrix in GP provides us with proportionality relation between random variables, in other words $x_1$ and $x_2$ are perfectly correlated if off-diagonal entry has $\rho=\pm 1$:...
1
vote
0answers
26 views

Dimensions of feature transfrom $\phi(x)$ for a kernel Support Vector Machines

Given a kernel function $K(x, x') = \langle \phi(x), \phi(x') \rangle$, how can we figure out the dimensions of the feature transform $\phi(x)$? For example, for $K(x, x') = (1+x^Tx')^M$
1
vote
0answers
36 views

Recent advances in the use of the spectra of kernel integrals following Yoshua Bengio's 2004 paper that links kernel PCA and spectral clustering?

In Yoshua Belgio's 2003 technical report http://www.iro.umontreal.ca/~lisa/pointeurs/TR1232.pdf, and subsequent 2004 paper http://www.iro.umontreal.ca/~lisa/pointeurs/bengio_eigenfunctions_nc_2004.pdf,...
0
votes
1answer
39 views

Comparing margin width of SVM with different kernels as a performance metric

Assume we have applied SVM with different kernels to a problem, Alongside the performance metrics like accuracy, precision, etc, can we compare the margin size to decide which kernel is the best?
3
votes
1answer
26 views

Periodicity of random kitchen sink feature mappings

In various papers, e.g. Random Features for Large-Scale Kernel Machines, Rahimi and Recht introduce the now popular methodology wherein a "low rank" approximation to a stationary, PSD, kernel $K(x,y) =...
0
votes
0answers
25 views

$\Sigma_*$ in Gaussian Process Prediction Formula

So posterior predictive distribution of the Gaussian process is given by the following equation. $\mathbf{f}_*$ given new input $\mathbf{X}_*:$ $$p(\mathbf{f}_* \lvert \mathbf{X}_*,\mathbf{X},\...
1
vote
1answer
46 views

Expression for Derivative of Hyperparameter of Kernel with respect to New Data

I would like to determine how the hyperparameter will change when a new data is observed and the GP is updated with this new data. Considering the following predictive distribution of the GP: $$\mu(...
1
vote
0answers
24 views

Should Kernel Ridge Regression with linear kernel yield same results as Ridge regression?

I'm comparing the performance of different regressors from scikit-learn for fitting some data. I would have expected that Ridge regression and Kernel Ridge regression both yield the same model/...
1
vote
0answers
47 views

Effect of Gamma and C on distant points in SVM

(I am aware of the following, already answered questions: this,and this,as well as others, and IMHO they are not related) I am trying to precisely understand the behavior of SVM's with regards to the ...
0
votes
0answers
27 views

Why is the input feature space infinite-dimensional after applying the kernel transformation in Kernel-PCA?

Kernel-PCA is an unsupervised technique that can be used to compress data. It has some hyper-parameters, for example the choice of the type of kernel (radial, linear, sigmoid, etc.). According to the ...
1
vote
1answer
38 views

What is the expanded representation, $\phi(X)$, required to obtain the RBF kernel?

For the two-dimensional case, where $\boldsymbol X=[x_1, x_2]$ and its corresponding expanded represetation $\boldsymbol\phi(X)= [1, \sqrt2 x_1, \sqrt2 x_2, x_1^2, x_2^2, \sqrt2x_1x_2]$, we can ...
0
votes
0answers
17 views

Covariate Shift in the case of Kernel Regression

Kernel Regression, in its most simplistic form (Nadaraya-Watson kernel regression), can be viewed as a conditional density estimation problem. Let $\hat{p}(y, x) = N^{-1}\sum_i K(y, y_i)K(x, x_i)$ be ...
0
votes
0answers
11 views

Informative Vector Machine for multiple classes

Reading about the GPLVM -Gaussian Process Latent Variable Models- in the Neil D. Lawrence 2003 paper I understood how the dimensionality reduction is performed. From my understanding, the algorithm is ...

1
2 3 4 5
12