Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Kernel methods are used in machine learning to generalize linear techniques to nonlinear situations, especially SVMs, PCA, and GPs. Not to be confused with [kernel-smoothing], for kernel density estimation (KDE) and kernel regression.

0
votes
0answers
20 views

random kitchen sinks as approximation to kernel machine

In the paper Rahimi, Ali, and Benjamin Recht. "Random features for large-scale kernel machines." Advances in neural information processing systems. 2008. the author introduces a way to ...
1
vote
0answers
75 views

Uniqueness of Reproducing Kernel Hilbert Spaces

Digging in the definition of Reproducing Kernel Hilbert Spaces (RKHS) I came across the following example taken from pages 49-51 of [1]: Given the kernel $k(x,y) = \langle x,y\rangle^2$, with $x,y\in ...
0
votes
0answers
20 views

How to choose hyper-parameter for Gaussian Process kernels?

I'm trying to fit Gaussian Process in scikit-learn, and start with using kernel = RBF_1 + RBF_2 + whitekernel(sum of two RBF kernels with different length_scale and ...
2
votes
0answers
12 views

Why do we need the gamma parameter in the polynomial kernel of SVMs?

The polynomial kernel is sometimes defined as just: $$ K(x,y):=(\left<x,y\right>+c)^d $$ with two parameters: the degree $d$ and constant coefficient $c$. But others (e.g., libsvm, and sklearn ...
1
vote
0answers
11 views

Precomputed Kernels for Support Vector Machines (SVM)

To calculate the linear kernel matrix for some training matrix X with dimensions n x d where d is the number of features and n is the number of data points, we can simply do: $X * X^T$. The result is ...
0
votes
0answers
10 views

Multiple kernel learning on gram matrices

l'm looking for a Multiple kernel learning algorithm such as simple MKL that do the following: Given 6 features matrices ...
0
votes
1answer
29 views

Finding the feature map corresponding to a specific Kernel? (Polynomial Kernels)

I am just getting into machine learning and I am kind of confused about how to show the corresponding feature map for a kernel. For example, how would I show the following feature map for this kernel? ...
1
vote
0answers
14 views

Prove that given kernel is valid and find the relevant mapping

Understanding Machine Learning: From Theory to Algorithms, Section 16.6, Question 4 is For $x>z$, I formulate my kernel matrix as $K = [x \quad z;z \quad z]$ which gives the cofactors as $x, z(x-...
0
votes
0answers
17 views

Proving a given kernel is not valid [duplicate]

$K(x,z) = e^{\gamma ||x-z||^2} \quad \gamma >0$ I took $x_1 = [1\quad 0]^T$ and $x_2 = [0\quad 2]^T$ which gives $K(x_1,x_2) = 5 = K(x_2,x_1)$ and $K(x_1,x_1) = K(x_2,x_2) = 0$ Thus the Kernel ...
1
vote
1answer
51 views

Checking if a kernel is valid

The kernel is $K(x,z) = \sum_{i=1}^D (x_i+z_i)$ My approach was trying to express $K = \phi(x)^T\phi(z) = (x_1 x_2 ... x_D \quad 1 1 ...1)(1 1 ...1\quad z_1 z_2 ... z_D )^T$ where $\phi$ is 2Dx1 ...
0
votes
1answer
24 views

Kernel ridge regression with matrix-vector data set $S := \{ X_i, y_i \}_{i=1}^{N}$?

Please notice that this question was asked in MO, but it seems that it doesn't interest MO community. So, I have got a comment to post in this community in the hope that I may get some attention to ...
1
vote
0answers
14 views

Use the features selected with RFE SVM linear for prediction of SVM rbf

I was wondering if the features selected with RFE with SVM linear kernel are still "good" features when we use a non linear model, like SVM rbf kernel. This comes in practice when you want to use SVM ...
1
vote
0answers
24 views

Kernel ridge regression with matrix-vector data set $S := \{ X_i, y_i \}_{i=1}^{N}$? [duplicate]

Background: For Kernel ridge regression, I have normally come across the data-set given in vector and scalar form, i.e., $\overline{S}:= \{x_i, \overline{y}_i \}$, where $x_i \in M_{n,1}(\mathbb{R})$ ...
2
votes
0answers
32 views

Kernel function for use in Kernel-PCA given a known piecewise linear true data generating process

If I know that a multivariate dataset has a piecewise-linear data generating process with known knots (or breakpoints), then what is the appropriate kernel function to use in Kernel-PCA? For example, ...
1
vote
0answers
14 views

How to combine multiple kernels of large sample datasets?

I have multiple large sample datasets in matrix format (each has 15000 rows and 5-50 columns) corresponding to different experiments. Each matrix contains the same number of samples(rows) but the ...
1
vote
1answer
15 views

Kernel functions with vector output

Kernel functions are used commonly with SVMs to make classification of non linearly separable data possible - i.e. the Kernel function provides the linear separability. But from looking at Kernel ...
0
votes
1answer
95 views

How to use the squared exponential kernel with multidimensional vector inputs?

I'm constructing an optimization (Bayesian optimization) algorithm using Java code. I have created the program, but the similarity values between inputted vectors in the kernel equation does not ...
0
votes
0answers
38 views

Is a kernel a correlation or a covariance function?

I am reading this paper on multi-fidelity optimization, where I came across an introductory section on kriging a.k.a. Gaussian Process regression (see Figure below). It confused me about the notion of ...
4
votes
0answers
113 views

Fourier transform of a Gaussian process

I would like to discuss and ask a question regarding the Fourier transform of a Gaussian process, if it makes sense. For that purpose, let me describe the following situation. Let $z(s)$ be a ...
3
votes
0answers
60 views

Understanding kernel PCA when the target space is infinite-dimensional

The PCA optimization problem is known as $$ \max_{U \in \mathbb{R}^{d\times r}, U^TU = I} tr(U^T\Sigma U), $$ where $\Sigma$ is a covariance matrix of a dataset $\{x_1,\dots,x_n\} \subset \mathbb{R}^d$...
3
votes
0answers
43 views

Are null space of matrix and kernel function same?

I have recently started learning about machine learning and have come across kernels and null spaces. I understand that null space is the set of all vectors that satisfy the equation A.v = 0 (Where A ...
1
vote
0answers
22 views

kernel publications

What are some contemporary papers that provide the reader with a complex overview of kernel functions used nowadays in machine learning?
1
vote
0answers
18 views

eigenvalue perturbation theory for kernel function

Let $S=\{x_i\}_{i=1}^n$ be a set of training examples, and let $K\in \mathbb{S}^n_+$ be the kernel matrix induced by $S$ and some kernel function $k$ (i.e., $K_{ij}=k(x_i,x_j)$). I was wondering how ...
1
vote
0answers
54 views

How to calculate RKHS norm of a function under given kernel transformation

This was a question asked before in mathoverflow but not yet got answered. I have the same problem when reading Srinivas et al (2010) [appendix B]'s paper. Here are my problems: Definitions: ...
0
votes
0answers
65 views

Kernel function from polynomial basis functions

In chapter 3 & 6 of Bishop's Pattern Recognition and Machine Learning, he showed that the equivalent kernel based on eqn (3.62) $$ k(x,x') = \beta \phi(x)^T (\alpha I + \beta \Phi^T \Phi )^{-1}\...
0
votes
0answers
70 views

Do Bayesian Optimization GP-UCB algorithm always converged for any continuous function in theory or practice?

Recently,I am studying the paper of Gaussian Process Optimization in the Bandit Setting, Srinivas. In theorem 3, they state: Let $\delta\in(0,1)$. Assume that the true underlying f lies in the RKHS ...
3
votes
1answer
64 views

Are Matérn class kernels universal kernels or not?

This is a question that I can't find the solution. I don't know it is a open question or it is a well-known result that can be attained from several lemmas. Here are the definition of Matérn class ...
2
votes
1answer
61 views

What is the motivation or objective for adopting Kernel methods? Is kernel trick a feature engineering method?

I come to know that kernel methods can be used in not only SVM but also many machine learning algorithms. I understanding that in SVM, the reason for using kernel trick is that some data are linearly ...
0
votes
0answers
31 views

What does it mean to have Covariance > 1 in Gaussian Processes? (Or Cov(x, x) != 1?)

The sum of two kernels is a kernel. [. . .] The product of of two kernels is a kernel. - Gaussian Processes for Machine Learning, Section 4.2.4 I can quite easily see how the product would work: ...
0
votes
1answer
62 views

Gaussian Process instability with more datapoints

I'm working my way through Rasmussen and Williams' classical work Gaussian Process for Machine Learning, and attempting to implement a lot of their theory in Python. I've attempted to fit a sin(x) ...
1
vote
0answers
49 views

Kernels property: integral of kernel product $\propto k(x,y)$

Let $k$ be a kernel function (symmetric and semi-positive definite function). Does the following relationship hold: $\int_{-\infty}^{+\infty}k(x,u)k(y,u) du \propto k(x,y)$ ? Or for what type of ...
3
votes
1answer
30 views

RKHS for polynomial kernel

Say we have a polynomial kernel of degree two: $k(x,x')=\langle x,x' \rangle^2$ for $X=\mathbb{R}^2$. I know that a feature map $\phi(x)=(x_1^2,\sqrt{2}x_1x_2,x_2^2)$ exist. What I want to know is ...
0
votes
0answers
38 views

Denoising with Kernel PCA, with handwritten digit denoising

When using Kernel pca and denoising handwritten images, basically every number gets denoised very well, and just with 1 PC, we have a clean denoised image. Yet, the number 7 gets somewhat an extra ...
0
votes
0answers
21 views

Polynomial kernel addition (Finding the mapping function)

I was given two polynomial kernels $K_1 (x,y)=(α_1 x\cdot y+β_1)^d$ and $K_1 (x,y)=(α_2 x\cdot y+β_2)^d$ (such that $a_1,a_2\beta_1,\beta_2\ne0$ ), with the corresponding mappings $φ_1$ and $φ_2$. ...
0
votes
0answers
53 views

SVM: Maximal number of components kernel PCA versus linear PCA

I'm comparing the Support Vector Machines (SVM) formulation of linear PCA with kernel PCA. I know that in linear PCA, the maximum number of principal components is equal to the dimension of the input ...
0
votes
1answer
75 views

How to tune bandwidth in machine learning kernel model?

Gaussian kernel $k(x,y) = \exp(-\lVert x-y \rVert^2/\sigma^2)$ has a hyperparameter $\sigma$. I know grid search cross validation, but this would require a lot of computation since computational ...
1
vote
0answers
55 views

Sum of two Poly Kernels

I have a questions from assignment about sum polynomial Kernels : If I have Poly Kernel $k_1$ =$(α_1 x\cdot y+β_1)^d$ and mapping $\varphi_1$ and kernel $k_2$ =$(α_2 x\cdot y+β_2)^d$ with mapping $\...
0
votes
0answers
71 views

Non-linear SVM and Curse of dimensionality

In Non-Linear SVM , the kernel function allows to avoid issues associated with the curse of dimensionality problem since the dot product in the kernel function is performed in the original space. $...
2
votes
0answers
90 views

A kernel $K_1$ maps from $R^n$ to $R^m (m>n)$ and is linearly separable. Given a different kernel $K=K_1+K_2$ will we still find a linear classifier?

Just started learning about SVM, kernels and all those things. Got stuck on this question (homework question): Consider a kernel $K_1$ and its corresponding mapping $φ_1$ that maps from the lower ...
0
votes
0answers
11 views

How to choose the bandwidth of Gaussian kernel in kernel trick-related models?

The no-brainer way is to run all possible bandwidth and see which one generalize the result the best. Is there any other ways?
0
votes
0answers
176 views

How does the shape of a decision boundary in relate between the original and kernel feature space?

I'm trying to get my head around the mathematics and implementation of SVM and hopefully gain some intuition into how kernels work and perhaps being able to, with a bit more confidence, define my own ...
1
vote
0answers
29 views

why kernel method is not overfitting?

A question comes to my mind and confuses me: Simply consider kernel method for regression. If kernel method is equivalently solving a infinite dimensional linear regression problem, then there should ...
11
votes
2answers
215 views

Does Mercer's theorem work in reverse?

A colleague has a function $s$ and for our purposes it is a black-box. The function measures the similarity $s(a,b)$ of two objects. We know for sure that $s$ has these properties: The similarity ...
0
votes
0answers
16 views

Procedure for Identifying Representative Documents

I have a large collection of documents, and would like to be able to select a subset of them that is representative of the whole. I have searched this question on here, Stack Overflow, and Google, ...
0
votes
1answer
39 views

Dropping the exponent of the Gaussian Kernel

I am wondering, for the Gaussian kernel $$k(x_n,x_m) = \exp(- \frac{\theta}{2} \lVert x_n-x_m\rVert^2)$$ whether we really need the exponent. What are the consequences of just using $$k(x_n,x_m) = ...
0
votes
0answers
28 views

Do kNN algorithms still work given a custom kernel function (or equivalently, distance function)?

Suppose I have $m$ feature vectors $\mathbf{x_1}, \mathbf{x_2}, ... \mathbf{x_m} \in \mathbb{R}^n$ and I define a custom kernel function $K(\mathbf{x_i}, \mathbf{x_j})$. For a given $i$, is it ...
0
votes
1answer
24 views

Does there exist a vector α st the equality holds?

A training set $(x_1,y_1),...,(x_m,y_m)$ is generic iff $x_i=x_j$ then $y_i=y_j$ and let's consider the following kernel $K_a(x,t)=\prod_{i=1}^n(1+(x_it_i)+(1-x_i)(1-t_i))$ Given a generic training ...
0
votes
0answers
53 views

Separability of dataset by sum of kernels

Suppose we have two kernels $K_1(x_1, x_2) = \langle \varphi_1 (x_1), \varphi_1( x_2) \rangle$ and $K_2(x_1, x_2) = \langle \varphi_2 (x_1), \varphi_2( x_2) \rangle$ and dataset $\{(x_1, y_1), \dots, (...
3
votes
1answer
54 views

The trace term in 2 Wassersteins metric for Gaussians

I was looking at the formula for 2 Wassersteins distance for Gaussian distribution on Wikipedia. https://en.wikipedia.org/wiki/Wasserstein_metric#Normal_distributions It satisfies all properties of a ...
2
votes
2answers
75 views

addition of a kernel and a function

I have this problem. My approach would be to take a Kernel matrix of $ \ k(x,y) $ as it is positive semidefinitive. And if I am able to make $ \ k'(x,y) $ a kernel matrix, then I can prove that it ...