Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [kernel-trick]

Kernel methods are used in machine learning to generalize linear techniques to nonlinear situations, especially SVMs, PCA, and GPs. Not to be confused with [kernel-smoothing], for kernel density estimation (KDE) and kernel regression.

1
vote
0answers
12 views

Kernel function with a feature space equipped with an inner product that is not the dot product

Premise: A function $K: \mathbb R^d \times \mathbb R^d \to \mathbb R$ is called a kernel function on $\mathbb{R}^d$ if there exists a Hilbert space $\mathcal{H}$ and a map $\phi: \mathbb R^d \...
1
vote
1answer
23 views

What is the intuition behind changing the dot product for another inner product in SVM?

I understand that, when classifying with a SVM using a non-linear kernel, we are basically changing the dot product for a "custom" inner product. Is there some reason for working with a different ...
0
votes
0answers
4 views

Predictors significance and interpretation in kernel methods (and Gaussian processes models)

Assuming one would want to evaluate the "significance" of features in non parametric model as GP (regression). Opposed to (linear) SVM or LDA, in which one would be able to somehow "interpret" the ...
1
vote
1answer
34 views

Is this a valid Gaussian Process kernel?

$\mathcal{K}\Big( \; (x,y), (x',y') \; \Big) = \sigma_f^2 \exp{ \frac{(x-x')^2}{2l^2 \cdot (y+y')^2} } $, where $l > 0$ The variance associated with each training point (given by a vector) is a ...
0
votes
0answers
12 views

Transforming the Kernel principal components to original space

According to my understanding, we obtain the kernel/gram matrix eigenvectors/values in kernel PCA. We can use the kernel matrix for transforming the data however is there a way to transform those ...
1
vote
1answer
46 views

Is a polynomial kernel ridge regression really equivalent to performing a linear regression on those expanded features?

Say we have a dataset, X, which is Nx2 where N is the number of examples and 2 is the number of dimensions "features". If we were to run a kernel ridge regression (or SVM or whatever) on these ...
0
votes
0answers
21 views

How do I know how to combine kernels in e.g. Gaussian Process Regressions

Looking for optimal parameters usually is a pain for most Machine Learning tasks. In most cases, however, we can perform a grid search to find out about which parameters do the job better or worse. ...
1
vote
0answers
34 views

How to prove 1-norm radial function is kernel?

How should I prove that is a valid kernel: $K(x,y)=exp(-\alpha||x-y||_1) $ As I understand, there are three ways to prove that prove $K(x,y)=<\phi(x) ,\phi(y) >$ prove $\sum_{j,k=1}^n a_j\,...
1
vote
0answers
35 views

Kernel trick: computationally inexpensive

I am reading this post about the kernel trick. The author claims that a calculation of a dot product of two vectors will need 100.000.000 multiplications given that both vectors are of a dimension (...
0
votes
1answer
25 views

Kernelize Linear Regression

We can kernelize Ridge regression as shown in these notes: https://www.ics.uci.edu/~welling/classnotes/papers_class/Kernel-Ridge.pdf. However would it be possible to find a vector $\boldsymbol\alpha$...
0
votes
0answers
14 views

Are there kernel functions available for categorical variables where matches between different variables would also raise the similarity?

For my master thesis I have to apply bayesian optimization on the development of modular endolysins. This endolysin consists of 3 building blocks that are linked together (variables). Each of these ...
0
votes
0answers
22 views

SVM training yields too many (or no) support vectors

So I implemented a support vector machine, using either a linear kernel or the rbf-kernel. I trained and tested it on a two dimensional set of data and everything seems to be working fine. However, ...
1
vote
1answer
113 views

Probabilistic Interpretation of Radial Basis Function

I was wondering if someone could flesh out the probabilistic interpretation of using the Radial Basis Function to compute the probability between an observation and some reference value. My question ...
0
votes
1answer
33 views

Using gradient descent to train dual formulation of Kernel SVM

I've seen other posts about using gradient descent for the primal form, but not the dual form. In this book, the author discusses using (projected) gradient descent for the dual form: http://ciml....
0
votes
0answers
41 views

If I center my kernel does it no longer remain positive semidefinite?? If so why is it being used in algorithms like kernel pca?

If I center my kernel then can it still be used in operations where a positive semi-definite kernel is required such as SVM and ridge regression? I am centering my kernel as follows: $$K_c(\mathbf{t}...
1
vote
2answers
121 views

Is a kernel function basically just a mapping?

I'm currently studying machine learning (support vector machines to be more specific), and I was wondering how exactly I should understand what a kernel function is. I've read other questions on this ...
1
vote
1answer
49 views

Choice of Gaussian process in non-parametric regression

I have been trying to understand non-parametric regression using Gaussian processes (GP), which are used to represent prior distributions over the space of functions. The linear model considered is $$ ...
1
vote
0answers
37 views

random kitchen sinks as approximation to kernel machine

In the paper Rahimi, Ali, and Benjamin Recht. "Random features for large-scale kernel machines." Advances in neural information processing systems. 2008. the author introduces a way to ...
4
votes
1answer
129 views

Uniqueness of Reproducing Kernel Hilbert Spaces

Digging in the definition of Reproducing Kernel Hilbert Spaces (RKHS) I came across the following example taken from pages 49-51 of [1]: Given the kernel $k(x,y) = \langle x,y\rangle^2$, with $x,y\in ...
1
vote
0answers
70 views

How to choose hyper-parameter for Gaussian Process kernels?

I'm trying to fit Gaussian Process in scikit-learn, and start with using kernel = RBF_1 + RBF_2 + whitekernel(sum of two RBF kernels with different length_scale and ...
2
votes
0answers
84 views

Why do we need the gamma parameter in the polynomial kernel of SVMs?

The polynomial kernel is sometimes defined as just: $$ K(x,y):=(\left<x,y\right>+c)^d $$ with two parameters: the degree $d$ and constant coefficient $c$. But others (e.g., libsvm, and sklearn ...
1
vote
0answers
52 views

Precomputed Kernels for Support Vector Machines (SVM)

To calculate the linear kernel matrix for some training matrix X with dimensions n x d where d is the number of features and n is the number of data points, we can simply do: $X * X^T$. The result is ...
0
votes
0answers
31 views

Multiple kernel learning on gram matrices

l'm looking for a Multiple kernel learning algorithm such as simple MKL that do the following: Given 6 features matrices ...
0
votes
1answer
252 views

Finding the feature map corresponding to a specific Kernel? (Polynomial Kernels)

I am just getting into machine learning and I am kind of confused about how to show the corresponding feature map for a kernel. For example, how would I show the following feature map for this kernel? ...
1
vote
0answers
21 views

Prove that given kernel is valid and find the relevant mapping

Understanding Machine Learning: From Theory to Algorithms, Section 16.6, Question 4 is For $x>z$, I formulate my kernel matrix as $K = [x \quad z;z \quad z]$ which gives the cofactors as $x, z(x-...
1
vote
1answer
76 views

Checking if a kernel is valid

The kernel is $K(x,z) = \sum_{i=1}^D (x_i+z_i)$ My approach was trying to express $K = \phi(x)^T\phi(z) = (x_1 x_2 ... x_D \quad 1 1 ...1)(1 1 ...1\quad z_1 z_2 ... z_D )^T$ where $\phi$ is 2Dx1 ...
0
votes
1answer
76 views

Kernel ridge regression with matrix-vector data set $S := \{ X_i, y_i \}_{i=1}^{N}$?

Please notice that this question was asked in MO, but it seems that it doesn't interest MO community. So, I have got a comment to post in this community in the hope that I may get some attention to ...
1
vote
0answers
28 views

Use the features selected with RFE SVM linear for prediction of SVM rbf

I was wondering if the features selected with RFE with SVM linear kernel are still "good" features when we use a non linear model, like SVM rbf kernel. This comes in practice when you want to use SVM ...
1
vote
0answers
24 views

Kernel ridge regression with matrix-vector data set $S := \{ X_i, y_i \}_{i=1}^{N}$? [duplicate]

Background: For Kernel ridge regression, I have normally come across the data-set given in vector and scalar form, i.e., $\overline{S}:= \{x_i, \overline{y}_i \}$, where $x_i \in M_{n,1}(\mathbb{R})$ ...
2
votes
0answers
44 views

Kernel function for use in Kernel-PCA given a known piecewise linear true data generating process

If I know that a multivariate dataset has a piecewise-linear data generating process with known knots (or breakpoints), then what is the appropriate kernel function to use in Kernel-PCA? For example, ...
1
vote
0answers
32 views

How to combine multiple kernels of large sample datasets?

I have multiple large sample datasets in matrix format (each has 15000 rows and 5-50 columns) corresponding to different experiments. Each matrix contains the same number of samples(rows) but the ...
1
vote
1answer
24 views

Kernel functions with vector output

Kernel functions are used commonly with SVMs to make classification of non linearly separable data possible - i.e. the Kernel function provides the linear separability. But from looking at Kernel ...
1
vote
1answer
517 views

How to use the squared exponential kernel with multidimensional vector inputs?

I'm constructing an optimization (Bayesian optimization) algorithm using Java code. I have created the program, but the similarity values between inputted vectors in the kernel equation does not ...
0
votes
0answers
52 views

Is a kernel a correlation or a covariance function?

I am reading this paper on multi-fidelity optimization, where I came across an introductory section on kriging a.k.a. Gaussian Process regression (see Figure below). It confused me about the notion of ...
4
votes
0answers
230 views

Fourier transform of a Gaussian process

I would like to discuss and ask a question regarding the Fourier transform of a Gaussian process, if it makes sense. For that purpose, let me describe the following situation. Let $z(s)$ be a ...
3
votes
0answers
73 views

Understanding kernel PCA when the target space is infinite-dimensional

The PCA optimization problem is known as $$ \max_{U \in \mathbb{R}^{d\times r}, U^TU = I} tr(U^T\Sigma U), $$ where $\Sigma$ is a covariance matrix of a dataset $\{x_1,\dots,x_n\} \subset \mathbb{R}^d$...
3
votes
0answers
67 views

Are null space of matrix and kernel function same?

I have recently started learning about machine learning and have come across kernels and null spaces. I understand that null space is the set of all vectors that satisfy the equation A.v = 0 (Where A ...
1
vote
0answers
22 views

kernel publications

What are some contemporary papers that provide the reader with a complex overview of kernel functions used nowadays in machine learning?
1
vote
0answers
21 views

eigenvalue perturbation theory for kernel function

Let $S=\{x_i\}_{i=1}^n$ be a set of training examples, and let $K\in \mathbb{S}^n_+$ be the kernel matrix induced by $S$ and some kernel function $k$ (i.e., $K_{ij}=k(x_i,x_j)$). I was wondering how ...
1
vote
0answers
126 views

How to calculate RKHS norm of a function under given kernel transformation

This was a question asked before in mathoverflow but not yet got answered. I have the same problem when reading Srinivas et al (2010) [appendix B]'s paper. Here are my problems: Definitions: ...
0
votes
0answers
80 views

Kernel function from polynomial basis functions

In chapter 3 & 6 of Bishop's Pattern Recognition and Machine Learning, he showed that the equivalent kernel based on eqn (3.62) $$ k(x,x') = \beta \phi(x)^T (\alpha I + \beta \Phi^T \Phi )^{-1}\...
0
votes
0answers
159 views

Do Bayesian Optimization GP-UCB algorithm always converged for any continuous function in theory or practice?

Recently,I am studying the paper of Gaussian Process Optimization in the Bandit Setting, Srinivas. In theorem 3, they state: Let $\delta\in(0,1)$. Assume that the true underlying f lies in the RKHS ...
3
votes
1answer
90 views

Are Matérn class kernels universal kernels or not?

This is a question that I can't find the solution. I don't know it is a open question or it is a well-known result that can be attained from several lemmas. Here are the definition of Matérn class ...
2
votes
1answer
94 views

What is the motivation or objective for adopting Kernel methods? Is kernel trick a feature engineering method?

I come to know that kernel methods can be used in not only SVM but also many machine learning algorithms. I understanding that in SVM, the reason for using kernel trick is that some data are linearly ...
0
votes
0answers
42 views

What does it mean to have Covariance > 1 in Gaussian Processes? (Or Cov(x, x) != 1?)

The sum of two kernels is a kernel. [. . .] The product of of two kernels is a kernel. - Gaussian Processes for Machine Learning, Section 4.2.4 I can quite easily see how the product would work: ...
0
votes
1answer
252 views

Gaussian Process instability with more datapoints

I'm working my way through Rasmussen and Williams' classical work Gaussian Process for Machine Learning, and attempting to implement a lot of their theory in Python. I've attempted to fit a sin(x) ...
1
vote
0answers
63 views

Kernels property: integral of kernel product $\propto k(x,y)$

Let $k$ be a kernel function (symmetric and semi-positive definite function). Does the following relationship hold: $\int_{-\infty}^{+\infty}k(x,u)k(y,u) du \propto k(x,y)$ ? Or for what type of ...
3
votes
1answer
72 views

RKHS for polynomial kernel

Say we have a polynomial kernel of degree two: $k(x,x')=\langle x,x' \rangle^2$ for $X=\mathbb{R}^2$. I know that a feature map $\phi(x)=(x_1^2,\sqrt{2}x_1x_2,x_2^2)$ exist. What I want to know is ...
0
votes
0answers
47 views

Denoising with Kernel PCA, with handwritten digit denoising

When using Kernel pca and denoising handwritten images, basically every number gets denoised very well, and just with 1 PC, we have a clean denoised image. Yet, the number 7 gets somewhat an extra ...
0
votes
0answers
29 views

Polynomial kernel addition (Finding the mapping function)

I was given two polynomial kernels $K_1 (x,y)=(α_1 x\cdot y+β_1)^d$ and $K_1 (x,y)=(α_2 x\cdot y+β_2)^d$ (such that $a_1,a_2\beta_1,\beta_2\ne0$ ), with the corresponding mappings $φ_1$ and $φ_2$. ...