Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [kernel-trick]

Kernel methods are used in machine learning to generalize linear techniques to nonlinear situations, especially SVMs, PCA, and GPs. Not to be confused with [kernel-smoothing], for kernel density estimation (KDE) and kernel regression.

0
votes
0answers
9 views

What are other nonlinear transformation methods except Sigmoid, ReLU, Tanh etc?

One advantage of the MLP neural networks is the nonlinear transformation used on raw features. The popular ones used are the activation functions like Sigmoid, Tanh, ReLU, Leaky ReLU, etc. They are of ...
0
votes
0answers
15 views

Linear separation in higher dimension

I am having a problem comprehending with the relation of kernel, weight and linear separation. I have a case where I am given a kernel k1. that has a corresponding mapping phi1. And we know that ...
0
votes
0answers
23 views

Reproducing kernels: how do I numerically compute the decomposition?

Suppose I'm given a kernel, $$ K(x,y) : \mathbb{R} \times \mathbb{R} \rightarrow \mathbb{R} $$ In order to describe/understand the (unique) associated RKHS, I seek its eigenfunctions, as per Mercer'...
0
votes
0answers
25 views

R - Gamma estimates in Kernel Ridge Regression

I am running a Kernel Ridge Regression in R. Mathematically, the minimization problem to be solved is the following: $$ \min_{\boldsymbol{\beta} \in \mathbb{R}^{d}} \ \sum_{i = 1}^{n} (y_{i} - \left \...
0
votes
1answer
24 views

When kernels are not useful in SVM?

In SVM using kernels we map the original features to the higher, transformer space (feature mapping) and then perform linear SVM in this higher space. But when kernels are not useful? I could not find ...
0
votes
1answer
50 views

RBF kernel mapping

I was reading that the Gaussian/RBF kernel maps its input onto the surface of normalized hypersphere. Our RBF kernel given by: $k(x,z) = exp(\frac{- ||x-z||^2}{2\sigma^2})$ Can anyone explain why ...
1
vote
0answers
8 views

Is there a representer theorem for unsupervised learning (to justify kernel density estimation)?

In supervised learning, we get a representer theorem by considering regularized losses of the following form: In Kernel Density Estimation, we simply directly assume densities of the form Could this ...
1
vote
0answers
8 views

Can SVM with Gaussian RBF kernel separate all kinds of data theoretically?

Gaussian is well known because its corresponding feature mapping is to infinite dimension. So with finite number of training data, is that the case that we can achieve zero training error with some ...
0
votes
0answers
20 views

Should the predicted variance in Gaussian process regression include experimental error?

Suppose I have an experiment where I measure the temperature of water in a cup, $y$, as a function of time, $x$. My measurement is normally distributed with an experimental uncertainty given by $\...
0
votes
0answers
29 views

Gaussian process vs. Bayesian linear regression / computational cost in weight space

Gaussian process (GP) regression with the linear covariance function $$k(x_i, x_j) = \sigma_0^2 + \sigma_1^2 x_i x_j + \delta(i=j)$$ can be seen as a Bayesian linear regression (BLR) model $$ y_i = ...
0
votes
0answers
17 views

Earth Movers Distance and Maximum Mean Discrepency

By Kantorovich-Rubinstein duality the Earth Movers Distance (EMD)/Wasserstein Metric is equivalent to Maximum Mean Discrepancy (MMD) correct? See here for a more thorough explanation. Why then does ...
1
vote
0answers
16 views

How to account for experimental errors when computing the derivative of a Gaussian process?

When applying Gaussian process regression upon training data, the covariance function can be generally given in the form: $\Sigma_{i,j} = k(x_i, x_j) + \sigma(x_i) \delta_{i,j}$, where $k$ is a ...
0
votes
0answers
13 views

Denoising and pre-images in Kernel PCA

In "Pattern Recognition and Machine Learning" by Bishop, the following problem about Kernel PCA is laid out : In linear PCA, we can approximate data points by projecting them onto the $L < D$-...
1
vote
0answers
22 views

Kernel function with a feature space equipped with an inner product that is not the dot product

Premise: A function $K: \mathbb R^d \times \mathbb R^d \to \mathbb R$ is called a kernel function on $\mathbb{R}^d$ if there exists a Hilbert space $\mathcal{H}$ and a map $\phi: \mathbb R^d \...
1
vote
1answer
31 views

What is the intuition behind changing the dot product for another inner product in SVM?

I understand that, when classifying with a SVM using a non-linear kernel, we are basically changing the dot product for a "custom" inner product. Is there some reason for working with a different ...
0
votes
0answers
8 views

Predictors significance and interpretation in kernel methods (and Gaussian processes models)

Assuming one would want to evaluate the "significance" of features in non parametric model as GP (regression). Opposed to (linear) SVM or LDA, in which one would be able to somehow "interpret" the ...
1
vote
1answer
40 views

Is this a valid Gaussian Process kernel?

$\mathcal{K}\Big( \; (x,y), (x',y') \; \Big) = \sigma_f^2 \exp{ \frac{(x-x')^2}{2l^2 \cdot (y+y')^2} } $, where $l > 0$ The variance associated with each training point (given by a vector) is a ...
0
votes
0answers
21 views

Transforming the Kernel principal components to original space

According to my understanding, we obtain the kernel/gram matrix eigenvectors/values in kernel PCA. We can use the kernel matrix for transforming the data however is there a way to transform those ...
1
vote
1answer
139 views

Is a polynomial kernel ridge regression really equivalent to performing a linear regression on those expanded features?

Say we have a dataset, X, which is Nx2 where N is the number of examples and 2 is the number of dimensions "features". If we were to run a kernel ridge regression (or SVM or whatever) on these ...
0
votes
0answers
32 views

How do I know how to combine kernels in e.g. Gaussian Process Regressions

Looking for optimal parameters usually is a pain for most Machine Learning tasks. In most cases, however, we can perform a grid search to find out about which parameters do the job better or worse. ...
1
vote
0answers
37 views

How to prove 1-norm radial function is kernel?

How should I prove that is a valid kernel: $K(x,y)=exp(-\alpha||x-y||_1) $ As I understand, there are three ways to prove that prove $K(x,y)=<\phi(x) ,\phi(y) >$ prove $\sum_{j,k=1}^n a_j\,...
1
vote
0answers
37 views

Kernel trick: computationally inexpensive

I am reading this post about the kernel trick. The author claims that a calculation of a dot product of two vectors will need 100.000.000 multiplications given that both vectors are of a dimension (...
0
votes
1answer
28 views

Kernelize Linear Regression

We can kernelize Ridge regression as shown in these notes: https://www.ics.uci.edu/~welling/classnotes/papers_class/Kernel-Ridge.pdf. However would it be possible to find a vector $\boldsymbol\alpha$...
0
votes
0answers
18 views

Are there kernel functions available for categorical variables where matches between different variables would also raise the similarity?

For my master thesis I have to apply bayesian optimization on the development of modular endolysins. This endolysin consists of 3 building blocks that are linked together (variables). Each of these ...
0
votes
0answers
33 views

SVM training yields too many (or no) support vectors

So I implemented a support vector machine, using either a linear kernel or the rbf-kernel. I trained and tested it on a two dimensional set of data and everything seems to be working fine. However, ...
1
vote
1answer
119 views

Probabilistic Interpretation of Radial Basis Function

I was wondering if someone could flesh out the probabilistic interpretation of using the Radial Basis Function to compute the probability between an observation and some reference value. My question ...
0
votes
1answer
88 views

Using gradient descent to train dual formulation of Kernel SVM

I've seen other posts about using gradient descent for the primal form, but not the dual form. In this book, the author discusses using (projected) gradient descent for the dual form: http://ciml....
0
votes
0answers
51 views

If I center my kernel does it no longer remain positive semidefinite?? If so why is it being used in algorithms like kernel pca?

If I center my kernel then can it still be used in operations where a positive semi-definite kernel is required such as SVM and ridge regression? I am centering my kernel as follows: $$K_c(\mathbf{t}...
1
vote
2answers
156 views

Is a kernel function basically just a mapping?

I'm currently studying machine learning (support vector machines to be more specific), and I was wondering how exactly I should understand what a kernel function is. I've read other questions on this ...
1
vote
1answer
233 views

Choice of Gaussian process in non-parametric regression

I have been trying to understand non-parametric regression using Gaussian processes (GP), which are used to represent prior distributions over the space of functions. The linear model considered is $$ ...
2
votes
0answers
45 views

random kitchen sinks as approximation to kernel machine

In the paper Rahimi, Ali, and Benjamin Recht. "Random features for large-scale kernel machines." Advances in neural information processing systems. 2008. the author introduces a way to ...
4
votes
1answer
145 views

Uniqueness of Reproducing Kernel Hilbert Spaces

Digging in the definition of Reproducing Kernel Hilbert Spaces (RKHS) I came across the following example taken from pages 49-51 of [1]: Given the kernel $k(x,y) = \langle x,y\rangle^2$, with $x,y\in ...
1
vote
0answers
108 views

How to choose hyper-parameter for Gaussian Process kernels?

I'm trying to fit Gaussian Process in scikit-learn, and start with using kernel = RBF_1 + RBF_2 + whitekernel(sum of two RBF kernels with different length_scale and ...
2
votes
0answers
128 views

Why do we need the gamma parameter in the polynomial kernel of SVMs?

The polynomial kernel is sometimes defined as just: $$ K(x,y):=(\left<x,y\right>+c)^d $$ with two parameters: the degree $d$ and constant coefficient $c$. But others (e.g., libsvm, and sklearn ...
1
vote
0answers
66 views

Precomputed Kernels for Support Vector Machines (SVM)

To calculate the linear kernel matrix for some training matrix X with dimensions n x d where d is the number of features and n is the number of data points, we can simply do: $X * X^T$. The result is ...
0
votes
1answer
346 views

Finding the feature map corresponding to a specific Kernel? (Polynomial Kernels)

I am just getting into machine learning and I am kind of confused about how to show the corresponding feature map for a kernel. For example, how would I show the following feature map for this kernel? ...
1
vote
0answers
24 views

Prove that given kernel is valid and find the relevant mapping

Understanding Machine Learning: From Theory to Algorithms, Section 16.6, Question 4 is For $x>z$, I formulate my kernel matrix as $K = [x \quad z;z \quad z]$ which gives the cofactors as $x, z(x-...
1
vote
1answer
86 views

Checking if a kernel is valid

The kernel is $K(x,z) = \sum_{i=1}^D (x_i+z_i)$ My approach was trying to express $K = \phi(x)^T\phi(z) = (x_1 x_2 ... x_D \quad 1 1 ...1)(1 1 ...1\quad z_1 z_2 ... z_D )^T$ where $\phi$ is 2Dx1 ...
0
votes
1answer
96 views

Kernel ridge regression with matrix-vector data set $S := \{ X_i, y_i \}_{i=1}^{N}$?

Please notice that this question was asked in MO, but it seems that it doesn't interest MO community. So, I have got a comment to post in this community in the hope that I may get some attention to ...
1
vote
0answers
34 views

Use the features selected with RFE SVM linear for prediction of SVM rbf

I was wondering if the features selected with RFE with SVM linear kernel are still "good" features when we use a non linear model, like SVM rbf kernel. This comes in practice when you want to use SVM ...
1
vote
0answers
24 views

Kernel ridge regression with matrix-vector data set $S := \{ X_i, y_i \}_{i=1}^{N}$? [duplicate]

Background: For Kernel ridge regression, I have normally come across the data-set given in vector and scalar form, i.e., $\overline{S}:= \{x_i, \overline{y}_i \}$, where $x_i \in M_{n,1}(\mathbb{R})$ ...
2
votes
0answers
49 views

Kernel function for use in Kernel-PCA given a known piecewise linear true data generating process

If I know that a multivariate dataset has a piecewise-linear data generating process with known knots (or breakpoints), then what is the appropriate kernel function to use in Kernel-PCA? For example, ...
1
vote
0answers
43 views

How to combine multiple kernels of large sample datasets?

I have multiple large sample datasets in matrix format (each has 15000 rows and 5-50 columns) corresponding to different experiments. Each matrix contains the same number of samples(rows) but the ...
1
vote
1answer
26 views

Kernel functions with vector output

Kernel functions are used commonly with SVMs to make classification of non linearly separable data possible - i.e. the Kernel function provides the linear separability. But from looking at Kernel ...
1
vote
1answer
823 views

How to use the squared exponential kernel with multidimensional vector inputs?

I'm constructing an optimization (Bayesian optimization) algorithm using Java code. I have created the program, but the similarity values between inputted vectors in the kernel equation does not ...
0
votes
0answers
66 views

Is a kernel a correlation or a covariance function?

I am reading this paper on multi-fidelity optimization, where I came across an introductory section on kriging a.k.a. Gaussian Process regression (see Figure below). It confused me about the notion of ...
4
votes
0answers
297 views

Fourier transform of a Gaussian process

I would like to discuss and ask a question regarding the Fourier transform of a Gaussian process, if it makes sense. For that purpose, let me describe the following situation. Let $z(s)$ be a ...
3
votes
0answers
85 views

Understanding kernel PCA when the target space is infinite-dimensional

The PCA optimization problem is known as $$ \max_{U \in \mathbb{R}^{d\times r}, U^TU = I} tr(U^T\Sigma U), $$ where $\Sigma$ is a covariance matrix of a dataset $\{x_1,\dots,x_n\} \subset \mathbb{R}^d$...
3
votes
0answers
87 views

Are null space of matrix and kernel function same?

I have recently started learning about machine learning and have come across kernels and null spaces. I understand that null space is the set of all vectors that satisfy the equation A.v = 0 (Where A ...
1
vote
0answers
23 views

kernel publications

What are some contemporary papers that provide the reader with a complex overview of kernel functions used nowadays in machine learning?