Questions tagged [kernel-trick]
Kernel methods are used in machine learning to generalize linear techniques to nonlinear situations, especially SVMs, PCA, and GPs. Not to be confused with [kernel-smoothing], for kernel density estimation (KDE) and kernel regression.
743
questions
0
votes
0
answers
6
views
SVM Kernels feature map and feature space [closed]
Can someone help me to find the feature space of kernel K(x,y) = exp(-|x-y|^2)
0
votes
0
answers
22
views
Non-stationary Random Fourier Features
Random Fourier Features (RFFs) were introduced by A. Rahimi and B. Recht in their 2007 publication Random Features for Large-Scale Kernel Machines. RFFs are based on Bochner's theorem, which applies ...
- 1
2
votes
0
answers
20
views
Identifiability of models on RKHS
I have just started learning about using reproducing kernel hilbert spaces for regularisation in machine learning. I am looking for some examples of reproducing kernels that produce identifiable and ...
- 41
0
votes
0
answers
46
views
Feature maps of the chi-squared kernel
The additive chi-squared kernel for histograms is defined as
$$K(x,y)= \sum_{i=1}^n \frac{2x_i y_i}{x_i + y_i}$$ Is this kernel positive definite on histograms? And if so, is there a known expression ...
- 249
0
votes
0
answers
17
views
Method of evaluating the feature map of a polynomial kernel feature mapping
I'm attempting to implement an adaptive kernel Kalman filter following this paper https://arxiv.org/abs/2203.08300, but I'm struggling to find a method of evaluating the feature mapping for a ...
3
votes
0
answers
62
views
Is the transformation implied by a positive-type kernel well-defined?
I’ve been trying to get my head around the particularity of the Hilbert space that a positive-type (equiv. positive definite) kernel represents an inner product on, and was hoping for some help in ...
- 339
2
votes
0
answers
18
views
In Gaussian Process Regression, what kinds of information can you *not* put in the kernel as opposed to the mean?
For example, suppose you want to learn some structure for the mean and then you also have some kernel. Is is sometimes not possible to put most things in the kernel? For example, consider ...
- 269
0
votes
0
answers
15
views
Optimal kernel regression with random target functions
The classic kernel regression problem fixes a target function $f^*$ that we seek to learn, and says that for a dataset/observations $D = \{x_i, y_i\}_{i=1}^n$ where $y(x) = f^*(x) + \epsilon$, for ...
- 173
2
votes
0
answers
15
views
Confused about kernel methods [duplicate]
I understand that kernel methods are used to exploit nonlinearity in a data set. For example, let $\mathbf{x} = \begin{bmatrix}x_1\\x_2 \end{bmatrix}$. We can define the feature map $\phi(\mathbf{x}) =...
- 8,457
1
vote
0
answers
30
views
Kernelization vs pre-defined basis functions: which one is better and why?
I am learning about kernels and how linear models can use them to model nonlinear data. Consider, for example, linear regression for nonlinear function $y(\textbf{x})$. The idea is to project the ...
- 217
1
vote
0
answers
34
views
Advantage of kernel methods over explicit feature construction
Kernel regression on data $x$ is equivalent to linear regression on transformed features $\phi(x)$ where $\phi$ is determined by the kernel $K(x_i, x_j)$. For example, when $K(x_i, x_j) = (1 + x_i^T ...
- 1,813
0
votes
0
answers
26
views
Is kernelization just a bases transform?
I am trying to understand the concept of kernelization, however, it appears to me that explanations from the machine learning community are rather confusing. For example, in the Bishop book (eq. 6.3-6....
- 217
1
vote
0
answers
40
views
Are low-rank kernel approximations implementing implicit regularization?
Consider a kernel estimation problem as follows. We have functions $f^* \sim GP(0, C^*)$ drawn from a Gaussian process. We want to construct a kernel $K$ that does well in regressing functions drawn ...
- 173
1
vote
0
answers
26
views
Mathematical steps for solving the Kernel trick for Support Vector Machines (SVM)s
I am trying to understand the math behind the Support Vector Machines (SVM) Kernel Trick but there isn't any real source online for the mathematical steps to I can follow to see how its solved. One ...
- 11
0
votes
0
answers
6
views
How are kernel similarities viable replacements in SVMs?
Support vector machine (SVM) is a supervised learning algorithm. It draws hyperplanes to separate data points of different classes. The objective function involves inner products of pairs of feature ...
- 249
0
votes
0
answers
61
views
Is there a connection between the Kernels in Statistics and Linear Algebra?
According to this question, the etymology of the terms is related, and Kernel is used to mean the "core" of something. In general it seems to refer to an unchanging transformation at the ...
- 279
0
votes
0
answers
30
views
Expected value of applying the product of the sigmoid-like function to a normal distribution
Let $\sigma : \mathbb{R}\to\mathbb{R}$ be a sigmoid-like function, $b \sim \mathcal{N}(0, 1)$, $x_1 \in \mathbb{R} $, and $x_2 \in \mathbb{R}$.
Can we obtain a closed-form solution of $\mathbb{E}[\...
- 11
0
votes
0
answers
21
views
Proving that the linear kernel is a valid kernel [duplicate]
I want to convince myself that $K(x,z) = x^Tz$ (the linear kernel) is a valid kernel using two conditions of Mercer's Theorem.
First: it is easy to show symmetry, $K(x,z) = K(z,x) = x^Tz = z^Tx$ since ...
- 8,457
4
votes
1
answer
38
views
Parameters of Gaussian process automatic relevance determination
In section 6.4.4 of "Pattern Recognition and Machine Learning" written by Christopher M. Bishop, it is said that, in automatic relevance determination (ARD), there is a separate parameter ...
- 183
1
vote
0
answers
35
views
Is the product of the sigmoid output positive definite kernel?
$$K(x_i, x_j) = \sigma(x_i)*\sigma(x_j),$$ where $\sigma: \mathbb{R}\to\mathbb{R}$ is a sigmoid function. Is this kernel positive definite? Thank you for your help.
- 11
7
votes
5
answers
3k
views
Would a machine learning classifier algorithm be able to determine whether a number is odd or even?
I was testing out some classifier algorithms in scikit but wasn't able to find a classifier (linear or non-linear) that managed to provide good prediction on whether an input number is odd or even. ...
- 181
0
votes
0
answers
19
views
Is the product of infinitely many valid kernel functions still a valid kernel function?
I know it is true for product of two kernels. But I'm not sure if it is the case for infinitely many valid kernel functions. I tried to follow the proof for product of two kernels, but I don't know if ...
- 183
0
votes
0
answers
36
views
Linear regression using kernel techniques
I am looking for a gentle introduction to kernel techniques and how it is applied to linear regression. Currently, I am reading this source from Stanford engineering (by Andrew Ng), whose link I can't ...
- 8,457
0
votes
0
answers
17
views
Excercise 6.1 and its solution in Bishop's PRML, Question 9
The problem comes from Exercise 6.1 of "Pattern Recognition and Machine Learning" by Christopher M. Bishop:
Consider the dual formulation of the least squares linear regression
problem ...
- 183
0
votes
0
answers
15
views
Excercise 6.1 and its solution in Bishop's PRML, Question 8
The problem comes from Exercise 6.1 of "Pattern Recognition and Machine Learning" by Christopher M. Bishop:
Consider the dual formulation of the least squares linear regression
problem ...
- 183
0
votes
0
answers
21
views
Exercise 6.1 and its solution in Bishop's PRML, Question 7
The problem comes from Exercise 6.1 of "Pattern Recognition and Machine Learning" by Christopher M. Bishop:
Consider the dual formulation of the least squares linear regression
problem ...
- 183
0
votes
0
answers
7
views
Excercise 6.1 and its solution in Bishop's PRML, Question 6
The problem comes from Exercise 6.1 of "Pattern Recognition and Machine Learning" by Christopher M. Bishop:
Consider the dual formulation of the least squares linear regression
problem ...
- 183
0
votes
0
answers
14
views
Excercise 6.1 and its solution in Bishop's PRML, Question 5
The problem comes from Exercise 6.1 of "Pattern Recognition and Machine Learning" by Christopher M. Bishop:
Consider the dual formulation of the least squares linear regression
problem ...
- 183
0
votes
0
answers
14
views
Excercise 6.1 and its solution in Bishop's PRML, Question 4
The problem comes from Exercise 6.1 of "Pattern Recognition and Machine Learning" by Christopher M. Bishop:
Consider the dual formulation of the least squares linear regression
problem ...
- 183
0
votes
0
answers
14
views
Excercise 6.1 and its solution in Bishop's PRML, Question 3
The problem comes from Exercise 6.1 of "Pattern Recognition and Machine Learning" by Christopher M. Bishop:
Consider the dual formulation of the least squares linear regression
problem ...
- 183
0
votes
0
answers
15
views
Excercise 6.1 and its solution in Bishop's PRML, Question 2
The problem comes from Exercise 6.1 of "Pattern Recognition and Machine Learning" by Christopher M. Bishop:
Consider the dual formulation of the least squares linear regression
problem ...
- 183
1
vote
1
answer
29
views
Excercise 6.1 and its solution in Bishop's PRML, Question 1
The problem comes from Exercise 6.1 of "Pattern Recognition and Machine Learning" by Christopher M. Bishop:
Consider the dual formulation of the least squares linear regression
problem ...
- 183
0
votes
0
answers
29
views
How to know if kernel PCA can be useful for clustering and how to know which kernel to use (data with low collinearity)?
I have a dataset with ~13.000 samples and ~200 variables. The data includes a target variables, dividing the samples into 3 clusters, The cumulative explained variance (CEV) increases very slowly ...
- 1
0
votes
0
answers
65
views
How to prove valid kernel by constructing a feature map?
I just wanted to make sure my reasoning seemed thorough:
Q: Suppose we have x ∈X,z ∈X, g : X→R. Prove that k(x,z) = g(x) ×g(z)
is a valid kernel by constructing a feature map Φ(·) and show that k(x,z) ...
- 1
0
votes
0
answers
25
views
In which situation will clustering gives similar results as classification?
Suppose you have a labeled data set $(X,Y)$ where $Y$ is the target variable taking its value from $\{1,...,K \}$. Imagine you run a classification model on this dataset, which gives you good ...
- 13
0
votes
0
answers
12
views
Quantifying the amount of information of a temporal signal after a kernel trick
We commonly use kernel trick, like in Neural Net, Support Vector Machine, etc, to map input signal to a high-dimensional feature spaces. Such mapping has enabled us to perform non-linear ...
- 1
1
vote
1
answer
95
views
Is this difference of two kernels also a kernel?
If we have two linear kernels $k1$, $k2$ over $R^2$ defined as:
$k1(x,y) = x^\mathsf{T}
\begin{bmatrix}
3 &1 \\
1 &5\\
\end{bmatrix}y$ and $k2(x,y)=x^\mathsf{T}y$ for all $x,y\in R^2$.
...
- 13
0
votes
0
answers
8
views
is there any conversion of the weights of the kernelized primal of a svm to the feature weights
In this question by mini quark, the answer by firebug shows a way to use kernels in the primal, I have performed a variation of the primal SVM, and I am wondering if there is any way that with the ...
1
vote
0
answers
18
views
Kernels and Support Vector Machines - any specific motivation?
In many applications, I have seen the combination of Kernels and Support Vector Machines. Why not combine Kernels with Linear or Logistic Regression instead? Is there any explicit reason to combine ...
- 2,608
1
vote
2
answers
105
views
Non linearly separable data to linearly separable data in the same dimension or lower dimension
In continuation to this question, Non linear to linearly seperable
I am trying to understand if we do not want to transform to a higher dimension is it possible to still find a feature map that maps ...
3
votes
0
answers
95
views
Equivalence between Gaussian Process Regression and Kernel Ridge Regression
Consider the model
$$ y(\mathbf{X}) = f(\mathbf{X}) + \epsilon, $$
where $\mathbf{X}$ is a given $n\times D$ matrix, and where $\epsilon \sim \mathcal{N}(0, \sigma^{2}I_{N})$ is iid Gaussian and is ...
- 311
3
votes
2
answers
225
views
Prove that ${f(x,y) = x^Txx^Tyy^Ty}$ is a valid kernel
The rules I can use without a proof are the following ones:
The thing that I know that a kernel is valid if it is symmetric, e.g. ${f(x,y) = f(y,x)}$ and if ${c^TKc >= 0}$, for each vector ${c \...
- 217
0
votes
0
answers
26
views
Why we need RKHS? [duplicate]
I'm pretty informed of the concept of RKHS and relative facts like Hilbert spaces or kernel function.
But why we relate kernel function to linear algebra and raise specificly the concept of RKHS?
Is ...
1
vote
0
answers
26
views
Would removing covariates correlated with model residuals (errors) from training improve prediction in machine-learning or deep-learning? [closed]
Would removing covariates correlated with model residuals (on training data) improve prediction in machine-learning or deep-learning or make the weights calculated more interpretable or useable to ...
user318514
0
votes
0
answers
35
views
Is it possible to apply the kernel trick to a "mahalnobis distance learner" such as GLS?
1.https://arxiv.org/pdf/0804.1441.pdf
2.https://www.sciencedirect.com/science/article/abs/pii/S0925231210001165
These papers describe kernelizing a mahalanobis distance learner.
I am interested in ...
user318514
1
vote
0
answers
39
views
Is there a multilinear kernel principal components analysis?
PCA can be extended to kPCA using the kernel trick.
MPCA is a multilinear extension of PCA that involves multiple matrices for the different modes of the data tensor.
Can MCPA be similarly extended ...
- 6,135
1
vote
0
answers
40
views
Is there a difference between kernel PCA with a non-linear kernel vs PCA with a non-linear change of variables?
I see that kernel PCA with a linear kernel is the same as PCA.
On Wikipedia's introduction of the kernel to PCA they suggest that there exists a non-trivial arbitrary choice of map $\Phi$ that is ...
- 6,135
0
votes
0
answers
13
views
Objective of maximal margin classifier with quadratic decision boundary
What is the objective for minimization for a maximal margin classifier with quadratic decision boundary of the form $ x^{T}Ax + b^Tx + c =0 $?
- 273
3
votes
0
answers
81
views
Relation between generalization bounds of Kernel Ridge Regression and largest eigenvalue of the kernel Gram matrix
Consider a positive-definite, symmetric function $k(x_1, x_2)$ which is used, given the dataset $\{(x_i, y_i)\}_{i=1}^m$, to construct the Gram matrix $K = [k(x_i, x_j)]_{i,j \in 1, ..., m}$.
What is ...
- 41
1
vote
1
answer
68
views
How to derive the three matrices of SVD from eigenvalue decomposition in Kernel PCA?
Kernel PCA is usually done via eigenvalue decomposition of the Kernel Matrix $\mathbf{K}$ and standard PCA via SVD of the input $\mathbf{X}$.
In standard PCA as far as I know we can derive $\mathbf{S}...
- 23