Questions tagged [kernel-trick]

Kernel methods are used in machine learning to generalize linear techniques to nonlinear situations, especially SVMs, PCA, and GPs. Not to be confused with [kernel-smoothing], for kernel density estimation (KDE) and kernel regression.

Filter by
Sorted by
Tagged with
3
votes
1answer
114 views

What does representer theorem in machine learning tells us?

In reference to the Representer Theorem in machine learning, Why this is so important? Somehow, this theorem justifies the importance of Kernels in machine learning, i.e. the Kernel trick - a more ...
1
vote
1answer
23 views

Was Kernel Regression Invented to Address the Problems with Higher Order Polynomial Regressions?

I had this thought today : We all know that higher order polynomials (e.g. polynomial regression models) have a tendency of overfitting the data and performing poorly (i.e. generalizing) to new data. ...
2
votes
1answer
48 views

Number of features in SVM Kernel matrix

I'm trying to implement the SVM algorithm manually, and I've succeeded in doing so with the case of a linear kernel (no kernel function). Now I'd like to add a gaussian kernel to the algorithm, but I'...
0
votes
0answers
138 views

A new method for processing music scores?

I have developed a method and python script: https://github.com/githubuser1983/algorithmic_python_music which allows the user to input a midi file and then chose a few numbers as parameters, and the ...
0
votes
1answer
26 views

Kernel design for Gaussian processes with multiple inputs

How can I design a kernel function when there are multiple input variables and their degree of influence on the covariance with the target variable is different from each other? For example, if the ...
2
votes
0answers
54 views

Which popular learning-based algorithms cannot be kernelized?

Question: Out of the most popular learning-based algorithms, which cannot be kernelized? I am aware that "popular" is rather ambiguous, I cannot give an exact search space but it should ...
0
votes
0answers
26 views

Sampling operator for vector valued functions

I have a question regarding learning vector valued functions. Let be the theoretical playground a Reproducing kernel Hilbert space $\mathcal{H}$ with kernel $k$. If we want to learn a scalar function $...
0
votes
0answers
13 views

What if I transform data into other space instead using kernel methods

I have a very small training dataset of 30 points (excluding test), the datasets is 3 dimensional (a 30 elements set of 3 element column vectors). The output that I want to predict is 1 dimensional (a ...
0
votes
1answer
106 views

Find the lowest number of samples for which the kernel matrix is singular

The question is as follows: Let $X$ be a matrix where the entries are natural numbers between 1 and $m$ $( X ∈ \{1 , 2 , 3 , . . . , m\}^{d×n} )$, $m >> 0$, $d$ is the number of features and $n$ ...
1
vote
0answers
18 views

How to create a weighted composite covariance function using GPML toolbox in MATLAB?

I'm trying to create a composite covariance function to model my data. Specifically I want to create a kernel which is weighted between @covSEard & ...
2
votes
0answers
42 views

What is the most suitable kernel for categorical features?

I am trying to train a Gaussian process regression model based on some data with a large amount of categorical features. Each data point can be represented by a vector of strings. Right now I am using ...
1
vote
1answer
47 views

Multi-kernel Gaussian Process model?

In Gaussian Process (GP), the kernel (covariance function) is used to measure the similarity between every two points. Currently I am using the Squared Exponential (SE) kernel to measure the ...
2
votes
1answer
80 views

Prove that multiplication with positive semidefinite matrix is a kernel

Let $A \in \mathbb{R}^{d \times d}$ be a symmetric and positive semi-definite matrix. Prove that $$ k(\textbf{x}, \textbf{x}') = \textbf{x}^T A \textbf{x}' $$ is a kernel. My first thought when I saw ...
1
vote
1answer
50 views

Prove that linear combination of kernels is not a kernel for real-valued coefficients

Let's say that $k_1(\textbf{x}, \textbf{x}')$, $k_2(\textbf{x}, \textbf{x}')$: $\mathbb{R}^d \times \mathbb{R}^d \rightarrow \mathbb{R}$ are kernels. It can then be proven that the linear combination ...
0
votes
0answers
10 views

Has a valid kernel to be positive in one dimension?

During my ML exercises, I have come across the kernel below: $$ k(x,y) = sin(x)sin(y),\ x,y \in \mathbb{R}$$ The text is asking me if this is a valid kernel. My current knowledge would reply "yes&...
1
vote
0answers
24 views

Why is the posterior of a neural network gaussian process equal to the posterior of a neural network in the limit of infinite width layers?

Its noted in this paper, Deep Neural Networks as Gaussian Processes, that "a single-layer fully-connected neural network with an i.i.d. prior over its parameters is equivalent to a Gaussian ...
0
votes
0answers
19 views

Influence variance Kernel on GPR prediction

Take a gaussian kernel such that $K(x,y) = \sigma_f^{2}\exp^{-\gamma||x-y||^2}$. We have two hyperparameters: $\sigma_f$ and $\gamma$. In GPR framework, the prediction at an unobserved point has the ...
1
vote
0answers
35 views

Equivalence RKHS Minimization

Take a kernel $K$. You want to use this kernel in a regression context. I read that the kernel rigde regression solution with kernel $K$ is equivalent to redefine the kernel $K$ as $K_{eq}(x,x') = K(x,...
0
votes
1answer
26 views

Is it efficient to use kernel trick in primal form of SVM?

I know we can use Kernel trick in the primal form of SVM. So the hypothesis will be - and optimization objective - We can optimize the above equation using gradient descent, but in this equation ...
2
votes
0answers
37 views

Kernel trick to logistic regression

Why can't I apply the kernel trick in logistic regression? My reasoning is: in SVM the logit is: $z = \sum_i \alpha_i K(x_i, x) + b$ Where K is the kernel function. In logistic regression you have ...
0
votes
0answers
20 views

Link between RKHS reconstruction and SVM

Simply put, my question is the following : I know Reproducing Kernel Hilbert Space Reconstruction (RKHS)-type methods like RBF reconstruction and I've used them in the past. I understand Support ...
0
votes
0answers
34 views

Centering in kernel pca

Below is how I am applying kernel pca. But my data in X is not mean centered (it does not have zero mean). In this regard, is below code correct (for pca we need mean centered data...what about kernel ...
3
votes
2answers
70 views

Idea of the kernel trick

I'm reading this article and I can't really grasp the idea of this so-called kernel trick. So far, what is present, is: $ \Phi(x)^T * \Phi(y) = \sum x_ix_jy_iy_j$ and $ k(x, y) = (x^T*y)^2 = \sum ...
2
votes
1answer
40 views

Kernel Ridge Regression and RKHS

Say you have $N$ observations of the function $F$. You want to predict the value of $F$ at an unknown point $x*$. Assume there is no noise in the observations. Given a kernel $K$, one is looking for ...
0
votes
0answers
30 views

Inner Product RKHS and regression

Say you want to solve the kernel ridge regression as follows: $\min\limits_{f\in H_K} \left[\lambda||f||^{2}_{H_K} + \sum\limits_{i=1}^{N}(y_i - f(x_i))^2\right]$, where $\lambda > 0$. We know how ...
0
votes
0answers
21 views

Why is the Kernel Trick Matrix Symmetric Even When the Training and Test Set Differ in Size?

Ridge Regression can be expressed as $$\hat{y} = (\mathbf{X'X} + a\mathbf{I}_d)^{-1}\mathbf{X}x$$ where $\hat{y}$ is the predicted label, $\mathbf{I}_d$ the $d \times d$ identify matrix, $\mathbf{x}$ ...
1
vote
1answer
65 views

About Kernel normalization in regression

Is it a good idea to normalize your kernel in the Gaussian Process Regression framework? By normalization, I mean $K_{norm}(x,y) = \frac{K(x,y)}{\sqrt{K(x,x)\times K(y,y)}}$. For instance, take a ...
0
votes
0answers
8 views

Kernel PCA embedding with the matrix inverse instead of square root

Given a set of points $X = \{x_1, x_2, \cdots, x_m\}$, one way of defining the Kernel PCA embedding for a new point $z$ is $K^{-1/2} [k(z, x_1), k(z, x_2), \cdots, k(z, x_m)]$ where $k$ is the kernel ...
0
votes
0answers
8 views

Extracting feature weights after fitting SVC with pre-computed linear kernel

I'm using sklearn's SVC with a linear kernel to train and predict brain states from functional MRI data. Upon completion, I want to extract the feature weights to identify which of these contain the ...
1
vote
0answers
19 views

Effect of the distribution of inputs GPR

I would like to know the effect of the sampling of the inputs parameters while wanting to build a metamodel using a Gaussian Process Regression. Say you want to build an metamodel by using $N$ ...
1
vote
1answer
29 views

Assumptions on inputs Gaussian Process Regression

Let $F:\mathcal{X} \to \mathcal{Y}$ a function one seeks to approximate. You have $N$ observations of this function and you want to predict the value at some other points. In the Gaussian Process ...
0
votes
0answers
9 views

What is the purpose of using two different vectors when computing polynomial kernel?

I'm having a hard time understanding the fundamental reason of using two different vectors when computing this mapping. For example, if we have two separate vectors $x_1$ and $x_2$, each having two ...
0
votes
0answers
20 views

Multicollinearity in Kernel Ridge Regression using 1-hot features

I am working on an ML model using Kernel Ridge Regression method. My features include a categorical 1-Hot encoded vector, and a proper $\mathbb{R}^n$ vector derived from the 1-hot vector (in embedding ...
0
votes
0answers
18 views

Alternative of the Nyström method

Say you want to obtain the eigenvalues/vectors of the integral operator associated to a kernel $K$. I know there is the Nyström method to obtain an approximation of these eigenvalues/vectors. What are ...
1
vote
1answer
90 views

Mercer's theorem and eigenfunctions

Consider a kernel $K$ which satisfies the conditions of Mercer's theorem. We know that $K(x,y) = \sum\limits_{i=1}^{+\infty}\lambda_ie_i(x)e_i(y)$ where $\lambda_i$ and $e_i$ are the eigenvalues/...
0
votes
0answers
35 views

Pre Image Problem in Kernel PCA: Understanding a Paper

Synopsis I want to implement the method for finding pre-images of a feature vector in the feature space after applying kernel methods described in https://www.aaai.org/Papers/ICML/2003/ICML03-055.pdf ...
0
votes
0answers
34 views

Linear kernel SVM performs well on non-linear separable data

I am working on an exercise for my course in SVMs. We are given a 2D-dataset and we are supposed to construct a SVM model that classifies the data as good as possible. When I plot the training set it ...
4
votes
1answer
78 views

Why use RBF kernel if less is needed?

I have seen online theorem's such as Cover's theorem Wikipedia which prove how given $p$ points in $\mathbb{R}^N$ the linear separability is almost certain as the fraction $\dfrac{p}{N}$ is kept close ...
0
votes
0answers
15 views

Kernel PCA for novelty detection and support estimation

For my thesis, I have been working for the past few months on a paper that takes KPCA as a tool to estimate the support of a distribution. However, it is very heavy on math and I would like external ...
0
votes
0answers
11 views

Polynomial Kernel and similarity across dimensions

I am using SVM to classify hand grasps, and as such I am reading up on the polynomial kernel. A section on the polynomial kernel in SKlearn reads: Conceptually, the polynomial kernels considers not ...
0
votes
0answers
38 views

Reproducing kernel - Eigenvalue problem

In page 9 of properties of SVM, the authors defined the reproducing kernel to be symmetric and positive definite. It is then stated that Given such a kernel $K$, a possible set of functions $\mathbf{\...
1
vote
1answer
22 views

count Number of unique common substrings using kernel

We have two strings consist of characters. we define kernel of two strings as number of unique common substrings. how to proof number of unique common substrings is a valid function for kernel? for ...
0
votes
0answers
14 views

Approximating a predictor with a kernel

Assume that some predictor $f$ is a kernel machine, but the kernel function $K(\cdot, \cdot)$ is unknown. Is there a way to recover the kernel $K(\cdot, \cdot)$ that "best approximates" $f$? ...
2
votes
0answers
23 views

In the broadest sense, what is a "kernel"?

In MCMC sampling methods, a transition kernel, as found in Metropolis(/Hastings) algorithm, is the comparison of the likelihood of the current position and the likelihood of the proposed position. ...
0
votes
0answers
36 views

Cross validation for kernel regression

I've been reading about the Kernel trick, where we we can obtain a prediction by calculating: $\hat{y} = y(K(x,x)+\lambda I_n)^{−1} K(x,\hat{x})$, where $K(x,z) = (1+xz)^p$. If I want to tune the ...
1
vote
1answer
42 views

What is it called when you find the best fit in an RKHS to some training data?

Suppose I have a series of labelled training inputs $(x_i, y_i)$, and a kernel function $k$ on the input domain, with a corresponding RKHS $H$. Now form the Gram matrix $A$, where $A_{ij}=k(x_i, x_j)$....
2
votes
1answer
65 views

Prove that the following matrix is positive definite

We define $K_{\mathbf{a}, \mathbf{b}}$ as the $n \times m$ matrix whose $ij^{th}$ entry is $\kappa(a_{i}, b_{j})$ Where, $\kappa$ is a (positive definite) kernel function. Here, $\mathbf{a}_{i}, \...
0
votes
0answers
32 views

Is $k(x, x') = 1$ if $x = x' $ and $k(x, x') = 0$ otherwise a valid kernel function?

Is $k(x, x') = 1$ if $x = x' $ and $k(x, x') = 0$ otherwise a valid kernel function? I can't prove the Gram matrix is positive semi-definite. I was wondering if we can find a basis function $\phi: X \...
1
vote
1answer
51 views

Why is there a discrepancy between the eigenvalues of the covariance matrix (PCA) and the eigenvalues of the kernel matrix (kernel PCA)?

I've done PCA on my data matrix $ \mathbf{X} $ which gives me i.a. the eigenvalues $ \lambda $ and eigenvectors $ v $ of the data covariance matrix $ C=\mathbf{X}^T \mathbf{X} $. I'm now extending my ...
2
votes
1answer
31 views

Computing sample variance

Anyone could help with this question on my homework? I managed to substitute f into the expression, and express the denominator as a summation of kernels. However, I am not sure what should I do with ...

1
2 3 4 5
14