Questions tagged [kernel-trick]

Kernel methods are used in machine learning to generalize linear techniques to nonlinear situations, especially SVMs, PCA, and GPs. Not to be confused with [kernel-smoothing], for kernel density estimation (KDE) and kernel regression.

Filter by
Sorted by
Tagged with
2 votes
2 answers
397 views

Universal approximation of Gaussians

Can gaussian kernels reproduce non continuous L2 integrable functions? ( Do non continuous L2 integrable functions lie in the RKHS constructed by a Gaussian Kernel?) Edit: I think my question is being ...
user avatar
  • 65
-1 votes
0 answers
31 views

Why are Kernels said to be a measure of Covariance? [duplicate]

I have always heard that Kernels are said to be a measure of Covariance - intuitively, I can somewhat understand why this argument is being made; however, I would like to confirm if my understanding ...
user avatar
  • 5,650
0 votes
0 answers
15 views

What kernel to use for image classification from pre-trained CNN feature extractor

Suppose I have a pre-trained CNN feature extractor and I connect those to a soft margin SVM, what is the recommended kernel to use to replace $x_n^Tx$ in SVM? My dataset comprises of pictures of ...
user avatar
  • 287
1 vote
0 answers
11 views

A question about linear inference in random Fourier feature kernels

In Ali Rahimi's and Ben Recht's paper "Random Features for Large-Scale Kernel Machines," there is a line near the bottom of the introduction which I can not reason about... In addition to ...
user avatar
  • 619
0 votes
0 answers
20 views

Estimating the Probability in a statistical analysis

I am carrying out a statistical analysis where I run the simulation (Matlab) 5000 times, to get 5000 results. The objective is to estimate the probability of having a result that is less than or equal ...
user avatar
1 vote
0 answers
32 views

Information preserved in the kernel mean embedding

I have recently been introduced to the kernel mean embedding of distributions, that is the map $$\mu: \mathcal{M}^{1}_{+}(X) \rightarrow \mathcal{H} \\ \mu(P) := \int \phi(x) dP(x)$$ where $K$ is a ...
user avatar
0 votes
0 answers
12 views

Would l-1 regularization with kernel trick induce sparsity on feature map's features?

Would l-1 regularization with kernel trick induce sparsity on the infinite dimensional feature map's features in the case of gaussian kernel?
user avatar
  • 224
0 votes
0 answers
7 views

Would logistic regression/support vector-machine with l-2 regularization and early stopping regularization cause underfitting?

Would early stopping regularization combined with l-2 regularization or in logistic regression/support vector machine cause underfitting? Does a kernel-trick affect what combination of regularization ...
user avatar
  • 224
0 votes
0 answers
15 views

Is the exponentiated sum of squared norms a valid kernel function?

Define $$k(x, y) = e^{-(||x||^2 + ||y||^2)}$$ Is this a valid kernel function? My guess is yes, with the feature map $\phi(x) := e^{-||x||^2} \in \mathbb{R}_{>0}$. Then $k(x,y) = \phi(x) \cdot \phi(...
user avatar
0 votes
0 answers
21 views

Proof of polynomial kernel using positive semi-definiteness

I was asked to prove that $K$ is a valid kernel: $K(\vec{u}, \vec{v}) = \sum_{i=0}^M {a_i(G(\vec{u}, \vec{v}))^i}$ with $a_i \ge 0$, given that $G$ is a valid kernel. I tried to prove it using ...
user avatar
  • 133
2 votes
1 answer
373 views

In machine learning, for a kernel function k, is sqrt(k) also a valid kernel function?

For some kernel function k which by definition has a symmetric and positive semidefinite kernel matrix K, is the new kernel function sqrt(k) also a valid kernel function? If we use Mercer's theorem, ...
user avatar
0 votes
0 answers
20 views

Kernel Multinomial Logistic regression in R

I have a dataset of graphs/networks, and associated labels in R. I have read literature on graph kernels (e.g. Borgwardt et al), and can implement these with the ...
user avatar
0 votes
0 answers
16 views

Kernelized Decision Trees

I came across a simple example that shows where decision trees may have difficulty solving a classification problem efficiently: "[...] For example, if we have a two-class problem and the ...
user avatar
0 votes
0 answers
64 views

fitrgp: custom Kernel

I want to using Gaussian processes to dataset using a custom kernel. My kernel has one parameter 'beta' that I want to optimize. The problem is that the parameter 'beta' is not changing at all during ...
user avatar
1 vote
0 answers
86 views

Square Root of kernel function validity [closed]

If k1(x,z) and k2(x,z) are valid kernels, then is k(x,z) a valid kernel, where k(x,z) = sqrt(k1(x,z)k2(x,z)) Prove using mercers theorem
user avatar
  • 19
0 votes
0 answers
69 views

Proving validity of kernels

If k1(x,z) and k2(x,z) are valid kernels, then is k(x,z) a valid kernel, where k(x,z) = a1k1(x,z) - a2k2(x,z) (where a1, a2 > 0 are real numbers) I don't think this is true, but I am having trouble ...
user avatar
  • 19
3 votes
1 answer
58 views

Kernel transformation in Machine Learning

I understand kernels allow us to linearly separate non-linearly separable data in a higher-dimensional space. Given a feature vector $\bar x = [x1,x2,..xn]^T$, we can apply the transformation $\phi(\...
user avatar
  • 31
0 votes
0 answers
45 views

Why are radial basis functions so different from classic inner product?

I was studying SVM with kernel tricks and it seems that the kernel is a modified dot product. A simple kernel would be $K(x,y) = <x,y>^2$. I understand how this is a modification of the dot ...
user avatar
  • 1
1 vote
0 answers
29 views

Prove that $k(u, v) = \tilde{k}(\phi(u), \phi(v))$ [closed]

Given that $\phi : \mathcal{X} → \mathcal{X}′$, prove that $k(u, v) = \tilde{k}(\phi(u), \phi(v))$. I've seen similar proofs where if $\phi : \mathcal{X} → \mathcal{X}$, the transformation is simply ...
user avatar
  • 31
1 vote
0 answers
72 views

Prove that $k(x,y) = f(x)\tilde{k}(x,y)f(y)$ is a valid kernel

Given that $f: \chi \rightarrow \mathbb{R}$, prove that $k(x,y) = f(x)\tilde{k}(x,y)f(y)$ is a valid kernel, using only the fact that addition and multiplication yields valid kernels. My approach was ...
user avatar
  • 31
4 votes
1 answer
97 views

Why Are Neural Networks Considered "Expensive" to Train?

Recently, I was looking at the optimization functions required in training Kernel Based Methods compared to Neural Networks. 1) Kernel Methods: For instance, I was looking at the optimization in ...
user avatar
  • 5,650
0 votes
0 answers
12 views

How do you obtain extra-dimensional values using the kernel trick in SVMs?

To preface, I've been reading about SVMs and the kernel trick for the better part of an hour, and I think I understand what it is trying to accomplish fairly well, but what I don't understand is the ...
user avatar
  • 1
1 vote
0 answers
27 views

Prove that the variance of a Gaussian Process is minimum on its train data points

I want to prove that the variance of a Gaussian Process (GP) is the lowest on any one of its $p$ training data points. The prior distribution for a zero-mean GP prior, with kernel function $k(x, x')$ ...
user avatar
  • 155
1 vote
2 answers
145 views

Theoretical Speculations as to Why Neural Networks have Replaced Kernel-Based Methods

I have been reading about the history of statistical and machine learning algorithms, and am particularly interested in the reasons as to why neural networks have "replaced" kernel-based ...
user avatar
  • 5,650
0 votes
1 answer
104 views

scikit-learn SVC with custom precomputed kernel matrix uses too much memory

I need to implement a custom kernel for the sklearn.svm.SVC learner. My custom kernel consists in multiplying every element of the kernel matrix except the main ...
user avatar
  • 1
1 vote
0 answers
29 views

What is representer theorem?

Representer theorem states that "a minimizer of a regularized empirical risk functional defined over a reproducing kernel Hilbert space can be represented as a finite linear combination of kernel ...
user avatar
  • 255
0 votes
0 answers
32 views

kernel PCA similarity matrix analogy

The standard explanation to linear PCA begins with the covariance matrix. That is, for a dataset $D$ of dimension $N \times d$, the covariance matrix is given as $\sum = \frac{D^{T}D}{N}$ where the ...
user avatar
  • 207
0 votes
0 answers
46 views

Is a kernel just a symmetric, positive semi-definite, and continuous matrix?

From page 9 of these course notes, A function $k : \mathcal{X} \times \mathcal{X} \mapsto \mathbb{R}$ is a kernel if $k$ is symmetric: $k(x,y) = k(y,x)$ $k$ gives rise to a positive semi-definite &...
user avatar
  • 2,906
0 votes
1 answer
109 views

How do Kernels Relate to Norms

The Euclidean Norm is associated with the linear kernel $K(x,y) = x^Ty$ as follows: If we denote by $\phi$ its feature map, we get $$\|\phi(x) - \phi(y)\|^2 = K(x,x) + K(y,y) - 2k(x,y) = \|x-y\|^2_{2}$...
user avatar
3 votes
1 answer
318 views

representer theorem in kernel ridge regression for slightly modified loss function

It is very well known that for a problem of the form $$\min_f\frac{1}{n}\sum_{i=1}^n (y_i -f(x_i))^2 + \lambda \|f\|^2_\mathcal{H}, \quad\lambda\ge0$$ for a $f$ in a RKHS $\mathcal{H}$ has a solution $...
user avatar
  • 363
1 vote
0 answers
44 views

Why are odd-degreed polynomial kernels slower than those with even degrees for SVM?

I have been using one-class support vector classifiers to extract features for multinomial classification. I noticed that fitting time is much longer when the degree of the polynomial kernel is odd. ...
user avatar
  • 111
4 votes
1 answer
265 views

Why not solve SVM with gradient descent instead of quadratic programming?

How is SVM optimization implemented in packages like Scikit-Learn? Clearly, SVM is a quadratic programming problem but why not just use gradient descent to update the parameters? Is it because we want ...
user avatar
  • 118
2 votes
1 answer
30 views

Kernel pca and kernel svm

PCA is a way to reduces dimension and complexities, but is it ok to use kernel PCA with radial basis function and then use kernel SVM using the same.
user avatar
2 votes
0 answers
55 views

Why do we try to "Reproduce" Hilbert Spaces in Statistics?

I am trying to better understand why people are interested in "reproducing" Hilbert Spaces in Statistics and Machine Learning. I (think) understand the general idea behind Hilbert Spaces. ...
user avatar
  • 5,650
1 vote
1 answer
57 views

Non-Ridge Kernelized Regression?

Every presentation that I have seen for kernelized regression focuses on finding $$\underset{f \in \mathcal{H}_k}{\min} \sum_{i=1}^{n}(y_i-f(\mathbf{x}_i))^2+\lambda \|f\|^2_{\mathcal{H}_k}.$$ Here, $\...
user avatar
0 votes
0 answers
24 views

Approximating RKHS norm with samples

I have a function $f'$, sampled from a Gaussian process prior with a known kernel $f' \sim N(0,K)$. Is it possible to find/approximate a solution for $min_w||f_w-f'||^2_H$ if I can calculate both $...
user avatar
1 vote
0 answers
25 views

Why isn't the reproducing kernel map unique?

I am working on a project using kernel PCA with a gaussian kernel, and I am trying to understand a part of the theory. According Mercer's thereom, I know that since the RBF kernel is PDS, there exists ...
user avatar
  • 361
3 votes
1 answer
117 views

Would Support Vector Machines work on arbitrary Hilbert spaces?

I have a few questions, Has SVM ever been used to classify points in a Hilbert space other than $\mathbb{R}^n$? Say $\ell^2$ or $L^2$? The concepts involved in the derivation (like the margin) all ...
user avatar
0 votes
1 answer
81 views

Why do my GaussianProcessRegressor prediction results converge to 0?

I am using sklearn GaussianProcessRegressor to predict a time series. The kernel I use is this: ...
user avatar
1 vote
0 answers
23 views

Explanation of the multiplier of gaussian process kernels in sklearn document

I have read the basic materials about gaussian process regression and understand its ideas. https://scikit-learn.org/stable/modules/gaussian_process.html However, when I look into the sklearn page, I ...
user avatar
0 votes
0 answers
30 views

Gaussian Process Covariance Guaranteed to be PSD?

I have a question regarding a proof to show that the covariance matrix of a Gaussian process is Positive SemiDefinite (PSD). Given the equation, $cov(\bar{f}) = K_{**} - K_{*f}K_{ff}^{-1}K_{f*}$ how ...
user avatar
0 votes
0 answers
27 views

How to calculate eigen-decomposition for a positive kernel in R^2 using R

Suppose $K$ is a continuous positive semi-definite kernel on $\mathcal{T} = [0,1]\times[0,1] \subseteq \mathbb{R}^2$. By Mercer Theorem, there is a eigen-decomposition of $K$ such that $$ K(x,y) = \...
user avatar
  • 1
0 votes
0 answers
25 views

Fit a kernel density function

I'm working on fitting a kernel density estimator and setting the correct bandwidth. The most popular technique is to minimise the following function. (see https://en.wikipedia.org/wiki/...
user avatar
0 votes
0 answers
27 views

Does a gaussian kernel suffer from the curse of dimensionality?

Some embedding methods map a data vector in original space to a new space with significantly high dimension and then calculate dot product between these mapped high dimensional vectors. Don't they ...
user avatar
3 votes
1 answer
32 views

Machine learning kernel with complex feature map

I have a question regarding my machine learning lecture where we had to decide whether $$K(x,y)=x_1y_1-x_2y_2$$ is a valid kernel (e.g. for a SVM). My intuition would say that it is a valid kernel ...
user avatar
0 votes
1 answer
113 views

Proving the function is a kernel

I have an exercise in my book, which I'm not sure if I have answered correctly. Here's the exercise: For the function $K: \mathbb{R}^2\times \mathbb{R}^2\to \mathbb{R}$ such that $$K(x,t)=x^TDt$$, ...
user avatar
0 votes
1 answer
48 views

Can different kernels be used when performing Gaussian Process Regression?

Given the equations for exact Gaussian process regression: \begin{equation} \bar{\boldsymbol{f}_*} = \boldsymbol{m}(X_*) + K_{*f}(K_{ff} + \sigma^2I_N)^{-1}K_{f*}(\boldsymbol{y} - \boldsymbol{m}(X)), \...
user avatar
1 vote
1 answer
124 views

How to derive an inverse of Gaussian Kernel

As an example, say I have a function (Gaussian process kernel): $$K(x_i, x_j) \equiv \exp(-\alpha |x_i-x_j|^2)+\beta \delta_{ij}$$ Is there a way to analytically express $K^{-1}(x_i,x_j)$, s.t. ...
user avatar
  • 261
0 votes
0 answers
22 views

Positive semi-definite kernel function in GPR

In the Gaussian Progress regression context, must the kernel function be positive semi-definite or positive definite? In many papers, such as in the book of Rasmussen [2006], it is not really clear if ...
user avatar
  • 161
0 votes
0 answers
29 views

Kernel trick in feature space

I am working on KPCA based fault detection. I have question concerning the kernel trick in the feature space. We all know that the dot product in the feature space is computed using kernel function. I ...
user avatar

1
2 3 4 5
14