Questions tagged [kernel-trick]

Kernel methods are used in machine learning to generalize linear techniques to nonlinear situations, especially SVMs, PCA, and GPs. Not to be confused with [kernel-smoothing], for kernel density estimation (KDE) and kernel regression.

692 questions
Filter by
Sorted by
Tagged with
397 views

Universal approximation of Gaussians

Can gaussian kernels reproduce non continuous L2 integrable functions? ( Do non continuous L2 integrable functions lie in the RKHS constructed by a Gaussian Kernel?) Edit: I think my question is being ...
• 65
31 views

Why are Kernels said to be a measure of Covariance? [duplicate]

I have always heard that Kernels are said to be a measure of Covariance - intuitively, I can somewhat understand why this argument is being made; however, I would like to confirm if my understanding ...
• 5,650
15 views

What kernel to use for image classification from pre-trained CNN feature extractor

Suppose I have a pre-trained CNN feature extractor and I connect those to a soft margin SVM, what is the recommended kernel to use to replace $x_n^Tx$ in SVM? My dataset comprises of pictures of ...
• 287
1 vote
11 views

A question about linear inference in random Fourier feature kernels

In Ali Rahimi's and Ben Recht's paper "Random Features for Large-Scale Kernel Machines," there is a line near the bottom of the introduction which I can not reason about... In addition to ...
• 619
20 views

Estimating the Probability in a statistical analysis

I am carrying out a statistical analysis where I run the simulation (Matlab) 5000 times, to get 5000 results. The objective is to estimate the probability of having a result that is less than or equal ...
• 143
1 vote
32 views

Information preserved in the kernel mean embedding

I have recently been introduced to the kernel mean embedding of distributions, that is the map $$\mu: \mathcal{M}^{1}_{+}(X) \rightarrow \mathcal{H} \\ \mu(P) := \int \phi(x) dP(x)$$ where $K$ is a ...
12 views

Would l-1 regularization with kernel trick induce sparsity on feature map's features?

Would l-1 regularization with kernel trick induce sparsity on the infinite dimensional feature map's features in the case of gaussian kernel?
• 224
7 views

Would logistic regression/support vector-machine with l-2 regularization and early stopping regularization cause underfitting?

Would early stopping regularization combined with l-2 regularization or in logistic regression/support vector machine cause underfitting? Does a kernel-trick affect what combination of regularization ...
• 224
15 views

• 31
45 views

Why are radial basis functions so different from classic inner product?

I was studying SVM with kernel tricks and it seems that the kernel is a modified dot product. A simple kernel would be $K(x,y) = <x,y>^2$. I understand how this is a modification of the dot ...
1 vote
29 views

Prove that $k(u, v) = \tilde{k}(\phi(u), \phi(v))$ [closed]

Given that $\phi : \mathcal{X} → \mathcal{X}′$, prove that $k(u, v) = \tilde{k}(\phi(u), \phi(v))$. I've seen similar proofs where if $\phi : \mathcal{X} → \mathcal{X}$, the transformation is simply ...
• 31
1 vote
72 views

Prove that $k(x,y) = f(x)\tilde{k}(x,y)f(y)$ is a valid kernel

Given that $f: \chi \rightarrow \mathbb{R}$, prove that $k(x,y) = f(x)\tilde{k}(x,y)f(y)$ is a valid kernel, using only the fact that addition and multiplication yields valid kernels. My approach was ...
• 31
97 views

Why Are Neural Networks Considered "Expensive" to Train?

Recently, I was looking at the optimization functions required in training Kernel Based Methods compared to Neural Networks. 1) Kernel Methods: For instance, I was looking at the optimization in ...
• 5,650
12 views

How do you obtain extra-dimensional values using the kernel trick in SVMs?

To preface, I've been reading about SVMs and the kernel trick for the better part of an hour, and I think I understand what it is trying to accomplish fairly well, but what I don't understand is the ...
1 vote
27 views

Prove that the variance of a Gaussian Process is minimum on its train data points

I want to prove that the variance of a Gaussian Process (GP) is the lowest on any one of its $p$ training data points. The prior distribution for a zero-mean GP prior, with kernel function $k(x, x')$ ...
• 155
1 vote
145 views

Theoretical Speculations as to Why Neural Networks have Replaced Kernel-Based Methods

I have been reading about the history of statistical and machine learning algorithms, and am particularly interested in the reasons as to why neural networks have "replaced" kernel-based ...
• 5,650
104 views

scikit-learn SVC with custom precomputed kernel matrix uses too much memory

I need to implement a custom kernel for the sklearn.svm.SVC learner. My custom kernel consists in multiplying every element of the kernel matrix except the main ...
1 vote
29 views

What is representer theorem?

Representer theorem states that "a minimizer of a regularized empirical risk functional defined over a reproducing kernel Hilbert space can be represented as a finite linear combination of kernel ...
• 255
32 views

kernel PCA similarity matrix analogy

The standard explanation to linear PCA begins with the covariance matrix. That is, for a dataset $D$ of dimension $N \times d$, the covariance matrix is given as $\sum = \frac{D^{T}D}{N}$ where the ...
• 207
46 views

Is a kernel just a symmetric, positive semi-definite, and continuous matrix?

From page 9 of these course notes, A function $k : \mathcal{X} \times \mathcal{X} \mapsto \mathbb{R}$ is a kernel if $k$ is symmetric: $k(x,y) = k(y,x)$ $k$ gives rise to a positive semi-definite &...
• 2,906
109 views

25 views

Fit a kernel density function

I'm working on fitting a kernel density estimator and setting the correct bandwidth. The most popular technique is to minimise the following function. (see https://en.wikipedia.org/wiki/...
27 views

Does a gaussian kernel suffer from the curse of dimensionality?

Some embedding methods map a data vector in original space to a new space with significantly high dimension and then calculate dot product between these mapped high dimensional vectors. Don't they ...
32 views

Machine learning kernel with complex feature map

I have a question regarding my machine learning lecture where we had to decide whether $$K(x,y)=x_1y_1-x_2y_2$$ is a valid kernel (e.g. for a SVM). My intuition would say that it is a valid kernel ...
• 203
113 views

Proving the function is a kernel

I have an exercise in my book, which I'm not sure if I have answered correctly. Here's the exercise: For the function $K: \mathbb{R}^2\times \mathbb{R}^2\to \mathbb{R}$ such that $$K(x,t)=x^TDt$$, ...
• 203
48 views

Can different kernels be used when performing Gaussian Process Regression?

Given the equations for exact Gaussian process regression: \bar{\boldsymbol{f}_*} = \boldsymbol{m}(X_*) + K_{*f}(K_{ff} + \sigma^2I_N)^{-1}K_{f*}(\boldsymbol{y} - \boldsymbol{m}(X)), \...
1 vote
124 views

How to derive an inverse of Gaussian Kernel

As an example, say I have a function (Gaussian process kernel): $$K(x_i, x_j) \equiv \exp(-\alpha |x_i-x_j|^2)+\beta \delta_{ij}$$ Is there a way to analytically express $K^{-1}(x_i,x_j)$, s.t. ...
• 261