# Questions tagged [kernel-trick]

Kernel methods are used in machine learning to generalize linear techniques to nonlinear situations, especially SVMs, PCA, and GPs. Not to be confused with [kernel-smoothing], for kernel density estimation (KDE) and kernel regression.

756 questions
Filter by
Sorted by
Tagged with
121 views

### In the contex of Kernel regression why do we define the feature map as equal to the Kernel $\varphi(x)=k(\cdot ,x)$?

I have a notational confusion I am trying to clear up. In the context of Kernel regression the following relationship between the kernel and the feature map is defined: Consider a positive-definite ...
• 1,445
1 vote
19 views

### Derivation of dual formulation of support vector regression

I'm trying to derive the dual formulation of epsilon-insensitive support vector regression. I think my derivation is correct, but I can't match it up to a result for the dual that I've seen given in ...
• 235
1 vote
29 views

### What is the best way to use Gaussian Processes to approximate highly non-stationary functions?

Gaussian process regression has trouble approximating functions with "kinks". So, what is the most widely used method to deal with this problem? I have found many proposed methods, including ...
• 43
32 views

### What are the "tricks" in machine learning? [closed]

I have come across a few different "tricks" in machine learning methodology, which I list below along with my rudimental understandings. The Kernel Trick: This is used in Support Vector ...
• 409
21 views

### SVM Kernel to compare histograms as input vectors

In lecture 7 of CS229 by Andrew Ng he mentions at the very end a specific Kernel that allows an SVM to "classify" how similar two histograms are, such as the demographics of 2 countries. He ...
• 33
17 views

### Using MMD for Feature Selection with Linear Regression: Valid Approach?

I'm using Maximum Mean Discrepancy (MMD) for feature selection (i.e., to select the features that minimize the dissimilarity between the training and testing datasets). I'm aware that MMD introduces ...
• 135
38 views

### Covariance inversion for Gaussian process

Background Let $x=f(u_x)\in\mathbb{R}$ and let $y=[f(u_y^1)\cdots f(u_y^{N})]\in\mathbb{R}^N$ for some function $f:u \in \mathbb{R}\mapsto \mathbb{R}$. Given $y$, $u_x$, $u_{y}^1,\dots, u_{y}^{N}$, I ...
• 373
24 views

### How to find K in kernel trick?

How does one go about finding the kernel when using the so-called "kernel trick?" Here is an example from quora: Simple Example: x = (x1, x2, x3); y = (y1, y2, y3). Then for the function f(...
• 21
7 views

### Computing Test Loss in Kernel Ridge Regression

In Kernel Ridge regression we have the standard loss function $$L(\beta) = \|Y-K\beta\|_2^2 + \alpha \beta^T K \beta$$ Here, $K$ is the kernel (gram) matrix. If I compute $\beta$ on a training set, so ...
• 1,501
10 views

### Estimation of bivariate function with one variable being constricted

Suppose the following classical supervised regression setting, $$y_{i} = f(x_{i}) + \epsilon_{i}, \quad i=1,\cdots,n,$$ where $\epsilon_{i}$ are i.i.d. zero mean Gaussian noise. The above regression ...
• 11
15 views

### Can I find the explicit feature map that generates exponent of a kernel?

Let's say I have a kernel $K$, and another kernel of the form : $$K' = e^K$$ now I know how to prove K' is a kernel, I can do it using taylor expansion of $e^x$ around $0$, but let's say if I want ...
• 123
32 views

### Does solution to ridge regression still minimizes the cost function when lambda is <=0?

This was a homework problem where I was asked to find explicit expression that minimises the cost function. I found the solution as : $\hat{\theta} = (X^TX + \lambda I)^{-1}X^Ty$ Now the problem ...
• 123
10 views

### What is normalized winning frequency in kernel self organizing map(SOM)?

In the k-means based kernel SOM, proposed by MacDonald and Fyfe (2000), the update of the mean is based on a soft learning algorithm mi(t + 1) = mi(t) + Λ[φ(x) − mi(t)] where Λ is the normalized ...
22 views

### theoretical question: why is RBF the 'best' kernel

I am trying to understand why the RBF kernel is usually used in many research papers doing kernel tricks. To reduce the scope, we can focus on linear regression (thus effectively, increasing the ...
• 9,217
19 views

### normalized dual activation function for neural tangent kernel

Let $\phi$ be an activation function. In this lecture note, The author assumes that the dual activation function, denoted as $\check{\phi}$ is normalized such that $\check{\phi}(1)=1$. How can it be ...
40 views

### How is the weight vector calculated when using kernel trick for ridge regression

Im trying to understand how kernelized ridge regression works, and how we manage to first transform, and subsequently learn on higher-dimensional features without explicitly having to calculate them. ...
• 33
47 views

### How to use random kitchen sinks for $\sigma \neq 1$?

The RBF kernel is given by $$k(x,y) = \exp\left(-\frac{\| x - y \|_2^2}{2 \sigma^2}\right)$$ where $\sigma$ is the length-scale parameter. I want to use the random kitchen sinks method to create a ...
1 vote
22 views

### RKHS inclusion relationship of the Erf network's NTK

In the referenced paper, it is stated that for ReLU networks, the Reproducing Kernel Hilbert Space (RKHS) of the Neural Tangent Kernels (NTK) remains unchanged regardless of the model's depth. I am ...
185 views

• 125
100 views

### Clarifying the difference between various regression methods called "kernel' or "Bayesian"

I want to understand the pairwise relationship between four types of regression: Bayesian Linear Regression, Gaussian Process Regression, Kernel Regression (Nadaraya-Watson), and Kernel Ridge ...
18 views

### Calculating the Orthogonal Distance to Kernel PCA subspace (with a new data)

I am studying Kernel PCA methods and now I'm trying to calculate orthogonal distances (OD) on the feature space. What I've found is, you can calculate ODs with a kernel trick if you are interested in ...
32 views

• 145
1 vote
15 views

### Given a psd matrix $Q$ and a kernel function $f(y_i, y_j)$, how do I find $Y \in \mathbb{R}^{n \times d}$ that best approximates $Q$? [duplicate]

The question is basically the title. I have a matrix $Q$ that I know is positive semi-definite. I now want to find the $Y$ that approximates this matrix under some kernel function $f(y_i, y_j)$. I ...
1 vote
251 views

### Non-stationary Random Fourier Features

Random Fourier Features (RFFs) were introduced by A. Rahimi and B. Recht in their 2007 publication Random Features for Large-Scale Kernel Machines. RFFs are based on Bochner's theorem, which applies ...
• 51
35 views

### Identifiability of models on RKHS

I have just started learning about using reproducing kernel hilbert spaces for regularisation in machine learning. I am looking for some examples of reproducing kernels that produce identifiable and ...
• 51
59 views

### Feature maps of the chi-squared kernel

The additive chi-squared kernel for histograms is defined as $$K(x,y)= \sum_{i=1}^n \frac{2x_i y_i}{x_i + y_i}$$ Is this kernel positive definite on histograms? And if so, is there a known expression ...
73 views

### Method of evaluating the feature map of a polynomial kernel feature mapping

I'm attempting to implement an adaptive kernel Kalman filter following this paper https://arxiv.org/abs/2203.08300, but I'm struggling to find a method of evaluating the feature mapping for a ...
68 views

### Is the transformation implied by a positive-type kernel well-defined?

I’ve been trying to get my head around the particularity of the Hilbert space that a positive-type (equiv. positive definite) kernel represents an inner product on, and was hoping for some help in ...
• 349
25 views

### In Gaussian Process Regression, what kinds of information can you *not* put in the kernel as opposed to the mean?

For example, suppose you want to learn some structure for the mean and then you also have some kernel. Is is sometimes not possible to put most things in the kernel? For example, consider ...
• 312
I understand that kernel methods are used to exploit nonlinearity in a data set. For example, let $\mathbf{x} = \begin{bmatrix}x_1\\x_2 \end{bmatrix}$. We can define the feature map \$\phi(\mathbf{x}) =...