0
$\begingroup$

Ridge Regression can be expressed as $$\hat{y} = (\mathbf{X'X} + a\mathbf{I}_d)^{-1}\mathbf{X}x$$ where $\hat{y}$ is the predicted label, $\mathbf{I}_d$ the $d \times d$ identify matrix, $\mathbf{x}$ the object we're trying to find a label for, and $\mathbf{X}$ the $n \times d$ matrix of $n$ objects $\mathbf{x}_i = (x_{i,1}, ..., x_{i,d})\in \mathbb{R}^d$ such that:

$$ \mathbf{X} = \begin{pmatrix} x_{1,1} & x_{1,2} & \ldots & x_{1,d}\\ x_{2,1} & x_{2,2} & \ldots & x_{2,d}\\ \vdots & \vdots & \ddots & \vdots\\ x_{n,1} & x_{1,2} &\ldots & x_{n,d} \end{pmatrix} $$

We can kernelise this as follows: $$\hat{y} = (\mathbf{\mathcal{K}} + a\mathbf{I}_d)^{-1} \mathbf{k}$$

where $\mathbf{\mathcal{K}}$ is the $n \times n$ matrix of kernel functions $K$

$$\mathcal{K} = \begin{pmatrix} K(\mathbf{x}_1,\mathbf{x}_1) & K(\mathbf{x}_1,\mathbf{x}_2) & \ldots & K(\mathbf{x}_1,\mathbf{x}_n)\\ K(\mathbf{x}_2,\mathbf{x}_1) & K(\mathbf{x}_2,\mathbf{x}_2) & \ldots & K(\mathbf{x}_2,\mathbf{x}_n)\\ \vdots & \vdots & \ddots & \vdots\\ K(\mathbf{x}_n,\mathbf{x}_1) & K(\mathbf{x}_n,\mathbf{x}_2) &\ldots & K(\mathbf{x}_n,\mathbf{x}_n) \end{pmatrix} $$

and $\mathbf{k}$ the $n \times 1$ column vector of kernel functions $K$

$$\mathbf{k} = \begin{pmatrix} K(\mathbf{x}_1,\mathbf{x})\\ K(\mathbf{x}_2,\mathbf{x}) \\ \vdots \\ K(\mathbf{x}_n,\mathbf{x}) \end{pmatrix}$$

What about the case where the training data, $\textbf{x} \in \mathbb{R}^{40}$ and $\textbf{y}$ is the test data $\in \mathbb{R}^{10}$?

But the Kernel, $\mathcal{K}$ loses its symmetry matrix? Does this make sense? You take the dot product between training and test data and do not have a a square matrix.

$\endgroup$
5
  • 1
    $\begingroup$ It is not. The kernel must be symmetric to learn the coefficients, but the functional form of the prediction doesn't require a symmetric K $\endgroup$
    – Firebug
    May 14, 2021 at 17:06
  • $\begingroup$ If you had 40 training points, and 10 test points, how do you even define the kernel? $\endgroup$
    – user318514
    May 14, 2021 at 17:07
  • 1
    $\begingroup$ A 40x10 matrix. $\endgroup$
    – Firebug
    May 14, 2021 at 17:07
  • $\begingroup$ When you do gradient descent, the train and test sizes aren't equal, so you can't learn the coefficients, with unequal train and test sizes? $\endgroup$
    – user318514
    May 14, 2021 at 17:12
  • 2
    $\begingroup$ You learn coefficients using only the train data (so a 40x40 matrix in you example) $\endgroup$
    – Firebug
    May 14, 2021 at 17:25

1 Answer 1

1
$\begingroup$

During training, the kernel trick matrix is symmetric.

During prediction, the kernel trick matrix is a rectangle. # Columns in prediction kernel trick matrix equals test samples. # Rows equals samples during training.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.