Skip to main content
deleted 1 character in body
Source Link
kjetil b halvorsen
  • 82.8k
  • 32
  • 201
  • 663

It might help to specialise this back to linear regression, which has all the features you don't like, but is easier.

First, $M<N$

We have $N$ observations of two variables $y$ and $x$, which are both centred. The function $\phi: \mathbb{R}^N\to\mathbb{R}^N$ is the identity, and $M=1$

The penalised OLS loss is

$$L=\left(\sum_{i=1}^n (wx_i-y_i)^2\right) + \lambda w^2$$

The dual form is $w=\Phi^Ta$, where $\Phi_{ij}=(x_i)_j=x_i$. That is, $w=\sum_{i=1}^N x_ia_i$.

The gradient is zero when $$a= \frac{1}{\lambda}(y-\Phi\cdot w)$$ or $$ a_i= \frac{1}{\lambda}(y_i-x_i w)$ a_i= \frac{1}{\lambda}(y_i-x_i w)$. That is, the gradient is zero when the fitted values fall on a line, and the line has the optimal slope. The apparent freedom to specify the $N$$N$-vector $a$$a$ is illusory -- at the optimum it has to be in the column space of $\Phi$$\Phi$, which is only $M$$M$-dimensional

Now $M>N$:

We still have linear regression and $\phi$ as the identity, but we now have $M>N$ predictors $x$. Still, $\Phi_{ij}=x_{ij}$

The dual form is $w_j=\sum_{i=1}^N x_{ij}a_i$.

The gradient is zero when $$a= \frac{1}{\lambda}(y-\sum_{j=1}^M\Phi_{ij}w_j)$$

It's true that $w$ is of higher dimension than $a$, but $\Phi\cdot w$ isn't, and that's all we see. If $M>N$ there will be multiple $w$s that could give the same fitted values; the problem of finding $w$ is underdetermined. But that was true in the original least-squares formulation as well -- it's not a problem introduced by the dual formulation.

It might help to specialise this back to linear regression, which has all the features you don't like, but is easier.

First, $M<N$

We have $N$ observations of two variables $y$ and $x$, which are both centred. The function $\phi: \mathbb{R}^N\to\mathbb{R}^N$ is the identity, and $M=1$

The penalised OLS loss is

$$L=\left(\sum_{i=1}^n (wx_i-y_i)^2\right) + \lambda w^2$$

The dual form is $w=\Phi^Ta$, where $\Phi_{ij}=(x_i)_j=x_i$. That is, $w=\sum_{i=1}^N x_ia_i$.

The gradient is zero when $$a= \frac{1}{\lambda}(y-\Phi\cdot w)$$ or $$ a_i= \frac{1}{\lambda}(y_i-x_i w) That is, the gradient is zero when the fitted values fall on a line, and the line has the optimal slope. The apparent freedom to specify the $N$-vector $a$ is illusory -- at the optimum it has to be in the column space of $\Phi$, which is only $M$-dimensional

Now $M>N$:

We still have linear regression and $\phi$ as the identity, but we now have $M>N$ predictors $x$. Still, $\Phi_{ij}=x_{ij}$

The dual form is $w_j=\sum_{i=1}^N x_{ij}a_i$.

The gradient is zero when $$a= \frac{1}{\lambda}(y-\sum_{j=1}^M\Phi_{ij}w_j)$$

It's true that $w$ is of higher dimension than $a$, but $\Phi\cdot w$ isn't, and that's all we see. If $M>N$ there will be multiple $w$s that could give the same fitted values; the problem of finding $w$ is underdetermined. But that was true in the original least-squares formulation as well -- it's not a problem introduced by the dual formulation.

It might help to specialise this back to linear regression, which has all the features you don't like, but is easier.

First, $M<N$

We have $N$ observations of two variables $y$ and $x$, which are both centred. The function $\phi: \mathbb{R}^N\to\mathbb{R}^N$ is the identity, and $M=1$

The penalised OLS loss is

$$L=\left(\sum_{i=1}^n (wx_i-y_i)^2\right) + \lambda w^2$$

The dual form is $w=\Phi^Ta$, where $\Phi_{ij}=(x_i)_j=x_i$. That is, $w=\sum_{i=1}^N x_ia_i$.

The gradient is zero when $$a= \frac{1}{\lambda}(y-\Phi\cdot w)$$ or $ a_i= \frac{1}{\lambda}(y_i-x_i w)$. That is, the gradient is zero when the fitted values fall on a line, and the line has the optimal slope. The apparent freedom to specify the $N$-vector $a$ is illusory -- at the optimum it has to be in the column space of $\Phi$, which is only $M$-dimensional

Now $M>N$:

We still have linear regression and $\phi$ as the identity, but we now have $M>N$ predictors $x$. Still, $\Phi_{ij}=x_{ij}$

The dual form is $w_j=\sum_{i=1}^N x_{ij}a_i$.

The gradient is zero when $$a= \frac{1}{\lambda}(y-\sum_{j=1}^M\Phi_{ij}w_j)$$

It's true that $w$ is of higher dimension than $a$, but $\Phi\cdot w$ isn't, and that's all we see. If $M>N$ there will be multiple $w$s that could give the same fitted values; the problem of finding $w$ is underdetermined. But that was true in the original least-squares formulation as well -- it's not a problem introduced by the dual formulation.

Source Link
Thomas Lumley
  • 46.8k
  • 1
  • 59
  • 168

It might help to specialise this back to linear regression, which has all the features you don't like, but is easier.

First, $M<N$

We have $N$ observations of two variables $y$ and $x$, which are both centred. The function $\phi: \mathbb{R}^N\to\mathbb{R}^N$ is the identity, and $M=1$

The penalised OLS loss is

$$L=\left(\sum_{i=1}^n (wx_i-y_i)^2\right) + \lambda w^2$$

The dual form is $w=\Phi^Ta$, where $\Phi_{ij}=(x_i)_j=x_i$. That is, $w=\sum_{i=1}^N x_ia_i$.

The gradient is zero when $$a= \frac{1}{\lambda}(y-\Phi\cdot w)$$ or $$ a_i= \frac{1}{\lambda}(y_i-x_i w) That is, the gradient is zero when the fitted values fall on a line, and the line has the optimal slope. The apparent freedom to specify the $N$-vector $a$ is illusory -- at the optimum it has to be in the column space of $\Phi$, which is only $M$-dimensional

Now $M>N$:

We still have linear regression and $\phi$ as the identity, but we now have $M>N$ predictors $x$. Still, $\Phi_{ij}=x_{ij}$

The dual form is $w_j=\sum_{i=1}^N x_{ij}a_i$.

The gradient is zero when $$a= \frac{1}{\lambda}(y-\sum_{j=1}^M\Phi_{ij}w_j)$$

It's true that $w$ is of higher dimension than $a$, but $\Phi\cdot w$ isn't, and that's all we see. If $M>N$ there will be multiple $w$s that could give the same fitted values; the problem of finding $w$ is underdetermined. But that was true in the original least-squares formulation as well -- it's not a problem introduced by the dual formulation.