Skip to main content
clarification of bias regularisation
Source Link
seanv507
  • 7.3k
  • 1
  • 23
  • 32

$f(b,c):=\sum_i^m(y_i-c-b^Tx_i)^2+\lambda b^T b$ is equivalent to $g(d,e):=\sum_i^m(y_i-e-d^T (x_i-\bar x))^2+\lambda d^T d$ under the change of variables $d=b,e=c+b^T \bar x$

ie $f(b,c)=g(b,c+b^T\bar x)$.

Therefore they have the same minimisers [same constraints on (b,c) vs (d,e)]. But this change of variables corresponds to using centred or uncentred data.

It should be noted that this only works when the regularisation is not on the constant term. Although regularisation is typically performed as above, some software also penalises the constant/bias term.

$f(b,c):=\sum_i^m(y_i-c-b^Tx_i)^2+\lambda b^T b$ is equivalent to $g(d,e):=\sum_i^m(y_i-e-d^T (x_i-\bar x))^2+\lambda d^T d$ under the change of variables $d=b,e=c+b^T \bar x$

ie $f(b,c)=g(b,c+b^T\bar x)$.

Therefore they have the same minimisers [same constraints on (b,c) vs (d,e)]. But this change of variables corresponds to using centred or uncentred data.

$f(b,c):=\sum_i^m(y_i-c-b^Tx_i)^2+\lambda b^T b$ is equivalent to $g(d,e):=\sum_i^m(y_i-e-d^T (x_i-\bar x))^2+\lambda d^T d$ under the change of variables $d=b,e=c+b^T \bar x$

ie $f(b,c)=g(b,c+b^T\bar x)$.

Therefore they have the same minimisers [same constraints on (b,c) vs (d,e)]. But this change of variables corresponds to using centred or uncentred data.

It should be noted that this only works when the regularisation is not on the constant term. Although regularisation is typically performed as above, some software also penalises the constant/bias term.

clarified answer
Source Link
seanv507
  • 7.3k
  • 1
  • 23
  • 32

To prove b' =b'' you take any solution of the first problem and show that it$f(b,c):=\sum_i^m(y_i-c-b^Tx_i)^2+\lambda b^T b$ is also minimisingequivalent to $g(d,e):=\sum_i^m(y_i-e-d^T (x_i-\bar x))^2+\lambda d^T d$ under the second problemchange of variables $d=b,e=c+b^T \bar x$

ie $f(b,c)=g(b,c+b^T\bar x)$.

Therefore they have the same minimisers [same constraints on (b'b, c''c) and vice versavs (d,e)]. By adding the bt xbarBut this change of variables corresponds to your cusing centred or uncentred data.

To prove b' =b'' you take any solution of the first problem and show that it is also minimising the second problem (b', c'') and vice versa. By adding the bt xbar to your c

$f(b,c):=\sum_i^m(y_i-c-b^Tx_i)^2+\lambda b^T b$ is equivalent to $g(d,e):=\sum_i^m(y_i-e-d^T (x_i-\bar x))^2+\lambda d^T d$ under the change of variables $d=b,e=c+b^T \bar x$

ie $f(b,c)=g(b,c+b^T\bar x)$.

Therefore they have the same minimisers [same constraints on (b,c) vs (d,e)]. But this change of variables corresponds to using centred or uncentred data.

Source Link
seanv507
  • 7.3k
  • 1
  • 23
  • 32

To prove b' =b'' you take any solution of the first problem and show that it is also minimising the second problem (b', c'') and vice versa. By adding the bt xbar to your c