1
$\begingroup$

I'm trying to understand why centering myregressor at a nonzero value improves the performance.

I always thought that the variance of the feature, and its correlation with the labels, would affect performance. But I cannot imagine why shifting the mean would have such an affect on performance? please see the example below: (change the seed in rng(2222) if desired).

In MATLAB I have the following code:

clear all;
clc;
rng(22222);

%%
N=10000;
cenetered_nonzero_errors=zeros(N,1);
centered_zero_errors=zeros(N,1);
for k=1:N
    num_x = 3;
    num_y = 8;
    A =[ 1 4 0
    2 0 1
    -2 -2 3
    -1 1 -4
    -3 1 1
    0 -2 2
    3 2 3
    0 -4 -6 ]; 
    x = [-8; 20; 5];
    y = A*x;
    for i = 1:num_y
        y(i) = y(i)+randn;
    end;
    x_ls = A\y;

    A_nonzero = [A ones(num_y,1)+randn(num_y,1)];
    x_ls_nonzero = A_nonzero\y;
    norm(x-x_ls);
    norm(x-x_ls_nonzero(1:3));
    cenetered_nonzero_errors(k)=norm(x-x_ls_nonzero(1:3));

    A_zero = [A randn(num_y,1)];
    x_ls_zero = A_zero\y;

    norm(x-x_ls_zero(1:3));
    centered_zero_errors(k)=norm(x-x_ls_zero(1:3));
end

%%
display('centered nonzero')
mean(cenetered_nonzero_errors)
display('centered zero')
mean(centered_zero_errors)

The output is the following:

centered nonzero

ans =

    0.2664

centered zero

ans =

    0.2882

As you can see, the performance is better (on average) with the first one, where the data is centered at zero. The performance is worse with the data not centered at zero. The error is measured by the norm of the difference between the true x and the approximate x computed by least squares.

Why does centering the regressor at a nonzero value improve performance?

Related: How to interpret whether centering improves linear regression model or not?

$\endgroup$
2
  • 2
    $\begingroup$ Please, show, in addition to your code, the results and possibly plots, to highlight the problem. (Do you think everybody uses matlab to understand what you did?) $\endgroup$
    – ttnphns
    Sep 21, 2018 at 17:25
  • $\begingroup$ If necessary, give the data itself, too. $\endgroup$
    – ttnphns
    Sep 21, 2018 at 17:26

0