I am working on some syntatic data for Error In Variable model for some research. Currently I have a single independent variable, and I am assuming I know the variance for the true value of the dependent variable.
So, with this information, I can achieve an unbiased estimator for the coefficient of the dependent variable.
The model:
$\tilde{x} = x + e_1$
$y = 0.5x -10 + e_2$
Where:
$e_1\text{~}N(0,\sigma^2)$ for some $\sigma$
$e_2\text{~}N(0,1)$
Where the values of $y,\tilde{x}$ are known for each sample only, and also the standard deviation of the real value of $x$ for the sample is known: $\sigma_x$.
I get the biased ($\hat{\beta}$) coefficient using OLS, and then making adjustments using:
$\beta' = \hat{\beta} * \frac{\hat{\sigma}_\tilde{x}^2}{\sigma_x^2}$
I see that my new, unbiased estimator for the coefficient is much better (closer to the real value) with this model, but the MSE is getting worse than using the biased estimator.
What is happening? I expected an ubiased estimator to yield better results than the biased one.
Matlab code:
reg_mse_agg = [];
fixed_mse_agg = [];
varMult = 1;
numTests = 60;
for dataNumber=1:8
reg_mses = [];
fixed_mses = [];
X = rand(1000,1);
X(:,1) = X(:,1) * 10;
X(:,1) = X(:,1) + 5;
varX = var(X);
y = 0.5 * X(:,1) -10;
y = y + normrnd(0,1,size(y));
origX = X;
X = X + normrnd(0,dataNumber * varMult ,size(X));
train_size = floor(0.5 * length(y));
for t=1:numTests,
idx = randperm(length(y));
train_idx = idx(1:train_size);
test_idx = idx(train_size+1:end);
Xtrain = X(train_idx,:);
ytrain = y(train_idx);
Xtest = X(test_idx,:);
ytest = y(test_idx);
b = OLS_solver(Xtrain, ytrain);
%first arg of evaluate returns MSE, working correctly.
[ reg_mse, ~ ] = evaluate( b,Xtest,ytest);
reg_mses = [reg_mses ; reg_mse];
varInd = var(Xtrain);
varNoise = varInd - varX;
bFixed = [0 0]';
bFixed(1) = b(1) * varInd / varX;
bFixed(2) = mean(ytrain - bFixed(1)*Xtrain);
[fixed_mse,~ ] = evaluate( bFixed,Xtest,ytest);
fixed_mses = [fixed_mses ; fixed_mse];
dataNumber * varMult
b
bFixed
end
reg_mse_agg = [reg_mse_agg , reg_mses];
fixed_mse_agg = [fixed_mse_agg , fixed_mses];
end
mean(reg_mse_agg)
mean(fixed_mse_agg)
Results:
biased estimator's MSE:
ans =
Columns 1 through 7
1.2171 1.6513 1.9989 2.3914 2.5766 2.6712 2.5997
Column 8
2.8346
Unbiased estimator's MSE:
ans =
Columns 1 through 7
1.2308 2.0001 2.9555 4.9727 7.6757 11.3106 14.4283
Column 8
11.5653
In addition, printing the values of b
and bFixed
- I see that bFixed
is indeed closer to the real values of 0.5,-10
than the biased estimator (as expected).
P.S. The results of the unbiased being worse than the biased estimator are statistical significant - the test for it is omitted from the code, since it is a simplification of the "full version" code.
UPDTAE: I added a test that checks $\sum_{\text{for each test}}{(\hat{\beta}-\beta)^2}$ and $\sum_{\text{for each test}}{(\beta'-\beta)^2}$, and the biased estimator is indeed significantly worse (larger value) than the unbiased one according to this metric, even though the MSE of the biased estimator (on test-set) is significantly better.
Where $\beta=0.5$ is the real coefficient of the dependent variable, $\hat{\beta}$ is the biased estimator for $\beta$, and $\beta'$ is the unbiased estimator for $\beta$.
This I believe shows that the reason for the results is NOT the higher variance of the unbiased estimator, as it is still closer to the real value.
Credit: Using Lecture notes of Steve Pischke as resource
b
andbFixed
, but explained what they show. $\endgroup$