Regression performance is often evaluated by means of cross-validation. However, classical cross-validation only regards the mean of the identified parameters. How can one quantify the quality of the uncertainty estimate, i.e., the validity of the estimated parameter covariance?
An illustrative example: ordinary least squares (OLS) assumes constant error covariance (homoskedasticity), and performing OLS on heteroskedastic data is known to yield unbiased parameter estimates but biased covariance estimates. One solution is to perform weighted OLS with weights $w_i = 1/\sigma_i$. Of course many more combinations of statistical properties of the data and different estimators are conceivable, all leading to different parameter covariance estimates. Thus the question arises in practice: how can I validate and compare the uncertainty estimates provided by different estimation procedures, in order to select the estimator with the most reliable uncertainty estimate?
Example code (Matlab): y = 0.1*x1 - 0.2*x2 + 1
rng(1); x1 = (0:.001:10)-5; x2 = detrend(x1-sin(x1)); % generate second, uncorrelated covariate sigma_eps = linspace(1, 10, length(x1)); % heteroskedastic error variance y = 0.1*x1 +--0.2*x2 + 1 + sigma_eps.*randn(size(x1)); ols = fitlm([x1; x2; ones(size(x1))]', y', 'Intercept', false) weights = 1./ sigma_eps; wls = fitlm([weights.*x1; weights.*x2; weights.*ones(size(x1))]', (weights.*y)', 'Intercept', false)
ols = Linear regression model: y ~ x1 + x2 + x3 Estimated Coefficients: Estimate SE tStat pValue _________ ________ ________ __________ x1 0.087713 0.021037 4.1694 3.0802e-05 x2 -0.076851 0.085888 -0.89478 0.37092 x3 0.95685 0.060736 15.754 2.9584e-55 ... wls = Linear regression model: y ~ x1 + x2 + x3 Estimated Coefficients: Estimate SE tStat pValue ________ ________ _______ __________ x1 0.081754 0.017311 4.7227 2.3587e-06 x2 -0.15835 0.059521 -2.6604 0.0078181 x3 0.95645 0.060051 15.927 2.021e-56
The true parameters are within 2*SE of the estimated parameters in all cases, hence nothing appears to be severely off. The aim of the question is to find a method to validate the SE estimates empirically.