3
$\begingroup$

In linear regression modelling, we obtain the standard error of estimated coefficients, and we use them to draw inferences about the corresponding population-level coefficients, which are unknown. Besides this important role of standard errors, I was wondering what information they convey about the predictor's variability and its explanatory power. Although we assume predictor variables are fixed (not random), they practically exhibit variability, and sometimes we tend to prefer (for variable selection) more variable predictors than those with low variability. For this reason, I ran a simple simulation to correlate the variability of predictors and the standard error of their estimated coefficient. Please find the codes below to reproduce my results.

set(1345)
n <- 500 # sample size
p <- 30  # number of predictors (all continuous)
range.x <- 1:p  # The range of variables in uniform distribution


# simulating predictors
X <- sapply(1:p, function(j){
  runif(n, 0, j)
})

bet <- c(0, runif(p, 0.25, 0.3)) # true regression coefficients

# simulate the response (you know what I'm doing)
Y <- cbind(1, X) %*% as.matrix(bet) + rnorm(n, 0, 1)

fit.lm <- lm(Y~X) 
se.coef <- diag(vcov(fit.lm))[-1]


plot(range.x, se.coef, pch=19) 

In the simulation procedure, I simulated each predictor from a uniform distribution between 0 and p, where p id the index predictor p (p=,1, 2, ..., 30). That means each predictor has a different range, hence different variability between predictors. The rest of the simulation is very simple and does not require explanation. If not, please let me know.

enter image description here

The relationship looks fascinating, i.e. lower standard error means large variability of the predictor. And I wonder if I can interpret it as such? That is, a low standard error of estimated coefficients translates to a high explanatory power of the corresponding predictor for the response variable.

$\endgroup$

3 Answers 3

1
$\begingroup$

This is an extension of what is seen in single-predictor OLS. Under the assumption of normal error terms:

the estimator of the slope coefficient will itself be normally distributed with mean $\beta$ and variance $\sigma^2/\sum(x_i-\bar x)^2$, where $\sigma^2$ is the variance of the error terms

Other things being equal, a higher variance in $x$ values gives more precise estimates of the slope $\beta$. This makes some intuitive sense, as if you only have a narrow distribution of $x$ values you have a correspondingly narrow distribution of $y$ values and thus less information to get a good estimate of the slope, their ratio after subtracting their mean values.

$\endgroup$
0
$\begingroup$

If you take the R2 as a measure of predictive ability, this is indeed the case.

For a simple linear model $y_i = X_i \beta + e_i$, the standard error can be estimated as

$$ \hat{\sigma} = \frac{1}{N} \sum_{i=1}^{N} \hat{e}_i^2 $$

where $\hat{e}_i = y_i - X_i \hat{\beta}$ with $\hat{\beta}$ the OLS estimate.

The R2 is now defined as

$$ R^2 = 1 - \frac{\sum_{i} \hat{e}_i^2}{\sum_{i} (y_i - \bar{y})^2} $$

where $\bar{y}$ is the mean of all $y_i$.

So if the standard error of the OLS estimate $\hat{e}_i^2$ increases, R2 decreases. If the standard error decreases, the predictive power increases.

$\endgroup$
1
  • $\begingroup$ That is a good argument but I think you are relating the mean-square error to the predictive power of predictors. However, my question is about the relationship between the standard error of the estimated coefficient to the explanatory power of the corresponding predictor. I agree the standard error of the coefficients is also a function of the mean square error, but I need an explicit association. $\endgroup$
    – Alemu
    Dec 23, 2021 at 14:10
0
$\begingroup$

The asymptotic variance of $\hat\beta$ is $$ \sigma^2[E(X_tX_t')]^{-1} $$ Since your predictors are independent, this is a diagonal matrix with elements inverse to $$ E(X^2_{t,j})=Var(X_{t,j})+E(X_{t,j})^2 $$ By properties of the uniform, we have, for your DGP $$ E(X^2_{t,j})=j^2/12+j/2, $$ such that the standard errors indeed decrease in $j$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.