1
$\begingroup$

For a linear regression fit for a problem with p variables X_i ranging between 0 and 1, where p>20 (I don't know if that is relevant or not), and the number of samples is about 1000, I wanted to estimate the variance contribution for each of the variables using the regression coefficients. If I understood correctly var(A*X) = A^2*var(X), and therefore I thought that taking the square of the regression coefficients and multiplying that with the variance for each of the variables should give a vector containing all the variance contributions of the different variables.

The problem is that I expected the sum of those variance contributions to be equal to the sum of the total regression model variance, but it isn't. The sum of the variances is some times up to 50% larger, then the total variance of the regression model.

here some Matlab like pseudo code to explain the problem more in detail.

X    %sample matrix    
Y    %output sample matrix 

 Linmodel=polyfitn(X,Y)   %fit a model    
 for ii=1:nr_colsX
      VARCONT(ii)=var(Linmodel.COEF(ii)*X(:,ii))) %variance of the contributions    
      VARRC(ii)=(Linmodel.COEF(ii)).^2*var(X(:,ii))  %variance based on Reg. Coef.    
 end

 SVC=sum(VARCONT(ii))  %sum of the variance contributions    
 SVSRC=sum(VARRC)    
 VY=var(Y)        %sum of the variance of the samples    
 VYmod=var(polyvaln(Linmodel,X))    %sum of the variance of the model on the samples    
 XR=rand(100000,nr_colsX)     %sum of the variance of the model with large number of samples    
 VYmodR=var(polyvaln(Linmodel,XR))

for one of the models that are supposed to be almost linear, VY is almost equal to VYmod. but SVC is about 50% largen then that. and VYmodR comes more in the direction of SVC.

1) Could some body please explain me why the sum of the variance contributions from the Regression coefficients, can be quite a bit larger then the variance of the regression model?

2) If this is so as it seems to be the case should there then not be some sort of upper bound for the sum of the square of the regression coefficients, such that the sum of their squares, multiplied by the variances of the input, should not be larger then the total variance of the output? Because it seems strange to me that the output of an interpolation model could result in a larger variance, then the variance of the data output points used for the interpolation.

Any help is highly appreciated, but any help that is written in a way, such that also just a silly engineer as me can understand it is appreciated even more.

$\endgroup$
2
$\begingroup$

Let $X_1$ and $X_2$ denote two random variables with variances $\sigma^2_{X1}$ and $\sigma^2_{X2}$. Let $Y = X_1 + X_2$. The variance of $Y$ is then equal to $\sigma^2_{X1} + \sigma^2_{X2} + \rho \sigma_{X1} \sigma_{X2}$, where $\rho$ is the correlation between $X_1$ and $X_2$. So, unless the two variables are uncorrelated, the sum of the variances will not be equal to variance of $Y$. Depending on the sign of $\rho$, the actual variance of $Y$ may be larger or smaller than the sum of $\sigma^2_{X1}$ and $\sigma^2_{X2}$.

$\endgroup$
  • $\begingroup$ Thank you for your fast reply. The input variable matrix X was created using a Sobol lptau sequence generator, link, however also with other pseudo random number generators, the problem remains. I understand that there could be a little bit correlation between my X1 X2...X50 because no random number generator is perfect. But is it really feasible that the final sum is over 50% off just because of such correlations? $\endgroup$ – Sarmes Dec 7 '13 at 16:13
  • $\begingroup$ If $n$ is large, then this is probably not the real source of the issue. Let me see if I understand you correctly. You are computing $\sum b_j^2 Var(x_j)$ and find that it does not equal $Var(y)$, correct? Even if all of the X's were perfectly uncorrelated, then this is not surprising. You need to compute $\sum b_j^2 Var(x_j) + Var(e)$ (i.e., you also need to add the variance of the residuals) if you want to get $Var(y)$. $\endgroup$ – Wolfgang Dec 7 '13 at 17:24
  • $\begingroup$ Well i would like to compare $SUM[(b_j)^2*Var(x_j)]$, with both Var(y), but especially with Var(Ymod) where i mean with Ymod The results that come from the linear fit, when i use it on the original samples. The difference do to the Var(e) term i anticipated, so i expected $SUM[(b_j)^2Var(x_j)]$ to be smaller, but never bigger then Var(Y). and i especially did not expect it to be larger then Var(Ymod). $\endgroup$ – Sarmes Dec 7 '13 at 17:36
  • $\begingroup$ I don't understand what you mean by 'Ymod' and 'the results that come from the linear fit, when i use it on the original samples.' Can you actually write down how 'Ymod' is computed? $\endgroup$ – Wolfgang Dec 7 '13 at 17:48
  • $\begingroup$ Y are the output values of the experiment. With Y_mod i mean the predicted values for Y by the linear regression. $Ymod = r_0 + \sum_{i=1}^n r_i*X_i$ where i mean with r the regression coefficients, and with X_i (column) vectors of sample matrix X where the rows are the different samples, but each column would be a collection of all the values of the first variable from the set of samples. $\endgroup$ – Sarmes Dec 7 '13 at 18:05
2
$\begingroup$

Assuming that the regressors are deterministic, and so they do not have a variance of their own (or alternatively that we are considering the variance of the fitted dependent variable conditional on the specific realizations of the regressors that comprise our sample of observations), we have (for a sample of $n$ observations and for $k+1$ regressors including the constant)

$$\hat y_i\mid \mathbf x_i = \hat b_0 + \hat b_1x_{1i} + ... \hat b_kx_{ki},\qquad i=1,...,n$$ and so

$$\begin{align}\operatorname {Var}(\hat y_i\mid \mathbf x_i) =& \operatorname {Var}(\hat b_0)+\operatorname {Var}(\hat b_1)x_{1i}^2+...+\operatorname {Var}(\hat b_1)x_{ki}^2 +...\\ +&2x_{1i}\operatorname {Cov}(\hat b_0, \hat b_1)+2x_{2i}\operatorname {Cov}(\hat b_0, \hat b_2)+...+2x_k\operatorname {Cov}(\hat b_0, \hat b_k) \\ +&2x_{1i}x_{2i}\operatorname {Cov}(\hat b_1, \hat b_2)+....+2x_{1i}x_{ki}\operatorname {Cov}(\hat b_1, \hat b_k)\\ +&...+ 2x_{k-1}x_k\operatorname {Cov}(\hat b_{k-1}, \hat b_k) \end{align} $$

In other words, we need to take into account the covariances between the estimated coefficients, covariances that in general are not zero, because the estimated coefficients are functions of the same data (and even if the $x$'s are deterministic, "inside" each coefficient estimate lurks the same stochastic error term). Note that some covariances may be positive while others negative.

Obviously the above magnitude is different for each observation $i$, which is only natural since it depends on the specific values of the $x_i$'s. One can then average over the whole sample, i.e. consider $$\hat E[\operatorname {Var}(\hat y_i\mid \mathbf x_i)] = \frac 1n \sum_{i=1}^n\operatorname {Var}(\hat y_i \mid \mathbf x_i)$$ while keeping in mind the variance-decomposition formula, which in general reads $$\operatorname {Var}(Y) = E[\operatorname {Var}(Y\mid X)] + \operatorname {Var}[E(Y\mid X)]$$

$\endgroup$
  • $\begingroup$ Could you explain your notation a bit more to me. In your notation the roof on the b's probably means vector, right? Is my case then not different because the regression coefficients are scalars, and my x1,x2...xn are vectors, or do i misunderstand you? or do you denote the regression coefficients as x..? but then b0 is difficult to understand. $\endgroup$ – Sarmes Dec 7 '13 at 17:12
  • $\begingroup$ The "roof" indicates that it is the estimated coefficient not the "true" unknown value. So the $\hat b$'s are scalars.I have made some changes in my answer I hope it is clear to you now. $\endgroup$ – Alecos Papadopoulos Dec 7 '13 at 19:04
  • $\begingroup$ Thanks for the clarification. comparing your answer and wolfgangs answer, i still dont understand the idea behind $Cov(b_i,b_k)$ if the b are scalar values. From Wolfgans answere i get the impression that i could exactly calculate the correlation or covariance terms, because i know the original X's. how are your $2x_i*x_jCov(b_i,b_j)$ terms related to his ρσX1σX2 terms? $\endgroup$ – Sarmes Dec 7 '13 at 19:16
  • $\begingroup$ The $\hat b_i$'s may be scalar values, but as I wrote, they are functions, not constants (constants are the unknown values that we are trying to estimate through the estimators): $\hat b_i = h_i(y_1,...,y_n, x_{11},...x_{1n},...,x_{k1},...,x_{kn})$ Now, all these (or at least the $y_i$'s are realizations of random variables, hence $h_i()$ is a function of random variables and so a random variable itself, and so the concept of covariance $Cov(h_i,h_j)$ is meaningful. $\endgroup$ – Alecos Papadopoulos Dec 7 '13 at 19:25
  • $\begingroup$ Thanks for your answer. I am still not sure how to calculate the covariance for such scalars that are functions $Cov(b_j,b_i)$. For the example of Wolfgang i understand that var(Y)=the total sum of $cov(X_i,X_j)$. But there $Y=X_1+X_2$, now it would be $Y^=b_1*X_1+b_2*X_2$ Would it be then correct to say that Var(Y^)=the total sum over $cov(b_i*X_,i,b_j*X_j))$? Such that the "$xi∗xjCov(bi,bj)$" and similar terms could be calculated by doing $Cov(b_i*X_i,b_j*X_j)$? were X_1 and X_2 are column vectors, of the sample matrix X were each row is a sample. $\endgroup$ – Sarmes Dec 9 '13 at 14:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.