Return to Answer

In the regression model:

$$y_i=\mathbf{x}_i'\beta+u_i$$

the usual assumption is that $(y_i,\mathbf{x}_i,u_i)$, $i=1,...,n$ is an iid sample. Under assumptions that $E\mathbf{x}_iu_i=0$ and $E(\mathbf{x}_i\mathbf{x}_i')$ has full rank, the ordinary least squares estimator:

$$\widehat{\beta}=\left(\sum_{i=1}^n\mathbf{x}_i\mathbf{x}_i'\right)^{-1}\sum_{i=1}\mathbf{x}_iy_i$$

is consistent and asymptotically normal. The expected covariance between a residual and the response variable then is:

$$Ey_iu_i=E(\mathbf{x}_i'\beta+u_i)u_i=Eu_i^2$$

If we furthermore assume that $E(u_i|\mathbf{x}_1,...,\mathbf{x}_n)=0$ and $E(u_i|\mathbf{x}_1,...,\mathbf{x}_n)=\sigma^2$, we can calculate the expected covariance between $y_i$ and its regression residual:

$$\begin{align*} Ey_i\widehat{u}_i&=Ey_i(y_i-\mathbf{x}_i'\widehat{\beta})\\\\ &=E(\mathbf{x}_i'\beta+u_i)(u_i-\mathbf{x}_i(\widehat{\beta}-\beta))\\\\ &=E(u_i^2)\left(1-E\mathbf{x}_i' \left(\sum_{j=1}^n\mathbf{x}_j\mathbf{x}_j'\right)^{-1}\mathbf{x}_i\right) \end{align*}$$

Now to get the correlation we need to calculate $\text{Var}(y_i)$ and $\text{Var}(\hat{u}_i)$. It turns out that

$$\text{Var}(\hat u_i)=E(y_i\hat{u}_i),$$

hence

$$\text{Corr}(y_i,\hat u_i)=\sqrt{1-E\mathbf{x}_i' \left(\sum_{j=1}^n\mathbf{x}_j\mathbf{x}_j'\right)^{-1}\mathbf{x}_i}$$

Now the term $\mathbf{x}_i' \left(\sum_{j=1}^n\mathbf{x}_j\mathbf{x}_j'\right)^{-1}\mathbf{x}_i$ comes from diagonal of the hat matrix $H=X(X'X)^{-1}X'$, where $X=[\mathbf{x}_i,...,\mathbf{x}_N]'$. The matrix $H$ is idempotent, hence it satisfies a following property

$$\text{trace}(H)=\sum_{i}h_{ii}=\text{rank}(H),$$

where $h_{ii}$ is the diagonal term of $H$. The $\text{rank}(H)$ is the number of linearly independent variables in $\mathbf{x}_i$, which is usually the number of variables. Let us call it $p$. The number of $h_{ii}$ is the sample size $N$. So we have $N$ nonnegative terms which should sum up to $p$. Usually $N$ is much bigger than $p$, hence a lot of $h_{ii}$ would be close to the zero, meaning that the correlation between the residual and the response variable would be close to 1 for the bigger part of observations.

The term $h_{ii}$ is also used in various regression diagnostics for determining influential observations.

In the regression model:

$$y_i=\mathbf{x}_i'\beta+u_i$$

the usual assumption is that $(y_i,\mathbf{x}_i,u_i)$, $i=1,...,n$ is an iid sample. Under assumptions that $E\mathbf{x}_iu_i=0$ and $E(\mathbf{x}_i\mathbf{x}_i')$ has full rank, the ordinary least squares estimator:

$$\widehat{\beta}=\left(\sum_{i=1}^n\mathbf{x}_i\mathbf{x}_i'\right)^{-1}\sum_{i=1}\mathbf{x}_iy_i$$

is consistent and asymptotically normal. The expected covariance between a residual and the response variable then is:

$$Ey_iu_i=E(\mathbf{x}_i'\beta+u_i)u_i=Eu_i^2$$

If we furthermore assume that $E(u_i|\mathbf{x}_1,...,\mathbf{x}_n)=0$ and $E(u_i^2|\mathbf{x}_1,...,\mathbf{x}_n)=\sigma^2$, we can calculate the expected covariance between $y_i$ and its regression residual:

$$\begin{align*} Ey_i\widehat{u}_i&=Ey_i(y_i-\mathbf{x}_i'\widehat{\beta})\\\\ &=E(\mathbf{x}_i'\beta+u_i)(u_i-\mathbf{x}_i(\widehat{\beta}-\beta))\\\\ &=E(u_i^2)\left(1-E\mathbf{x}_i' \left(\sum_{j=1}^n\mathbf{x}_j\mathbf{x}_j'\right)^{-1}\mathbf{x}_i\right) \end{align*}$$

Now to get the correlation we need to calculate $\text{Var}(y_i)$ and $\text{Var}(\hat{u}_i)$. It turns out that

$$\text{Var}(\hat u_i)=E(y_i\hat{u}_i),$$

hence

$$\text{Corr}(y_i,\hat u_i)=\sqrt{1-E\mathbf{x}_i' \left(\sum_{j=1}^n\mathbf{x}_j\mathbf{x}_j'\right)^{-1}\mathbf{x}_i}$$

Now the term $\mathbf{x}_i' \left(\sum_{j=1}^n\mathbf{x}_j\mathbf{x}_j'\right)^{-1}\mathbf{x}_i$ comes from diagonal of the hat matrix $H=X(X'X)^{-1}X'$, where $X=[\mathbf{x}_i,...,\mathbf{x}_N]'$. The matrix $H$ is idempotent, hence it satisfies a following property

$$\text{trace}(H)=\sum_{i}h_{ii}=\text{rank}(H),$$

where $h_{ii}$ is the diagonal term of $H$. The $\text{rank}(H)$ is the number of linearly independent variables in $\mathbf{x}_i$, which is usually the number of variables. Let us call it $p$. The number of $h_{ii}$ is the sample size $N$. So we have $N$ nonnegative terms which should sum up to $p$. Usually $N$ is much bigger than $p$, hence a lot of $h_{ii}$ would be close to the zero, meaning that the correlation between the residual and the response variable would be close to 1 for the bigger part of observations.

The term $h_{ii}$ is also used in various regression diagnostics for determining influential observations.

In the regression model:

$$y_i=\mathbf{x}_i'\beta+u_i$$

the usual assumption is that $(y_i,\mathbf{x}_i,u_i)$, $i=1,...,n$ is an iid sample. Under assumptions that $E\mathbf{x}_iu_i=0$ and $E(\mathbf{x}_i\mathbf{x}_i')$ has full rank, the ordinary least squares estimator:

$$\widehat{\beta}=\left(\sum_{i=1}^n\mathbf{x}_i\mathbf{x}_i'\right)^{-1}\sum_{i=1}\mathbf{x}_iy_i$$

is consistent and asymptotically normal. The expected covariance between a residual and the response variable then is:

$$Ey_iu_i=E(\mathbf{x}_i'\beta+u_i)u_i=Eu_i^2$$

If we furthermore assume that $E(u_i|\mathbf{x}_1,...,\mathbf{x}_n)=0$ and $E(u_i|\mathbf{x}_1,...,\mathbf{x}_n)=\sigma^2$, we can calculate the expected covariance between $y_i$ and its regression residual:

$$\begin{align*} Ey_i\widehat{u}_i&=Ey_i(y_i-\mathbf{x}_i'\widehat{\beta})\\\\ &=E(\mathbf{x}_i'\beta+u_i)(u_i-\mathbf{x}_i(\widehat{\beta}-\beta))\\\\ &=E(u_i^2)\left(1-E\mathbf{x}_i' \left(\sum_{j=1}^n\mathbf{x}_j\mathbf{x}_j'\right)^{-1}\mathbf{x}_i\right) \end{align*}$$

Now to get the correlation we need to calculate $Var(y_i)$ and $Var(\hat{u}_i)$. It turns out that

$$Var(\hat u_i)=E(y_i\hat{u}_i),$$

hence

$$Corr(y_i,\hat u_i)=\sqrt{1-E\mathbf{x}_i' \left(\sum_{j=1}^n\mathbf{x}_j\mathbf{x}_j'\right)^{-1}\mathbf{x}_i}$$

Now the term $\mathbf{x}_i' \left(\sum_{j=1}^n\mathbf{x}_j\mathbf{x}_j'\right)^{-1}\mathbf{x}_i$ comes from diagonal of the hat matrix $H=X(X'X)^{-1}X'$, where $X=[\mathbf{x}_i,...,\mathbf{x}_N]'$. The matrix $H$ is idempotent, hence it satisfies a following property

$$trace(H)=\sum_{i}h_{ii}=rank(H),$$

where $h_{ii}$ is the diagonal term of $H$. The $rank(H)$ is the number of linearly independent variables in $\mathbf{x}_i$, which is usually the number of variables. Let us call it $p$. The number of $h_{ii}$ is the sample size $N$. So we have $N$ nonnegative terms which should sum up to $p$. Usually $N$ is much bigger than $p$, hence a lot of $h_{ii}$ would be close to the zero, meaning that the correlation between the residual and the response variable would be close to 1 for the bigger part of observations.

The term $h_{ii}$ is also used in various regression diagnostics for determining influential observations.

In the regression model:

$$y_i=\mathbf{x}_i'\beta+u_i$$

the usual assumption is that $(y_i,\mathbf{x}_i,u_i)$, $i=1,...,n$ is an iid sample. Under assumptions that $E\mathbf{x}_iu_i=0$ and $E(\mathbf{x}_i\mathbf{x}_i')$ has full rank, the ordinary least squares estimator:

$$\widehat{\beta}=\left(\sum_{i=1}^n\mathbf{x}_i\mathbf{x}_i'\right)^{-1}\sum_{i=1}\mathbf{x}_iy_i$$

is consistent and asymptotically normal. The expected covariance between a residual and the response variable then is:

$$Ey_iu_i=E(\mathbf{x}_i'\beta+u_i)u_i=Eu_i^2$$

If we furthermore assume that $E(u_i|\mathbf{x}_1,...,\mathbf{x}_n)=0$ and $E(u_i|\mathbf{x}_1,...,\mathbf{x}_n)=\sigma^2$, we can calculate the expected covariance between $y_i$ and its regression residual:

$$\begin{align*} Ey_i\widehat{u}_i&=Ey_i(y_i-\mathbf{x}_i'\widehat{\beta})\\\\ &=E(\mathbf{x}_i'\beta+u_i)(u_i-\mathbf{x}_i(\widehat{\beta}-\beta))\\\\ &=E(u_i^2)\left(1-E\mathbf{x}_i' \left(\sum_{j=1}^n\mathbf{x}_j\mathbf{x}_j'\right)^{-1}\mathbf{x}_i\right) \end{align*}$$

Now to get the correlation we need to calculate $\text{Var}(y_i)$ and $\text{Var}(\hat{u}_i)$. It turns out that

$$\text{Var}(\hat u_i)=E(y_i\hat{u}_i),$$

hence

$$\text{Corr}(y_i,\hat u_i)=\sqrt{1-E\mathbf{x}_i' \left(\sum_{j=1}^n\mathbf{x}_j\mathbf{x}_j'\right)^{-1}\mathbf{x}_i}$$

Now the term $\mathbf{x}_i' \left(\sum_{j=1}^n\mathbf{x}_j\mathbf{x}_j'\right)^{-1}\mathbf{x}_i$ comes from diagonal of the hat matrix $H=X(X'X)^{-1}X'$, where $X=[\mathbf{x}_i,...,\mathbf{x}_N]'$. The matrix $H$ is idempotent, hence it satisfies a following property

$$\text{trace}(H)=\sum_{i}h_{ii}=\text{rank}(H),$$

where $h_{ii}$ is the diagonal term of $H$. The $\text{rank}(H)$ is the number of linearly independent variables in $\mathbf{x}_i$, which is usually the number of variables. Let us call it $p$. The number of $h_{ii}$ is the sample size $N$. So we have $N$ nonnegative terms which should sum up to $p$. Usually $N$ is much bigger than $p$, hence a lot of $h_{ii}$ would be close to the zero, meaning that the correlation between the residual and the response variable would be close to 1 for the bigger part of observations.

The term $h_{ii}$ is also used in various regression diagnostics for determining influential observations.

In the regression model:

$$y_i=\mathbf{x}_i'\beta+u_i$$

the usual assumption is that $(y_i,\mathbf{x}_i,u_i)$, $i=1,...,n$ is an iid sample. Under assumptions that $E\mathbf{x}_iu_i=0$ and $E(\mathbf{x}_i\mathbf{x}_i')$ has full rank, the ordinary least squares estimator:

$$\widehat{\beta}=\left(\sum_{i=1}^n\mathbf{x}_i\mathbf{x}_i'\right)^{-1}\sum_{i=1}\mathbf{x}_iy_i$$

is consistent and asymptotically normal. The expected covariance between a residual and the response variable then is:

$$Ey_iu_i=E(\mathbf{x}_i'\beta+u_i)u_i=Eu_i^2$$

If we furthermore assume that $E(u_i|\mathbf{x}_1,...,\mathbf{x}_n)=0$, we can calculate the expected covariance between $y_i$ and its regression residual:

$$\begin{align*} Ey_i\widehat{u}_i&=Ey_i(y_i-\mathbf{x}_i'\widehat{\beta})\\\\ &=E(\mathbf{x}_i'\beta+u_i)(u_i-\mathbf{x}_i(\widehat{\beta}-\beta))\\\\ &=E(u_i^2)\left(1-\mathbf{x}_i \left(\sum_{j=1}^n\mathbf{x}_j\mathbf{x}_j'\right)^{-1}\mathbf{x}_i\right) \end{align*}$$

In the regression model:

$$y_i=\mathbf{x}_i'\beta+u_i$$

the usual assumption is that $(y_i,\mathbf{x}_i,u_i)$, $i=1,...,n$ is an iid sample. Under assumptions that $E\mathbf{x}_iu_i=0$ and $E(\mathbf{x}_i\mathbf{x}_i')$ has full rank, the ordinary least squares estimator:

$$\widehat{\beta}=\left(\sum_{i=1}^n\mathbf{x}_i\mathbf{x}_i'\right)^{-1}\sum_{i=1}\mathbf{x}_iy_i$$

is consistent and asymptotically normal. The expected covariance between a residual and the response variable then is:

$$Ey_iu_i=E(\mathbf{x}_i'\beta+u_i)u_i=Eu_i^2$$

If we furthermore assume that $E(u_i|\mathbf{x}_1,...,\mathbf{x}_n)=0$ and $E(u_i|\mathbf{x}_1,...,\mathbf{x}_n)=\sigma^2$, we can calculate the expected covariance between $y_i$ and its regression residual:

$$\begin{align*} Ey_i\widehat{u}_i&=Ey_i(y_i-\mathbf{x}_i'\widehat{\beta})\\\\ &=E(\mathbf{x}_i'\beta+u_i)(u_i-\mathbf{x}_i(\widehat{\beta}-\beta))\\\\ &=E(u_i^2)\left(1-E\mathbf{x}_i' \left(\sum_{j=1}^n\mathbf{x}_j\mathbf{x}_j'\right)^{-1}\mathbf{x}_i\right) \end{align*}$$

Now to get the correlation we need to calculate $Var(y_i)$ and $Var(\hat{u}_i)$. It turns out that

$$Var(\hat u_i)=E(y_i\hat{u}_i),$$

hence

$$Corr(y_i,\hat u_i)=\sqrt{1-E\mathbf{x}_i' \left(\sum_{j=1}^n\mathbf{x}_j\mathbf{x}_j'\right)^{-1}\mathbf{x}_i}$$

Now the term $\mathbf{x}_i' \left(\sum_{j=1}^n\mathbf{x}_j\mathbf{x}_j'\right)^{-1}\mathbf{x}_i$ comes from diagonal of the hat matrix $H=X(X'X)^{-1}X'$, where $X=[\mathbf{x}_i,...,\mathbf{x}_N]'$. The matrix $H$ is idempotent, hence it satisfies a following property

$$trace(H)=\sum_{i}h_{ii}=rank(H),$$

where $h_{ii}$ is the diagonal term of $H$. The $rank(H)$ is the number of linearly independent variables in $\mathbf{x}_i$, which is usually the number of variables. Let us call it $p$. The number of $h_{ii}$ is the sample size $N$. So we have $N$ nonnegative terms which should sum up to $p$. Usually $N$ is much bigger than $p$, hence a lot of $h_{ii}$ would be close to the zero, meaning that the correlation between the residual and the response variable would be close to 1 for the bigger part of observations.

The term $h_{ii}$ is also used in various regression diagnostics for determining influential observations.