# Influence functions and OLS

I am trying to understand how influence functions work. Could someone explain in the context of a simple OLS regression

$$y_i = \alpha + \beta \cdot x_i + \varepsilon_i$$

where I want the influence function for $\beta$.

• There isn't a specific question here yet: do you want to see how the influence function is computed? Do you want a specific empirical example? A heuristic explanation of what it means? – whuber Mar 16 '11 at 14:29
• If you look up Frank Critchley's 1986 paper "influence functions in principal components" (can't remember the exact name of the paper). He defines the influence function for ordinary regression here (which may or may not prove my answer wrong). – probabilityislogic Mar 17 '11 at 12:05

Influence functions are basically an analytical tool that can be used to assess the effect (or "influence") of removing an observation on the value of a statistic without having to re-calculate that statistic. They can also be used to create asymptotic variance estimates. If influence equals $I$ then asymptotic variance is $\frac{I^2}{n}$.

The way I understand influence functions is as follows. You have some sort of theoretical CDF, denoted by $F_{i}(y)=Pr(Y_{i}<y_{i})$. For simple OLS, you have

$$Pr(Y_{i}<y_{i})=Pr(\alpha+\beta x_{i} + \epsilon_{i} < y_{i})=\Phi\left(\frac{y_{i}-(\alpha+\beta x_{i})}{\sigma}\right)$$ Where $\Phi(z)$ is the standard normal CDF, and $\sigma^2$ is the error variance. Now you can show that any statistic will be a function of this CDF, hence the notation $S(F)$ (i.e. some function of $F$). Now suppose we change the function $F$ by a "little bit", to $F_{(i)}(z)=(1+\zeta)F(z)-\zeta \delta_{(i)}(z)$ Where $\delta_{i}(z)=I(y_{i}<z)$, and $\zeta=\frac{1}{n-1}$. Thus $F_{(i)}$ represents the CDF of the data with the "ith" data point removed. We can do a taylor series of $F_{(i)}(z)$ about $\zeta=0$. This gives:

$$S[F_{(i)}(z,\zeta)] \approx S[F_{(i)}(z,0)]+\zeta\left[\frac{\partial S[F_{(i)}(z,\zeta)]}{\partial \zeta}|_{\zeta=0}\right]$$

Note that $F_{(i)}(z,0)=F(z)$ so we get: $$S[F_{(i)}(z,\zeta)] \approx S[F(z)]+\zeta\left[\frac{\partial S[F_{(i)}(z,\zeta)]}{\partial \zeta}|_{\zeta=0}\right]$$

The partial derivative here is called the influence function. So this represents an approximate "first order" correction to be made to a statistic due to deleting the "ith" observation. Note that in regression the remainder does not go to zero asymtotically, so that this is an approximation to the changes you may actually get. Now write $\beta$ as:

$$\beta=\frac{\frac{1}{n}\sum_{j=1}^{n}(y_{j}-\overline{y})(x_{j}-\overline{x})}{\frac{1}{n}\sum_{j=1}^{n}(x_{j}-\overline{x})^2}$$

Thus beta is a function of two statistics: the variance of X and covariance between X and Y. These two statistics have representations in terms of the CDF as:

$$cov(X,Y)=\int(X-\mu_x(F))(Y-\mu_y(F))dF$$ and $$var(X)=\int(X-\mu_x(F))^{2}dF$$ where $$\mu_x=\int xdF$$

To remove the ith observation we replace $F\rightarrow F_{(i)}=(1+\zeta)F-\zeta \delta_{(i)}$ in both integrals to give:

$$\mu_{x(i)}=\int xd[(1+\zeta)F-\zeta \delta_{(i)}]=\mu_x-\zeta(x_{i}-\mu_x)$$ $$Var(X)_{(i)}=\int(X-\mu_{x(i)})^{2}dF_{(i)}=\int(X-\mu_x+\zeta(x_{i}-\mu_x))^{2}d[(1+\zeta)F-\zeta \delta_{(i)}]$$

ignoring terms of $\zeta^{2}$ and simplifying we get: $$Var(X)_{(i)}\approx Var(X)-\zeta\left[(x_{i}-\mu_x)^2-Var(X)\right]$$ Similarly for the covariance $$Cov(X,Y)_{(i)}\approx Cov(X,Y)-\zeta\left[(x_{i}-\mu_x)(y_{i}-\mu_y)-Cov(X,Y)\right]$$

So we can now express $\beta_{(i)}$ as a function of $\zeta$. This is:

$$\beta_{(i)}(\zeta)\approx \frac{Cov(X,Y)-\zeta\left[(x_{i}-\mu_x)(y_{i}-\mu_y)-Cov(X,Y)\right]}{Var(X)-\zeta\left[(x_{i}-\mu_x)^2-Var(X)\right]}$$

We can now use the Taylor series:

$$\beta_{(i)}(\zeta)\approx \beta_{(i)}(0)+\zeta\left[\frac{\partial \beta_{(i)}(\zeta)}{\partial \zeta}\right]_{\zeta=0}$$

Simplifying this gives:

$$\beta_{(i)}(\zeta)\approx \beta-\zeta\left[\frac{(x_{i}-\mu_x)(y_{i}-\mu_y)}{Var(X)}-\beta\frac{(x_{i}-\mu_x)^2}{Var(X)}\right]$$

And plugging in the values of the statistics $\mu_y$, $\mu_x$, $var(X)$, and $\zeta=\frac{1}{n-1}$ we get:

$$\beta_{(i)}\approx \beta-\frac{x_{i}-\overline{x}}{n-1}\left[\frac{y_{i}-\overline{y}}{\frac{1}{n}\sum_{j=1}^{n}(x_{j}-\overline{x})^2}-\beta\frac{x_{i}-\overline{x}}{\frac{1}{n}\sum_{j=1}^{n}(x_{j}-\overline{x})^2}\right]$$

And you can see how the effect of removing a single observation can be approximated without having to re-fit the model. You can also see how an x equal to the average has no influence on the slope of the line. Think about this and you will see how it makes sense. You can also write this more succinctly in terms of the standardised values $\tilde{x}=\frac{x-\overline{x}}{s_{x}}$ (similarly for y):

$$\beta_{(i)}\approx \beta-\frac{\tilde{x_{i}}}{n-1}\left[\tilde{y_{i}}\frac{s_y}{s_x}-\tilde{x_{i}}\beta\right]$$

• So the story is about the influence of additional data point? I more used to the impulse response for the time series data, in statistical context all influence would be described by marginal effect or (better choice) beta coefficient from standardized regression. Well I really need more context to judge the question and answer, but this one is nice, I think (+1 not yet but awaiting). – Dmitrij Celov Mar 16 '11 at 13:43
• @dmitrij - That is what was implied (or what I inferred) from the link - it is about the robustness properties of a statistic. Influence functions are slightly more general than 1 data point - you can redefine the delta function to be a sum of them (so many observations). I would think of it as a "cheap Jacknife" to some degree - because you don't require re-fitting of the model. – probabilityislogic Mar 17 '11 at 12:02

Here is a super general way to talk about influence functions of a regression. First I'm going to tackle one way of presenting influence functions:

Suppose $F$ is a distribution on $\Sigma$. The contaminated distribution function, $F_\epsilon(x)$ can be defined as: $$F_\epsilon(x)=(1-\epsilon)F+\epsilon\delta_x$$ where $\delta_x$ is the probability measure on $\Sigma$ which assigns probability 1 to $\{x\}$ and 0 to all other elements of $\Sigma$.

From this we can define the influence function fairly easily:

The influence function of $\hat{\theta}$ at $F$, $\psi_i:\mathcal{X}\to\Gamma$ is defined as: $$\psi_{\hat{\theta},F}(x)=\lim\limits_{\epsilon\to 0}\dfrac{\hat{\theta}(F_\epsilon(x))-\hat{\theta}(F)}{\epsilon}$$

From here it's possible to see that an influence function is the Gateaux derivative of $\hat\theta$ at $F$ in the direction of $\delta_x$. This makes the interpretation of influence functions (for me) a little bit clearer: An influence function tells you the effect that a particular observation has on the estimator.

The OLS estimate is a solution to the problem:

$$\hat\theta=\arg\min_\theta E[(Y-X\theta)^T(Y-X\theta)]$$

Imagine a contaminated distribution which puts a little more weight on observation $(x,y)$:

$$\hat\theta_\epsilon = \arg\min_\theta (1-\epsilon)E[(Y-X\theta)^T(Y-X\theta)]+\epsilon (y-x\theta)^T(y-x\theta)$$

Taking first order conditions:

$$\left\{(1-\epsilon)E[X^TX]+\epsilon x^Tx\right\}\hat\theta_\epsilon = (1-\epsilon)E[X^TY]+\epsilon x^Ty$$

Since the influence function is just a Gateaux derivative we can now say:

$$-(E[X^TX]+x^Tx)\hat\theta_\epsilon + E[X^TX]\psi_{\theta}(x,y) = -E[X^TY] + x^Ty$$

At $\epsilon=0$, $\hat\theta_\epsilon=\hat\theta=E[X^TX]^{-1}E[X^TY]$, so:

$$\psi_{\theta}(x,y)=E[X^TX]^{-1}x^T(y-x\theta)$$

The finite sample counterpart of this influence function is:

$$\psi_{\theta}(x,y)=\left(\dfrac{1}{N}\sum_i X_i^TX_i\right)^{-1}x^T(y-x\theta)$$

In general I find this framework (working with influence functions as Gateaux derivatives) easier to deal with.