# Derivative of a linear model

Question
Is there such concept in econometrics/statistics as a derivative of parameter $\hat{b_{p}}$ in a linear model with respect to some observation $X_{ij}$?
By derivative I mean $\frac{\partial \hat{b_{p}}}{\partial X_{ij}}$ - how would change parameter $\hat{b_{p}}$ if we changed $X_{ij}$?

Motivation
I was thinking about a situation when we have some uncertainty in data (e.g. results from a survey) and we have enough money to obtain precise results only in a single observation, which observation should we choose?
My intuition is saying that we should choose the observation that might change parameters the most, which is equivalent to the highest value of the derivative. If there are any other concepts feel free to write about them.

-
 Hey @tomek, do you mean parameter $b_p$ or parameter estimate $\hat{b_p}$? – suncoolsu Jan 26 '11 at 18:34 @suncoolsu: I menat parameter estimate $\hat{b_{p}}$ – Tomek Tarczynski Jan 26 '11 at 19:05

@onestop points in the right direction. Belsley, Kuh, and Welsch describe this approach on pp. 24-26 of their book. To differentiate with respect to an observation (and not just one of its attributes), they introduce a weight, perform weighted least squares, and differentiate with respect to the weight.

Specifically, let $\mathbb{X} = X_{ij}$ be the design matrix, let $\mathbf{x}_i$ be the $i$th observation, let $e_i$ be its residual, let $w_i$ be the weight, and define $h_i$ (the $i$th diagonal entry in the hat matrix) to be $\mathbf{x}_i (\mathbb{X}^T \mathbb{X})^{-1} \mathbf{x}_i^T$. They compute

$$\frac{\partial b(w_i)}{\partial w_i} = \frac{(\mathbb{X}^T\mathbb{X})^{-1} \mathbf{x}_i^T e_i}{\left[1 - (1 - w_i)h_i\right]^2},$$

whence

$$\frac{\partial b(w_i)}{\partial w_i}\Bigg|_{w_i=1} = (\mathbb{X}^T\mathbb{X})^{-1} \mathbf{x}_i^T e_i.$$

This is interpreted as a way to "identify influential observations, ... provid[ing] a means for examining the sensitivity of the regression coefficients to a slight change in the weight given to the ith observation. Large values of this derivative indicate observations that have large influence on the calculated coefficients." They suggest it can be used as an alternative to the DFBETA diagnostic. (DFBETA measures the change in $b$ when observation $i$ is completely deleted.) The relationship between the influence and DFBETA is that DFBETA equals the influence divided by $1 - h_i$ [equation 2.1 p. 13].

-
 As usual great answer, I was looking exactly for this. – Tomek Tarczynski Jan 27 '11 at 8:43 The book that You mentioned is available for free in google books, well at least most of it. Nevertheless, have You heard about differentiation with respect to a single attribute? I'm asking because I was 'playing' with matrices lately and I have found that it's quite simple to calculate and I was courious if that is anything new... – Tomek Tarczynski Jan 27 '11 at 11:18 @Tomek Differentiation w.r.t. a single attribute ought to give its coefficient (when there are no interactions). Of course, since the solutions are functions of all the data, you could differentiate w.r.t any particular $x_{ij}$. It should be straightforward to work out. (In fact, finding the solution for OLS would be sufficiently general.) I haven't read B,K,& W in 20 years, so my memory is dim, but I don't recall that they went down to this level of detail and they definitely don't do this computation in the section I quoted. – whuber♦ Jan 27 '11 at 16:22

I guess this would come under the heading of regression diagnostics. I haven't seen this precise statistic before, but something that comes fairly close is DFBETAij, which is the the change in regression coefficient i when the jth observation is omitted divided by the estimated standard error of coefficient i.

The book that defined this and many other regression diagnostics (perhaps too many) is:

Belsley, D. A., E. Kuh, and R. E. Welsch. (1980). Regression Diagnostics: Identifying Influential Data and Sources of Collinearity. New York: Wiley. ISBN 0471691178

-
 @onestep: This is very good answer to the problem described in the motivation, I haven't heard about it earlier and it seems to be really useful. Thanks! – Tomek Tarczynski Jan 27 '11 at 8:45