I am trying to understand the weighted least squares estimation method, and I'd really appreciate it if you could shed some light on me. Let me explain my problem briefly:

Consider a linear model in a matrix form as $y=\beta x +e$ with $e \sim \mathcal{N} (0, \sigma^2I)$. To find an estimate of $x$, the weighted linear least squares estimator gives $$ \hat{x} = (\beta^tW\beta)^{-1} \beta^tW y, $$ where $W$ is the weight matrix with $w_{ii} = \sigma^{-2}$.

Assume that $\beta$ is known (and fixed). How sensitive is the WLS estimator ($\hat{x}$) with respect to the distortions of $y$? What is the relationship between the entries of $\hat{x}$ and $y$? Are there any relationships for the changes of $y$ such that $\hat{x}$ keeps the same values?


Preliminary confusions: There are a number of preliminary confusions in your question that need to be resolved before answering it. Firstly, the WLS estimator you have written is an estimator for $\beta$ and it is a function of the design matrix $x$. You seem to have stated this in an inverted form, as an estimator for $x$ that depends on $\beta$. (Remember that the $x$ values are taken as known in regression and so they are not estimated.) Moreover, if you assume that $\beta$ is known, as you have posited here, there is no estimation problem to begin with, and so WLS estimation does not apply.

Answering the question you meant to ask: The WLS estimator is an estimator for the unknown parameter $\beta$ in a model where this is taken to be unknown (and usually used when we have heteroscedastic error terms). Correctly stated, this estimator is:

$$\hat{\beta} = (x^\text{T} W x)^{-1} x^\text{T} W y = Ay \quad \quad \quad A \equiv (x^\text{T} W x)^{-1} x^\text{T} W.$$

If the weights in the are fixed by the design matrix $x$ then the hat matrix is fully determined by $x$ and so the estimator is a linear function of the response vector $y$. It is therefore easy to quantify the sensitivity of the estimator to distortions in the response variables. These are given by the partial derivatives:

$$\frac{\partial \hat{\beta}_k}{\partial y_i} = A_{i,k}.$$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.