Sensitivity of weighted least squares (WLS) method

Question

I am trying to understand the weighted least squares estimation method, and I'd really appreciate it if you could shed some light on me. Let me explain my problem briefly:

Consider a linear model in a matrix form as $y=\beta x +e$ with $e \sim \mathcal{N} (0, \sigma^2I)$. To find an estimate of $x$, the weighted linear least squares estimator gives $$ \hat{x} = (\beta^tW\beta)^{-1} \beta^tW y, $$ where $W$ is the weight matrix with $w_{ii} = \sigma^{-2}$.

Assume that $\beta$ is known (and fixed). How sensitive is the WLS estimator ($\hat{x}$) with respect to the distortions of $y$? What is the relationship between the entries of $\hat{x}$ and $y$? Are there any relationships for the changes of $y$ such that $\hat{x}$ keeps the same values?

Answer 1

Preliminary confusions: There are a number of preliminary confusions in your question that need to be resolved before answering it. Firstly, the WLS estimator you have written is an estimator for $\beta$ and it is a function of the design matrix $x$. You seem to have stated this in an inverted form, as an estimator for $x$ that depends on $\beta$. (Remember that the $x$ values are taken as known in regression and so they are not estimated.) Moreover, if you assume that $\beta$ is known, as you have posited here, there is no estimation problem to begin with, and so WLS estimation does not apply.

Answering the question you meant to ask: The WLS estimator is an estimator for the unknown parameter $\beta$ in a model where this is taken to be unknown (and usually used when we have heteroscedastic error terms). Correctly stated, this estimator is:

$$\hat{\beta} = (x^\text{T} W x)^{-1} x^\text{T} W y = Ay \quad \quad \quad A \equiv (x^\text{T} W x)^{-1} x^\text{T} W.$$

If the weights in the are fixed by the design matrix $x$ then the hat matrix is fully determined by $x$ and so the estimator is a linear function of the response vector $y$. It is therefore easy to quantify the sensitivity of the estimator to distortions in the response variables. These are given by the partial derivatives:

$$\frac{\partial \hat{\beta}_k}{\partial y_i} = A_{i,k}.$$