Least squares with variable errors

Question

I am trying to understand the classical linear model

$$ Y=X\beta +\varepsilon$$ where instead we have that $\epsilon_k\sim N(0,a_k\sigma^2)$ are independent where $a_k>0$ are given. I want to recover all the standard information we get from least squares (LS estimator, residual sum squares and of an unbiased estimator for $\sigma^2$ . Setting $A=Diag(a_1,...,a_n)$, I have derived the weighted least squares estimator to be $\hat{\beta}=(X^TA^{-1}X)^{-1}X^TA^{-1}Y$. I am trying to convert this back to an OLS equation in order to get the residual sum of squares and an unbiased estimator of $\sigma^2$. Naively setting $Z=X^TA^{-1}$ causes issues so this won't work.

Computing $\hat{\beta}$ is simple only because the $a_1,..a_n$ are known. Can we still compute it even if they were unknown? What if the $a_1,...a_n$ all depended on another parameter $\eta$, how would we be able to solve for the least squares estimator? For instance Newton-Raphson would no longer be applicable.

Answer 1

There are many questions in here, so I will focus on the case of $a_1, a_2, \dotsc, a_n$ being unknown. But this is also many cases, specifically

1. $a_k$ unknown and varies in a way unrelated to known (measured) factors. That must be an untypical case, and could be modeled as $a_k$ being random variables from some unknown distribution. The overall result would be that the error term is not longer normally distributed, but could be modeled with some other error distribution.

2. The more interesting case is that $a_k$ is a function, maybe of the expectation $\mu$, maybe some covariates directly. Today that could be treated by modeling the variance (or often in practice the logarithm of variance) directly. In R that is implemented in the package gamlss and an example is at Are there better approaches than the weighted mean?.

Another approach which could be used is ordinary least squares but with robust standard errors. If the $a_k$ cannot be well estimated that could well be better. Here is an interesting & relevant paper.