Wikipedia says the standard way to estimate the parameter covariance $M^\beta$ for weighted least squares (WLS) is $$ M^\beta = \chi_\nu^2 (X^TWX)^{-1} $$ with $$ \chi_\nu^2=\frac{r^TWr}{\nu} $$ and the degrees of freedom $$ \nu=n-m $$ (n=number of samples, m=number of fitted parameters, r=residuals, W=diagonal weight matrix, X=data).

My question is about the formula for the degrees of freedom, i.e., the normalization of the reduced chi squared statistic.

Assume I have a bunch of samples with weights 1 and only one with weight 2. Since in WLS, weights correspond to the sample measurement precision, this should be the exact same scenario as simply observing the sample with weight 2 twice (independently).* Thus, WLS can be "emulated" by doing OLS with an extra repeated measurement sample. This yields identical parameter values $\beta$ but, using the above formula for the degrees of freedom, a different $\chi_\nu^2$.

I can achieve equivalence of the two parameter covariance estimates by choosing $$ \nu = \text{sum(diag(}W)) - m, $$ i.e., replacing $n$ in the formula by the sum of the weights. (This obviously reduces to the standard case for all-unit weights.) However, I am confused that the standard formula (which is also the one implemented by, e.g., lm in R and fitlm in Matlab) does not seem to "work" for this simple test case. What am I missing or misunterstanding?

There are many related questions already, e.g., on DOFs or on weighted sample variances.

*Precisions of multiple Gaussian observations are additive... two observations with the same mean and precision 1 are equivalent to one observation with precision 2. The only clear reference for this I could find is this paper, Table 2, Eq. (II.1).

  • $\begingroup$ It's a good question -- but the motivational example misses an important point. When an observation is repeated, the two values are perfectly correlated. This violates OLS assumptions, so one shouldn't expect a standard formula for DF to apply. $\endgroup$
    – whuber
    Jul 26, 2021 at 19:52
  • $\begingroup$ @whuber Thanks, I somewhat expected that counter-argument after reading through the other threads. I do think these are separate issues, but maybe this is where I'm wrong? I would argue that it's not a real repeated measurement; I only add it to "emulate" WLS using OLS, assuming that it's completely independent of the original measurement. If I assumed that the added measurement was perfectly correlated with the original one, this would defeat the whole purpose: in that case, no new information is gained at all and I could also simply omit the added measurement, right? $\endgroup$
    – Eike P.
    Jul 26, 2021 at 20:09

1 Answer 1


Very good question.

But based on what, do you claim "this should be the exact same scenario"?

An observation (assuming the last WLOG) with a weight of 2, mathematically, means that the disturbance for that observation carries half of the variance. Suppose some higher being offers you a deal where you could have the opportunity to get an extra observation but both the old last and the new observation would get the regular variance. Would you take this deal?

The answer is yes, as there is more information under the new deal. You would do no worse just by taking the average of both x and y over the old last and the new extra. But there is also intra-variability of data between them that carries information (and this is the extra information).

Now suppose you take the deal and, by miracle, the new observation happens to coincide with old last in terms of x and y. Then there is no intra-variability extra information in this particular case (or this particular point in the sample space, if you want to be technical). Then the new point estimate would not change.

But the new estimator is indeed more accurate and should have a smaller standard error, in the classical statistical sense (i.e. frequencist sense across all the "parallel universes", or sample points, that could have been).


The above attempted answer, trying to explain intuitively why the OLS/GLS formulae would indicate that two observations with regular variance is more informative than a single observation with half of the variance, is flawed. My claim, that you "can do no worse" by taking the average (and throwing away the extra information), is wrong.

The informational value of an observation increases with a smaller disturbance variance, but also decreases with a smaller variance on the independent variable. So smoothing over the two observations would indeed halve the disturbance variance but would also make the independent variable less insightful. So, if you throw away the extra intra-variability information (e.g. differences within the Xs and Ys), you would be strictly worse off than the original situation of a single observation with half of the variance.

My further attempt to repair this, by utilizing the extra intra-variability information, so far just becomes an alternate way to derive those OLS/WOLS formulae and is no more intuitive than the formulae themselves.

I will think about this more later. Sorry.

  • $\begingroup$ Sorry, I can't (yet) follow. What exactly is the extra piece of information the OLS estimator gets from 2 measurements at variance 1 vs. 1 measurement at variance 0.5? I.e., what do you mean by "intra-variability extra information"? $\endgroup$
    – Eike P.
    Jul 26, 2021 at 21:05
  • $\begingroup$ Jhin, see edit above. $\endgroup$ Jul 26, 2021 at 21:47
  • $\begingroup$ Thanks for the edit and the thought you already put into this! $\endgroup$
    – Eike P.
    Jul 26, 2021 at 22:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.