6
$\begingroup$

And yet another for the long list of degrees-of-freedom questions!

Given an i.i.d. sample $x_1,..., x_n$ from an arbitrary real-valued distribution with expectation $\mu$, the sample mean can be written as $$\bar x = \sum_i \frac{1}{n} x_i.$$ One can say that the degrees of freedom for $\bar x$ is $n-1$, as there are $n$ data points and one parameter being estimated.

Now suppose that some twisted statistician decides that the weights $1/n$ are not good enough, and replaces $\bar x$ with $$\bar x_w = \sum_i w_i x_i,$$ where $w = (w_1, ..., w_n)$ is a pre-determined (i.e., not related to the data) vector of nonnegative weights with $\sum_i w_i = 1$.

What is the degrees of freedom for $\bar x_w$? References to versions of this problem in peer-reviewed work would be of particular interest.

$\endgroup$
  • 2
    $\begingroup$ Good question. I would think this would be related to the effective sample size, which is (Sum of the weights)^2 / (Sum of the (Weights^2)) -- perhaps minus 1. But I'm intentionally putting this as a comment, not an answer. $\endgroup$ – zbicyclist Sep 26 '13 at 4:08
  • $\begingroup$ @zbicyclist, yes. The formula you give simplifies (with my notation) as the sum in the numerator is exactly 1. One over the sum of squares was my best guess so far, but I don't have rigorous support for it. Thanks for giving it a name at least -- I had not heard of 'effective sample size.' $\endgroup$ – zkurtz Sep 26 '13 at 4:32
  • 2
    $\begingroup$ In the absence of any particular statistical procedure, "degrees of freedom" has little or no meaning. About the only meaning it could reasonably have is the dimension of the manifold of possible values of the sample conditioned on the weighted mean. That (obviously) is still $n-1$. Whether an "effective sample size"--however it may be computed--is useful, accurate, or even meaningful depends on how it will be used in a procedure. $\endgroup$ – whuber Sep 26 '13 at 15:37
  • 1
    $\begingroup$ Right, that [nonexistence] is exactly my claim. "Degrees of freedom" is often a way of calibrating some kind of approximation. An indication of what I mean by that can be gleaned from the exposition of DF in a $\chi^2$ context at stats.stackexchange.com/a/17148/919. $\endgroup$ – whuber Sep 26 '13 at 16:13
  • 1
    $\begingroup$ If the $x_i$ are from a population with variance $V$ then the variance of $\bar x_w$ is $V/n'$, where $n' = (\sum w)^2 / \sum w^2$ plays the role of the sample size and is perhaps the primary justification for calling it the "effective sample size". $\endgroup$ – Ray Koopman Sep 29 '13 at 0:17
2
$\begingroup$

This is wrong: (as correctly pointed out by @zkurtz)

I think the answer given by @zbicyclist as a comment above is quite sensible. One way to rationalize it is as follows. If you arrange your sample $x_1,\ldots,x_n$ and you "regress" on a vector whose $i$-th element is $w_i / \sum w_i^2$ (with no intercept), you get as the estimate $\hat\beta = \sum_iw_ix_i$.

The degrees of freedom of such regression would be the trace of the matrix $X(X'X)^{-1}X'$; replacing $X$ by the column vector whose $i$ element is as given above, those degrees of freedom turn out to be $(\sum_iw_i)^2/\sum_iw_i^2$. Of course if the $w_i$ are constrained to add up to 1, this coincides with your guess of $1/\sum_iw_i^2$.

As references to lend support to such "degrees of freedom" I think it is interesting Hastie-Tibshirani(1990) Generalized Additive Models, Chapman & Hall, section 3.5. (They give alternatives to the trace of the "hat" matrix.)

This may be right:

Hastie-Tibshirani(1990) cited above propose alternative definitions of "degrees of freedom used" in a general non-parametric smoother $\hat{\boldsymbol{x}} = S \boldsymbol{x}$ as follows: i) trace$(S)$, ii) trace$(S^TS)$ and iii) trace$(2S-S^TS)$. They draw on the analogy with a linear model, in which $S = X^T(X^TX)^{-1}X^T$, whose trace is $p$, the number of parameters (throughout I consider the full rank case). Since $S = X^T(X^TX)^{-1}X^T$ is symmetric idempotent, $S^TS$ and $(2S-S^TS)$ are equal to $S$, so the three definitions give the same answer in the linear regression case.

In the case of the question asked, we may consider $\boldsymbol{\hat{x}} = W \boldsymbol{x}$ where $\boldsymbol{\hat{x}}$ is the weighted mean multiplied by a column vector $\boldsymbol{1}$ and $W$ is a symmetric matrix of weights, each of whose rows is equal to the set of weights used.

If we adopt the definition i) above, the number of degrees of freedom used would be 1 (assuming $\sum_iw_i = 1$, if we adopt ii) it would be $n\sum_iw_i^2$. (In the case $w_i = 1/n$ for all $i$ (ordinary average), this produces 1 as it should.)

I find such definition to have some intuitive appeal, but totally aggree with @whuber that the name "degrees of freedom" (used in the smooth or fit) is abuse of language. I do not believe there is a non-controversial definition.

On this topic I have also found interesting Hodges, J. S. and Sargent, D. J. (2001) Counting Degrees of Freedom in Hierarchical and Other Richly-Parameterised Models, Biometrika, vol. 88, p.367-379. There are many other papers dealing with counting "equivalent parameters" (or "degrees of freedom used") in different situations.

$\endgroup$
  • $\begingroup$ This seems deep. I'm having trouble replicating your trace calculation. The trace is the sum of n terms. What do you get for the first term in this sum? $\endgroup$ – zkurtz Sep 26 '13 at 16:00
  • $\begingroup$ My guess: the trace had better be 1 (and not one over the sum of squares) because you've essentially set up a linear regression with one predictor variable, and one corresponding parameter. The trace should be the number of parameters in the model, which is the degrees of freedom in the model, not the degrees of freedom that remain in the data. $\endgroup$ – zkurtz Sep 26 '13 at 16:02
  • $\begingroup$ You are right. I was thinking of $I-X(X'X)^{-1}X'$ but that is inessential. I will clear the mess tomorrow (European time) and try to reproduce the computations I too hastily made this morning. Thank you for pointing out the obvious. $\endgroup$ – F. Tusell Sep 26 '13 at 18:38
  • $\begingroup$ I edited my answer above. Hope it now makes some sense. $\endgroup$ – F. Tusell Sep 28 '13 at 14:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.