14
$\begingroup$

The Welch t-test for unequal variances (also known as Welch–Satterthwaite or Welch-Aspin) generally has a non-integer degrees of freedom. How should these degrees of freedom be quoted when reporting the results of the test?

"It is conventional to round down to the nearest integer before consulting standard t tables" according to various sources* - which makes sense as this direction of rounding is conservative.** Some older statistical software would do this too (e.g. Graphpad Prism before version 6) and some online calculators still do. If this procedure had been used, reporting the rounded-down degrees of freedom seems appropriate. (Though using some better software might be even more appropriate!)

But the vast majority of modern packages make use of the fractional part so in this case it seems the fractional part should be quoted. I can't see it being appropriate to quote to more than two decimal places, as a thousandth of a degree of freedom would only have negligible impact on the p-value.

Looking around Google scholar, I can see papers quoting the df as a whole number, with one decimal place, or with two decimal places. Are there any guidelines on how much accuracy to use? Also, if the software used the full fractional part, should the quoted df be rounded down to the desired number of figures (e.g. $7.5845... \rightarrow 7.5$ to 1 d.p. or $\rightarrow 7$ as a whole number) as was appropriate with the conservative calculation, or as seems more sensible to me, rounded conventionally (to the nearest) so that $7.5845... \rightarrow 7.6$ to 1 d.p. or $\rightarrow 8$ to the nearest whole?

Edit: aside from knowing the most theoretically sound way of reporting non-integer df, it would also be good to know what people do in practice. Presumably journals and style guides have their own requirements. I would be curious what influential style guides like the APA require. From what I can discern (their manual is not freely available online), the APA have a general preference that almost everything should appear to two decimal places, except p-values (which may be two or three d.p.) and percentages (rounded to the nearest percent) - that covers regression slopes, t statistics, F statistics, $\chi^2$ statistics and so on. This is quite illogical, bearing in mind that the second decimal place occupies a very different significant figure, and suggests quite different precision, in 2.47 than in 982.47, but might explain the number of Welch df with two decimal places I saw in my unscientific sample.

$*$ e.g. Ruxton, G.D. The unequal variance t-test is an underused alternative to Student's t-test and the Mann–Whitney U test, Behavioral Ecology (July/August 2006) 17 (4): 688-690 doi:10.1093/beheco/ark016

$**$ Though the Welch-Satterthwaite approximation itself may or may not be conservative, and in a case where it is not conservative, rounding down the degrees of freedom is no guarantee of compensating overall.

$\endgroup$
  • 1
    $\begingroup$ I have not studied actual practice--which is why this is a comment and not an answer--but I would expect it to be based on judgment related to reporting significant figures. For relatively high df, often a change in the first decimal place would not change the p-value at all (to the level of precision reported), so rounding to an integer is fine. For very low df $\nu$ and extreme values of $t$, the derivative $|\frac{\partial}{\partial\nu}F_\nu(t)|$ can exceed $0.01$, suggesting in such cases that $\nu$ should be reported to only one less significant figure than $p$ itself. $\endgroup$ – whuber Dec 19 '14 at 23:43
  • $\begingroup$ @whuber That is indeed a useful observation, particularly when taken together with Glen_b's answer. How low is "very low" for $\nu$? (My suspicion from the sample of papers I came across is that much "actual practice" may not be the same thing as "good practice"! I suspect robotic following of guidelines has as much effect as judgment, which is why it would be interesting to know what common reporting guidelines are.) $\endgroup$ – Silverfish Dec 19 '14 at 23:53
10
+50
$\begingroup$

I have not studied actual practice, so this reply cannot address that aspect of the question. As a general principle I would expect the treatment of significant digits in reporting the degrees of freedom (df) to be based on judgment related to significant figures.

The principle is to be consistent: use the precision in one quantity that is appropriate for the precision used in another one that is related to it. Specifically, when reporting values $x$ and $y=f(x)$ when $x$ is given to the nearest multiple of a small value $h$ (such as $h=\frac{1}{2}\times 10^{-6}$ for six places after the decimal point), the relative precision in $y$ as mediated by the function $f$ is

$$\sup_{-h \le k \le h} |f(x+k) - f(x)| \approx h | \frac{d}{dx} f(x) |.$$

The approximation applies when $f$ is continuously differentiable on the interval $[x-h, x+h]$.

In the present application, $y$ is the $p$-value, $x$ is the degrees of freedom $\nu$, and

$$y = f(x) = f(\nu) = F_\nu(t)$$

where $t$ is the Welch-Satterthwaite statistic and $F_\nu$ is the CDF of the Student $t$ distribution with $\nu$ degrees of freedom.

For relatively high df $\nu$, often a change in the first decimal place would not change the p-value at all (to the level of precision reported), so rounding to an integer is fine ($h=1/2$ but $h|\frac{d}{dx}f(x)|$ is very small). For very low df and extreme values of the statistic $t$, the magnitude of the derivative $|\frac{\partial}{\partial\nu}F_\nu(t)|$ can exceed $0.01$, suggesting in such cases that $\nu$ should be reported to only one less decimal place than $p$ itself.

See for yourself with this labeled contour plot of the magnitude of the derivative for the lowest (reasonable) df and ranges of $|t|$ that would be of interest (because they can lead to low p-values).

Figure

The labels show the base-10 logarithm of the derivative. Thus, at points between $-k$ and $-(k+1)$ on this plot, changing the reported df in the $j^\text{th}$ place after the decimal point will likely change the reported p-value only in the $(j+k)^\text{th}$ and later places. For example, suppose you are rounding the p-value to $10^{-6}$ (six decimal places). Consider the statistics $\nu=2.5$ and $t=8$. These are located near the $-3$ log contour. Therefore, $\nu$ should be reported to $6+(-3)=3$ decimal places.

The light blue areas, for the largest $k$, are the ones of concern, because they show where small changes in $\nu$ have the greatest effects on the p-value.

Contrast this with the situation for higher df (from $4$ to $30$ shown):

Figure 2

The influence of $\nu$ on the precision of $p$ quickly wanes as $\nu$ increases.

$\endgroup$
  • 1
    $\begingroup$ This was a very helpful contribution for establishing by what principles one ought to round the degrees of freedom (+50!); I hope a later answerer can fill in the gaps about actual practice. $\endgroup$ – Silverfish Jan 6 '15 at 23:13
7
$\begingroup$

It is conventional to round down to the nearest integer before consulting standard t tables

The reason that was a convention is because tables don't have noninteger df. There's no reason to do it otherwise.

which makes sense as this adjustment is conservative.

Well, the statistic doesn't actually have a t-distribution, because he squared denominator doesn't actually have a scaled chi-squared distribution. It's an approximation that may or may not be conservative in some particular instance -- rounding df down may not be certain to be conservative when we consider the exact distribution of the statistic in a particular instance.

(by interpolation or by actually crunching the numbers for the t-distribution with that df?)

p-values from t-distributions (applying the cdf to a t-statistic) can be computed by a variety of pretty accurate approximations, so they're effectively calculated rather than interpolated.

I can't see it being appropriate to quote to more than two decimal places

I agree.

Are there any guidelines on how much accuracy to use?

One possibility might be to investigate how accurate the Welch-Satterthwaite approximation for the p-value is in that general region of variance ratios and not quote substantially more relative accuracy than that would suggest was in the d.f. (keeping in mind that the df on the chi-squared in the square of the denominator are just giving an approximation to something that isn't chi-squared anyway).

$\endgroup$
  • 1
    $\begingroup$ I should have clarified re "rounding down is conservative". The Welch-Satterthwaite approximation itself may or may not be conservative. But the process of rounding down surely is - if the approximation was not conservative to start with, it is at least less bad after rounding down. In contrast, rounding up (e.g. "7.5845 rounds nearest to 8") is definitely not a conservative adjustment. I could do with finding a better way to phrase this, but I hope my point is clear! $\endgroup$ – Silverfish Nov 21 '14 at 14:33
  • $\begingroup$ "One possibility might be to investigate how accurate the Welch-Satterthwaite approximation for the p-value is in that general region of variance ratios" - this is very sensible and seems to be the principled approach. Is this something commonly done? Some hints for implementation would be nice. In practice I suspect journal style guidelines often have the final say on the matter! But I don't know what they say - there was definitely a variety of practice in the papers my search turned up. $\endgroup$ – Silverfish Nov 21 '14 at 14:42
  • $\begingroup$ To try to avoid confusion to future readers I have attempted to clarify re conservative rounding in the question body. Thanks for picking that up. $\endgroup$ – Silverfish Nov 21 '14 at 14:47
  • $\begingroup$ I don't think anything like it is commonly done, but I don't think that means it shouldn't be. How much of explaining why one rounds/truncates to a certain point gets into the paper would clearly depend on the journal/editor/referees. $\endgroup$ – Glen_b Nov 21 '14 at 14:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.