Skip to main content
clarified requirement for normality assumption in t-test, with additional links
Source Link
EdM
  • 101.5k
  • 11
  • 102
  • 303

There is a marvelous thread on this site: "Is normality testing 'essentially useless'?". It indicates that your take on this matter is essentially correct.

If you have a small sample size then you won't have enough power to detect a true deviation from normality; $p \ge 0.05$ in a normality test doesn't demonstrate normality, it simply doesn't rule it out at that (arbitrary) probability cutoff.

If you have a very large sample size then you typically will rule out normality.

And in either case you've used the data to choose the test, violating the assumptions of the test that you choose.

The one place where you aren't quite right is the statement that "if we only care about mean values, we can always perform a T-test." Yes, you can always calculate a t statistic but the significance test for that t-statistic does assume normally distributed data,* even in tests that allow for different variances between groups. It turns out that t-tests often work well enough when the normality assumption doesn't hold. As another answer to your question notes, transformations of data often can help make data conform better to that assumption.

Also, you don't comment on your company's use of multiple t-tests when there are multiple means to compare. That leads to multiple-comparison issues that might be more troublesome than choosing the tests for significant differences based on an initial normality test.

A Frank Harrell notes in one answer on that first thread, non-parametric tests have reasonably high power even if data are normally distributed, they work well without the normality assumption, and generalize to ordinal cumulative probability models when something more than a 2-sample comparison is needed.

So a more defensible strategy, if you have reason to believe that normality assumptions will be violated so strongly as to pose a problem for t-tests and the like, is just to go with non-parametric tests.


*This linked page notes 3 required assumptions: that the sample mean estimate be appropriately normally distributed, that the sample variance estimate be $\chi^2$ with appropriate degrees of freedom, and that these estimates be independent. What it doesn't make immediately clear is that the third assumption is characteristic only of normal distributions. See this answer, linked in a comment by @COOLSerdash, for more details.

There is a marvelous thread on this site: "Is normality testing 'essentially useless'?". It indicates that your take on this matter is essentially correct.

If you have a small sample size then you won't have enough power to detect a true deviation from normality; $p \ge 0.05$ in a normality test doesn't demonstrate normality, it simply doesn't rule it out at that (arbitrary) probability cutoff.

If you have a very large sample size then you typically will rule out normality.

And in either case you've used the data to choose the test, violating the assumptions of the test that you choose.

The one place where you aren't quite right is the statement that "if we only care about mean values, we can always perform a T-test." Yes, you can always calculate a t statistic but the significance test for that t-statistic does assume normally distributed data, even in tests that allow for different variances between groups. It turns out that t-tests often work well enough when the normality assumption doesn't hold. As another answer to your question notes, transformations of data often can help make data conform better to that assumption.

Also, you don't comment on your company's use of multiple t-tests when there are multiple means to compare. That leads to multiple-comparison issues that might be more troublesome than choosing the tests for significant differences based on an initial normality test.

A Frank Harrell notes in one answer on that first thread, non-parametric tests have reasonably high power even if data are normally distributed, they work well without the normality assumption, and generalize to ordinal cumulative probability models when something more than a 2-sample comparison is needed.

So a more defensible strategy, if you have reason to believe that normality assumptions will be violated so strongly as to pose a problem for t-tests and the like, is just to go with non-parametric tests.

There is a marvelous thread on this site: "Is normality testing 'essentially useless'?". It indicates that your take on this matter is essentially correct.

If you have a small sample size then you won't have enough power to detect a true deviation from normality; $p \ge 0.05$ in a normality test doesn't demonstrate normality, it simply doesn't rule it out at that (arbitrary) probability cutoff.

If you have a very large sample size then you typically will rule out normality.

And in either case you've used the data to choose the test, violating the assumptions of the test that you choose.

The one place where you aren't quite right is the statement that "if we only care about mean values, we can always perform a T-test." Yes, you can always calculate a t statistic but the significance test for that t-statistic does assume normally distributed data,* even in tests that allow for different variances between groups. It turns out that t-tests often work well enough when the normality assumption doesn't hold. As another answer to your question notes, transformations of data often can help make data conform better to that assumption.

Also, you don't comment on your company's use of multiple t-tests when there are multiple means to compare. That leads to multiple-comparison issues that might be more troublesome than choosing the tests for significant differences based on an initial normality test.

A Frank Harrell notes in one answer on that first thread, non-parametric tests have reasonably high power even if data are normally distributed, they work well without the normality assumption, and generalize to ordinal cumulative probability models when something more than a 2-sample comparison is needed.

So a more defensible strategy, if you have reason to believe that normality assumptions will be violated so strongly as to pose a problem for t-tests and the like, is just to go with non-parametric tests.


*This linked page notes 3 required assumptions: that the sample mean estimate be appropriately normally distributed, that the sample variance estimate be $\chi^2$ with appropriate degrees of freedom, and that these estimates be independent. What it doesn't make immediately clear is that the third assumption is characteristic only of normal distributions. See this answer, linked in a comment by @COOLSerdash, for more details.

added 27 characters in body
Source Link
EdM
  • 101.5k
  • 11
  • 102
  • 303

There is a marvelous thread on this site: "Is normality testing 'essentially useless'?". It indicates that your take on this matter is essentially correct.

If you have a small sample size then you won't have enough power to detect a true deviation from normality; $p \ge 0.05$ in a normality test doesn't demonstrate normality, it simply doesn't rule it out at that (arbitrary) probability cutoff.

If you have a very large sample size then you typically will rule out normality.

And in either case you've used the data to choose the test, violating the assumptions of the test that you choose.

The one place where you aren't quite right is the statement that "if we only care about mean values, we can always perform a T-test." Yes, you can always performcalculate a t-test statistic but the significance tests dotest for that t-statistic does assume normally distributed data, even in tests that allow for different variances between groups. It turns out that t-tests often work well enough when the normality assumption doesn't hold. As another answer to your question notes, transformations of data often can help make data conform better to that assumption.

Also, you don't comment on your company's use of multiple t-tests when there are multiple means to compare. That leads to multiple-comparison issues that might be more troublesome than choosing the tests for significant differences based on an initial normality test.

A Frank Harrell notes in one answer on that first thread, non-parametric tests have reasonably high power even if data are normally distributed, they work well without the normality assumption, and generalize to ordinal cumulative probability models when something more than a 2-sample comparison is needed.

So a more defensible strategy, if you have reason to believe that normality assumptions will be violated so strongly as to pose a problem for t-tests and the like, is just to go with non-parametric tests.

There is a marvelous thread on this site: "Is normality testing 'essentially useless'?". It indicates that your take on this matter is essentially correct.

If you have a small sample size then you won't have enough power to detect a true deviation from normality; $p \ge 0.05$ in a normality test doesn't demonstrate normality, it simply doesn't rule it out at that (arbitrary) probability cutoff.

If you have a very large sample size then you typically will rule out normality.

And in either case you've used the data to choose the test, violating the assumptions of the test that you choose.

The one place where you aren't quite right is the statement that "if we only care about mean values, we can always perform a T-test." Yes, you can always perform a t-test but the significance tests do assume normally distributed data, even in tests that allow for different variances between groups. It turns out that t-tests often work well enough when the normality assumption doesn't hold. As another answer to your question notes, transformations of data often can help make data conform better to that assumption.

Also, you don't comment on your company's use of multiple t-tests when there are multiple means to compare. That leads to multiple-comparison issues that might be more troublesome than choosing the tests for significant differences based on an initial normality test.

A Frank Harrell notes in one answer on that first thread, non-parametric tests have reasonably high power even if data are normally distributed, they work well without the normality assumption, and generalize to ordinal cumulative probability models when something more than a 2-sample comparison is needed.

So a more defensible strategy, if you have reason to believe that normality assumptions will be violated so strongly as to pose a problem for t-tests and the like, is just to go with non-parametric tests.

There is a marvelous thread on this site: "Is normality testing 'essentially useless'?". It indicates that your take on this matter is essentially correct.

If you have a small sample size then you won't have enough power to detect a true deviation from normality; $p \ge 0.05$ in a normality test doesn't demonstrate normality, it simply doesn't rule it out at that (arbitrary) probability cutoff.

If you have a very large sample size then you typically will rule out normality.

And in either case you've used the data to choose the test, violating the assumptions of the test that you choose.

The one place where you aren't quite right is the statement that "if we only care about mean values, we can always perform a T-test." Yes, you can always calculate a t statistic but the significance test for that t-statistic does assume normally distributed data, even in tests that allow for different variances between groups. It turns out that t-tests often work well enough when the normality assumption doesn't hold. As another answer to your question notes, transformations of data often can help make data conform better to that assumption.

Also, you don't comment on your company's use of multiple t-tests when there are multiple means to compare. That leads to multiple-comparison issues that might be more troublesome than choosing the tests for significant differences based on an initial normality test.

A Frank Harrell notes in one answer on that first thread, non-parametric tests have reasonably high power even if data are normally distributed, they work well without the normality assumption, and generalize to ordinal cumulative probability models when something more than a 2-sample comparison is needed.

So a more defensible strategy, if you have reason to believe that normality assumptions will be violated so strongly as to pose a problem for t-tests and the like, is just to go with non-parametric tests.

Source Link
EdM
  • 101.5k
  • 11
  • 102
  • 303

There is a marvelous thread on this site: "Is normality testing 'essentially useless'?". It indicates that your take on this matter is essentially correct.

If you have a small sample size then you won't have enough power to detect a true deviation from normality; $p \ge 0.05$ in a normality test doesn't demonstrate normality, it simply doesn't rule it out at that (arbitrary) probability cutoff.

If you have a very large sample size then you typically will rule out normality.

And in either case you've used the data to choose the test, violating the assumptions of the test that you choose.

The one place where you aren't quite right is the statement that "if we only care about mean values, we can always perform a T-test." Yes, you can always perform a t-test but the significance tests do assume normally distributed data, even in tests that allow for different variances between groups. It turns out that t-tests often work well enough when the normality assumption doesn't hold. As another answer to your question notes, transformations of data often can help make data conform better to that assumption.

Also, you don't comment on your company's use of multiple t-tests when there are multiple means to compare. That leads to multiple-comparison issues that might be more troublesome than choosing the tests for significant differences based on an initial normality test.

A Frank Harrell notes in one answer on that first thread, non-parametric tests have reasonably high power even if data are normally distributed, they work well without the normality assumption, and generalize to ordinal cumulative probability models when something more than a 2-sample comparison is needed.

So a more defensible strategy, if you have reason to believe that normality assumptions will be violated so strongly as to pose a problem for t-tests and the like, is just to go with non-parametric tests.