Skip to main content
emphasis
Source Link
Underminer
  • 4.2k
  • 1
  • 23
  • 44

Given a big enough sample size, a test will always show significant results, unless the true effect size is exactly zero, as discussed here. In practice, the true effect size is not zero, so gathering more data will eventually be able to detect the most minuscule differences.

The (IMO) facetious answer from Fisher was in response to a relatively trivial question that at its premise is conflating 'significant difference' with 'practically relevant difference'.

It would be equivalent to a researcher coming into my office and asking "I weighed this lead weight labeled '25 gram' and it measured 25.0 grams. I believe it to be mislabeled, what should I do?" To which I could answer, "Get a more precise scale."

I believe the go-get-more-data approach is appropriate if the initial test is woefully underpoweredThat being said, I believe the go-get-more-data approach is appropriate if the initial test is woefully underpowered to detect the magnitude of difference that is practically relevant.

Given a big enough sample size, a test will always show significant results, unless the true effect size is exactly zero, as discussed here. In practice, the true effect size is not zero, so gathering more data will eventually be able to detect the most minuscule differences.

The (IMO) facetious answer from Fisher was in response to a relatively trivial question that at its premise is conflating 'significant difference' with 'practically relevant difference'.

It would be equivalent to a researcher coming into my office and asking "I weighed this lead weight labeled '25 gram' and it measured 25.0 grams. I believe it to be mislabeled, what should I do?" To which I could answer, "Get a more precise scale."

I believe the go-get-more-data approach is appropriate if the initial test is woefully underpowered to detect the magnitude of difference that is practically relevant.

Given a big enough sample size, a test will always show significant results, unless the true effect size is exactly zero, as discussed here. In practice, the true effect size is not zero, so gathering more data will eventually be able to detect the most minuscule differences.

The (IMO) facetious answer from Fisher was in response to a relatively trivial question that at its premise is conflating 'significant difference' with 'practically relevant difference'.

It would be equivalent to a researcher coming into my office and asking "I weighed this lead weight labeled '25 gram' and it measured 25.0 grams. I believe it to be mislabeled, what should I do?" To which I could answer, "Get a more precise scale."

That being said, I believe the go-get-more-data approach is appropriate if the initial test is woefully underpowered to detect the magnitude of difference that is practically relevant.

Source Link
Underminer
  • 4.2k
  • 1
  • 23
  • 44

Given a big enough sample size, a test will always show significant results, unless the true effect size is exactly zero, as discussed here. In practice, the true effect size is not zero, so gathering more data will eventually be able to detect the most minuscule differences.

The (IMO) facetious answer from Fisher was in response to a relatively trivial question that at its premise is conflating 'significant difference' with 'practically relevant difference'.

It would be equivalent to a researcher coming into my office and asking "I weighed this lead weight labeled '25 gram' and it measured 25.0 grams. I believe it to be mislabeled, what should I do?" To which I could answer, "Get a more precise scale."

I believe the go-get-more-data approach is appropriate if the initial test is woefully underpowered to detect the magnitude of difference that is practically relevant.