1
$\begingroup$

From my perspective, the reason p-value of a statistical test isn't useful in large sample scenario is because it will change according to the scale.

E.g. let's focus on chi-square test. In a chi-square test, the chi-square score changes linearly w.r.t. the scale of the frequencies (I mean, multiply 10 for each cell in the contingency table, the chi-square will also be 10 times larger). So when the data size is larger, even it has the same ratio/proportion, the p-value becomes smaller.

Thus, in this sense, why don't we just set a standard value, e.g. 1000 or 10000, and do a linear transform to normalize the total sample size to this value before we do the chi-square test. Then I suppose the p-value makes sense again. And scores like Cramer's V or Cohen's h or odds ratio remains the same.

What's wrong with my idea? (I don't know why the number in the table must be counts for chi-square. If this is a problem, maybe think of other tests like ANOVA)

Any idea/thought is appreciated:)

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

The point is that the sample size is actually informative. If you have the same relative departures from row-column independence, but a 10 times larger sample size, you have much more evidence against the null hypothesis of row-column independence. This is rightly reflected in your $\chi^2$ statistic and corresponding p-value.

Artificially changing the sample size as you propose would incorrectly change the certainty on the estimates.

Finally, the entries in the contingency table are counts by construction. The fact that you have counts yields an "automatic" assumption that the variance equals the mean, needed to standardize the residuals in the calculation of the $\chi^2$ statistic.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks! I think it's a very good point. It's definitely invalid to normalize when data size is small. The sample size is very informative in this case. But is it still the same when the sample is already large enough? For large sample size, the problem is actually the significance indicated by p-value is exaggerated. I don't think it makes a huge difference between 10,000 and 100,000 samples, I guess, the growth of information from the sample size will decrease as the sample size becomes larger. Maybe once it's larger than certain threshold, the sample size doesn't matter that much. $\endgroup$
    – G. Yu
    Aug 29, 2018 at 14:46
  • $\begingroup$ My point is: the information we get from sample size is not a linear function of the sample size. In classical statistical measure tools, e.g. Cramer's V for chi-square, it normalized the chi-squared score by number of samples, which (I think) assumes a linear relationship between information gain and sample size. But I believe it's invalid for large sample. Is the information difference between 10 and 100 samples the same as 100,000 and 1,000,000 samples? Maybe not. Finding the real relationship and normalizing correspondingly provides a way to better use these statistical tools for big data. $\endgroup$
    – G. Yu
    Aug 29, 2018 at 15:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.