The trade-off between p-value and sample size

Question

This question stems from an existing question, in which I tried to compute the p-value of some features over a large dataset (2,000,000 articles and 15-20 topics). Each article must belong to only one topic. I want to investigate whether articles with different topic have different levels of reading time. For example, whether users tend to spend more time on business articles than entertainment articles.

I found that when I included the whole population, the p-values were extremely low. Decreasing sample size led to higher p-values. So my question is (if I stick to use p-value) how large the sample size is better to be in my case? 200 articles in one topic?

Answer 1

I'm afraid statistics is not done that way. Sample size calculation involves three groups of component:

1. The designated chance of calling a statistical significance while in fact there isn't any, known as type I error. It's conventionally set at 5% and it's related to the p-value < 0.05 threshold. Additionally, the other is the designated chance of missing a statistical significance while in fact there is one, known as type II error. It's conventionally set at 20%. The chance of not committing a type II error is called "power" and if type II error rate is 20%, power is 80%.

2. The second component is called the "effect size." It comes in many shapes and forms but generally it's the strength of association or magnitude of difference you wish to use the statistics to show if such phenomenon is likely a fluke or not.

3. The third component is the sample size.

In order to make a decision on any one of these three, the other two need to be known (or assumed). So, generally a question on sample size would be something like:

How many samples do I need to detect a difference of 10 units or more (with a SD of 7.5 units) in total blood cholesterol between the mice fed with low-fat diet and the mice fed with control diet, with type I error rate set at 5% and power at 80%? And I'd like the ratio of test to control be 1:1.

Then statisticians would be able to help you. Currently, your question does not have effect sizes. (What does "evaluate the significance of each topic" actually measure? Do you mean chance of being downloaded? Viewed? Cited? Funded?) And it does not have the declared error rates. So, it's unlikely for anyone to come up with a sample size.

Answer 2

If you "include the whole population" then it is arguable as to whether you need or should use p-values at all. p-values are about inference. Inference goes from a sample to a population. If you have the whole population then I (and others) would argue that there is no inference to be done.

p-values on large samples are, as you've found, often small. You need to estimate the effect size.