Big Sample size, Small coefficients, significant results. What should I do?

Question

I did some quantitative research and I used Rank-Order logistic regression in Stata. The the independent variables have almost 0 p-value which shows they have significant effect on dependent variable. But, since the sample size is big (35000 records) and coefficients are so small (e.g. 0.0001) then it shows that there is no relationship because when sample size is so big everything can get significant. I tested the model with only 5000 records as well and I got the significant result as well. What do you recommend me to do? should I use small sample size then the reviewers of my paper will not point to the problem of big sample size... or is there any other way that I can report my results and show that in fact the variables have significant effect? I will appreciate any help. Thanks

Answer 1

I think it's been asked before. It's useful to realize that, without a prespecified sample size and alpha level, the $p$-value is just a measure of the sample size you ultimately wind up with. Not appealing. An approach I use is this: at what sample size would a 0.05 level be appropriate? Scale accordingly. For instance, I feel the 0.05 level is often suited to problems where there are 100 observations. That is: I would say WOW that is an interesting finding if it had a 1/20 chance of being a false positive. So if you had a sample size of 5,000, that's 50 times larger than 100. So divide your 0.05 level by 50 and come up with 0.001 as a significance level. This is in line with what Fisher advocated: don't do significance testing with p-values, compare them to the power of the study. The sample size is the simplest/rawest measure of the study's power. An overpowered study with a conventional 0.05 cut off makes absolutely no sense.

Usually, it is never advisable to choose a significance cut-off after viewing data and results. One might believe it might be kosher to arbitrarily choose a more stringent significance criterion post hoc. Actually, it only deceives readers into thinking you ran a better controlled trial than you did. Think of it this way: if you observed p = 0.04, you wouldn't be asking this question; the analysis would be a tidy inferential package.

Another way to look at it is this: just report the CI and that the analysis was statistically significant. For instance, you might have a 95% CI for a hazard ratio that goes from (0.01, 0.16) - the null is 1. It suffices to say that the p-value is really freakin' small, so you don't need to clutter the page displaying p=0.0000000023 (don't do this... only show p out to its precision, if 3 decimal places show p < 0.001 and never round to 0.000 - that shows you don't know what a p-value means.).

Answer 2

You have encountered the gulf between "statistically significant" and "meaningful". As you point out, with sufficient sample size, you can assign statistical significance to arbitrarily small differences - there is no difference too small that can't be called "significant" with large enough N. You need to use domain knowledge to determine what is a "meaningful" difference. You might find, for example, that a new drug increases a person's lifespan by 10 seconds - even though you can be very confident that that increase is not due to random variation in your data, it's hardly a meaningful increase in lifespan.

Some of this will come from knowing about your problem and what people in the field consider meaningful. You could also try to think of future studies that might replicate your findings, and the typical N that they might use. If future studies will likely have a much lower N, you could calculate the effect size needed to replicate your findings in data of that size, and only report significant, meaningful, and feasibly reproducible results.

Answer 3

When you have many samples and the observed effect is very small (small for the specified application), you can safely conclude that the independent variables do not have an important effect on the dependent variable. The effect size can be “statistically significant” and unimportant at the same time.

Using small sample size and ignoring the results from the large samples is inappropriate. You owe that to the people that read your paper and design some new experiments based on your observations.

Answer 4

I think you should decide on an "expected minimal effect size", i.e. the minimal coefficients you care to include in your model. Say, do you care about coefficients less than 0.0001, or 1, or 100? To clarify, the effect size is the degree to which the null hypothesis is false, or how large the coefficient actually is. It's a parameter of the population. On the other hand, the expected minimal effect size is the minimal amount of departure from the null you care to detect. It's a parameter of the test.

Now that you have the sample size $N = 35000$, as well as some expected minimal effect size, a power analysis should reveal the relationship between $\alpha$ and $\beta$ given there parameters. Next, make another decision about how to balance your significance level and power by choosing a pair of $\alpha$ and $\beta$. (Technically, all these parameters must be decided before looking at the data, but at this point, I guess you can just pretend you didn't see them.) Then, carry out your test, compare $p$ with $\alpha$, and draw a conclusion accordingly.

By the way, I believe there are no reasons to exclude any record, unless you are doing cross-validation, for example. More data generally leads to more accurate inference, and additionally, discarding sample points in a selective manner may introduce bias.