-4
$\begingroup$

P-values denote statistical significance. Real world data is claims data or EMR data or any data that is not clinical trial data basically. Now, there is a world view that p-values do not hold merit in the real world scenario. Is there a resource/article/blog post that supports the viewpoint that Yes p-values are important to be reported in real world data studies.

I have googled enough and hit roadblock. I am hoping experienced statisticians may have their insights to share at the least if not a resource.

Thanks!

$\endgroup$
13
  • 2
    $\begingroup$ Since there are literally millions of such "real world" studies that do report p-values, I think I must be misreading your question, because you cannot possibly be suggesting that all these studies are reporting "unimportant" results. But how else are we supposed to understand your question? $\endgroup$
    – whuber
    Sep 4, 2018 at 21:07
  • 1
    $\begingroup$ @Kon Your question should have some clarification; it may help to focus your question to ask something more specific about p-values. At the least you would need to address a few points: People have said many different things about p-values. Please provide some kind of context for your remarks. What are the statements or claims being made that prompt your question (i.e. what do they actually say?); (ii) are you familiar with the definition of a p-value? $\endgroup$
    – Glen_b
    Sep 4, 2018 at 23:56
  • 2
    $\begingroup$ "Real world data is... any data that is not clinical trial data basically." Wow. I suggest sorting out this definition before any discussion is possible. $\endgroup$
    – juod
    Sep 5, 2018 at 1:19
  • $\begingroup$ @Juod: Clinical trial data is from studies that are conducted in controlled settings..real world data is not any such controlled settings. That is why it is called: Real World data. I am curious as to what is wrong with my layman termed definition.. $\endgroup$ Sep 5, 2018 at 16:31
  • 1
    $\begingroup$ What counts as *necessary*/Who decides necessity? For example, (i) If a journal requires p-values in reporting statistical results, is that necessity? (ii) if the FDA required a p-value, would that count as necessary for your purpose? On the other hand, if I am just doing prediction I don't need p-values at all. If I'm working in a Bayesian framework, I don't need p-values for anything. If I work in a pure Neyman-Pearson framework, one can reject a null, or fail to reject it, without ever calculating a p-value (though this might avoid none of the problems). Do these imply a lack of necessity? $\endgroup$
    – Glen_b
    Sep 5, 2018 at 22:30

2 Answers 2

1
$\begingroup$

Not sure if this answers your question but its possible for p-values to be less meaningful as indicators of signal in large data. Sample sizes have an inverse relationship with p-values, so as sample sizes get big p-values get small. When conducting inference on very large samples, you may very well get a tiny p-value suggesting a difference between two groups for your outcome measure, though the effect size may be irrelevant for any practical purpose. EMR data can be very large, so its possible you may see some tiny p-values in this setting detecting differences between groups that are practically meaningless. Again not sure if this helps answer your question but it is something to be aware of when working with large data.

$\endgroup$
0
$\begingroup$

Okay I will take a crack at the revised questions: "What are the scenarios that deem necessity of reporting a p-value. Where it is mandatory to report p-value. Why is it necessary to report p-value. Why are they important. Why do we care?"

Short answers: No empirical scenario requires it. In practice, certain journals and fields of research mandate the reporting of p-values for publication/credibility. It is not necessary to report a p-value. I do not think we really should care about them at all.

Now for some longer answers.

There is not a methodology in which p-values are required for results to be important. It doesn't matter whether you are running experiments with subatomic particles, using survey data, or running a field experiment. None of these scenarios require p-values for you to estimate effects. As scientists, we care about estimation and prediction no matter the methodology, and p-values do not help with either (more on this below).

As was stated in the comments, p-values are terribly misunderstood, even by experts. They are used inappropriately as a simple heuristic to gauge whether or not your results are meaningful. While this is quite short-sighted, it certainly helps coordinate expectations and reduces the number of papers viable for publication (making the lives of the editors easier). This is why many academic journals and fields effectively mandate them to be present.

The last three questions are all closely related. Let's start with the definition of a p-value in the context of a simple linear regression. The p-value tells the researcher how likely it is that he observed some data under the assumption that the null hypothesis is true. So what does having statistical significance actually mean? It means that it is unlikely that we observed the given data if the null hypothesis is true. Note, this does NOT tell us that the null model is more or less likely than the alternative model! This is a very common mistake. P-values are not valid metrics for model comparisons and can only be used to interpret the likelihood of observing specific data sets. With this information in mind, it follows that p-values are also unable to make claims about the predictive capabilities of the null relative to the alternative. Finally, for completeness, just note the obvious fact that p-values do not help in obtaining estimates of the magnitude or spread of the magnitude.

In summary, the p-value doesn't help us with estimation or model selection. More often than not, researchers misuse p-values, believing them to be some indication of the null hypothesis being more or less likely. In my experience, people do not have any great argument for their use other than it is simply the procedure of a certain field. So it seems to me that the answer to your last three questions is this: we should not care about p-values because they really do not tell us anything.

Let me drive home this point with one last p-value fact that is often overlooked. If you have enough data, you will obtain significance. This is due to the (somewhat silly) nature of the null hypothesis. That is, the null assumes that the effect size is exactly zero. In almost all cases, this is certainly untrue. There will be some effect, just a small one. So what a p-value does in practice is set up a straw-man requirement that can always be met provided you have enough data. This fact should give any researcher who uses p-values pause. Intuitively, it suggests that p-values are not a useful metric for anything because they are simply a function of the size of the data.

Scientists are interested in estimation and prediction, and p-values do not help with either. If you are interested in model comparison, you should use something like AIC or BIC. You could also leave the frequentist framework all together and use Bayesian methods to compute Bayes Factors (which are approximated by BIC).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.