# When not to look at p-value

I am trying to indentify correlations between the columns of a dataset I have. I am using the pearson correlation index, and am looking at the p-value I get from the "correlation test" of R. I know that, in theory, the p-value is the probability of the correlation being due to chance. Now, what I wanted to know, can I always look at this value to have a good metric of how certain the relation between the variables is, or are there any assumptions that have to be statisfied?

## EDIT

I'll add some context in order to better explain the question

I am actually studying the relationship between some variables I extracted from a tv series (eg number of character, number of cuts, IMDB rating etc.) and wanted to see if there is any kind of relation between them. In order to do this, I thought correlation might be a nice idea. Some variables have got interesting correlations, with very little p-values, and reading online it seemed like a good idea looking at this value.

After the comments I am not sure I really understood what this value means, but reading online I found this: "The p-value is the probability that the null hypothesis is true. In our case, it represents the probability that the correlation between x and y in the sample data occurred by chance" (source).

• " I know that, in theory, the p-value is the probability of the correlation being due to chance." This is not true. The p-value is the probability under the assumption that there is no correlation that a correlation as large or larger (in absolute value) than the one actually observed will occur in data. Jun 18 at 10:21
• What kind of "relation" between variables are you interested in? There are certain nonlinear relations that are not connected to correlation at all. Jun 18 at 10:23
• Use the p-value to see whether your observed correlations are compatible with the idea that there is no relation at all (in case of insignificance). This, however, doesn't mean that indeed there is no relation. Jun 18 at 10:24
• I think what Jonathan meant by the p-value being the probability of a correlation due to chance is that the null of the test is that there is correlation. So if we reject the null, i.e. the p-value is small, and there still appears to be some sort of correlation, we can infer that this observed correlation is due to chance and not actually in the data, otherwise we shouldn't be able to reject the null. So, while this way of looking at the p-value is not generally a good way, it might be true here specifically.
– DLTS
Jun 18 at 10:48
• The explanation of p-values in that source is incorrect. Unfortunate, but not that surprising. This thread and wikipedia's article both contain more accurate descriptions, which Christian has already summarized. In this case the p-value is the probability that, if there truly were no correlation, we happen to see a correlation at least this large just by random chance. I'm afraid I don't understand @DLTS 's comment Jun 18 at 13:48

If you are looking for correlations, your first look should not be at the p-value, but at the value of the correlation coefficient $$r$$. High correlation means high $$|r|$$, not high p-value.

The p-value only tells you how likely the observed correlation value is under the assumption that the correlation is exactly zero. In a situation that has a correlation different from zero (which is the case in almost all situations), you can make the p-value arbitrarily small simply by collecting more data, however small the correlation is.

The R function cor.test also provides a confidence interval for the correlation coefficient:

> cor.test(faithful$$waiting, faithful$$eruptions)
[...]
95 percent confidence interval:
0.8756964 0.9210652


This is more informative than merely a p-value because it includes the information about the actual value of the correlation.

Before you look at anything at all you should have a clear goal or research question. You need a reason why you want to look at the correlations. What are you gonna do with it?

For instance,

1. Do you want to make some tv-show yourself and see what parameters can influence the success? Then you have to deal with the problem of spurious relationships.

2. Do you just want to make a statistical model, to predict the success of a show based on some features? (in this case the question about causality is not important as in the first case) Then you need to take care that you are not overfitting. You are looking at multiple features this brings along the problem of multiple comparisons. Along with using correction formulas you can also use a cross validation method to find an optimal model (and you don't care about the p-value bit you care about the best model).

3. Do you want to do hypothesis testing? Then another issue is that multivariate tests can be made more powerful when you use a linear model instead of seperate correlations. Individual correlations can be small while a combined model is very significant (See for an explanation in this question which is about the opposite case Can ANOVA be significant when none of the pairwise t-tests is? ) Also some variable may become significant when it is corrected for another variable (related is When is MANOVA most useful ).

can I always look at this value to have a good metric of how certain the relation between the variables is, or are there any assumptions that have to be statisfied?

As described in the above three points there are situations where you want to look at it differently.

But if correlations are what you after, then you also need to consider the distribution of the data points. Asside from independence of seperate points (I don't think this is true for the IMDB database and there might be all sort of bias in sampling), it is also important that the distribution is normal distributed (if you use the standard computation of the p-value). The Kendall rank correlation coefficient and associated tests is an alternative.

It sounds like you might want to do exploratory data analysis instead of a deep dive into the meaning of p-values. Basically, make plots, lots of them, to understand the patterns in your data.

It might help to keep in mind to following advice from [1]:

Quick tips to improve your [regression] modeling

1. Think about variation and replication (ie., patterns you observe in one dataset won't necessarily replicate in another dataset about the same phenomenon)
2. Forget about statistical significance and p-values
3. Graph the relevant and not the irrelevant; make many graphs

There are 7 more steps (see below) but the first 3 steps are most relevant to you since you seem to be at the start of your investigation.

Notes

• The Pearson's correlation coefficient measures strength and direction of a linear relationship. So by choosing Pearson's correlation out of hand you've implicitly decided to see if there is a linear relationship between the variables. Again, plotting the data might turn out to be more informative than computing correlations.
• You plan to run multiple statistical tests with the goal of discovering interesting hypotheses. It's important to correct for doing all those tests because each test has a probability of making a false discovery (type I error). The probability of making false discoveries grows quickly with multiple testing. This is an advanced topic. You can learn more about it here.

Quick tips to improve your [regression] modeling, continued

1. Interpret regression coefficients as comparisons
2. Understand statistical methods using simulations
3. Fit many models
4. Set up a computational workflow
5. Use transformations
6. Do causal inference in a targeted way, not as a byproduct of a large regression
7. Learn methods through live examples

[1] A. Gelman, J. Hill, and A. Vehtari. Regression and Other Stories. Cambridge University Press, 2020.

• (+1) for point 2., although the whole point of introducing p-values was to take care of point 1. in an automated way ;-) Jun 21 at 14:38
• @cdalitz Where is the fun in automating EDA? At the very early exploratory stage, it tends to be hard to report p-values with a straight face. Either the p-values are not corrected for multiple testing at all, or they are corrected only in the reported analysis but other (unreported) analyses were done as well. Jun 21 at 15:10
• Sorry, I did not want to start a hair splitting thread... Jun 21 at 15:14

Documentation forcor.test in R says "By default, probabilities for method="spearman" and method="kendall" are found by normal theory. If normal=="FALSE", then repetitive calls are made to cor.test. ..."

What attention have you given to normality of data based on "(number of characters, number of cuts, IMDB rating etc.)"?

From what is said in your Question and in some of the Comments, I would wonder which P-values should be taken seriously which should not.

• I'm pretty sure "normal theory" there refers to the usual normal approximation of the null distribution of those (nonparametric) test statistics. Those null distributions and their approximations don't rely on normality for the original variables (only on continuity, usually, but for the normal approximation there's also a variance adjustment for ties) Jun 20 at 2:29