When not to look at p-value

Question

I am trying to indentify correlations between the columns of a dataset I have. I am using the pearson correlation index, and am looking at the p-value I get from the "correlation test" of R. I know that, in theory, the p-value is the probability of the correlation being due to chance. Now, what I wanted to know, can I always look at this value to have a good metric of how certain the relation between the variables is, or are there any assumptions that have to be statisfied?

EDIT

I'll add some context in order to better explain the question

I am actually studying the relationship between some variables I extracted from a tv series (eg number of character, number of cuts, IMDB rating etc.) and wanted to see if there is any kind of relation between them. In order to do this, I thought correlation might be a nice idea. Some variables have got interesting correlations, with very little p-values, and reading online it seemed like a good idea looking at this value.

After the comments I am not sure I really understood what this value means, but reading online I found this: "The p-value is the probability that the null hypothesis is true. In our case, it represents the probability that the correlation between x and y in the sample data occurred by chance" (source).

Answer 1

Documentation forcor.test in R says "By default, probabilities for method="spearman" and method="kendall" are found by normal theory. If normal=="FALSE", then repetitive calls are made to cor.test. ..."

What attention have you given to normality of data based on "(number of characters, number of cuts, IMDB rating etc.)"?

From what is said in your Question and in some of the Comments, I would wonder which P-values should be taken seriously which should not.

Answer 2

If you are looking for correlations, your first look should not be at the p-value, but at the value of the correlation coefficient $r$. High correlation means high $|r|$, not high p-value.

The p-value only tells you how likely the observed correlation value is under the assumption that the correlation is exactly zero. In a situation that has a correlation different from zero (which is the case in almost all situations), you can make the p-value arbitrarily small simply by collecting more data, however small the correlation is.

The R function cor.test also provides a confidence interval for the correlation coefficient:

> cor.test(faithful$waiting, faithful$eruptions)
[...]
95 percent confidence interval:
 0.8756964 0.9210652

This is more informative than merely a p-value because it includes the information about the actual value of the correlation.

Answer 3

It sounds like you might want to do exploratory data analysis instead of a deep dive into the meaning of p-values. Basically, make plots, lots of them, to understand the patterns in your data.

It might help to keep in mind to following advice from [1]:

Quick tips to improve your [regression] modeling

1. Think about variation and replication (ie., patterns you observe in one dataset won't necessarily replicate in another dataset about the same phenomenon)
2. Forget about statistical significance and p-values
3. Graph the relevant and not the irrelevant; make many graphs

There are 7 more steps (see below) but the first 3 steps are most relevant to you since you seem to be at the start of your investigation.

Notes

• The Pearson's correlation coefficient measures strength and direction of a linear relationship. So by choosing Pearson's correlation out of hand you've implicitly decided to see if there is a linear relationship between the variables. Again, plotting the data might turn out to be more informative than computing correlations.
• You plan to run multiple statistical tests with the goal of discovering interesting hypotheses. It's important to correct for doing all those tests because each test has a probability of making a false discovery (type I error). The probability of making false discoveries grows quickly with multiple testing. This is an advanced topic. You can learn more about it here.

---

Quick tips to improve your [regression] modeling, continued

4. Interpret regression coefficients as comparisons
5. Understand statistical methods using simulations
6. Fit many models
7. Set up a computational workflow
8. Use transformations
9. Do causal inference in a targeted way, not as a byproduct of a large regression
10. Learn methods through live examples

[1] A. Gelman, J. Hill, and A. Vehtari. Regression and Other Stories. Cambridge University Press, 2020.

Answer 4

Before you look at anything at all you should have a clear goal or research question. You need a reason why you want to look at the correlations. What are you gonna do with it?

For instance,

1. Do you want to make some tv-show yourself and see what parameters can influence the success? Then you have to deal with the problem of spurious relationships.

2. Do you just want to make a statistical model, to predict the success of a show based on some features? (in this case the question about causality is not important as in the first case) Then you need to take care that you are not overfitting. You are looking at multiple features this brings along the problem of multiple comparisons. Along with using correction formulas you can also use a cross validation method to find an optimal model (and you don't care about the p-value bit you care about the best model).

3. Do you want to do hypothesis testing? Then another issue is that multivariate tests can be made more powerful when you use a linear model instead of seperate correlations. Individual correlations can be small while a combined model is very significant (See for an explanation in this question which is about the opposite case Can ANOVA be significant when none of the pairwise t-tests is? ) Also some variable may become significant when it is corrected for another variable (related is When is MANOVA most useful ).

can I always look at this value to have a good metric of how certain the relation between the variables is, or are there any assumptions that have to be statisfied?

As described in the above three points there are situations where you want to look at it differently.

But if correlations are what you after, then you also need to consider the distribution of the data points. Asside from independence of seperate points (I don't think this is true for the IMDB database and there might be all sort of bias in sampling), it is also important that the distribution is normal distributed (if you use the standard computation of the p-value). The Kendall rank correlation coefficient and associated tests is an alternative.