I'm currently reading a paper which utilises a multiple regression and reports the adjusted $R^2$ with a p-value, and I'm wondering what this p-value refers to. Can you calculate p-values for adjusted $R^2$? It separately reports the p-values of the t-statistics for each exogenous variable so it can't be t-statistics.
As POC notes, it might be possible to obtain a $p$ value through bootstrapping. Alternatively, one could run a permutation test. However, if the authors of the paper did something like this, one would expect that they noted it in the text.
What is indeed common is reporting the $p$ value of an $F$ test for the ANOVA between a given model and an intercept-only model. It may well be that the authors' software reported the $F$ statistic (with its degrees of freedom), the $R^2$ and the $p$ value for the $F$ test, and the authors only reported $R^2$ and $p$.
Alternatively, given the relationships between $R^2$, sums of squared residuals, and $F$ statistics as given in the other answers, it appears to be straightforward to translate back and forth between all these statistics, and their associated $p$ values. It's just quite uncommon to report $R^2$ and its $p$ value, and people usually rather report $F$ statistics and their $p$ values.
What is a little more concerning to me is the "power" column, which I fear refers to post hoc power, which is problematic. It simulates understanding we do not have, because if a significant effect was found, then the study was by definition sufficiently powerful to detect this effect as significant. Post hoc power reasons in a circle. The proper place for power calculations is in planning a study and its sample size, using effect sizes "we would be sorry to miss". On post hoc power, I like to recommend Hoenig & Heisey (2001).
$R^2$ and adjusted $R^2$ are computed based on a decomposition of the sum of squares, $SS_{total} = SS_{model}+SS_{residual}$, and can be related to F-statistics in ANOVA which is based on the same sum of squares values.
For simple linear regression an example is given here: Is there a relation between the p-values of coefficients and the $R^2$ in an OLS regression?
It separately reports the p-values of the t-statistics for each exogenous variable so it can't be t-statistics.
The models contain several variables for which you can seperately compute p-values, by computing t-values, or computing ANOVA by comparing with a model that drops a single factor (unless the factor being a categorical variable, this should give the same result).
The p-value for the entire model is an ANOVA comparison between a null model and the entire model. In such a case $R^2$ can be related to the F statistic and it's p-value in a similar way as the simple linear regression linked above.
(In the case of the 'log-dimensionless jerk' the model is actually just a simple linear regression and the 'p-value of the parameter', is the same as the 'p-value of the entire model'.)
Example computation in R
### generate some data
x1 = c(1,5,3,7,2,5,4,9,0)
x2 = c(1,1,1,1,0,0,0,0,0)
set.seed(1)
noise = rnorm(9)
### simulate a measurement
y = 2 + x1 + x2 + noise
### compute two models
mod_full = lm(y ~ 1 + x1 + x2) ## intercept plus both variables
mod_null = lm(y ~ 1) ## only intercept
### compare the full model with an intercept only model
anova(mod_full,mod_null)
# p-value = 0.0001191
### provide p-values of individual variables
summary(mod_full)
# x1 p-value = 3.99e-05
# x2 p-value = 0.2013
The p-values for the variables are evaluated with a t-test and result in
Call:
lm(formula = y ~ 1 + x1 + x2)
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 1.8019 0.5646 3.191 0.0188 *
x1 1.1151 0.1045 10.672 3.99e-05 ***
x2 0.8171 0.5694 1.435 0.2013
The p-value for the model is evaluated with an ANOVA test and results in
Model 1: y ~ 1 + x1 + x2
Model 2: y ~ 1
Res.Df RSS Df Sum of Sq F Pr(>F)
1 6 4.323
2 8 87.867 -2 -83.544 57.976 0.0001191
car::Anova(mod_full, type = "III").
$\endgroup$
Commented
Mar 22 at 8:57
The adjusted coefficient of determination is a monotonically increasing function of the F-statistic, so both statistics give the same evidentiary ordering in a goodness-of-fit test. Specifically, it can be shown that:
$$\begin{align} F &= \frac{df_\text{Res}}{df_\text{Reg}} \cdot \frac{R^2}{1-R^2} \\[6pt] &= \frac{df_\text{Res}}{df_\text{Reg}} \cdot \frac{(n-1) - (n-p-1)(1-R_\text{Adj}^2)}{(n-p-1)(1-R_\text{Adj}^2)} \\[6pt] &= \frac{df_\text{Res}}{df_\text{Reg}} \cdot \frac{p + (n-p-1)R_\text{Adj}^2}{(n-p-1) (1-R_\text{Adj}^2)}, \\[6pt] \end{align}$$
which gives:
$$R_\text{Adj}^2 = \frac{(n-p-1) df_\text{Reg} \cdot F - p \cdot df_\text{Res}}{(n-p-1) df_\text{Reg} \cdot F + (n-p-1) df_\text{Res}}.$$
Since this is a monotonically increasing function of $F$, the goodness-of-fit test for the $R_\text{Adj}^2$ statistic ought to be identical to the standard F-test.
Assuming the authors used R (I did not read the paper!), they likely used the cor.test function with default parameters to obtain those numbers. Here is the relevant section from the online help:
If ‘method’ is ‘"pearson"’, the test statistic is based on Pearson's product moment correlation coefficient ‘cor(x, y)’ and follows a t distribution with ‘length(x)-2’ degrees of freedom if the samples follow independent normal distributions. If there are at least 4 complete pairs of observation, an asymptotic confidence interval is given based on Fisher's Z transform.
