3
$\begingroup$

I come from a primarily Bayesian background when using performing statistical analysis. In the context of linear regression, I would look at the posterior distributions for each regression coefficient. If a given coefficient was distributed ~N(1,1), I might not be terribly convinced of the associated variable's influence on the dependent variable.

From a frequentist perspective, regression coefficients often have p-values assigned. And I'm not quite sure what to make of this coming from a bayesian, full posterior interpretation. Is the idea that there's only an x% chance that we would have observed this specific regression coefficient inferred by accident?

$\endgroup$
2

3 Answers 3

19
$\begingroup$

A p value is the probability of observing a test statistic as or more extreme than the researcher's own test statistic, assuming the null hypothesis, and an assumed distribution model are both true.

So when you see frequentist statistical software provide regression output that includes something like:

name estimate S.E. t (or z) p value
cons 3.0 0.015 200 <0.001
beta 0.5 0.18 2.78 0.003

The default null hypothesis for regression coefficients in most stats software I am familiar with is $\text{H}_{0}\text{: }\beta = 0$, with $\text{H}_{\text{A}}\text{: }\beta \ne 0$. You should read the $p = 0.003$ as $p = P(|T_{\text{df}}|\ge |t|)$ given $\text{H}_{0}$, or in plain language: the probability of observing a value of $\widehat{\beta}$ as or more extreme than 0.5 is 0.003, assuming the null hypothesis is true (and assuming Student's t distribution truly describes the distribution of your test statistic). If a priori you had an $\alpha =0.01$, then you would interpret that p value as evidence against the null hypothesis, and for the alternative hypothesis (i.e. you found evidence that $\beta \ne 0$).

$\endgroup$
6
  • 1
    $\begingroup$ Isn't the distribution model part of the null hypothesis? A complete null hypothesis would be something like "All the variables have IID normal distributions" (if the means are identical, then $\beta=0$, and if the SD are the same, then $\hat \beta$ will have a Student's t distribution). $\endgroup$ Aug 19, 2021 at 2:54
  • 1
    $\begingroup$ I think the null hypothesis and the assumptions of the test are typically considered to be two different things. That's certainly the way I think about it, & I've never seen a book that introduced the null as, say, "$H_0: \mu_1 = \mu_2\ \&\ \mu_2-\mu_1 \sim \mathcal N(0, \sigma^2_{\mu_2-\mu_1})$". $\endgroup$ Aug 19, 2021 at 13:21
  • $\begingroup$ @Alexis: I think it makes sense to mention that the $H_0$ for a specific $\beta$ implies that all other variables are still in the model (if there is more than one) and can be nonzero. $\endgroup$ Aug 19, 2021 at 13:44
  • $\begingroup$ @Acccumulation: You are right that the full specification of the $H_0$ would require all other model assumptions (iid normal should hold for error terms, not for "all variables"). However, the test statistic is constructed to specifically distinguish $\beta=0$ from $\beta\neq 0$. Technically the rest of the assumptions go into both $H_0$ and $H_A$. $\endgroup$ Aug 19, 2021 at 13:48
  • $\begingroup$ @Acccumulation I like to think of the distribution model as part of the null hypothesis. However, not all statisticians agree with that (like the lovely gung - Reinstate Monica), so I do not get pushy about it. :) $\endgroup$
    – Alexis
    Aug 19, 2021 at 16:46
0
$\begingroup$

These numbers are unreliable if your predictors are correlated.

Suppose for example that the value to be predicted is the sum of two hidden N(0,1) random variables $Y=Z_1+Z_2$ and $Z_2$ is unobserved, and you have observations of $X_1 = Z_1+N(0, 0.01), X_2 = Z_1+N(0, 0.01)$

Then the linear regression is degenerate. Any solution is equally good, in which the coefficients of $X_1, X_2$ sum to one. No magic with p-values can help you choose.

Perfect degeneracy like this is rare, but some level of correlation is common. Take home lesson: correlation is not causation.

$\endgroup$
4
  • 2
    $\begingroup$ Sorry, how does the “take home lesson” follow from the earlier claims? $\endgroup$ Aug 18, 2021 at 15:50
  • $\begingroup$ "Any solution is equally good, in which the coefficients of X1,X2 sum to one." Do you mean "All solutions in which the coefficient of $X_1$ and $X_2$ sum to one are equally good"? $\endgroup$ Aug 19, 2021 at 2:57
  • 4
    $\begingroup$ @AryaMcCarthy, well, the answer correlates with the take home lesson, but it doesn't cause it. $\endgroup$
    – justhalf
    Aug 19, 2021 at 9:59
  • 1
    $\begingroup$ "These numbers are unreliable if your predictors are correlated." These numbers are perfectly reliable if they are not interpreted as being more than they actually are. They are not meant to be "magic to help choosing". $\endgroup$ Aug 19, 2021 at 13:50
0
$\begingroup$

In most regression software you can also request confidence intervals for the regression coefficients. The confidence interval is a set of plausible true values of the parameter, given the observed data. A 90% confidence interval contains all the hypotheses for the parameter under which the statistic is not significant at the two-sided 10% level. While this does not have the same interpretation as a 90% credible interval, it serves the same purpose. If the 90% confidence interval contains zero (two-sided p-value testing $H_0:\beta=0$ is greater than 0.10) you would not be convinced at the two-sided 10% level of the association between the covariate and the dependent variable. If the 90% confidence interval excludes zero (two-sided p-value testing $H_0:\beta=0$ is less than 0.10) this could be interpreted as evidence (not proof) of an association at the two-sided 10% level (or one-sided 5% level).

If you use something like Proc MCMC you are used to getting kernel density estimates for the posterior distribution of a parameter. You can analogously produce confidence curves using the point estimate and standard error. These curves show p-values and confidence intervals of all levels for hypotheses concerning the parameter.

The upper-tailed p-value for a standard issue Wald test as a function of the hypothesis would take the form

$$H(\beta,\boldsymbol{x})=1-\Phi\bigg(\frac{\hat{\beta}-\beta}{\hat{\text{se}}}\bigg).$$

Analogously, the lower-tailed p-value is $$H^{-}(\beta,\boldsymbol{x})=\Phi\bigg(\frac{\hat{\beta}-\beta}{\hat{\text{se}}}\bigg).$$

The corresponding confidence curve, $C(\beta,\boldsymbol{x})$, can then be defined as

$$C(\beta,\boldsymbol{x})=H(\beta,\boldsymbol{x})\text{, if $\beta < \hat{\beta}$ }$$ $$C(\beta,\boldsymbol{x})=H^{-}(\beta,\boldsymbol{x})\text{, if $\beta > \hat{\beta}$ }$$.

The confidence curve will look like a teepee or a pyramid on the parameter space depicting ex-post sampling probability of the experiment (frequentist confidence).

If, say, you have an estimated beta coefficient of 0.5 and an estimated standard error of 0.18, the figure below depicts the corresponding confidence curve for inference on the unknown fixed true $\beta$.

enter image description here

data norm;
do beta=-0.6 to 1.6 by 0.001;
beta_hat=0.5;
se=0.18;

C_lower=1-cdf('normal',(beta_hat-beta)/se,0,1); if beta gt beta_hat then C_lower=.;
C_upper=cdf('normal',(beta_hat-beta)/se,0,1); if beta lt beta_hat then C_upper=.;

output;
end;
run;

proc sql noprint;
select max(beta)
into: lower
from norm
where . lt C_lower le 0.05;

select min(beta)
into: upper
from norm
where . lt C_upper le 0.05;
quit;

data norm;
set norm;
lower=&lower.;
upper=&upper.;
yscatter=-0.035;
run;

ods escapechar="^";
ods graphics / height=3in width=6in border=no;
proc sgplot data=norm noautolegend;
series x=beta y=C_lower / lineattrs=(color=darkblue) name="cc" legendlabel="Confidence curve (one-sided p-value) with 90% confidence interval";
series x=beta y=C_upper / lineattrs=(color=darkblue);
scatter x=beta_hat y=yscatter / xerrorlower=lower xerrorupper=upper errorbarattrs=(color=darkblue) markerattrs=(color=darkblue);
refline 0 / axis=y;
yaxis max=0.6 label="p-value" min=0 offsetmin=0.1;
xaxis label="^{unicode beta}" min=-0.5 max=1.5;
keylegend "cc";
run;
$\endgroup$
2
  • $\begingroup$ "all the hypotheses for the parameter that are not significant at the two-sided 10% level" Hypotheses do not have significance level. The precise statement is "all the hypotheses under which the statistic is not significant at the two-sided 10% level". $\endgroup$ Aug 19, 2021 at 3:00
  • $\begingroup$ Thank you! Duly noted and corrected. $\endgroup$ Aug 19, 2021 at 13:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.