32
$\begingroup$

When I perform a linear regression in some software packages (for example Mathematica), I get p-values associated with the individual parameters in the model. For, instance the results of a linear regression that produces a result $ax+b$ will have a p-value associated with $a$ and one with $b$.

  1. What do these p-values mean individually about those parameters?

  2. Is there a general way to compute parameters for any regression model?

  3. Can the p-value associated with each parameter be combined into a p-value for the whole model?

To keep this question mathematical in nature, I am seeking only the interpretation of p-values in terms of probabilities.

Improve this question
$\endgroup$
16
  • $\begingroup$ Gavin's answer in the question @cardinal linked to says it well. $\endgroup$
    – J. M. is not a statistician
    Commented Aug 29, 2011 at 1:26
  • 6
    $\begingroup$ @zyx, there is nothing advanced about the OP's questions. These are very common questions for which, in my opinion, stats.SE is more appropriate---and to which the participants there are more attuned to, as well. Math.SE and MO are both excellent resources for probability questions, but much less so for statistical ones. The OP's questions lean much more toward the latter. $\endgroup$
    – cardinal
    Commented Aug 29, 2011 at 1:36
  • $\begingroup$ @cardinal: I've followed stats.SE since the start of the public beta. Out of 4800+ questions to date I was not able to locate one that asks or answers item 3 from the OP, which is odd if this is a "very common" query. Nor have I seen conceptually precise answers to item 1 on the few times it came up. I think these things should be posted to math.SE and MO periodically to engage the attention of a larger audience, not migrated within minutes to stats.SE. It doesn't hurt to also ask on stat.SE but turning the latter into the sole place where stats can be discussed is not helpful. $\endgroup$
    – zyx
    Commented Aug 29, 2011 at 2:21
  • $\begingroup$ There is now a thread about math.SE to stats.SE migrations in meta.math.SE. $\endgroup$
    – zyx
    Commented Aug 29, 2011 at 2:42
  • $\begingroup$ (Some comments referenced above were lost in migration. They are visible at the original math.SE posting, linked below next to the words "migrated from...") $\endgroup$
    – zyx
    Commented Aug 29, 2011 at 6:43

2 Answers 2

Reset to default
15
$\begingroup$

  1. The p-value for $a$ is the p-value in a test of the hypothesis "$\alpha = 0$" (usually a 2-sided $t$-test). The p-value for $b$ is the p-value in a test of the hypothesis "$\beta = 0$" (also usually a 2-sided $t$-test) and likewise for any other coefficients in the regression. The probability models for these tests are determined by the one assumed in the linear regression model. For least-squares linear regression, the pair ($a,b$) follows a bivariate normal distribution centered on the true parameter values ($\alpha, \beta$), and the hypothesis test for each coefficient is equivalent to $t$-testing whether $\alpha = 0$ (resp. $\beta=0$) based on samples from a suitable normal distribution [of one variable, i.e., the distribution of $a$ or $b$ alone]. The details of which normal distributions appear are somewhat complicated and involve "degrees of freedom" and "hat matrices" (based on the notation $\hat{A}$ for some of the matrices that constantly appear in the theory of OLS regression).

  2. Yes. Usually it is done (and defined) by Maximum Likelihood Estimation. For OLS linear regression and a small number of other models there are exact formulas for estimating the parameters from the data. For more general regressions the solutions are iterative and numerical in nature.

  3. Not directly. A p-value is calculated separately for a test of the whole model, that is, a test of the hypothesis that all the coefficients (of the variables presumed to actually vary, so not including the coefficient of the "constant term" if there is one). But this p-value cannot usually be calculated from knowledge of the p-values of the coefficients.

Improve this answer
$\endgroup$
12
  • 2
    $\begingroup$ In your point (1.) there seems to be a bit of confusion between a parameter and an estimator. The $p$-value is associated with the estimator rather than the parameter and the estimators are bivariate normal, not the parameters (which, at least, in classical statistics are considered fixed). Also, your comments in point (3.) can lead to confusion since it is entirely possible (and quite common) for some of the individual $p$-values of regression estimates to be both larger and smaller than the joint $p$-value from the corresponding $F$-test. $\endgroup$
    – cardinal
    Commented Aug 29, 2011 at 12:22
  • $\begingroup$ @NRH: Sorry, Can you clarify your previous comment. I don't quite follow it (yet). :) $\endgroup$
    – cardinal
    Commented Aug 29, 2011 at 12:42
  • $\begingroup$ @cardinal: it seems more accurate to say that a p-value is associated to a hypothesis test. The parameters appear in the null hypothesis of the test and the pair (observed value of estimator, alternative hypothesis) then determine a p-value. The null hypotheses should be described using parameters, such as α=0 rather than estimators a=0 as was [carelessly] done in the original answer, now edited (thanks for pointing out the error). However, the supposedly confused or missing distinction "the estimators are bivariate normal, not the parameters" was stated explicitly in the answer. $\endgroup$
    – zyx
    Commented Aug 29, 2011 at 13:26
  • 1
    $\begingroup$ Sorry, I just couldn't resist. @zyx made a comment to the original post on math.SE that answers on stat.SE were often imprecise. I find that many answers are quite accurate though sometimes mathematical imprecise. That is in the nature of things. Statistical questions and answers can not always be reduced to precise mathematical statements. In particularly not the difficult ones. Yet the answer provided here is neither particularly accurate nor precise in my opinion. $\endgroup$
    – NRH
    Commented Aug 29, 2011 at 13:28
  • 3
    $\begingroup$ I think it would be nice if whoever downvoted supplied an explanatory comment. $\endgroup$
    – cardinal
    Commented Aug 29, 2011 at 14:28
1
$\begingroup$

wrt your first question: this depends on your software of choice. There are really two types of p-values that are used frequently in these scenarios, both typically based upon likelihood ratio tests (there are others but these are typically equivalent or at least differ little in their results).

It is important to realize that all of these p-values are conditional on (part of) the rest of the parameters. That means: Assuming (some of) the other parameter estimates are correct, you test whether or not the coefficient for a parameter is zero. Typically, the null hypothesis for these tests is that the coefficient is zero, so if you have a small p-value, it means (conditionally on the value of the other coefficients) that the coefficient itself is unlikely to be zero.

Type I tests test for the zeroness of each coefficient conditionally on the value of the coefficients that come before it in the model (left to right). Type III tests (marginal tests), test for the zeroness of each coefficient conditional on the value of all other coefficients.

Different tools present different p-values as the default, although typically you have ways of obtaining both. If you don't have a reason outside of statistics to include the parameters in some order, you will generally be interested in the type III test results.

Finally (relating more to your last question), with a likelihood ratio test you can always create a test for any set of coefficients conditional on the rest. This is the way to go if you want to test for multiple coefficients being zero at the same time (otherwise you run into some nasty multiple testing issues).

Improve this answer
$\endgroup$
7
  • $\begingroup$ Could you please elaborate on the conditionality you mentioned? In the univariate regression with $p$ predictors and an intercept, testing a hypothesis on a linear combination of parameters $\psi = c'\beta$ uses test statistic $t = \frac{\hat{\psi} - \psi_0}{\hat{\sigma} \sqrt{c' (X' X)^{-1} c}}$... $\endgroup$
    – caracal
    Commented Aug 29, 2011 at 14:09
  • $\begingroup$ Here $\hat{\psi} = c'\hat{\beta}$, with $\hat{\beta}$ being the vector of parameter estimates, and $c$ a vector of coefficients. $X$ is the design matrix, and $\hat{\sigma}$ is the residual standard error $||e||^2 / (n - (p+1))$, where $e$ is the vector of residuals from the supplied model. For the test of a single parameter $j$ being 0, $c$ is the $j$-th unit vector, and $\psi_0 = 0$. I don't see where model comparisons play a roll for $t$. $\endgroup$
    – caracal
    Commented Aug 29, 2011 at 14:09
  • $\begingroup$ The essence of the matter is captured for example here. Remember that anova is just a special case of regression. Basically, it comes down to this: if you do a test for zeroness of (the coefficient of) variable A in a model with or without variable B, you may get different results. Hence, the result is conditional on your model, the data (even for the values of variable B) and thus on the coefficients not in your test but in your model. Finding that idea in the maths may be somewhat harder :-) $\endgroup$
    – Nick Sabbe
    Commented Aug 29, 2011 at 14:58
  • $\begingroup$ True, but the anova hypotheses test whether all $p-1$ effect parameters corresponding to the $p$ groups of a factor are simultaneously 0. This hypothesis is different from the one about $c'\beta$ (here a single parameter $\beta_j$), and also uses a different test statistic: $F = \frac{(SS_{er} - SS_{eu}) / (df_{er} - df_{eu})}{SS_{eu} / df_{eu}}$ where $SS_{er}$ and $df_{er}$ are the residual sum of squares $||e_r||^2$ and their df for the restricted model, likewise $u$ for the unrestricted model. Obviously, this indeed depends on the choice for the restricted and unrestricted models. $\endgroup$
    – caracal
    Commented Aug 29, 2011 at 15:18
  • $\begingroup$ The continuous case should be completely equivalent to a dichotomous 0-1 encoded variable. $\endgroup$
    – Nick Sabbe
    Commented Aug 29, 2011 at 15:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.