14
$\begingroup$

In literature, both terms are often used synonymously or interwoven. I am now trying to find a clear distinction between both terms. From my point of view, a hypothesis is usually expressed via a model. So even if we test a null vs. alternative hypothesis, from my perspective we are doing model selection. Can someone give me an intuitive description of this distinction?

Improve this question
$\endgroup$
1
  • 3
    $\begingroup$ By hovering the mouse over the tags, you can see the wiki excerpts for hypothesis-testing and model-selection. Despite their brevity, they seem to do a good job answering your question. $\endgroup$
    – whuber ♦
    Jan 6, 2015 at 17:31

1 Answer 1

Reset to default
17
$\begingroup$

To me the distinction is that with hypothesis testing one is considering contrasts of model parameters and is not entertaining the thought of changing the model. For example, in ANOVA, people are smart enough not to convert a 4 degree of freedom $F$-test to a 3 d.f. $F$-test when comparing 5 groups and finding that two of the groups have similar means. People who formulating models often make the basic mistake of selecting which parameters should be in the model on the basis of statistical tests/comparisons, not realizing that this biases things (especially $\sigma^2$). The the example to which I just alluded, the unbiased estimate of $\sigma^2$ comes from the model having 5 regression parameters (overall intercept + 4 indicator variables).

Model selection often involves (dangerously) choosing

  1. among a set of competing model families or distributions
  2. which $X$s should be in the model
  3. how each $X$ should be modeled (e.g., consideration of nonlinear terms)
Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ Thanks for the answer. Let me give you an example where I am still unsure. Suppose you want to compare nested models with something like a likelihood ratio test. Are you now comparing models or are you also testing hypotheses? Your models might be based on hypotheses after all. $\endgroup$
    – fsociety
    Jan 6, 2015 at 17:50
  • 1
    $\begingroup$ @ph_singer You are doing both: the likelihood ratio test for nested models is itself a hypothesis test. To make an analogy, in cooking dinner, you might use an oven, but using an oven doesn't mean you're always cooking dinner. The oven, like a hypothesis test, is a tool; cooking dinner, like model selection, is a goal facilitated by the use of tools. $\endgroup$
    – heropup
    Jan 6, 2015 at 19:13
  • 1
    $\begingroup$ Nice question and answer. One issue is whether a non-rejection of the hypothesis should lead to adoption of the simpler model. I think not, in most situations. An exception is when one is checking a flexible model against an all-linear model and the test of nonlinearity with, say, 10 d.f., yields $P=.05$. I would feel safe in adopting the all-linear model. $\endgroup$
    – Frank Harrell
    Jan 6, 2015 at 20:40
  • $\begingroup$ Sorry that should be $P=0.50$ not $P=0.05$. $\endgroup$
    – Frank Harrell
    Jan 8, 2015 at 4:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.