7
$\begingroup$

This question is based on using a regression for statistical inference (not prediction).

I have conducted hierarchical (logistic mixed effects) regression.

The first model includes the predictors of interest for the study. I am interested in the interactions between Condition and 1) SPQ, 2) CAPS, 3) PDI. There are no significant interactions for any predictor.

performance ~ Condition * (SPQ + PDI + CAPS) + (1 | participant)

The second model includes a list of covariates that are of potential theoretical importance. I wanted to assess whether these affect the (non-significant) interactions from the first model.

performance ~ Condition * (SPQ + PDI + CAPS+ S1 + S2 + Age + IQ) + 
                 (1 | participant)

This is obviously a highly complex model, although it does converge. There are significant main effects of Age and IQ and a significant S1*Condition interaction.

How important is parsimony in this case? Should I use backward stepwise elimination comparing log likelihood statistics and report a less complex model?

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ Regarding the title question, it is certainly possible to have valid statistical inference in a model that is not parsimonious. There is no need to 'get to parsimony' to make the subsequent inferences valid. Moreover, if you were to use backwards selection, or otherwise decide which variables to consider based on what you see in your data, then your inferences would not be valid. $\endgroup$
    – gung - Reinstate Monica
    Commented Jan 15 at 19:57
  • 2
    $\begingroup$ Parsimony can be useful in many contexts, but achieving it by arbitrary methods is not necessarily a good way to get there, and such an approach may have substantial undesirable consequences; choose between potential approaches with some idea of what their properties are, rather than doing the first thing that you think of. If you must use data to do some form of variable selection (or model selection more generally), then even with a good method for doing so, you typically don't want to use the same data for inference or prediction using that selected model. $\endgroup$
    – Glen_b
    Commented Jan 16 at 1:34
  • $\begingroup$ The purpose of the parsimony rule is to deal with the fact that there are an infinite number of functions that fit your data perfectly, and which predict every possible combination of outputs at out-of-sample locations. How do we choose between them? Many of them represent "complex" relationships between predictors and response variables, so we can reduce our search space considerably by preferring "parsimonious" models. If you have reasons to prefer a more "complex" model, then there's nothing wrong with that, but it's probably a good idea to have reasons. $\endgroup$
    – Him
    Commented Jan 17 at 16:41

3 Answers 3

Reset to default
16
$\begingroup$

You ask two questions:

  1. How important is parsimony?
  2. Should you use backward stepwise elimination?

The second one is easy: No, you shouldn't. This has been discussed here many times. It doesn't work well. The output is going to be wrong: Standard errors will be too small, p values too low, and parameter estimates biased away from 0. See Frank Harrell's book Regression Modeling Strategies for details, examples, proofs, and recommendations. Also see this thread.

The first one is trickier. Some people overuse Ockham's Razor which is often paraphrased as

The simplest explanation is usually the best one.

(It's not clear exactly what William of Ockham said; he lived in the 14th century and the first written evidence of this razor is from centuries later. But he said something like that.)

But note that a) William said "usually". And b) There's no guarantee that he is right!

Nature is not always simple.

Many centuries later, Einstein said:

It can scarcely be denied that the supreme goal of all theory is to make the irreducible basic elements as simple and as few as possible without having to surrender the adequate representation of a single datum of experience.

which is often misquoted as "Make everything as simple as possible, but not simpler."

and then there's one of my favorite statistics quotes, from George Box:

All models are wrong, but some are useful.

How does all this relate to your question? Models (including regression models) are attempts to make nature simpler than it really is. How much simpler? That's a key question.

As for prediction, you can make models too simple just as easily as you can make them too complex. The model gets too wrong to be useful. However, you also risk overfitting with a very complex model. There are strategies to avoid this (e.g. running train, test, and validation subsets of the data). But how complex a model can be supported depends on the amount of data you have and its structure. I would say it also depends on how strong your theories are and how accurate your measures are.

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ Footnote to Peter and Frank: if you are interested in the theory of parsimony, the "Objective Razor" section on the Ockham's Razor Wikipedia page provides a good summary with links. $\endgroup$
    – ctwardy
    Commented Jan 16 at 18:20
  • 1
    $\begingroup$ Ah, I did not know about the Einstein misquote! Thank you! $\endgroup$
    – Galen
    Commented Jan 16 at 22:40
  • 1
    $\begingroup$ I think that what you refer to a "misquote" is somewhat tongue-in-cheek, in that it simplifies the original quote as much as possible while still conveying the main idea. Sort of like "brevity -> wit". $\endgroup$
    – Acccumulation
    Commented Jan 17 at 3:29
11
$\begingroup$

Consistent with what Peter said, the only really cohesive and inferential accuracy-respecting approach is to use a Bayesian model where priors are put on all parameters. Especially for interactions, using skeptical priors is the only real way to handle bias-variance tradeoffs while giving exact inference and not requiring an astronomical sample size. More here.

Improve this answer
$\endgroup$
6
$\begingroup$

Mixing model building (deciding which terms should be in the model, their form, switching between approaches etc.) with hypothesis testing/statistical inference is usually very difficult. E.g. doing backward stepwise elimination (or determining based on "significance" in the original model which terms to keep) would invalidate the standard errors and p-values you software shows you by default. There may be complex ways of somehow correcting for the model building, but having some extra terms in a model usually does not really get in the way of interpreting its results (or is there any particular issue with interpreting the results?).

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks. Could you clarify the meaning of 'switching approaches'? There's a package for building mixed models (buildmer by C Voeten: cran.r-project.org/web/packages/buildmer/vignettes/…). This balances the advice in the literature for using maximal models with parsimony, especially when such maximal models do not converge. There is no issue with interpretability or convergence with my model, but I wanted to check whether there is another reason to report a more parsimonious model (especially as my final model is complex and multiple predictors are far from significance). $\endgroup$
    – SilvaC
    Commented Jan 15 at 11:26
  • 1
    $\begingroup$ "Switch approaches" = e.g. "linear model if test for non-normal residuals not signif., switch to rank test" (or similar such ideas). Unless there's extensive simulation studies to show what is being done in the package works, I'm a little suspicious of the buildmer package approach. It's commonly done and invovles many things known to sometimes be problematic. $\endgroup$
    – Björn
    Commented Jan 16 at 9:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.