1
$\begingroup$

I ran a hierarchical regression comparing different models, in order of complexity. The first step was not significant (the simplest model vs a more complex model). However the next two steps were significant (a more complex model vs an even more complex model). Since the first step was not significant, I think I should stop here and select the simplest model (Model 1).

Below are the results from the hierarchical regression for model comparison

   Res.Df   RSS        Df   Sum of Sq     F      Pr(>F)    

1  45768    311819838                                  

2  45766    311802160  2    17678         1.2981 0.2730717   (n.s.) 

3  45764    311696461  2    105699        7.7612 0.0004265 ***p<0.01

4  45762    311613774  2    82687         6.0715 0.0023096 ***p<0.001

I also looked at the AIC and BIC values. The AIC follows the pattern for the p-values: Model 2 is not better than Model 1, but Models 3 and 4 are better (the change in AIC for each step is 1.4, -11.5, -8.1). The BIC suggests that Model 1 is the best (the change in BIC for each step is 18.9, 5.9, 9.3). According to the BIC, I also think I will select Model 1.

Can someone please explain to me why the more complex models show up as significantly better when the earlier model was not?

For reference, the models being tested are segmented regression models as follows

  • Model 1: A linear model (one slope); task_score ~ age
  • Model 2: A model with one breakpoint in a predictor variable (one psi estimate to mark the breakpoint) task_score~ age + psi1.age
  • Model 3: A model with two breakpoints (two psi estimates) task_score ~ age + psi1.age + psi2.age
  • Model 4: A model with three breakpoints (three psi estimates) task_score~ age + psi1.age + psi2.age + psi3.age

The hierarchical regression measured

  • Step 1: Model 1 vs Model 2
  • Step 2: Model 2 vs Model 3
  • Step 3: Model 3 vs Model 4

Note on segmented regression: A segmented regression (also called piecewise or changepoint regression) is a linear regression with an abrupt change in the x~y relationship, i.e., where the line is allowed to 'bend' at a given point (called a breakpoint). Examples of 1,2, and 3 breakpoint models are below.

An example of a 1 breakpoint model

An example of a 2 breakpoints model

An example of a 2 breakpoints model

Improve this question
$\endgroup$
3
  • $\begingroup$ How do you define the breakpoints and why are they hierarchical? $\endgroup$
    – Sextus Empiricus
    Jun 24, 2020 at 15:57
  • $\begingroup$ A breakpoint is a point in which the x~y relationship is allowed to change. You can think of it as a 'shift' or 'bend' in the regression line, or multiple lines joined together by breakpoints. I have added a definition and some images to my original post. $\endgroup$
    – Annalise
    Jun 24, 2020 at 18:35
  • $\begingroup$ They are hierarchical because each breakpoint creates an additional bend in the line, and requires an additional slope to be estimated. $\endgroup$
    – Annalise
    Jun 24, 2020 at 18:48

1 Answer 1

Reset to default
1
$\begingroup$

I think what you're describing is fairly common. More complex models often tend to outperform simpler models. This is not necessarily a problem. If you're adding new and meaningful variables to the equation, then this would absolutely make sense. It could also mean that the variables you included in the first model are just bad predictors. I'm not familiar with the U or psi notation, but I think a next step is to dive deeper into why the more complex models are performing better. I would look not only at the overall statistics, but also look at the individual regression weights and see which of your new predictors is doing a good job of predicting your criterion. That will help you get a better understanding of what's going on.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thank you the reply! It is very helpful. If more complex models outperform simpler models, should this occur in a stepwise manner (each layer of complexity offers a better fit than the previous model)? Or can a slightly more complex model not perform well, but an even more complex model perform well (i.e., the first layer of complexity is not better than the simple model, but the others are, which was the case here)? $\endgroup$
    – Annalise
    Jun 24, 2020 at 18:39
  • $\begingroup$ For your first question, the answer is not necessarily. The answer is more nuanced than more complexity = better model. It really just depends on what predictors you're adding to the model. If you start with the most meaningful predictors, then you'll likely find that adding predictors after that doesn't improve your model much. The answer to your second question is yes (like what you found). Again, it just depends on what predictors you're adding. $\endgroup$
    – Mathias Simmons
    Jun 26, 2020 at 12:03
  • $\begingroup$ For your results specifically, I think the reason that you're seeing more complexity improve your model fit is because you're allowing predictors that allow the relationship to vary at different points. Relationships are rarely a true straight line, so adding break points or letting your relationship change direction more (in curvilinear regression) will usually result in a better fit. I think your challenge is to determine whether additional breakpoints make sense theoretically. You need to determine if there is a good rationale for why the relationship would change at different ages. $\endgroup$
    – Mathias Simmons
    Jun 26, 2020 at 12:07
  • $\begingroup$ Sorry for the late reply, but thank you very much for your answer and additional responses. They were very helpful. $\endgroup$
    – Annalise
    Apr 23, 2021 at 14:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.