I ran a hierarchical regression comparing different models, in order of complexity. The first step was not significant (the simplest model vs a more complex model). However the next two steps were significant (a more complex model vs an even more complex model). Since the first step was not significant, I think I should stop here and select the simplest model (Model 1).
Below are the results from the hierarchical regression for model comparison
Res.Df RSS Df Sum of Sq F Pr(>F)
1 45768 311819838
2 45766 311802160 2 17678 1.2981 0.2730717 (n.s.)
3 45764 311696461 2 105699 7.7612 0.0004265 ***p<0.01
4 45762 311613774 2 82687 6.0715 0.0023096 ***p<0.001
I also looked at the AIC and BIC values. The AIC follows the pattern for the p-values: Model 2 is not better than Model 1, but Models 3 and 4 are better (the change in AIC for each step is 1.4, -11.5, -8.1). The BIC suggests that Model 1 is the best (the change in BIC for each step is 18.9, 5.9, 9.3). According to the BIC, I also think I will select Model 1.
Can someone please explain to me why the more complex models show up as significantly better when the earlier model was not?
For reference, the models being tested are segmented regression models as follows
- Model 1: A linear model (one slope); task_score ~ age
- Model 2: A model with one breakpoint in a predictor variable (one psi estimate to mark the breakpoint) task_score~ age + psi1.age
- Model 3: A model with two breakpoints (two psi estimates) task_score ~ age + psi1.age + psi2.age
- Model 4: A model with three breakpoints (three psi estimates) task_score~ age + psi1.age + psi2.age + psi3.age
The hierarchical regression measured
- Step 1: Model 1 vs Model 2
- Step 2: Model 2 vs Model 3
- Step 3: Model 3 vs Model 4
Note on segmented regression: A segmented regression (also called piecewise or changepoint regression) is a linear regression with an abrupt change in the x~y relationship, i.e., where the line is allowed to 'bend' at a given point (called a breakpoint). Examples of 1,2, and 3 breakpoint models are below.