1
$\begingroup$

I am interested in identifying the best of 3 physiologically reasonable models that fits my continuous data. Data is some measure derived from neurons recorded from 3 adjacent regions of brain tissue (marked by dotted lines in plots). Data has outliers (to be kept) and is n=26 to 50+ (depending on size of sliding window smoothing bin). Model is single predictor to single response.

Why model selection: Previous studies show that the y-value follows a linear trend as x increases. From raw data plots, I suspect that this trend is better characterized as a model with 3 separate lines. Hence I plot 3 different models with 1-3 segments (see figure below for a positive example). These are 1) linear model, 2) discontinuous piecewise two-segment model 3) discontinuous piecewise three-segment model. All are OLS linear regression models, no robust fitting is used.

Example of good fits: 3 candidate models with diff-AIC and r-val of each fitted line

I calculated AICc for these models, and was able to use diff-AICc to get the best of 3.

However, is there a safeguard/threshold that prevents me from selecting poorly fit linear models (see below), i.e. perhaps there is no real trend in data, but I can still pick the best model of 3?

Example of bad linear fits

Some possible solutions: Should I screen my models with some absolute goodness-of-fit statistics like RMSE (uncorrected for n params) or adj. R^2 (corrected for n params) before applying AIC?

Or can I calculate a null distribution of AIC values by shuffling the y-label and calculating AIC values, and only consider models whose AIC values pass a threshold?

Thanks!

$\endgroup$
  • 1
    $\begingroup$ Thanks, this is helpful. Your new details do complicate this question, though. For example, it's not clear why you assume that piecewise linear captures the nonlinearity best; there appears to be clear curvature in the data points, rather than an abrupt change in a linear trend. $\endgroup$ – mkt Aug 22 at 9:52
  • $\begingroup$ What is your goal? Are you just trying to describe trends in the data? Draw inferences? Make predictions? $\endgroup$ – mkt Aug 22 at 9:53
  • $\begingroup$ @mkt Yes there is a curvature, but a linear trend is used to 1) keep things simpler and 2) that's what a previous group used. But i agree, I should rethink this. Goal is to describe the trend, and draw the inference that the y-var is possibly organized in a segmented linear/linear fashion in the brain. $\endgroup$ – Richard Aug 22 at 9:54
  • $\begingroup$ Also, the second set of plots indicates that the Y-axis is bounded, which suggests that you may need to use a GLM with a different distribution. $\endgroup$ – mkt Aug 22 at 11:25
  • 1
    $\begingroup$ This seems to be little more than stepwise variable selection, and will bias the estimate of residual variance and make standard errors and p-values incorrect, plus confidence limits. $\endgroup$ – Frank Harrell Aug 22 at 11:44
1
$\begingroup$

I don't think this is the best use of information criteria. Given your small sample size, it's very possible for the parameter penalty to cause the simpler models to be selected as best if the data are noisy. The fact that there are outliers suggests that noise is a problem.

You mention that all 3 models are physiologically plausible, but there are very large differences in the shapes that those models imply. It therefore seems unlikely that they would all be equally plausible. I would decide on which model to use based on prior knowledge and plots of the raw data.

EDIT after question was updated with details and figures:

The plots indicate that none of the 3 models is a good description of the data. Moreover, the models also seem physiologically implausible, especially the piecewise models. I think it would be best to reconsider the set of models, keeping in mind the goal(s). If description is the primary goal, I would use a GAM, since these would deal well with the complex curved shape. If inference is desired, I think you need a better and more complex model, one that is grounded in the physiology of this system.

If you just want to establish whether "the y-var is possibly organized in a segmented linear/linear fashion in the brain", then I would argue that the plots you show are fairly good evidence against those possibilities. In theory, you could also compare these the three models you have against a fitted GAM, but you don't have enough data to make an information criterion-based comparison very informative. You could examine patterns in the residuals of the linear & piecewise linear to establish whether they are acceptable fits, and I am fairly certain that you would see autocorrelation in them, indicating that they are not appropriate.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.