Screening candidate models before AIC comparison?

I am interested in identifying the best of 3 physiologically reasonable models that fits my continuous data. Data is some measure derived from neurons recorded from 3 adjacent regions of brain tissue (marked by dotted lines in plots). Data has outliers (to be kept) and is n=26 to 50+ (depending on size of sliding window smoothing bin). Model is single predictor to single response.

Why model selection: Previous studies show that the y-value follows a linear trend as x increases. From raw data plots, I suspect that this trend is better characterized as a model with 3 separate lines. Hence I plot 3 different models with 1-3 segments (see figure below for a positive example). These are 1) linear model, 2) discontinuous piecewise two-segment model 3) discontinuous piecewise three-segment model. All are OLS linear regression models, no robust fitting is used. I calculated AICc for these models, and was able to use diff-AICc to get the best of 3.

However, is there a safeguard/threshold that prevents me from selecting poorly fit linear models (see below), i.e. perhaps there is no real trend in data, but I can still pick the best model of 3? Some possible solutions: Should I screen my models with some absolute goodness-of-fit statistics like RMSE (uncorrected for n params) or adj. R^2 (corrected for n params) before applying AIC?

Or can I calculate a null distribution of AIC values by shuffling the y-label and calculating AIC values, and only consider models whose AIC values pass a threshold?

Thanks!

• Thanks, this is helpful. Your new details do complicate this question, though. For example, it's not clear why you assume that piecewise linear captures the nonlinearity best; there appears to be clear curvature in the data points, rather than an abrupt change in a linear trend. – mkt Aug 22 at 9:52
• What is your goal? Are you just trying to describe trends in the data? Draw inferences? Make predictions? – mkt Aug 22 at 9:53
• @mkt Yes there is a curvature, but a linear trend is used to 1) keep things simpler and 2) that's what a previous group used. But i agree, I should rethink this. Goal is to describe the trend, and draw the inference that the y-var is possibly organized in a segmented linear/linear fashion in the brain. – Richard Aug 22 at 9:54
• Also, the second set of plots indicates that the Y-axis is bounded, which suggests that you may need to use a GLM with a different distribution. – mkt Aug 22 at 11:25
• This seems to be little more than stepwise variable selection, and will bias the estimate of residual variance and make standard errors and p-values incorrect, plus confidence limits. – Frank Harrell Aug 22 at 11:44