I am interested in identifying the best of 3 physiologically reasonable models that fits my continuous data. Data is some measure derived from neurons recorded from 3 adjacent regions of brain tissue (marked by dotted lines in plots). Data has outliers (to be kept) and is n=26 to 50+ (depending on size of sliding window smoothing bin). Model is single predictor to single response.
Why model selection: Previous studies show that the y-value follows a linear trend as x increases. From raw data plots, I suspect that this trend is better characterized as a model with 3 separate lines. Hence I plot 3 different models with 1-3 segments (see figure below for a positive example). These are 1) linear model, 2) discontinuous piecewise two-segment model 3) discontinuous piecewise three-segment model. All are OLS linear regression models, no robust fitting is used.
I calculated AICc for these models, and was able to use diff-AICc to get the best of 3.
However, is there a safeguard/threshold that prevents me from selecting poorly fit linear models (see below), i.e. perhaps there is no real trend in data, but I can still pick the best model of 3?
Some possible solutions: Should I screen my models with some absolute goodness-of-fit statistics like RMSE (uncorrected for n params) or adj. R^2 (corrected for n params) before applying AIC?
Or can I calculate a null distribution of AIC values by shuffling the y-label and calculating AIC values, and only consider models whose AIC values pass a threshold?