inconsistency in AIC and AICc - null and an alternative model

Question

I have a dataset containing one response variable, and 3 independent variables. There are 6 number of observations. I want to see, in AICc framework, which of these independent variables best explain my response variable. I have chosen AICc because my number of observation is low.

I have 8 models in my model set (all possible models, without interactions, plus a null model). FYI, those three independent variables are chosen from a set of more than 20 variables based on literature (working hypotheses), and after assessing collinearity.

AIC results seem to match with my thoughts and hypotheses. One of the models (with two independent variables) is the best model (has the lowest AIC). This model has Adj-R2 of ~0.60, which is among the highest in my model set. And for the fully saturated model (with all three independent variables), Adj.R2 and AIC do not improve, which matches with AIC penalty etc. But according to AICc, the null model is the best one (the lowest AICc), and the more variable is added to the model, the higher the AICc.

How it is possible to have completely different interpretation for AIC and AICc? I am not sure if it is reasonable to have the best AICc for the null model (which has Adj-R2=0), while another model that has Adj.R2 = 0.60 having one of the highest AICc. Any thoughts?

I should add that the parameter estimates (slope) are not high in none of the models (usually less than 0.02) and the intercept remains high and close to what it is in the null model (i.e., the intercept for null model is around 1.2, and even for the best model according to AIC, it remains around 0.8. This may suggest, I think, the variability in the response variable is not that much, and is not being explained by the independent variables. But as mentioned, Adj.R2 is 0.6 for the best model according to AIC, which shows there are some variation in the response variable (although not that much) and this variation is explained by the two variable.

Any input and thoughts will be appreciated. Mehdi

Answer 1

Having 6 data points associated with 3 explanatory variables and trying 8 different models is a classical case of "researcher degrees of freedom" abuse. I would urge you to first read about this topic first (despite not being directly associated with AICc). Two good papers to start are : Simons et al. 2011 "False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant" and "Degrees of Freedom in Planning, Running, Analyzing, and Reporting Psychological Studies: A Checklist to Avoid p-Hacking".

Back to the particulars of this use-case: Shrinkage / Regularisation methods is a great way to circumvent variable selection questions in a data driven way. The classical suggestion would be using something like LASSO. Nevertheless with just 6 data points we should consider employing a fully Bayesian approach using a Bayesian regression model that encapsulates our beliefs about the explanatory variables' effects into the priors of that model. There are many well-meaning rules-of-thumb about the ratio of number of observations over the number of data for regression and/or AIC to be relevant (e.g. see Burnham and Anderson (2004) Model Selection and Multi-Model Inference) but realistically arguing the use of AIC with 6 data points is irrelevant as the estimate of the underlying likelihood will be highly variable; the fact that AICc suggest the null model as optimal is unsurprising in that sense.