1
$\begingroup$

Can AIC values between different weighed models be compared to select the best model (ie the model with the lowest weighted AIC)?

For example, if my response variable is the 'Average Sales Per Person' where this equals to (Sum of Sales / Number of People), so naturally I would like to assign more weight to groups with higher number of people.

I run the following two models:

Gamma

glm(formula = AvrSales ~ x + y, family = Gamma(link = "identity"),weights=NumPeople)

Null deviance: 692476  on 2181  degrees of freedom   
Residual deviance: 574155  on 2171  degrees of freedom

AIC: 28802851

Normal

glm(formula = AvrSales ~ x + y, family = gaussian(link = "identity"), 
    weights = NumPeople)

Null deviance: 5.1948e+10  on 2181  degrees of freedom   
Residual deviance: 4.4921e+10  on 2171  degrees of freedom

AIC: 33158

Using AIC as a model selection criteria, would the Normal model then be selected since it has a lower AIC? Or are both models bad at explaining the response since the Residual Deviance is so high?


If I remove the weights, then the opposite conclusion can be reached.

Gamma (with out weights)

glm(formula = AvrSales ~ x + y, family = Gamma(link = "identity"))

Null deviance: 10903  on 2181  degrees of freedom   
Residual deviance: 10685  on 2171  degrees of freedom

AIC: 25738

Normal (with out weights)

glm(formula = AvrSales ~ x + y, family = gaussian(link = "identity"))

Null deviance: 847754154  on 2181  degrees of freedom   
Residual deviance: 824882849  on 2171  degrees of freedom

AIC: 34239

What makes sense here? If I must use weights inside my model, how is then possible to compare different models subjectively? Any advice, or references to literature would be greatly appreciated.

$\endgroup$
1
$\begingroup$

Your question seems to be more geared towards selecting the right response variable distribution rather than typical usage of AIC, which is for selecting the variables to include in the model. Typically we choose the correct response distribution based on characteristics of the response variable. I don't believe there is a definitive rule for this; in practice it's somewhat depends on your judgment and exploratory analysis. For example, we typically use the Gamma distribution when our data exhibits heteroscedasticity. Often in financial data this is the case, where the variance of your response can be expressed as a function of the mean. Also, the Gamma distribution is always nonzero, so again for your sales data this might be more appropriate. Also, note that we often use a log link instead of an identity link.

Your second model using Gaussian (or Normal) with a identity link is identical to an OLS; I presume you know this already. This means that all of the restrictive assumptions with OLS apply here too.

I've never been comfortable using AIC alone in selecting the response distribution. Maybe someone else can weigh in on the specifics here.


You'd probably get a more solid answer by using a cross-validation technique. I won't go into details here, but if you have enough data, why not use K-Fold CV to test which model fits better?

$\endgroup$
  • $\begingroup$ Thanks AdmiralWen, while I agree than simply looking at AIC is an incomplete way of picking a mode, I was more getting at what are some pitfalls of examing AIC results from a weighted model. If AIC is loosely translated as the likelihood of underlying distribution re-creating the sample data...what does then the weighted AIC translate? Is it simply the likelihood of underlying distribution recreating a the weighted sample data? Just looking for a better way to explain my self the interaction of weights and AIC/LL. Thanks for the help! -Senna $\endgroup$ – Senna Jul 11 '15 at 23:03
  • 1
    $\begingroup$ What makes model selection a worthwhile exercise in this setting? $\endgroup$ – Frank Harrell Jan 3 '18 at 12:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.