2
$\begingroup$

Can AIC values between different weighed models be compared to select the best model (ie the model with the lowest weighted AIC)?

For example, if my response variable is the 'Average Sales Per Person' where this equals to (Sum of Sales / Number of People), so naturally I would like to assign more weight to groups with higher number of people.

I run the following two models:

Gamma

glm(formula = AvrSales ~ x + y, family = Gamma(link = "identity"),weights=NumPeople)

Null deviance: 692476  on 2181  degrees of freedom   
Residual deviance: 574155  on 2171  degrees of freedom

AIC: 28802851

Normal

glm(formula = AvrSales ~ x + y, family = gaussian(link = "identity"), 
    weights = NumPeople)

Null deviance: 5.1948e+10  on 2181  degrees of freedom   
Residual deviance: 4.4921e+10  on 2171  degrees of freedom

AIC: 33158

Using AIC as a model selection criteria, would the Normal model then be selected since it has a lower AIC? Or are both models bad at explaining the response since the Residual Deviance is so high?


If I remove the weights, then the opposite conclusion can be reached.

Gamma (with out weights)

glm(formula = AvrSales ~ x + y, family = Gamma(link = "identity"))

Null deviance: 10903  on 2181  degrees of freedom   
Residual deviance: 10685  on 2171  degrees of freedom

AIC: 25738

Normal (with out weights)

glm(formula = AvrSales ~ x + y, family = gaussian(link = "identity"))

Null deviance: 847754154  on 2181  degrees of freedom   
Residual deviance: 824882849  on 2171  degrees of freedom

AIC: 34239

What makes sense here? If I must use weights inside my model, how is then possible to compare different models subjectively? Any advice, or references to literature would be greatly appreciated.

$\endgroup$

2 Answers 2

1
$\begingroup$

Your question seems to be more geared towards selecting the right response variable distribution rather than typical usage of AIC, which is for selecting the variables to include in the model. Typically we choose the correct response distribution based on characteristics of the response variable. I don't believe there is a definitive rule for this; in practice it's somewhat depends on your judgment and exploratory analysis. For example, we typically use the Gamma distribution when our data exhibits heteroscedasticity. Often in financial data this is the case, where the variance of your response can be expressed as a function of the mean. Also, the Gamma distribution is always nonzero, so again for your sales data this might be more appropriate. Also, note that we often use a log link instead of an identity link.

Your second model using Gaussian (or Normal) with a identity link is identical to an OLS; I presume you know this already. This means that all of the restrictive assumptions with OLS apply here too.

I've never been comfortable using AIC alone in selecting the response distribution. Maybe someone else can weigh in on the specifics here.


You'd probably get a more solid answer by using a cross-validation technique. I won't go into details here, but if you have enough data, why not use K-Fold CV to test which model fits better?

$\endgroup$
2
  • $\begingroup$ Thanks AdmiralWen, while I agree than simply looking at AIC is an incomplete way of picking a mode, I was more getting at what are some pitfalls of examing AIC results from a weighted model. If AIC is loosely translated as the likelihood of underlying distribution re-creating the sample data...what does then the weighted AIC translate? Is it simply the likelihood of underlying distribution recreating a the weighted sample data? Just looking for a better way to explain my self the interaction of weights and AIC/LL. Thanks for the help! -Senna $\endgroup$
    – Senna
    Commented Jul 11, 2015 at 23:03
  • 1
    $\begingroup$ What makes model selection a worthwhile exercise in this setting? $\endgroup$ Commented Jan 3, 2018 at 12:08
0
$\begingroup$

I think there are possible two questions here:

  1. Should you differential weight observations in your regression set up
  2. If you do weight observations, is AIC still a valid test criteria?

First, yes, you should differentially weight observations when each observation has contains a quantifiably different amount of experience or knowledge. In your example, yes, it makes sense to weight by Number of People.

Then, once you have decided to weight, all statistical model assessment should be on this weighted data, including AIC.

You AIC results tell you that if you care about predicting observations, or rows in your dataframe correctly, then Normal is better, but if you care about predicting correctly weighted by Number of People, then Gamma is better.

Your example here is hopefully contrived because you wouldn't use either a Normal or Gamma distribution, which take continuous variables along all real numbers (Normal) or all positive numbers (Gamma) to model a fraction Average Sales = Sales / Reps.

A better modeling setup would be Total Sales = N Reps * GLM(x1, x2) where the GLM can be interpreted as a productivity rate.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.