2
$\begingroup$

I am fitting two different models to the same data. In one model, there is one free parameter for three different experimental conditions. In another model, I fit three free parameters, one for each condition. I do this for 10 subjects in a dataset.

For each subject, the model with fewer free parameters has a higher BIC. But for every single subject, the difference in BIC is roughly the same (about 10). I find this very suspicious, since the BIC values themselves range from ~30 to ~1000.

I have never used BIC before, and would like to say that the model with one free parameter is better.

Improve this question
$\endgroup$
2
  • $\begingroup$ I am not sure what you are doing. I have never heard of using BIC on particular subjects. It would usually be used to compare the two models for all 10 subjects. A model fit to one subject would be perfect fit (but meaningless). Also, 3 parameters is rather a lot for 10 subjects. $\endgroup$
    – Peter Flom
    Commented Oct 29, 2013 at 17:48
  • $\begingroup$ I am using the Ratcliff diffusion model, and it is standard practice to fit the model to participants separately. It's not a perfect fit because each subject has ~1000 data points. There are actually more than 3 parameters, but I was trying to keep it simple: in one model, a single parameter is broken down into three and the others remain the same. $\endgroup$
    – Jeff
    Commented Oct 29, 2013 at 17:57

1 Answer 1

Reset to default
2
$\begingroup$

Given your further comment, I am not surprised at this result. BIC is a penalized log likelihood. It is useful for comparing models on one data set (here, each participant), but not for comparing across data sets.

What this result is telling you, in essence, is that the model fits very differently for different people, but that the amount of improvement in the fit by adding two parameters is about the same for each person.

Improve this answer
$\endgroup$
3
  • $\begingroup$ so you are saying that, indeed, i have justification for saying that the model with fewer parameters (higher BIC for all participants) is a "better" model, correct? this was my interpretation, but it seemed strange that the increase in BIC was nearly constant across participants. $\endgroup$
    – Jeff
    Commented Oct 29, 2013 at 19:10
  • 1
    $\begingroup$ Whether higher or lower is better depends on the implementation. In SAS I believe lower is better (it says which is better on the printout). In R, as well, a lower BIC is better. From the R documentation for AIC: "When comparing fitted objects, the smaller the AIC or BIC, the better the fit. " $\endgroup$
    – Peter Flom
    Commented Oct 29, 2013 at 19:24
  • $\begingroup$ Can´t you use the average square error for predicting numerical targets? If your target is binary, then you can use the cumulative lift or the ROC curve. This is more in line with the SAS output, but i´m sure you can do this with R. $\endgroup$
    – marbel
    Commented Dec 17, 2013 at 21:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.