meta user
network profile
| bio | website | stat.cmu.edu/~nmv |
|---|---|---|
| location | Pittsburgh, PA | |
| age | ||
| visits | member for | 1 year, 10 months |
| seen | May 20 at 15:50 | |
| stats | profile views | 14 |
|
Jul 22 |
comment |
Does there exist a model fit statistic (like AIC or BIC) that can be used for absolute instead of just relative comparisons? @whuber Apparently It only lets me notify one person, so please see my response above about my specific case. Your point is well taken about $R^2$ and the context of the data. |
|
Jul 22 |
comment |
Does there exist a model fit statistic (like AIC or BIC) that can be used for absolute instead of just relative comparisons? @drknexus I have 3 different exams that were given to 3 different groups of people and scored by 3 different sets of raters. I fit a few models (e.g. linear, Rasch, hierarchical Bayesian) on each set of exams separately. Within the same exam data-set I can compare among the models using AIC/BIC, but I can't compare across the data-sets. For example, on exam 1 Bayesian beats Rasch, but on exam 2 Rasch beats Bayesian. Is that because of poor fit (and high AIC/BIC variance) or because of good fit and there is something different about the two exams. |
|
Jul 22 |
awarded | Student |
|
Jul 22 |
comment |
Does there exist a model fit statistic (like AIC or BIC) that can be used for absolute instead of just relative comparisons? @whuber Wow, that's an awesome response to the $R^2$ question! But, its inadequacies aside, $R^2$ is used to say that their model is "good" in an "absolute" sense (e.g. "My $R^2$ is such-and-such which is better than what one normally sees..."). I'm looking for a more justified (and general) statistic than $R^2$ to accomplish the same purpose (e.g. "My MagicStatistic is such-and-such which is better...). My first naive thought was to do something like normalizing a k-fold cross validation score, but it doesn't seem like anyone has done such a thing (so its probably not a good idea). |
|
Jul 22 |
asked | Does there exist a model fit statistic (like AIC or BIC) that can be used for absolute instead of just relative comparisons? |
