Timeline for If summarizing stats from multiple models is it meaningful to report a mean AIC?
Current License: CC BY-SA 3.0
8 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jul 10, 2014 at 8:17 | comment | added | Sideshow Bob | Alas there is more I'm not saying. Each algorithm has a (different) number of calibration parameters so I want to use AIC to reflect the complexity of correctly choosing these. | |
| Jul 9, 2014 at 16:06 | comment | added | Frank Harrell | Since your regressions are simple and you don't have any apriori reason to assume linearity, I would just use Spearman's $\rho$ to describe the predictive discrimination for each combination of variables. | |
| Jul 9, 2014 at 10:01 | comment | added | Sideshow Bob | When I say different techniques I mean different predictor variables derived in different ways. I am producing a comparative study to determine which of a number of simulation algorithms best predict traffic flow. Each algorithm produces a variable X - the flow in the simulation - and I regress Y, the actual measured flow, against X (in four different datasets). So I am trying to produce a single AIC for each algorithm to show its effectiveness. This would somehow be a combination of the AICs of each X-Y regression in different areas for that algorithm. | |
| Jul 8, 2014 at 19:48 | comment | added | Frank Harrell | I think I understand the possible need for four different test datasets, but I'm not clear on why you are using different techniques as opposed to a single unified, flexible technique. | |
| Jul 8, 2014 at 17:01 | comment | added | Sideshow Bob | What about combining relative likelihoods - is it reasonable to take a mean of these? | |
| Jul 8, 2014 at 16:54 | comment | added | Sideshow Bob | I would like to produce a single summary statistic for each technique, to allow comparison of the effectiveness of each. Above I have referred to each technique as a "group of models". Within the group there is one model per test area. So that's why I have multiple models divided into groups. Does that make it any clearer? | |
| Jul 8, 2014 at 16:53 | comment | added | Sideshow Bob | In concrete terms: I am comparing different techniques for predicting vehicle flow. I try out each technique in four different areas i.e. on four different data sets. | |
| Jul 8, 2014 at 15:46 | history | answered | Frank Harrell | CC BY-SA 3.0 |