Timeline for Justification for and optimality of $R^2_{adj.}$ as a model selection criterion
Current License: CC BY-SA 4.0
12 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Oct 18, 2021 at 0:42 | comment | added | John Vandivier | @RichardHardy I just added items 5 and 6 - I don't expect these are what you are looking for either, but just in case! | |
| Oct 18, 2021 at 0:40 | history | edited | John Vandivier | CC BY-SA 4.0 |
link fix
|
| Oct 16, 2021 at 19:24 | comment | added | Richard Hardy | I do not remember Ben Lambert saying the 2k penalty is arbitrary. I hope he did not say that as that would simply be wrong. There is a clear theoretical justification for it in Akaiske's original paper and in later elaborations. I cannot comment on the relationship between AIC and DIC or WAIC as I do not know the latter two well enough. I am still looking for any optimality justification for $R^2_{adj.}$. The arguments you have provided are not quite what I am looking for. Nor are they on the level of concreteness that would lend itself for constructive criticism given my understanding. Sorry. | |
| Oct 16, 2021 at 19:13 | comment | added | John Vandivier | Now, if we abandon "I must use the ideal optimality justification" for "I must use a satisfactory optimality justification," then sure AIC seems fine - the gains from others will often be unimportant and AIC is relatively widely used, understood, and supported in code libraries. However, on exactly parallel grounds, we can appreciate adjusted r-squared. | |
| Oct 16, 2021 at 19:12 | comment | added | John Vandivier | Link for the audience here. The key statement in the video as I understand it wasn't to do with popularity; that's a result not a cause of utility. He says "The big step forward which DIC makes over AIC is applying a more general and useful measure of penalty." DIC penalty is related to data variance in contrast to the arbitrary 2k AIC penalty. It's hard for me to see how AIC can be preferred to either since it is a special case of either DIC or WAIC - a special case with little retrospective or comparative optimality justification. | |
| Oct 16, 2021 at 16:44 | comment | added | Richard Hardy | I cannot find the relevant thread anymore to post this comment to but I will do it here. Just wanted to let you know I watched Ben Lambert's YouTube video about AIC, DIC, WAIC and LOOCV and did not find it convincing enough to drop AIC in favor of DIC or WAIC. Statements in the video (e.g. about popularity or approximation quality of AIC) are too general to be correct even if they may hold in special cases (though I understand this is intentional due to the format of the lecture series). DIC and WAIC may function OK for Bayesian modeling but I do not see if they are applicable outside of that. | |
| Oct 13, 2021 at 20:41 | comment | added | John Vandivier | Sometimes I’m also interested in how dependent variables relate to each other as a secondary concern. Eg does X2 “partial out” X1 | |
| Oct 13, 2021 at 20:40 | comment | added | John Vandivier | Here I am trying to generally express coefficient hypothesis testing. Usually I care about direction of effect, significance, and importance. This general term leaves room to explore an independent factor, lags, interactions, marginal effects, and other computed, derived, or engineered features. Possibly including causal testing. I am mainly thinking about OLS at the moment. | |
| Oct 13, 2021 at 19:31 | comment | added | Richard Hardy | When you say "factor relations", what exactly do you mean? | |
| Oct 13, 2021 at 19:30 | comment | added | Richard Hardy | Note that under some assumptions, LOOCV is asymptotically equivalent to AIC. | |
| S Oct 13, 2021 at 19:25 | review | First answers | |||
| Oct 13, 2021 at 19:25 | |||||
| S Oct 13, 2021 at 19:25 | history | answered | John Vandivier | CC BY-SA 4.0 |