# Tag Info

0

Create a function cv_aicc <- function(fit, lambda = 'lambda.1se'){ whlm <- which(fit$lambda == fit[[lambda]]) with(fit$glmnet.fit, { tLL <- nulldev - nulldev * (1 - dev.ratio)[whlm] k <- df[whlm] n <- nobs return(list('AICc' = - tLL + 2 * k + 2 * k * (k + 1) / (n - k - 1), '...

1

Your intuition is correct; you are describing the fundamental questions behind thinking about "effective degrees of freedom". We should indeed put "less trust" in the AIC/BIC values when using correlated explanatory variables. Our AIC/BIC will potentially be negatively biased (i.e. we will assume that we have more explanatory variables ...

1

Computing it multiple times can be more useful for specific algorithms. Let's take an example : while a regression will do the same thing if you use it on the exact same data (ie if you split your train and test with a random state to each time have the exact same rows), an algorithm like random forest will randomly take few attributes to make the forest (...

1

First thing to do is to make sure that you're not overfitting. If there is no such strong signal, then averaging out performance metrics you mentioned make sense. And, producing a basic confidence intervals (or +- std intervals) out of different runs of the same algorithm is a common practice in research papers.

1

There are variations of the Bayesian Information Criterion that are appropriate for categorical data. For example, Bayesian criterion based model assessment for categorical data. Unfortunately, I dont know of any implementations.

Top 50 recent answers are included