Model selection is a problem of judging which model from some set performs best. Popular methods include $R^2$, AIC and BIC criteria, test sets, and cross-validation. To some extent, feature selection is a subproblem of model selection.

learn more… | top users | synonyms (1)

0
votes
0answers
16 views

Why is $R^2$ poor for AR model selection used for forecasting?

There is a related question here, about how to calculate the R-squared on a regression with ARIMA errors. I found the answer quite useful, and hoped for some elaboration, particularly on Rob's closing ...
0
votes
0answers
14 views

Model selection in nlme's

I can think of four ways to perform model selection nlme's: LRT, AIC, ...
0
votes
0answers
15 views

Number of parameters for AIC for a particular model

I know there have been a few well answered questions on this topic, but i have found myself in a bit of a special case this time. I am using AIC for model selection, and i am having trouble counting ...
1
vote
0answers
12 views

Does the Rank Probability Score tell how good a forecasting model is?

The ranked probability score (RPS) is a measure of how good forecasts that are expressed as probability distributions are in matching observed outcomes. Both the location and spread of the forecast ...
1
vote
0answers
17 views

Testing the variance part of a Generalized Linear Model out of sample

Suppose I have a response vector and an ANOVA design (for simplicity, assume it’s a one-way ANOVA with two treatments). A few Generalized Linear Models (Poisson, Negative Binomial, etc) are fitted to ...
6
votes
3answers
64 views

What is the upside of treating a factor as random in a mixed model?

I have a problem embracing the benefits of labeling a model factor as random for a few reasons. To me it appears like in almost all cases the optimal solution is to treat all of the factors as fixed. ...
1
vote
0answers
16 views

Choosing models with similar AIC values [duplicate]

I'm using a multinomial logistic regression analysis to examine deer behavioural responses to camera traps in terms of 7 predictors (both singly and their interactions). I have found that the model ...
0
votes
0answers
6 views

Looking for catalog of “mechanism classes” that give rise to specific curve shapes

(I apologize for the length of this post. I don't know how to frame the question more succinctly.) I have some experimental data, in the form of a collection of curves with fairly little noise, ...
1
vote
0answers
26 views

How can I use Akaike's Information Criterion to compare two models of multi-peaked emission spectra?

I have several photoluminescence emission spectra that I am trying to fit curves to. The spectra each have a slight baseline and four peaks. The independent variable $x_i$ is wavelength (converted to ...
0
votes
1answer
19 views

model specification problem

I have the following model that I would like to rebuild: $Y_{i,t}=a+bx_{i,t}+cx_{i,t-1}+e_{i,t}$ I' am wondering now whether this is the same as the model above: $Y_{i,t}=a+ d\Delta ...
2
votes
1answer
43 views

Cross-validation with dummy variables?

Does it make sense to use cross-validation with factor variables that have 3+ levels? When using bestglm, I get an error saying that it doesn't work with categorical variables. In the documentation ...
0
votes
0answers
9 views

Error when using glmulti with aicc?

Does anyone know the solution to the error I get when I run glmulti with the aicc criteria: Error in if (length(lesCrit) == confsetsize && minouN - minou >= -deltaM && : missing value ...
1
vote
0answers
11 views

Motivating likelihood ratio test vs Wald test for paper reviewer

I've got back reviews for a paper I've submitted, with the following problem. I have two logistic regression models, say y ~ A, and y ~ A + B, where B is a factor with several levels. I have ...
0
votes
1answer
21 views

How to get both MSE and R2 from a sklearn GridSearchCV?

I can use a GridSearchCV on a pipeline and specify scoring to either be 'MSE' or 'R2'. I can then access ...
2
votes
2answers
44 views

Is cross validation for validating a model or for selecting best model in different kinds of models?

I am confused about the concept of cross validation and its usage. As I read about cross validation before, it is a way of validating a model. I did cross validation in my project (developing ...
0
votes
0answers
18 views

Can I use cross-validation to select optimal parameters SEPARATELY?

I'm wondering if there is any math/stat theory out there to support or deny this idea: I am using cross-validation and building models over a vector of parameter values to then choose the optimal ...
2
votes
2answers
51 views

What interactions to include in my GLM model?

I realize this might be a too general question, so I'll describe what I'm doing right now first. I'm working for a virtual insurance company and I have this dataset. It has severity (meaning ...
3
votes
2answers
36 views

Generalized log likelihood ratio test for non-nested models

I understand that if I have two models A and B and A is nested in B then, given some data, I can fit the parameters of A and B using MLE and apply the generalized log likelihood ratio test. In ...
0
votes
1answer
27 views

Log-likelihood (and AIC) of robust nlrob model differs from standard nls model

Comparing models generated by nlrob to ones generated by nls, I've noticed that even though the models might be nearly identical, the log-likelihood of the models is sometimes significantly different, ...
0
votes
0answers
28 views

Iterative Addition of Variables to Model Based on P Value

Suppose I have 64 columns that I have chosen out of 500+ columns based on the fact that they have the highest pairwise correlation (is this a good way?). I take 16 of these columns and run a simple ...
1
vote
0answers
32 views

Model diagnostics for a glmmPQL in R mixed-effects model

Several texts (both online and published books) have been reviewed prior to asking this. What diagnostics are accepted as best practise for a generalised linear mixed-effects model fitted in R using ...
0
votes
0answers
31 views

Compare LMM GLMM (generalised linear mixed model, negative binomial) by numerical measure (AIC BIC, cross validation, R² squared) for model validation

How to compare results of generalized linear mixed model (GLMM, negative binomial) with a log transformed linear mixed model (multilevel, hierarchical) . I have a data set (counts), which is nested. ...
0
votes
0answers
12 views

What is a “concordance score” for regression coefficients?

I came across this "concordance score" in a set of slides called Penalized regression methods for ranking variables by effect size, with applications to genetic mapping studies, by Ji Zhu: $$ ...
1
vote
0answers
26 views

Proper scoring rules for observations with different supports

Suppose to have a bivariate variable $z_t=(x_t, y_t)$ indexed by $t=1,2, ..., T$. Suppose now that the two components have different support, i.e. in my specific problem $x_t \in \mathcal{S}$, where ...
1
vote
0answers
31 views

Model without endogeneity correction has lower AIC than one with correction

I have two models, one with endogeneity correction (includes correction terms obtained using Heckman) and one without. The correction terms are significant in the second stage model, yet the AIC/BIC ...
0
votes
0answers
31 views

Applying the Akaike Information Criterion to Data

I have some variables that I would like to run regressions on, to create a model, but I am unsure about how to actually AIC (or the BIC). Unfortunately I have not yet taken a mathematical statistics ...
1
vote
0answers
13 views

Variable coarsening in Naive Bayes

Say we have a binary classification problem that we solve with Naive Bayes. All features are categorical variables. Say we focus on a single feature that takes one of $N$ possible values. If $N$ is ...
0
votes
0answers
4 views

how to measure the model recovery performance

I have some simulated linear model $y = X\beta+\epsilon$ where $\beta$ is sparse. I am comparing different techniques to recover the structure of $\beta$. I have so 4 values True Positives, False ...
1
vote
1answer
36 views

How to deal with bouts of equilibrium in an online learning setting?

I have a kalman filter (a recursive least square filter, really) regressing over real-time streams of data. Because the data-generating process varies slightly over time I add an exponential ...
1
vote
0answers
89 views

using Root Mean Squared Error (RMSE) to compare models with different sample size

I'm using k-fold cross-validation to compare different models. I splitted my dataset in 6 chunks and used 4 random chunks as training set and the remaining 2 as a test set. Now I fitted n-different ...
0
votes
0answers
45 views

BRT analysis using count data

I have some problems with my BRT analysis. Introduction to the data: The dependent variable is count data of a specific palm species in SA, and the predictors consists of nine various kinds of ...
1
vote
0answers
30 views

Cross-Validation vs. AICc for LASSO

I was working on a research project in which I try to estimate the the individual contribution of a group of regional political leaders to local economic growth. The major challenge is that there is ...
3
votes
1answer
27 views

Non-constant standard deviation in residuals

I am fitting a model in the frequency domain, and my fit looks as follows: As you can see, the model function does not fit the data perfectly, especially in the higher frequencies. So, I examined ...
2
votes
1answer
98 views

Is it possible to use the Breusch-Pagan Lagrange multiplier test (xttest0) in Stata for unbalanced data?

Is it possible to use xttest0 in Stata with unbalanced panel data? I want to test whether the I should use pooled OLS or random effects estimation. What does this test actually do?
1
vote
0answers
33 views

Estimating a first order plus dead time model

The data generating process is given by the following differential equation: $y(t) = a + b u(t - \theta) + c \frac {dy} {dt}$ Now imagine having as data a long time series for both $y$ and $u$. If ...
0
votes
0answers
17 views

after using AIC, how to determine the contribution or effect size of a individual covariate?

I am confused and looking for advise. I have found myself in this same situation repeatedly in the last few months. I want to know if covariate X is influential or important. However, I also ...
0
votes
1answer
34 views

How to compare the significance of two models from two different datasets?

I have two different regression models which I learned from two different data sets. Is there any statistical method which shows the significance of models based on the number of parameters and cross ...
0
votes
0answers
20 views

What are the best criteria to select the model for Lasso regression?

I have two different formulations of the Lasso regression for the same problem. For each formulation, I selected the best model based on cross validation error. But Now, I want to compare two models ...
0
votes
0answers
11 views

Multiple predictors in a GAMM: which method for model selection?

Recently a comment of a reviewer made me ask myself questions about model selection. My data are disease counts on algae. I would like to test the relationship between disease counts and percentage ...
2
votes
1answer
70 views

“Better” goodness-of-fit tests than chi squared for histogram modeling?

I work on data from a mass spectrometer that produces billions upon billions of count histograms, and I need a good way to test whether these histograms are consistent with one or several model ...
1
vote
2answers
52 views

How to refer to AIC model-averaged parameters and confidence intervals

I am writing up results from regression analysis where I used AICc model averaging to arrive at my final parameter estimates. I am wondering how best to refer to these parameters and their 95% ...
12
votes
1answer
229 views

Why doesn't Wilks' 1938 proof work for misspecified models?

In the famous 1938 paper ("The large-sample distribution of the likelihood ratio for testing composite hypotheses", Annals of Mathematical Statistics, 9:60-62), Samuel Wilks derived the asymptotic ...
0
votes
0answers
33 views

Correction of variance structure in gls after selection of the fixed effects using R

I'm fitting a gls following these steps: I select the random effect holding the fixed part unchanged. I.e. I try different variance and correlation structures and random effect. Once I find my ...
0
votes
0answers
48 views

Is it statistically sound to use Lasso for variable selection even when $n\gg p$?

I have a classification task where $n\gg p$ (like 440000 vs. 23). I want to use Lasso (glmnet in R) to select the variables first, then use techniques like random ...
5
votes
0answers
75 views

If coefficient variance is incorrect (for a regression parameter), does that mean the model's log-likelihood is incorrect?

I am using logistic regression to estimate ~probability of a sample unit being used by an animal. Due to my sampling design it is unavoidable that there is overlap between 'used' sample units and ...
6
votes
1answer
103 views

How do we interpret model priors if models intersect?

Let $\Theta$ be some parameter space and $\Theta_1,\Theta_2,\dots,\Theta_s \subset \Theta$ be parameter subsets which represent competing models in some model selection procedure. There are no ...
1
vote
0answers
28 views

Model averaging effect sizes of Gamma family GLMs

I'm trying to get some model averaged effect sizes from a set of candidate models, all of them assuming a Gamma error distribution, according to the theory given by the book from Burnham and Anderson ...
1
vote
1answer
40 views

Hausman's test for all $\beta$s – comparing FE vs RE models

I fit several two level models in SAS using PROC MIXED: an empty model with multilevel structure (null), a model with a level 2 covariate (partial model), and a ...
2
votes
1answer
66 views

Different AIC values for the same model using step()?

I'm working with a GLM to try and optimize the model, and there are 152 predictive variables. A LOT of these are not significant, so I'm trying to figure out which ones to remove through use of the ...
0
votes
0answers
21 views

validation of a Zero Adjusted Gamma model

I am using a Zero Adjusted Gamma regression (two-part ZAGA) model to estimate the effect of a psychometric categorical factor ("expected recovery", with 3 levels) on cost associated with treatment ...