Model selection is a problem of judging which model from some set performs best. Popular methods include $R^2$, AIC and BIC criteria, test sets, and cross-validation. To some extent, feature selection is a subproblem of model selection.

learn more… | top users | synonyms (1)

3
votes
2answers
32 views

Generalized log likelihood ratio test for non-nested models

I understand that if I have two models A and B and A is nested in B then, given some data, I can fit the parameters of A and B using MLE and apply the generalized log likelihood ratio test. In ...
0
votes
1answer
24 views

Log-likelihood (and AIC) of robust nlrob model differs from standard nls model

Comparing models generated by nlrob to ones generated by nls, I've noticed that even though the models might be nearly identical, the log-likelihood of the models is sometimes significantly different, ...
0
votes
0answers
23 views

Iterative Addition of Variables to Model Based on P Value

Suppose I have 64 columns that I have chosen out of 500+ columns based on the fact that they have the highest pairwise correlation (is this a good way?). I take 16 of these columns and run a simple ...
1
vote
0answers
15 views

Model diagnostics for a glmmPQL in R mixed-effects model

Several texts (both online and published books) have been reviewed prior to asking this. What diagnostics are accepted as best practise for a generalised linear mixed-effects model fitted in R using ...
0
votes
0answers
16 views

Compare LMM GLMM (generalised linear mixed model, negative binomial) by numerical measure (AIC BIC, cross validation, R² squared) for model validation

How to compare results of generalized linear mixed model (GLMM, negative binomial) with a log transformed linear mixed model (multilevel, hierarchical) . I have a data set (counts), which is nested. ...
0
votes
0answers
9 views

What is a “concordance score” for regression coefficients?

I came across this "concordance score" in a set of slides called Penalized regression methods for ranking variables by effect size, with applications to genetic mapping studies, by Ji Zhu: $$ ...
1
vote
0answers
25 views

Proper scoring rules for observations with different supports

Suppose to have a bivariate variable $z_t=(x_t, y_t)$ indexed by $t=1,2, ..., T$. Suppose now that the two components have different support, i.e. in my specific problem $x_t \in \mathcal{S}$, where ...
1
vote
0answers
25 views

Model without endogeneity correction has lower AIC than one with correction

I have two models, one with endogeneity correction (includes correction terms obtained using Heckman) and one without. The correction terms are significant in the second stage model, yet the AIC/BIC ...
0
votes
0answers
25 views

Applying the Akaike Information Criterion to Data

I have some variables that I would like to run regressions on, to create a model, but I am unsure about how to actually AIC (or the BIC). Unfortunately I have not yet taken a mathematical statistics ...
1
vote
0answers
13 views

Variable coarsening in Naive Bayes

Say we have a binary classification problem that we solve with Naive Bayes. All features are categorical variables. Say we focus on a single feature that takes one of $N$ possible values. If $N$ is ...
0
votes
0answers
4 views

how to measure the model recovery performance

I have some simulated linear model $y = X\beta+\epsilon$ where $\beta$ is sparse. I am comparing different techniques to recover the structure of $\beta$. I have so 4 values True Positives, False ...
1
vote
1answer
35 views

How to deal with bouts of equilibrium in an online learning setting?

I have a kalman filter (a recursive least square filter, really) regressing over real-time streams of data. Because the data-generating process varies slightly over time I add an exponential ...
0
votes
0answers
52 views

using Root Mean Squared Error (RMSE) to compare models with different sample size

I'm using k-fold cross-validation to compare different models. I splitted my dataset in 6 chunks and used 4 random chunks as training set and the remaining 2 as a test set. Now I fitted n-different ...
0
votes
0answers
39 views

BRT analysis using count data

I have some problems with my BRT analysis. Introduction to the data: The dependent variable is count data of a specific palm species in SA, and the predictors consists of nine various kinds of ...
1
vote
0answers
25 views

Cross-Validation vs. AICc for LASSO

I was working on a research project in which I try to estimate the the individual contribution of a group of regional political leaders to local economic growth. The major challenge is that there is ...
3
votes
1answer
26 views

Non-constant standard deviation in residuals

I am fitting a model in the frequency domain, and my fit looks as follows: As you can see, the model function does not fit the data perfectly, especially in the higher frequencies. So, I examined ...
2
votes
1answer
63 views

Is it possible to use the Breusch-Pagan Lagrange multiplier test (xttest0) in Stata for unbalanced data?

Is it possible to use xttest0 in Stata with unbalanced panel data? I want to test whether the I should use pooled OLS or random effects estimation. What does this test actually do?
1
vote
0answers
32 views

Estimating a first order plus dead time model

The data generating process is given by the following differential equation: $y(t) = a + b u(t - \theta) + c \frac {dy} {dt}$ Now imagine having as data a long time series for both $y$ and $u$. If ...
0
votes
0answers
15 views

after using AIC, how to determine the contribution or effect size of a individual covariate?

I am confused and looking for advise. I have found myself in this same situation repeatedly in the last few months. I want to know if covariate X is influential or important. However, I also ...
0
votes
1answer
30 views

How to compare the significance of two models from two different datasets?

I have two different regression models which I learned from two different data sets. Is there any statistical method which shows the significance of models based on the number of parameters and cross ...
0
votes
0answers
18 views

What are the best criteria to select the model for Lasso regression?

I have two different formulations of the Lasso regression for the same problem. For each formulation, I selected the best model based on cross validation error. But Now, I want to compare two models ...
0
votes
0answers
9 views

Multiple predictors in a GAMM: which method for model selection?

Recently a comment of a reviewer made me ask myself questions about model selection. My data are disease counts on algae. I would like to test the relationship between disease counts and percentage ...
2
votes
1answer
58 views

“Better” goodness-of-fit tests than chi squared for histogram modeling?

I work on data from a mass spectrometer that produces billions upon billions of count histograms, and I need a good way to test whether these histograms are consistent with one or several model ...
1
vote
2answers
41 views

How to refer to AIC model-averaged parameters and confidence intervals

I am writing up results from regression analysis where I used AICc model averaging to arrive at my final parameter estimates. I am wondering how best to refer to these parameters and their 95% ...
12
votes
1answer
218 views

Why doesn't Wilks' 1938 proof work for misspecified models?

In the famous 1938 paper ("The large-sample distribution of the likelihood ratio for testing composite hypotheses", Annals of Mathematical Statistics, 9:60-62), Samuel Wilks derived the asymptotic ...
0
votes
0answers
30 views

Correction of variance structure in gls after selection of the fixed effects using R

I'm fitting a gls following these steps: I select the random effect holding the fixed part unchanged. I.e. I try different variance and correlation structures and random effect. Once I find my ...
0
votes
0answers
47 views

Is it statistically sound to use Lasso for variable selection even when $n\gg p$?

I have a classification task where $n\gg p$ (like 440000 vs. 23). I want to use Lasso (glmnet in R) to select the variables first, then use techniques like random ...
5
votes
0answers
74 views

If coefficient variance is incorrect (for a regression parameter), does that mean the model's log-likelihood is incorrect?

I am using logistic regression to estimate ~probability of a sample unit being used by an animal. Due to my sampling design it is unavoidable that there is overlap between 'used' sample units and ...
6
votes
1answer
103 views

How do we interpret model priors if models intersect?

Let $\Theta$ be some parameter space and $\Theta_1,\Theta_2,\dots,\Theta_s \subset \Theta$ be parameter subsets which represent competing models in some model selection procedure. There are no ...
1
vote
0answers
24 views

Model averaging effect sizes of Gamma family GLMs

I'm trying to get some model averaged effect sizes from a set of candidate models, all of them assuming a Gamma error distribution, according to the theory given by the book from Burnham and Anderson ...
1
vote
1answer
36 views

Hausman's test for all $\beta$s – comparing FE vs RE models

I fit several two level models in SAS using PROC MIXED: an empty model with multilevel structure (null), a model with a level 2 covariate (partial model), and a ...
2
votes
1answer
62 views

Different AIC values for the same model using step()?

I'm working with a GLM to try and optimize the model, and there are 152 predictive variables. A LOT of these are not significant, so I'm trying to figure out which ones to remove through use of the ...
0
votes
0answers
20 views

validation of a Zero Adjusted Gamma model

I am using a Zero Adjusted Gamma regression (two-part ZAGA) model to estimate the effect of a psychometric categorical factor ("expected recovery", with 3 levels) on cost associated with treatment ...
0
votes
0answers
16 views

how to generate samples for model selection testing on adaboost

I want to assess some statistical methods for model selection on binary classification using adaBoost. For this, I have to generate artificial samples (input data) and create an oracle that has the ...
1
vote
0answers
43 views

Comparing AIC among models with different amounts of data

I have a data set with many missing observations for certain parameters (NA values) in it. I have been performing model selection using AIC. Based on AIC scores I have reduce the model to the form ...
0
votes
0answers
27 views

how to generate data for model selection in machine learning

I want to test statistical methods for model selection in binary classification problems. In order to do so, I plan to generate data and then use some specific model (i.e. fix parameters for my model) ...
0
votes
1answer
53 views

What is a relevant level for percentage deviance explained in a GLM?

I am trying to sort out the unique effect of various environmental predictors on species occurrence (presence/absence data). I have been running glm models in R ...
1
vote
0answers
50 views

Model selection on variance parameters (and about REML)

Various descriptions on model selection on random effects of Linear Mixed Models instruct to use REML. I know difference between REML and ML at some level, but I don't understand why REML should be ...
0
votes
0answers
30 views

Can all the p-values from drop1 models be high even when the p-value for the full lm model is low

I have used the R lm() function to fit a model with one numeric and ten categorical variables. The p-value reported for the regression is very low: Residual standard error: 2.459 on 320 degrees of ...
0
votes
0answers
95 views

R: glmulti for mixed models returns several best models, Automated Model Selection (multilevel analysis, hierarchical model, nested data)

I searched the entire web including this forum on some help on how to use the glmulti package in order to identify the "optimal" fixed part of a mixed model with a given random part. However, I could ...
8
votes
4answers
366 views

Choosing a regression model

How can one objectively (read "algorithmically") select an appropriate model for doing a simple linear least-squares regression with two variables? For example, say the data seem to show a quadratic ...
0
votes
1answer
45 views

Crossvalidation and/or testdata. Always use both or can one exclude the other?

I'm trying to build a two class classifier on a dataset of around 570 samples. Im evaluating several classificiation stratigies (LDA, QDA,RDA, logistic, logistic with some additional ellements like ...
3
votes
1answer
80 views

How to discriminate between non-nested models?

So first this question asks "how I would discriminate between model(1) and model(2)". It appears that both models are non-nested so I would come up with a hybrid model consisting of Xt,Zt and Qt ...
2
votes
3answers
125 views

How to test a program that uses machine learning

I understand how to test programs that are either right or wrong. But what if permissible some inaccuracy? How to distinguish a bug in the implementation of bad classifier? Naive solution (baseline)? ...
1
vote
2answers
41 views

Bayes factor for selecting between two beta-distributions

I have two beta-distributions: $H_1 = Beta(\alpha_1, \beta_1) $ and $H_2 = Beta(\alpha_2, \beta_2) $ (parameters are known), and I'd like to estimate whether a new sample $D$ rather comes from ...
1
vote
0answers
30 views

Regression model selection when there are more variables than cases

I have a database with 200+ variables and less then 50 cases. I need to choose an optimal model that predicts one dependent variable. Are stepwise/lasso regressions still appropriate methods to ...
0
votes
0answers
46 views

Model selection of multinomial logistic regression for 2x3 contingency table

I want to analyze a 2x3 contingency table using multinomial logistic regression and I hope to be able to do it in Matlab or in R. I have looked around in old threads, but haven't been able to find a ...
3
votes
2answers
147 views

Statistical method for modelling of disease incidence for NGO

I am trying to provide help to an NGO in the non-profit sector that is running a disease screening program: The program visits thousands of villages a year. A village has a population (on average ...
0
votes
1answer
121 views

Choosing correct C and g parameters for libsvm

libsvm 3.18 Features: 10 I have used following, parameter range: ...
0
votes
0answers
15 views

testing for differences in means after an experimental change

I have a set of Mechanical Turk-like workers who have been paid $0.25 per task to complete tasks from a finite, but very large, pool of tasks, and I have data on how many tasks they've completed and ...