Model selection is a problem of judging which model from some set performs best. Popular methods include $R^2$, AIC and BIC criteria, test sets, and cross-validation. To some extent, feature selection is a subproblem of model selection.

learn more… | top users | synonyms (1)

0
votes
0answers
7 views

What is a “concordance score” for regression coefficients?

I came across this "concordance score" in a set of slides called Penalized regression methods for ranking variables by effect size, with applications to genetic mapping studies, by Ji Zhu: $$ ...
1
vote
0answers
21 views

Proper scoring rules for observations with different supports

Suppose to have a bivariate variable $z_t=(x_t, y_t)$ indexed by $t=1,2, ..., T$. Suppose now that the two components have different support, i.e. in my specific problem $x_t \in \mathcal{S}$, where ...
1
vote
0answers
22 views

Model without endogeneity correction has lower AIC than one with correction

I have two models, one with endogeneity correction (includes correction terms obtained using Heckman) and one without. The correction terms are significant in the second stage model, yet the AIC/BIC ...
0
votes
0answers
24 views

Applying the Akaike Information Criterion to Data

I have some variables that I would like to run regressions on, to create a model, but I am unsure about how to actually AIC (or the BIC). Unfortunately I have not yet taken a mathematical statistics ...
1
vote
0answers
11 views

Variable coarsening in Naive Bayes

Say we have a binary classification problem that we want to solve with Naive Bayes. All features are categorical variables. Say we focus on a single feature that takes one of $N$ possible values. If ...
0
votes
0answers
4 views

how to measure the model recovery performance

I have some simulated linear model $y = X\beta+\epsilon$ where $\beta$ is sparse. I am comparing different techniques to recover the structure of $\beta$. I have so 4 values True Positives, False ...
1
vote
1answer
35 views

How to deal with bouts of equilibrium in an online learning setting?

I have a kalman filter (a recursive least square filter, really) regressing over real-time streams of data. Because the data-generating process varies slightly over time I add an exponential ...
0
votes
0answers
42 views

using Root Mean Squared Error (RMSE) to compare models with different sample size

I'm using k-fold cross-validation to compare different models. I splitted my dataset in 6 chunks and used 4 random chunks as training set and the remaining 2 as a test set. Now I fitted n-different ...
0
votes
0answers
38 views

BRT analysis using count data

I have some problems with my BRT analysis. Introduction to the data: The dependent variable is count data of a specific palm species in SA, and the predictors consists of nine various kinds of ...
1
vote
0answers
23 views

Cross-Validation vs. AICc for LASSO

I was working on a research project in which I try to estimate the the individual contribution of a group of regional political leaders to local economic growth. The major challenge is that there is ...
3
votes
1answer
20 views

Non-constant standard deviation in residuals

I am fitting a model in the frequency domain, and my fit looks as follows: As you can see, the model function does not fit the data perfectly, especially in the higher frequencies. So, I examined ...
2
votes
1answer
48 views

Is it possible to use the Breusch-Pagan Lagrange multiplier test (xttest0) in Stata for unbalanced data?

Is it possible to use xttest0 in Stata with unbalanced panel data? I want to test whether the I should use pooled OLS or random effects estimation. What does this test actually do?
1
vote
0answers
32 views

Estimating a first order plus dead time model

The data generating process is given by the following differential equation: $y(t) = a + b u(t - \theta) + c \frac {dy} {dt}$ Now imagine having as data a long time series for both $y$ and $u$. If ...
0
votes
0answers
12 views

after using AIC, how to determine the contribution or effect size of a individual covariate?

I am confused and looking for advise. I have found myself in this same situation repeatedly in the last few months. I want to know if covariate X is influential or important. However, I also ...
0
votes
1answer
29 views

How to compare the significance of two models from two different datasets?

I have two different regression models which I learned from two different data sets. Is there any statistical method which shows the significance of models based on the number of parameters and cross ...
0
votes
0answers
18 views

What are the best criteria to select the model for Lasso regression?

I have two different formulations of the Lasso regression for the same problem. For each formulation, I selected the best model based on cross validation error. But Now, I want to compare two models ...
0
votes
0answers
8 views

Multiple predictors in a GAMM: which method for model selection?

Recently a comment of a reviewer made me ask myself questions about model selection. My data are disease counts on algae. I would like to test the relationship between disease counts and percentage ...
2
votes
1answer
50 views

“Better” goodness-of-fit tests than chi squared for histogram modeling?

I work on data from a mass spectrometer that produces billions upon billions of count histograms, and I need a good way to test whether these histograms are consistent with one or several model ...
1
vote
2answers
36 views

How to refer to AIC model-averaged parameters and confidence intervals

I am writing up results from regression analysis where I used AICc model averaging to arrive at my final parameter estimates. I am wondering how best to refer to these parameters and their 95% ...
12
votes
1answer
214 views

Why doesn't Wilks' 1938 proof work for misspecified models?

In the famous 1938 paper ("The large-sample distribution of the likelihood ratio for testing composite hypotheses", Annals of Mathematical Statistics, 9:60-62), Samuel Wilks derived the asymptotic ...
0
votes
0answers
29 views

Correction of variance structure in gls after selection of the fixed effects using R

I'm fitting a gls following these steps: I select the random effect holding the fixed part unchanged. I.e. I try different variance and correlation structures and random effect. Once I find my ...
0
votes
0answers
46 views

Is it statistically sound to use Lasso for variable selection even when $n\gg p$?

I have a classification task where $n\gg p$ (like 440000 vs. 23). I want to use Lasso (glmnet in R) to select the variables first, then use techniques like random ...
5
votes
0answers
73 views

If coefficient variance is incorrect (for a regression parameter), does that mean the model's log-likelihood is incorrect?

I am using logistic regression to estimate ~probability of a sample unit being used by an animal. Due to my sampling design it is unavoidable that there is overlap between 'used' sample units and ...
6
votes
1answer
102 views

How do we interpret model priors if models intersect?

Let $\Theta$ be some parameter space and $\Theta_1,\Theta_2,\dots,\Theta_s \subset \Theta$ be parameter subsets which represent competing models in some model selection procedure. There are no ...
1
vote
0answers
23 views

Model averaging effect sizes of Gamma family GLMs

I'm trying to get some model averaged effect sizes from a set of candidate models, all of them assuming a Gamma error distribution, according to the theory given by the book from Burnham and Anderson ...
1
vote
1answer
36 views

Hausman's test for all $\beta$s – comparing FE vs RE models

I fit several two level models in SAS using PROC MIXED: an empty model with multilevel structure (null), a model with a level 2 covariate (partial model), and a ...
2
votes
1answer
60 views

Different AIC values for the same model using step()?

I'm working with a GLM to try and optimize the model, and there are 152 predictive variables. A LOT of these are not significant, so I'm trying to figure out which ones to remove through use of the ...
0
votes
0answers
19 views

validation of a Zero Adjusted Gamma model

I am using a Zero Adjusted Gamma regression (two-part ZAGA) model to estimate the effect of a psychometric categorical factor ("expected recovery", with 3 levels) on cost associated with treatment ...
0
votes
0answers
16 views

how to generate samples for model selection testing on adaboost

I want to assess some statistical methods for model selection on binary classification using adaBoost. For this, I have to generate artificial samples (input data) and create an oracle that has the ...
1
vote
0answers
39 views

Comparing AIC among models with different amounts of data

I have a data set with many missing observations for certain parameters (NA values) in it. I have been performing model selection using AIC. Based on AIC scores I have reduce the model to the form ...
0
votes
0answers
27 views

how to generate data for model selection in machine learning

I want to test statistical methods for model selection in binary classification problems. In order to do so, I plan to generate data and then use some specific model (i.e. fix parameters for my model) ...
0
votes
1answer
50 views

What is a relevant level for percentage deviance explained in a GLM?

I am trying to sort out the unique effect of various environmental predictors on species occurrence (presence/absence data). I have been running glm models in R ...
1
vote
0answers
50 views

Model selection on variance parameters (and about REML)

Various descriptions on model selection on random effects of Linear Mixed Models instruct to use REML. I know difference between REML and ML at some level, but I don't understand why REML should be ...
0
votes
0answers
30 views

Can all the p-values from drop1 models be high even when the p-value for the full lm model is low

I have used the R lm() function to fit a model with one numeric and ten categorical variables. The p-value reported for the regression is very low: Residual standard error: 2.459 on 320 degrees of ...
0
votes
0answers
80 views

R: glmulti for mixed models returns several best models, Automated Model Selection (multilevel analysis, hierarchical model, nested data)

I searched the entire web including this forum on some help on how to use the glmulti package in order to identify the "optimal" fixed part of a mixed model with a given random part. However, I could ...
8
votes
4answers
365 views

Choosing a regression model

How can one objectively (read "algorithmically") select an appropriate model for doing a simple linear least-squares regression with two variables? For example, say the data seem to show a quadratic ...
0
votes
1answer
43 views

Crossvalidation and/or testdata. Always use both or can one exclude the other?

I'm trying to build a two class classifier on a dataset of around 570 samples. Im evaluating several classificiation stratigies (LDA, QDA,RDA, logistic, logistic with some additional ellements like ...
3
votes
1answer
78 views

How to discriminate between non-nested models?

So first this question asks "how I would discriminate between model(1) and model(2)". It appears that both models are non-nested so I would come up with a hybrid model consisting of Xt,Zt and Qt ...
2
votes
3answers
123 views

How to test a program that uses machine learning

I understand how to test programs that are either right or wrong. But what if permissible some inaccuracy? How to distinguish a bug in the implementation of bad classifier? Naive solution (baseline)? ...
1
vote
2answers
40 views

Bayes factor for selecting between two beta-distributions

I have two beta-distributions: $H_1 = Beta(\alpha_1, \beta_1) $ and $H_2 = Beta(\alpha_2, \beta_2) $ (parameters are known), and I'd like to estimate whether a new sample $D$ rather comes from ...
1
vote
0answers
29 views

Regression model selection when there are more variables than cases

I have a database with 200+ variables and less then 50 cases. I need to choose an optimal model that predicts one dependent variable. Are stepwise/lasso regressions still appropriate methods to ...
0
votes
0answers
45 views

Model selection of multinomial logistic regression for 2x3 contingency table

I want to analyze a 2x3 contingency table using multinomial logistic regression and I hope to be able to do it in Matlab or in R. I have looked around in old threads, but haven't been able to find a ...
3
votes
2answers
145 views

Statistical method for modelling of disease incidence for NGO

I am trying to provide help to an NGO in the non-profit sector that is running a disease screening program: The program visits thousands of villages a year. A village has a population (on average ...
0
votes
1answer
115 views

Choosing correct C and g parameters for libsvm

libsvm 3.18 Features: 10 I have used following, parameter range: ...
0
votes
0answers
15 views

testing for differences in means after an experimental change

I have a set of Mechanical Turk-like workers who have been paid $0.25 per task to complete tasks from a finite, but very large, pool of tasks, and I have data on how many tasks they've completed and ...
1
vote
1answer
59 views

Model fit is High but Ramsey RESET Test suggests omitted variables. What to do?

I'm trying to figure out what next steps to take. I created a model and ran OLS on a very large sample of data (over 400000 observations) and got an R-squared value of 0.80. So the model fit seems ...
1
vote
0answers
43 views

Testing if two models have a significant different forecast (error)

I have two different models which estimate a continuous variable $X$ depending on N categorial covariates $V_i$. Is there a way to test if both models produce significantly different results? I ...
2
votes
1answer
46 views

Interpreting LASSO tables in SAS

I have been working on LASSO in SAS lately, and I'm still trying to figure out how to work with the options, but my main question for which I have not been able to find an answer on the internet so ...
1
vote
0answers
50 views

Problem with Kullback–Leibler divergence criteria

I am using Kullback–Leibler divergence criteria for comparing my estimation and true density functions, but I have zero value on my estimation function when I have a testing set of size 10000, mostly ...
2
votes
1answer
33 views

What is an indicator in statistics and why is it used to refine models?

In a problem we are asked to "refine the fitted model by using an indicator for the outlier". What does it mean to use an indicator for an outlier?