Model selection is a problem of judging which model from some set performs best. Popular methods include $R^2$, AIC and BIC criteria, test sets, and cross-validation. To some extent, feature selection is a subproblem of model selection.

learn more… | top users | synonyms (1)

1
vote
0answers
6 views

Selecting optimal model when only smoothed data available?

I have a graph of some (highly nonlinear) experimental spectrum which is obtained by smoothing results of several repeated measurements obtained by different experimental methods. The graph also ...
1
vote
0answers
19 views

Follow-on to “Training with the full dataset after cross-validation” - sequential parameter estimation

Background: Here is the background for the question, both the question itself and the answer given by Dikran Marsupial. Training with the full dataset after cross-validation? It asks about after ...
0
votes
0answers
17 views

How to develop a robust procedure to select a predictive model

Imagine you have a matrix, M, of n input variables and m values per variable. There's also a ...
4
votes
2answers
162 views

Should train and test datasets have similar variance?

If variance of test dataset is lower than the one of the train dataset is it worth splitting the data? Since we know our dataset will always be limited is it fair to select models under the above ...
0
votes
0answers
8 views

Compressed Population Complexity in Minimum Description Length (MDL)

I am studying the MDL and found it is sum of model complexity and compressed population complexity. To my understanding, model complexity refers to number of bits to encode the model, which can be ...
2
votes
0answers
15 views

Posterior Predictive Checks

I understand what the posterior predictive distribution is, and I have been reading about posterior predictive checks, although it isn't clear to me what it does yet. What exactly is the posterior ...
1
vote
1answer
20 views

Empirical Bayes vs “non-informative” priors

I am familiar with the mechanics with both methods, but don't know what factors I should consider when choosing between these two approaches for adjusting a prior. I would imagine that, on a case by ...
1
vote
1answer
24 views

model comparison when alternatives are not all nested within one another

I am running a glmm with three fixed effects: opponent 1 size ("1") opponent 2 size ("2") opponent 1 size - opponent 2 size ("diff") I am unable to run all three variables in the model at once ...
0
votes
0answers
17 views

Determine which variable or variables is/are the most efficient to predict the outcome

I have a small dataset (n=74) with a +/- 50 variables, not the best data but I have to work with it. The variables are used to select a product. I want to determine which variable or variables is/are ...
0
votes
2answers
34 views

Logistic regression, Chi-square, and study design

I have a study in which I have developed a new predictor (binary) for a disease (also a binary variable). The study has two parts. In the first part, I want to test if my predictor is strongly ...
3
votes
1answer
68 views

Model selection: where to start? [closed]

For a general modeling problem, there are literally at least a dozen choices of statistical and algorithmic models to choose from. Off the top of my head, choices could be: regression (and its ...
1
vote
1answer
34 views

Comparing two models

I am interested in comparing two logistic regression models. The two models are nested: model 1 contains all predictors, and model 2 contains all predictors except 1. My goal is to test if removing ...
0
votes
0answers
26 views

selecting a link function for GLM's

If you don't care about using GLM model parameters to predict anything, but simply want to select the best-fitting model for your data, is it necessary to get into the theoretical debate as to which ...
1
vote
0answers
29 views

the role of AIC versus p-values in model selection

Let's say you are trying to choose between two models. One has two significant fixed effects. The other includes only one of the two fixed effects from the aforementioned model but has a lower AIC ...
1
vote
0answers
28 views

Model selection using mean AIC for very huge data sets

I want to select a model which best performs for a very huge data set. However, the data set is too large to calculate a model within reasonable time. If this is the case, is the following a ...
3
votes
1answer
113 views

How to choose between exponential and gamma distributions

I have same data and I would like to choose a model for it. To start with I fit an exponential distribution and a gamma distribution. Now I wanted to do a simple likelihood ratio test . However, I ...
0
votes
0answers
2 views

True discovery rate or intrinsic sensitivity?

I'm trying to evaluate my species distribution model by the rate of discovered\predicted localities. For example, my model at a stated threshold predicts 50 grid suitable cells. Then, after field ...
0
votes
0answers
24 views

GEE in SPSS: the case of ever-decreasing QICc values

I've been using GEEs in SPSS 22 to analyze my dissertation data and have discovered an interesting problem: when trying to figure out which subset of model factors have the lowest QICc, and therefore ...
0
votes
1answer
22 views

Determining if time has influence on correlation between x and y

Is there a sort of statistics model I could use to determine if time has an influence on the correlation between two sets of data? For example say I have a column $X$ and a column $Y$, both just ...
1
vote
2answers
77 views

Why is Lasso regression for high dimensional data better than Stepwise AIC?

I know Lasso eventually set some parameters to zero, acting like variable selection. I also read from paper talking about automated variable selection method like Stepwise AIC can be troublesome. So ...
0
votes
0answers
21 views

How to deal with categorical features.

Recently I am playing in the famous Big-Data website Kaggle. There is a Display Advertising Challenge. In this competition, you are giving a training file which include huge records. the records is ...
0
votes
0answers
23 views

Why is $R^2$ poor for AR model selection used for forecasting?

There is a related question here, about how to calculate the R-squared on a regression with ARIMA errors. I found the answer quite useful, and hoped for some elaboration, particularly on Rob's closing ...
0
votes
0answers
18 views

Model selection in nlme's

I can think of four ways to perform model selection nlme's: LRT, AIC, ...
0
votes
0answers
17 views

Number of parameters for AIC for a particular model

I know there have been a few well answered questions on this topic, but i have found myself in a bit of a special case this time. I am using AIC for model selection, and i am having trouble counting ...
3
votes
0answers
34 views

Understanding the Rank Probability Score

The ranked probability score (RPS) is a measure of how good forecasts that are expressed as probability distributions are in matching observed outcomes. Both the location and spread of the forecast ...
1
vote
1answer
65 views

Testing the variance part of a Generalized Linear Model out of sample

Suppose I have a response vector and an ANOVA design (for simplicity, assume it’s a one-way ANOVA with two treatments). A few Generalized Linear Models (Poisson, Negative Binomial, etc) are fitted to ...
12
votes
5answers
353 views

What is the upside of treating a factor as random in a mixed model?

I have a problem embracing the benefits of labeling a model factor as random for a few reasons. To me it appears like in almost all cases the optimal solution is to treat all of the factors as fixed. ...
1
vote
0answers
16 views

Choosing models with similar AIC values [duplicate]

I'm using a multinomial logistic regression analysis to examine deer behavioural responses to camera traps in terms of 7 predictors (both singly and their interactions). I have found that the model ...
0
votes
0answers
6 views

Looking for catalog of “mechanism classes” that give rise to specific curve shapes

(I apologize for the length of this post. I don't know how to frame the question more succinctly.) I have some experimental data, in the form of a collection of curves with fairly little noise, ...
1
vote
0answers
32 views

How can I use Akaike's Information Criterion to compare two models of multi-peaked emission spectra?

I have several photoluminescence emission spectra that I am trying to fit curves to. The spectra each have a slight baseline and four peaks. The independent variable $x_i$ is wavelength (converted to ...
0
votes
1answer
20 views

model specification problem

I have the following model that I would like to rebuild: $Y_{i,t}=a+bx_{i,t}+cx_{i,t-1}+e_{i,t}$ I' am wondering now whether this is the same as the model above: $Y_{i,t}=a+ d\Delta ...
2
votes
1answer
57 views

Cross-validation with dummy variables?

Does it make sense to use cross-validation with factor variables that have 3+ levels? When using bestglm, I get an error saying that it doesn't work with categorical variables. In the documentation ...
0
votes
0answers
11 views

Error when using glmulti with aicc?

Does anyone know the solution to the error I get when I run glmulti with the aicc criteria: Error in if (length(lesCrit) == confsetsize && minouN - minou >= -deltaM && : missing value ...
2
votes
0answers
18 views

Motivating likelihood ratio test vs Wald test for paper reviewer

I've got back reviews for a paper I've submitted, with the following problem. I have two logistic regression models, say y ~ A, and y ~ A + B, where B is a factor with several levels. I have ...
0
votes
1answer
33 views

How to get both MSE and R2 from a sklearn GridSearchCV?

I can use a GridSearchCV on a pipeline and specify scoring to either be 'MSE' or 'R2'. I can then access ...
2
votes
2answers
52 views

Is cross validation for validating a model or for selecting best model in different kinds of models?

I am confused about the concept of cross validation and its usage. As I read about cross validation before, it is a way of validating a model. I did cross validation in my project (developing ...
1
vote
1answer
30 views

Can I use cross-validation to select optimal parameters SEPARATELY?

I'm wondering if there is any math/stat theory out there to support or deny this idea: I am using cross-validation and building models over a vector of parameter values to then choose the optimal ...
2
votes
2answers
57 views

What interactions to include in my GLM model?

I realize this might be a too general question, so I'll describe what I'm doing right now first. I'm working for a virtual insurance company and I have this dataset. It has severity (meaning ...
3
votes
2answers
47 views

Generalized log likelihood ratio test for non-nested models

I understand that if I have two models A and B and A is nested in B then, given some data, I can fit the parameters of A and B using MLE and apply the generalized log likelihood ratio test. In ...
0
votes
1answer
32 views

Log-likelihood (and AIC) of robust nlrob model differs from standard nls model

Comparing models generated by nlrob to ones generated by nls, I've noticed that even though the models might be nearly identical, the log-likelihood of the models is sometimes significantly different, ...
0
votes
0answers
38 views

Iterative Addition of Variables to Model Based on P Value

Suppose I have 64 columns that I have chosen out of 500+ columns based on the fact that they have the highest pairwise correlation (is this a good way?). I take 16 of these columns and run a simple ...
1
vote
0answers
59 views

Model diagnostics for a glmmPQL in R mixed-effects model

Several texts (both online and published books) have been reviewed prior to asking this. What diagnostics are accepted as best practise for a generalised linear mixed-effects model fitted in R using ...
0
votes
0answers
49 views

Compare LMM GLMM (generalised linear mixed model, negative binomial) by numerical measure (AIC BIC, cross validation, R² squared) for model validation

How to compare results of generalized linear mixed model (GLMM, negative binomial) with a log transformed linear mixed model (multilevel, hierarchical) . I have a data set (counts), which is nested. ...
0
votes
0answers
17 views

What is a “concordance score” for regression coefficients?

I came across this "concordance score" in a set of slides called Penalized regression methods for ranking variables by effect size, with applications to genetic mapping studies, by Ji Zhu: $$ ...
1
vote
0answers
27 views

Proper scoring rules for observations with different supports

Suppose to have a bivariate variable $z_t=(x_t, y_t)$ indexed by $t=1,2, ..., T$. Suppose now that the two components have different support, i.e. in my specific problem $x_t \in \mathcal{S}$, where ...
1
vote
0answers
32 views

Model without endogeneity correction has lower AIC than one with correction

I have two models, one with endogeneity correction (includes correction terms obtained using Heckman) and one without. The correction terms are significant in the second stage model, yet the AIC/BIC ...
0
votes
0answers
34 views

Applying the Akaike Information Criterion to Data

I have some variables that I would like to run regressions on, to create a model, but I am unsure about how to actually AIC (or the BIC). Unfortunately I have not yet taken a mathematical statistics ...
1
vote
0answers
15 views

Variable coarsening in Naive Bayes

Say we have a binary classification problem that we solve with Naive Bayes. All features are categorical variables. Say we focus on a single feature that takes one of $N$ possible values. If $N$ is ...
0
votes
0answers
4 views

how to measure the model recovery performance

I have some simulated linear model $y = X\beta+\epsilon$ where $\beta$ is sparse. I am comparing different techniques to recover the structure of $\beta$. I have so 4 values True Positives, False ...
1
vote
1answer
36 views

How to deal with bouts of equilibrium in an online learning setting?

I have a kalman filter (a recursive least square filter, really) regressing over real-time streams of data. Because the data-generating process varies slightly over time I add an exponential ...