Model selection is a problem of judging which model from some set performs best. Popular methods include $R^2$, AIC and BIC criteria, test sets, and cross-validation. To some extent, feature selection is a subproblem of model selection.

learn more… | top users | synonyms (1)

0
votes
0answers
13 views

country-time dummies and specification error

Currently, I am doing research about the impact of free trade agreements to trade flow. I have panel data that consist of 86 countries from 1980-2012. I use panel data estimation : Pooled Least ...
0
votes
0answers
6 views

Trial with two sites, two years - lme model building and simplification (Q2)

I have got two questions on an agricultural field trial that was conducted at two sites in two conscutive years. Virtually everything was the same in all trials (variety, spacing, planting ...
0
votes
1answer
25 views

Model comparison with AIC based on different sample size

Let's assume I have two models M1 and M2: M1: y ~ x1 + x2 + x3 M2: y ~ x1 + x2 + x3 + x4 Since variable x4 has some missing values the sample size of M2 is ...
2
votes
4answers
69 views

How to deal with curvature in residuals plot

I am trying to do a multiple linear regression in R but am having some problems. I have a set up where I am trying to develop a multiple linear regression model for one variable (y) using six other ...
1
vote
0answers
11 views

How to obtain predicted values from a gamm() using averaged coefficients (MuMIn)?

I want to extract the predicted values from a gamm() whose coefficients have been averaged using the package MuMIn, but I'm getting an error. ...
-2
votes
0answers
20 views

Best ARIMA Model

What ARIMA model would be the best fit for the data provided?: ...
0
votes
2answers
27 views

Cox PH model selection and validation

I am trying to analyze my data using survival CoX PH in SPSS v.19 and also attempting to make different prediction models (without and with a biomarker of interest). I am a clinician (not a ...
1
vote
0answers
31 views

Leaps() or AIC for model selection

I am deciding how many predictors to include in my model - I currently have 4. When I use the leaps() function, the smallest value for the residual standard error ...
0
votes
2answers
57 views

Variable Selection [duplicate]

I am using R/RStudio to code a regression (and to optimize the function) over 50+ different variables. For the optimization to work I need to fit a higher order function (I am not sure to what degree ...
0
votes
0answers
17 views

Model selection based on bic.glm function output and K-fold cross validation

I'm looking for a logistic regression model with high prediction power, and the best model returned by the bic.glm function has a posterior probability of 0.348. I ...
1
vote
0answers
13 views

Influence.measure output interpretation in R

Playing with lm is a good way into understanding various statistical concepts. Assume we have a linear model 'Model1' for which we get the following results: for ...
0
votes
0answers
83 views

Criteria for choosing the most appropriate logistic regression model

I've fitted 16 logistic regression models to my data and I'm not sure as to which model to choose as my final model. I looked at a couple of things to help me choose my final model. 1) significance of ...
0
votes
0answers
25 views

How to interpret log-likelihood outputs from MASS::fitdistr (R)

AIM: Fit the best distribution to columns in a dataset (30k records) so that I can to go on to produce test data that is in a similar distribution. WHAT I'VE DONE SO FAR: Using R, I have found and ...
0
votes
0answers
18 views

Limited Dependent Variable Analysis

I am using a dependent variable that is the average of a rating. Customers use a one to ten scale regarding how much they liked a meal. I have the average rating (each meal is rated by 22 to 129 ...
0
votes
0answers
20 views

Model choice for nonnegative and positive continuous right skewed outcome

I am trying to analyze a set of nonnegative continuous non-integer data (i.e. the data points are not counts) that are mostly between 0 and 3 whose distribution is highly right-skewed even after log ...
3
votes
1answer
51 views

How to perform parameters tuning for machine learning?

I have a very basic question regarding parameter tuning using grid search. Typically some machine learning methods have parameters that need to be tuned using grid search. For example, in the ...
1
vote
1answer
37 views

Cost Benefit Analysis of Pre-screening Widgets for Faults before they Fail

I want to build a model that determines whether to pre-screen my widgets for defects. If I do pre-screen, it costs a fixed amount per check and I resolve the problem 100% of the time. If I don't ...
1
vote
1answer
51 views

Suggest models for prediction based on small sample data

I am not a traditional statistics guy. I am from an electrical engineering background. So, spare me for lack of jargon. The model is to be used for predicting agricultural output based on previous ...
1
vote
1answer
115 views

When is a proper scoring rule a better estimate of generalization in a classification setting?

A typical approach to solving a classification problem is to identify a class of candidate models, and then perform model selection using some procedure like cross validation. Typically one selects ...
4
votes
3answers
92 views

Selecting most important variable based on individual p-value vs. partial $R^2$

I'm trying to solve a problem where the goal is to find an association between children's cortisol values (y) against their mother's weekly cortisol averages (...
0
votes
0answers
27 views

Sequential Sum of Squares, degree of freedom when number of variables greater than number of samples

I would like to use the Sequential Sum of Squares test. But the degree of freedom for the denominator is (n - p - 1), where n = number of samples, and p = number of variables in the full model. What ...
0
votes
0answers
11 views

How to calculate performance statistics of continus learning model?

I have continuous weak stationary process that I need to map on logical result value (0,1). For example I want model that in 2 ways: logistic regression native Bayes classificator I want to know ...
0
votes
0answers
56 views

Output of dredge and model.avg functions of MuMIn

I am running a GLM of about 15 predictor variables and using dredge and model.avg functions of ...
1
vote
0answers
30 views

Choosing an appropriate model - What does this phrase mean?

I have a question where I am given a short story: An investigator wishes to evaluate the effect of medicine administered for colds on reaction time. Data represent the reaction times (in ...
0
votes
1answer
46 views

Including confounders in a model

Suppose that you are performing a linear regression examining the main effect $x_1$ and want to adjust for possible confounders $x_2, x_3, x_4$. Is it better to have an unadjusted model and a model ...
5
votes
1answer
126 views

Equivalence of AIC and p-values in model selection

In a comment to the answer of this question, it was stated that using AIC in model selection was equivalent to using a p-value of 0.154. I tried it in R, where I used a "backward" subset selection ...
0
votes
1answer
79 views

Superiority of LASSO over forward selection/backward elimination in terms of the cross validation prediction error of the model

I obtained three reduced models from a original full model using forward selection backward elimination L1 penalization technique (LASSO) For the models obtained using forward selection/backward ...
0
votes
0answers
77 views

Variable selection with principal components analysis

I ran principal components analysis in R on my data. All my regressors are continuous, non categorical variables, except gender which I excluded. I will add it and compare model 1 = PCA to model 2 = ...
1
vote
1answer
57 views

Linear model predictor selection. Which method to use ?

From what I understand, there are 3 main types of predictor selection method for linear models, namely, 1 Subset Selection, 2 Shrinkage and 3 Dimension Reduction. The subset selection includes the ...
0
votes
0answers
21 views

Cox regression with interaction giving NA values, how to interpret? + Choosing covariates based on p-values

I am working with a dataset with 12 covariates(categorical). The covariates have different Levels, some are from 0-5, and one has as much as 0-11 categories. I started with a univariate analysis to ...
1
vote
1answer
56 views

Does Akaike information criterion penalize model complexity any more than is necessary to avoid overfitting

The AIC penalizes complex models. Obviously a certain penalty for complex models is necessary to avoid overfitting of statistical models: otherwise we would favour a model which is simply a copy of ...
4
votes
2answers
138 views

Sparse parameters when computing AIC, BIC, etc

I'm designing large-scale, regularized logistic regression models with lots of sparse, binarized features. e.g. isUS, isFR, etc. As a result, a lot of the weights in the model are zero. I'm wondering ...
2
votes
1answer
61 views

How to select tuning parameter for regularized regressions for interpretation?

I'm using linear regression to predict a continuous variable using a large number (~200) of binary indicator variables. I have around 2,500 data rows. There are a couple of issues here: When I run ...
0
votes
0answers
27 views

Building prediction model with estimated predictor variables

I'm planning to use logistic regression with multiple (~5) predictor variables to predict whether something happens or not. I have two types of predictor variables: known (measurable) variables and ...
6
votes
0answers
121 views

Model selection with Firth logistic regression

In a small data set ($n\sim100$ ) that I am working with, several variables give me perfect prediction/separation. I thus use Firth logistic regression to deal with the issue. If I select the best ...
2
votes
3answers
65 views

Partial F-test vs Model Selection

I'm a first year statistics graduate student taking a course in regression. In the previous chapter we covered, we discussed partial F-tests for deciding whether to include a predictor variable. In ...
0
votes
0answers
80 views

Best model selection (after using several methods)

I have 30 variables and am trying to select the best model. I have run the following methods on a 'large' data set (having removed a smaller test set): OLS, best subset selection, stepwise ...
0
votes
0answers
33 views

Comparing 2D heat maps of observed data to 2D model predictions

From "How to ask a statistics question": PROBLEM you are trying to solve: Given two-dimensional heat maps of responses (DV), choose the 2D model (also a heat map, but can have different ranges of ...
4
votes
1answer
93 views

When correlation turns too high?

It is possible to find a lot online on intepreting correlation coefficients. But I find often difficult to decide when to drop a variable from a linear model because of its correlation with another ...
1
vote
0answers
19 views

Ensemble learning combined with item response theory

I am recently learning item response theory, which is always used to select some combinations of tests for students of different specific levels. Then I have an immature idea. Could I apply item ...
0
votes
1answer
122 views

Confused about multicollinearity, variable selection and interaction terms

I have run a few tests/methods on my data and am getting contradictory results. I have a linear model saying: reg1 = lm(weight = height + age + gender (categorical) + several other variables). If I ...
2
votes
1answer
81 views

Why do I get different BIC values when I use regsubsets and lm in R

I used regsubsets to find a model with lowest BIC; height is our D.V. , the code I typed is below: ...
1
vote
1answer
46 views

Divergence measure of two classifiers' performance?

I have two classifiers built with the same data. How can I measure divergence of these models? I found something like DIC but I don't know how to calculate this in R?
1
vote
0answers
175 views

Backward stepwise regression with cross validation in R

I would like to do model selection using backward stepwise procedure and cross validation. https://www.otexts.org/fpp/5/3 I have used stepAIC in ...
1
vote
1answer
55 views

AIC values and their use in stepwise model selection for a simple linear regression

The Wikipedia article for AIC says the following (emphasis added): As an example, suppose that there were three models in the candidate set, with AIC values 100, 102, and 110. Then the second ...
1
vote
0answers
16 views

Relation between Scoring rule and Loss function in Parameter estimation and model selection?

Initially, I had only heard of MLE and use it for almost everything, e.g. point estimate and model selection (with some penalty). Then, MSE appeared, which seems to play the same role as MLE does. I ...
2
votes
1answer
74 views

How to calculate model error (like MSE) for a multivariate proportional response?

I have data where the response is multivariate and proportional (rows [observations] sum to 1). I am modelling this response using a Dirichlet regression via the DirichletReg R package where the ...
1
vote
0answers
39 views

Can statistics be used to measure which model is the most accurate

I found three websites which lists the times of sunrises and sunsets in the place where I live. But as those times differ occasionally some minutes, I would like to know if there is a method to find ...
25
votes
3answers
420 views

AIC,BIC,CIC,DIC,EIC,FIC,GIC,HIC,IIC — Can I use them interchangeably?

On p. 34 of his PRNN Brian Ripley comments that "The AIC was named by Akaike (1974) as 'An Information Criterion' although it seems commonly believed that the A stands for Akaike". Indeed, when ...
5
votes
0answers
58 views

Can the Burnham-Anderson book on multimodel inference be recommended?

As motivated by the recent change of the default model selection statistic in the R's forecast package from AIC to AICc, I am curious whether the latter is indeed applicable wherever the former is. ...