Model selection is a problem of judging which model from some set performs best. Popular methods include $R^2$, AIC and BIC criteria, test sets, and cross-validation. To some extent, feature selection is a subproblem of model selection.
1
vote
2answers
72 views
Overfitting when using corrected AIC for model selection
I am using the corrected AIC to select the lag order in a simple AR(p) model. I chose the the AICc since my sample is fairly small (n=135). The AICc minimal model is the AR(15). To me it seems like an ...
6
votes
1answer
108 views
Can you compare AIC values as long as the models are based on the same dataset?
I am doing some forecasting in R using Rob Hyndman's forecast package. The paper belonging to the package can be found here.
In the paper, after explaining the automatic forecasting algorithms, the ...
1
vote
1answer
45 views
Is there a good specification error test against a generalized alternative?
Suppose I believe a sample is drawn from a population that is distributed according to some specified distributional family. I intend to estimate the parameters of the distribution using some ...
-1
votes
0answers
35 views
Model selection in weka based on error measures
I would like to select a good model using WEKA for my 48 points data set. Each point is a 4-tuple -- one response and three input variables (all are numeric values). I have identified that the three ...
0
votes
0answers
22 views
Should the number of parameters depend on the purpose of the model?
I am curious what are some arguments for/against increasing/decreasing the number of parameters depending on the purpose on the model.
I am currently building a model which will be used for ...
1
vote
1answer
46 views
Generalization of Heckman's two step procedure
Following my question Heckman sample selection vs. OLS about the meaning of Mills ratio I am wondering why some researchers estimate a generalization of Heckman instead of the actual Heckman's ...
1
vote
1answer
33 views
Heckman sample selection vs. OLS
If the mills ratio of a Heckman selection model (with/without exclusion restriction) is not significant, shall I prefer to estimate my model with OLS instead?
Or is it better to use the estimates from ...
1
vote
0answers
26 views
Is there a way to correct standard errors and/or prediction intervals for multiple comparison after doing backwards selection?
It is well known that most model selection algorithms can easily fall into a multiple comparison trap. To quote Friedman:
Consider developing a regression model in a context where substantive
...
2
votes
1answer
37 views
How to report the results of cross-validation for comparing two models?
I want to compare the predictive power of two models. For this, I calculated the difference in some measure of predictive performance over many cross-validation replications. Now I have a distribution ...
1
vote
1answer
32 views
Manually evaluating the DIC: very big number of effective parameters?
My problem is the following: I'm evaluating the fit to a function $f(\mathbf{x},\theta)$ via MCMC (because I have some priors on the parameters), and I'm trying to evaluate the DIC, given by:
...
0
votes
1answer
84 views
AIC, BIC, DIC, model selection criteria
I am trying to understand the difference between these parameters, and their application. Was hoping to get some correction/clarification to my statements. I have a training set and cross-validation ...
0
votes
0answers
15 views
Choosing prior distribution in LDA
how do you set prior distribution of K in LDA and can it be used for feature selection to improved selection accuracy of document. Abbey
0
votes
1answer
24 views
BMI at baseline & followup with exposure at baseline; model change or BMI at FUP? Control for BMI baseline?
For a prospective occupational cohort where everyone is exposed to one or more chemical agents, examining BMI at follow-up compared to a specific chemical exposure at baseline, is it necessary to ...
0
votes
0answers
26 views
Compare the predictive power of a model between datasets
I have two sets of continuous response data for the same group of species, but in different areas (area a and area b). I am building a model for each area separately, to predict the area-specific ...
0
votes
0answers
62 views
Using logistic regression to fit a weird dataset
I have the variables $Y$, $x_{m1}$, $x_{m2}$, $\ldots$, $x_{mn}$, $x_{p1}$, $x_{p2}$, $\ldots$, $x_{pn}$, $x_{q1}$, $x_{q2}$, $\ldots$, $x_{qn}$.
$Y$ is the class label (0 or 1). $x_{mj}$, $x_{pj}$ ...
0
votes
2answers
21 views
Selecting out main effects when locational-scale-invariance isn't an issue(?)
I'm starting a new thread to ask a specific question that I'm left with after reading this old, good thread:
Do all interactions terms need their individual terms in regression model?
The gist of ...
0
votes
0answers
27 views
complicated model without a huge dataset
I've got a set of data that I'm trying to model. Lots of the data is missing, so I'm using multiple imputation. I've got about 360 observations and 13 variables. I'm also using GAMs, but that ...
2
votes
2answers
80 views
ML with fastest classification speed
I have a data classification problem and I'm wondering what is the best machine learning approach to use for the particular constraints of my problem.
My constraints are as follows:
- the data ...
1
vote
0answers
51 views
Forward Stepwise selection
I am assuming the following model:
$Y = \beta X + \epsilon$
Here both $X$ and $Y$ are matrices. I fit the least squares model without any regularization and get the matrix $\beta$. I would like to ...
1
vote
1answer
56 views
SVM model selection for datasets with sharp corners
I'm working with an artificially generated dataset that is separated by many sharp corners. As an example, imagine an H-shape in a 3D (or higher-dimensional) space. Points within the H are positive, ...
2
votes
1answer
70 views
What is the dimension (k) of these regression models?
I am attempting to use Akaike's Information Criterion to select the most appropriate model for some data. This means I need to find the likelihood of my data under various models, compute the AIC ...
0
votes
2answers
84 views
Should I keep the interaction term?
I am doing research on mortality. I am running cox regression and I am adjusting for 40 variables which are proven in the literature to be related with mortality. My main exposure is X. In my final ...
4
votes
1answer
114 views
Two simple or one complex model, BIC and likelihood
I have a set of data points with a total number of Nt. I know a priori that the data comes from two distinct processes (distributions). I am trying to find the optimal model parameters together with ...
1
vote
1answer
95 views
Backward selection in a Cox regression model
My goal is to fit a cox regression model in SAS, for which I use the PROC PHREG statement. As I am still new to regression methods, I would appreciate a little of ...
3
votes
3answers
76 views
Analogous measure of AIC which uses the posterior distribution for model selection?
Suppose the following problem: I have $n$ models, $M_k$, each with parameters $\mathbf{\theta}_k$ for a data set $D$. There where previous observations of a subset of the parameters which are common ...
0
votes
0answers
54 views
How to perform step() when n < p in R?
I am trying to perform stepwise regression for variable selection in R.
In matlab, the stepwisefit function is able to work in ...
6
votes
5answers
305 views
What to do with collinear variables
Disclaimer: This is for a homework project.
I'm trying to come up with the best model for diamond prices, depending on several variables and I seem to have a pretty good model so far. However I have ...
1
vote
1answer
99 views
Interactions terms and higher order polynomials
If I were interested in fitting two-way interactions between a linear explanatory variable $a$ and another explanatory variable $b$ that has a quadratic relationship with the dependent variable $y$, ...
2
votes
1answer
80 views
Curvature terms and model selection
I am running a model selection analysis with a continuous dependent variable and a variety of continuous and categorical explanatory variables. For two of my continuous explanatory variables I am ...
0
votes
2answers
200 views
How to find how independent variables affect dependent variables
I'm a bit of a novice at maths and am trying to get my head around a problem.
I have 3 independent variables which affect 1 dependent variable. I want to create a 4D model which will give me the 4th ...
1
vote
0answers
38 views
How do I know if the differences in ICs among candidate models are significant?
I'm doing some exploratory modelling on a data set with 29 covariates and an additional 11 variables that are of interest to my research question.
My strategy is to develop a model with a subset of ...
1
vote
0answers
57 views
How to approach multilevel model?
I have kind of a general question, and am hoping someone can point me in the right direction.
I'll set up a fake example problem.
You have a length $10$ x $1$ column vector $\vec{x}$ and a $10$ ...
0
votes
0answers
55 views
When using lmer is a random intercept being estimated more than once if specified in seperate grouping factors?
I know there are a slew of lmer specification questions already floating around. Please let me know if this is a duplicate, or if it is deemed off-topic, and I'll delete it.
I am using a forward ...
1
vote
1answer
116 views
significant difference between coefficients
What is the correct way to test for significant differences in parameter estimates in the following case:
I have a dependent variable (height for age) and a independent variable (household state ...
5
votes
1answer
342 views
Omitted variable bias in linear regression
I have a philosophical question regarding omitted variable bias.
We have the typical regression model (population model)
$$
Y= \beta_0 + \beta_1X_1 + ... + \beta_nX_n + \upsilon,
$$
where the ...
0
votes
1answer
81 views
Partial correlation
I want to create a regression model to predict state crime rate. There are two variables among 10 ( Vi= # of violent crimes per 100,000 population, Vi2 = # of violent crimes per 10,000 population) ...
0
votes
0answers
46 views
Confusion while using lassoglm
I am trying to fit a logistic regression model with L1 regularization on my data. My data has just 12 examples with 150 features. So I used L1 regularization. Now when I use the lassoglm function like ...
3
votes
1answer
71 views
Is it possible to compare model fit for a gaussian vs binomial glm
I'm modelling some proportional data using a binomial GLM - ie fitted values are sigmoidal - but the data look like they may be better fit by a straight line (model validation plots of a simple linear ...
0
votes
0answers
61 views
How do I explain that software implemented model selection procedures should not be used unsupervised?
I know that people generally say that procedures which select a model based on information criterion lead to inconsistent model selections.
I read a paper by Leeb and Potscher (2005), MODEL SELECTION ...
3
votes
2answers
219 views
Prerequisites for AIC model comparison
What are exactly the prerequisites, that need to be fulfilled for AIC model comparison to work?
I just came around this question when I did comparison like this:
...
0
votes
0answers
42 views
Stat tools and methodologies for principal-agent problem
What are the most useful, popular or important statistical tools and/or methodologies for empirical study of 'agent-principal problem'?
A simple example will be very useful.
References or book ...
0
votes
2answers
144 views
Logistic regression and odds ratio
If the odds ratio is greater than one with an insignificant p value for a variable in logistic regression should the variable be kept in the model?
Can I select the variable with odds close to 1?
...
2
votes
1answer
141 views
Regression with 3 measurement points (in R)
I have a regression in which I try to understand how much variance of the metric dependent variable each of the regressors explains. I use the package R relaimpo (Grömping, 2006) for that purpose, ...
0
votes
1answer
122 views
Variable selection and logistic regression
I am using Matlab, I have a $600 \times 9$ matrix with each row representing the 9 features which I am trying to evaluate using logistic regression.
I understand that I need to perform feature ...
2
votes
0answers
45 views
Algorithm for selecting interactions without very many degrees of freedom (mgcv, gam)
I've got a semi-parametric model that I'm fitting with GAM's (mgcv in R). It is of the form
$$y = \theta + X'\beta + f(Z) + \text{Interactions!} + \epsilon$$
I've got 289 observations.
Many of ...
6
votes
1answer
335 views
Linear model with log-transformed response vs. generalized linear model with log link
In this paper titled "CHOOSING AMONG GENERALIZED LINEAR MODELS APPLIED TO MEDICAL DATA" the authors write:
In a generalized linear model, the mean is transformed, by the link
function, instead ...
1
vote
0answers
105 views
Selecting GAM with/without random effects - residual plots vs. AIC
My question is about fitting GAMs with a random effect in mgcv, using s(x, bs="re"). I understand that determining the random effects structure should occur before determining the fixed effects ...
0
votes
0answers
51 views
Test and train datatasets with prediction models and TunePareto package
I'm trying to chose the best prediction/classification model for a concrete problem. The methodology I've been asked to follow is this:
Separate the data into test and training.
Run a concrete model ...
1
vote
0answers
156 views
Comparing non-nested models with out of sample likelihood
I recently read a paper in which the authors claim that in order to compare the forecasting performance of two non-nested models, models A and B, a valid procedure is to fit models A and B on the same ...
4
votes
2answers
424 views
Relative variable importance with AIC
I am confused and just need some confirmation about calculating the relative variable importance value for the co-variates I used in AIC model selection procedures. I know that there is this one ...
