Model selection is a problem of judging which model from some set performs best. Popular methods include $R^2$, AIC and BIC criteria, test sets, and cross-validation. To some extent, feature selection is a subproblem of model selection.

learn more… | top users | synonyms (1)

2
votes
1answer
36 views

Cross-validation and logistic regression

I'm interested in building a set of candidate models in R for an analysis using logistic regression. Once I build the set of candidate models and evaluate their fit to the data using AICc (...
2
votes
1answer
26 views

How does step function selects best linear Models which includes polynomial effects and interaction effects in R?

I try to find "best" linear models with continuous and categorical covariables with Interaction Effect by BIC. The continuous covariables should have a quadratic effect on the response variable. ...
3
votes
2answers
52 views

Generate fake data consistent with adjusted R^2 pattern

Is it possible to specify a vector of adjusted $R^2$ values (or any other measure like AIC, BIC, $C_p$) for the set of all possible models in a data set, and then generate data that is consistent with ...
1
vote
1answer
29 views

Is there a default parameter choice for the spike-and-slab prior?

In the spike-and-slab prior, one needs to specify $h_{0j} = P(\beta_j=0)$, which demonstrates our prior belief about how likely $\beta_j$ to be an important predictor. Is there a default choice for ...
0
votes
1answer
17 views

Holdout set for image task

I need to validate whether one or two templates/shapes are present in an image. Fitting two templates has a better maximum likelihood then fitting one template which is a clear symptom of overfitting. ...
0
votes
1answer
63 views

How to use LASSO to select glm model gaussian

I have a small sample size n<20. I want to find which combination of 8 variables better predict y. I was using a stepAICc but it is suggested to away stepwise model selection. I have tried lars ...
0
votes
1answer
32 views

is there a way to plot best glm model in model selection

I have run this glm model y~poly(xa,2)+poly(xb,2)+... Then have found the best fitting model using AICc. The best fitting model has a subset of the ...
0
votes
0answers
28 views

Model selection in nonlinear fitting

Learning curves are fitted with multiple trendlines (exponential, power, logarithm). The fitting is performed by the Levenberg–Marquardt algorithm. So far so good. The question is, how to select the ...
2
votes
1answer
36 views

How is the Akaike Information Criteron applied for model with large number of predictors?

I am reading a paper (details not very relevant) which assumes that the market cost $C$ of a trade is related to $N$ predictors $X_1,\dots,X_N$ (page 25) through a linear relationship $$C = \beta_0 + ...
22
votes
3answers
2k views

Is p-value essentially useless and dangerous to use?

First are some background information. This article "The Odds, Continually Updated" from NY Times happened to catch my attention. To be short, it states that [Bayesian statistics] is proving ...
0
votes
0answers
6 views
0
votes
0answers
9 views

Does it make sense to report equally-fit, more complex, model, if it fits better a theory?

I have two (logistic) regression models for which the deviance is not significantly distinct (p = 0.7). One of them has education, gender and age explaining variable Y. In the other, I have added a ...
6
votes
2answers
120 views

Analyse ACF and PACF plots

I want to see if I am on the right track analysing my ACF and PACF plots: Background: (Reff: Philip Hans Franses, 1998) As both ACF and PACF show significant values, I assume that an ARMA-model ...
1
vote
1answer
63 views

The best model of an AICc-based model selection on a very small sample has an high number of predictors: does it make any sense?

I'm working with a very small sample size (N=14) and I'm using AICc to identify the most parsimonious model using a large number of possible predictors. Unexpectedly the best model has six predictors! ...
0
votes
0answers
26 views

Assesing the explanatory power of predictors, interactions and combination of terms

I have a model with 5 basic predictors and all interactions between the predictors themselves. Something like (I'm simplifying here, in reality I have many more variables): ...
0
votes
1answer
32 views

Comparisons of confirmatory and exploratory model in IRT

I am trying to decide whether a theoretically derived (i.e. confirmatory) IRT model fits the data better than some parsimonious (i.e. exploratory) IRT model. Specifically, I have 14 binary indicators ...
2
votes
0answers
14 views

Is it possible to compare the parsimony of models with the same number of parameters and explanatory variables?

Parsimony is often defined as the minimisation of unnecessary parameters or explanatory variables in a model. But models also have structure - functional forms that can change. Between two models that ...
2
votes
0answers
11 views

Half-Normal Plot of Coefficients from Binary Factorial Experiments

After I wrote this all up I debated whether or not I should post it because I think I know the answer to this question (after looking at the two models I'd end up with), but since I don't really know ...
0
votes
0answers
25 views

Modeling of Multivariate Data

Suppose I have a multivariate data set. For the sake of example, lets say that the dimension of my data set is $p=7$ and I have a matrix which contains samples of this multivariate data set. Now ...
1
vote
1answer
23 views

K-fold validation, how to use MSE and STD for model selection

When using K-fold validation for model selection I'm wondering what's the best approach to select a model using both the mean square error (MSE) and the standard deviation of errors among folds (STD). ...
0
votes
2answers
45 views

p value vs prediction error

In a lot of fields (like medicine) to check if a variable is related to an output is controlled if the p-value of that variable in a regression model is significant. For example: ...
1
vote
1answer
42 views

Marginal Likelihood in PYMC

I am using the PYMC toolbox in python in order to carry out a model selection problem using MCMC. What I would like to have for each model is the marginal log-likelihood (i.e. model evidence). The ...
1
vote
0answers
12 views

Testing sequentially nested models

Assume I have a simple model with (at least) four parameters $\beta_1, \beta_2, \beta_3, \beta_4$. If I would want to test $H_o: \beta_1 = \beta_2 \& \beta_3 = \beta_4$ by using the likelihood ...
0
votes
0answers
18 views

Model-selection and interactions: Glmnet dropping underlying terms

I have a dataset and want to do an Ancova with several explanatory variables (several factors and two covariates) and their interactions. I want to select the best model using glmnet and lasso ...
1
vote
3answers
38 views

How to compare 2 predictive models where one uses predictor with missing values

I am developing a model to predict y from a dataset (N=20,000) that contains x1, x2. Say I ...
9
votes
1answer
256 views

What is the difference between 'hypothesis testing' and 'model selection'?

In literature, both terms are often used synonymously or interwoven. I am now trying to find a clear distinction between both terms. From my point of view, a hypothesis is usually expressed via a ...
2
votes
1answer
37 views

Is it better to remove higher order interactions or least significant terms first in model simplification?

I have a mixed effects model with 3 explanatory factors and a full interaction set (including 3 way interaction). This is the full model. Factor 1 is time and I am interested in the change in the ...
2
votes
0answers
16 views

Am I using all subsets model selection appropriately?

This is the first time I am posting a question, so please excuse any etiquette violations and poorly worded questions! I am working on the analysis for a chapter of my thesis. I am examining the ...
2
votes
1answer
78 views

Mixed models and backward elimination

Let's say I have a data like this, and I'm trying to build a mixed model. ...
9
votes
2answers
168 views

Random- and fixed-effects structure in linear-mixed models

Consider the following data from a two-way within subjects design: ...
1
vote
1answer
45 views

Choosing the right ARIMA model in MATLAB

I have a problem regarding choosing the right model for historical data that I need to forecast. when drawing the ACF and PACf, a clear seasonality appears at lag 24 as you can see in the figure: I ...
0
votes
0answers
29 views

Calculating Log-Likelihood from Simulated Distribution

I want to perform some sort of model evaluation of a multivariate distribution with the property that it is difficult/impossible to calculate the likelihood (of the whole model, you can do it for ...
0
votes
1answer
28 views

Machine Learning approach to solve this selection problem?

I have a set of items S. items can be joined to groups consisting of up to x items. For a group of items i can derive a score Y using some unknown performance measure. What would be the most efficient ...
1
vote
1answer
67 views

Model uncertainty (model averaging) and R-Squared (R2)

Is it possible to calculate r-squared for an "average model"? Lets say I have 4 different response variables that I want to model to a set (or subset) of 4 independent variables. I'd then like to ...
8
votes
2answers
302 views

Best suggested textbooks on Bootstrap resampling?

I just wanted to ask which are in your opinion the best available books on bootstrap out there. By this I don't necessarily only mean the one written by its developers. Could you please indicate ...
1
vote
2answers
77 views

What model should I use for this?

Consider a dataset with 30 samples. We have a response $Y$ and 12 potential predictors $X_1, \cdots, X_{12}$. We fit two models. The first model $M_1$ includes only $X_1$ and $X_2$ and has an RSS of ...
0
votes
0answers
31 views

How to evaluate variable contribution to a prediction

I need to produce a logistic regression model that: Gives a ranked list of the most important factors Allows you to break out most important factors for each new observation scored Given that I'm ...
1
vote
1answer
172 views

what if response variable is 'yes or no' in R?

How to analyze above the data to predict the probability that people have disease with a model? Factors thought to influence infection include city, age, and diet. BUT, I don't know how to do ...
4
votes
1answer
112 views

Justifying and choosing a proper scoring rule

Most resources on proper scoring rules mention a number of different scoring rules like log-loss, Brier score or spherical scoring. However, they often don't give much guidance on the differences ...
0
votes
1answer
35 views

What is the correct order of these hierarchical priors?

I'm quite new to Bayesian data analysis, so this is most likely am easy question. I have the following model: a function f has two exponential rate parameters $\lambda_1$ and $\lambda_2$ and for some ...
0
votes
1answer
52 views

Structural equation modelling: model selection

I am currently trying to fit a structural equation model in R with the Lavaan package. I have this model that fits my data pretty good. This model is what I consider the full model, it has all paths ...
1
vote
0answers
27 views

Comparing a model with a latent variable to one without

I would like to test whether my 3 dependent variables all load onto an underlying latent variable or if a latent variable is not necessary to explain the relationship between the IVs and the DVs. In ...
3
votes
2answers
54 views

How to compare predictions from MLE-based regression Vs. predictions from bayesian regression?

Say I have two linear regression models that I want to use for predictions. Linear regression: \begin{equation} \mathbf{y} \sim \mathcal{N}(\mathbf{X^Tb}, \Sigma_y) \end{equation} Bayesian linear ...
0
votes
0answers
49 views

AIC rankings: Why would a global model rank lower than an intercept-only model?

I'm working with some real-world (i.e. potentially messy) data on the nesting ecology of several bird species. I'm attempting to relate the daily survival rate of nests to vegetation characteristics ...
1
vote
1answer
42 views

$R^2$ for mixed models = ICC?

I will be referring here to Nakagawa and Schielzeth (2013). As those authors state, $R^2$ for OLS regression could be defined as follows: $$R^2 = \frac{\sum^n_{i=1}(\bar{y} - ...
0
votes
0answers
12 views

Averaging against mixed-effect model

I have experimental data that contains information about 50 participants who performed a task in five different conditions (different set sizes). The result is the time spent on the task. My data is ...
0
votes
0answers
15 views

With posterior inclusion probability how do I settle on the final predictive model?

After using the spike-and-slab prior to perform Bayesian model selection, I get the posterior distribution of my variables, from which I calculate the inclusion probability for each variable. How do ...
1
vote
0answers
41 views

Selecting an appropriate VAR model

I would like to receive critical comments on an idea explained below. Suppose I have variables $x_1$ through $x_K$, and this is a time series setting. My aim is to forecast variable $x_1$. I know ...
0
votes
0answers
47 views

Advice for feature selection or feature extraction with semi-supervised learning

I am trying to solve a semi-supervised learning problem using LaplacianSVM. However, before applying LapSVM I would like either to perform feature selection or feature extraction. Furthermore, after ...
3
votes
0answers
49 views

Explain log likelihood behaviour

(This question is related to a previous one I made, here) I have a set of 2D observations (measured data) of sample size $N$: $$O = \{(x_1, y_1), (x_2, y_2), ..., (x_N, y_N)\}$$ I also have a model ...