Model selection is a problem of judging which model from some set performs best. Popular methods include $R^2$, AIC and BIC criteria, test sets, and cross-validation. To some extent, feature selection is a subproblem of model selection.

learn more… | top users | synonyms (1)

0
votes
0answers
5 views
0
votes
0answers
6 views

Does it make sense to report equally-fit, more complex, model, if it fits better a theory?

I have two (logistic) regression models for which the deviance is not significantly distinct (p = 0.7). One of them has education, gender and age explaining variable Y. In the other, I have added a ...
5
votes
2answers
107 views

Analyse ACF and PACF plots

I want to see if I am on the right track analysing my ACF and PACF plots: Background: (Reff: Philip Hans Franses, 1998) As both ACF and PACF show significant values, I assume that an ARMA-model ...
1
vote
1answer
48 views

The best model of an AICc-based model selection on a very small sample has an high number of predictors: does it make any sense?

I'm working with a very small sample size (N=14) and I'm using AICc to identify the most parsimonious model using a large number of possible predictors. Unexpectedly the best model has six predictors! ...
0
votes
0answers
25 views

Assesing the explanatory power of predictors, interactions and combination of terms

I have a model with 5 basic predictors and all interactions between the predictors themselves. Something like (I'm simplifying here, in reality I have many more variables): ...
0
votes
1answer
30 views

Comparisons of confirmatory and exploratory model in IRT

I am trying to decide whether a theoretically derived (i.e. confirmatory) IRT model fits the data better than some parsimonious (i.e. exploratory) IRT model. Specifically, I have 14 binary indicators ...
2
votes
0answers
14 views

Is it possible to compare the parsimony of models with the same number of parameters and explanatory variables?

Parsimony is often defined as the minimisation of unnecessary parameters or explanatory variables in a model. But models also have structure - functional forms that can change. Between two models that ...
2
votes
0answers
10 views

Half-Normal Plot of Coefficients from Binary Factorial Experiments

After I wrote this all up I debated whether or not I should post it because I think I know the answer to this question (after looking at the two models I'd end up with), but since I don't really know ...
0
votes
0answers
21 views

Modeling of Multivariate Data

Suppose I have a multivariate data set. For the sake of example, lets say that the dimension of my data set is $p=7$ and I have a matrix which contains samples of this multivariate data set. Now ...
1
vote
1answer
20 views

K-fold validation, how to use MSE and STD for model selection

When using K-fold validation for model selection I'm wondering what's the best approach to select a model using both the mean square error (MSE) and the standard deviation of errors among folds (STD). ...
0
votes
2answers
43 views

p value vs prediction error

In a lot of fields (like medicine) to check if a variable is related to an output is controlled if the p-value of that variable in a regression model is significant. For example: ...
1
vote
1answer
39 views

Marginal Likelihood in PYMC

I am using the PYMC toolbox in python in order to carry out a model selection problem using MCMC. What I would like to have for each model is the marginal log-likelihood (i.e. model evidence). The ...
1
vote
0answers
12 views

Testing sequentially nested models

Assume I have a simple model with (at least) four parameters $\beta_1, \beta_2, \beta_3, \beta_4$. If I would want to test $H_o: \beta_1 = \beta_2 \& \beta_3 = \beta_4$ by using the likelihood ...
0
votes
0answers
16 views

Model-selection and interactions: Glmnet dropping underlying terms

I have a dataset and want to do an Ancova with several explanatory variables (several factors and two covariates) and their interactions. I want to select the best model using glmnet and lasso ...
1
vote
3answers
37 views

How to compare 2 predictive models where one uses predictor with missing values

I am developing a model to predict y from a dataset (N=20,000) that contains x1, x2. Say I ...
9
votes
1answer
248 views

What is the difference between 'hypothesis testing' and 'model selection'?

In literature, both terms are often used synonymously or interwoven. I am now trying to find a clear distinction between both terms. From my point of view, a hypothesis is usually expressed via a ...
2
votes
1answer
34 views

Is it better to remove higher order interactions or least significant terms first in model simplification?

I have a mixed effects model with 3 explanatory factors and a full interaction set (including 3 way interaction). This is the full model. Factor 1 is time and I am interested in the change in the ...
2
votes
0answers
13 views

Am I using all subsets model selection appropriately?

This is the first time I am posting a question, so please excuse any etiquette violations and poorly worded questions! I am working on the analysis for a chapter of my thesis. I am examining the ...
2
votes
1answer
72 views

Mixed models and backward elimination

Let's say I have a data like this, and I'm trying to build a mixed model. ...
9
votes
2answers
163 views

Random- and fixed-effects structure in linear-mixed models

Consider the following data from a two-way within subjects design: ...
1
vote
1answer
41 views

Choosing the right ARIMA model in MATLAB

I have a problem regarding choosing the right model for historical data that I need to forecast. when drawing the ACF and PACf, a clear seasonality appears at lag 24 as you can see in the figure: I ...
0
votes
0answers
29 views

Calculating Log-Likelihood from Simulated Distribution

I want to perform some sort of model evaluation of a multivariate distribution with the property that it is difficult/impossible to calculate the likelihood (of the whole model, you can do it for ...
0
votes
1answer
28 views

Machine Learning approach to solve this selection problem?

I have a set of items S. items can be joined to groups consisting of up to x items. For a group of items i can derive a score Y using some unknown performance measure. What would be the most efficient ...
1
vote
1answer
61 views

Model uncertainty (model averaging) and R-Squared (R2)

Is it possible to calculate r-squared for an "average model"? Lets say I have 4 different response variables that I want to model to a set (or subset) of 4 independent variables. I'd then like to ...
8
votes
2answers
299 views

Best suggested textbooks on Bootstrap resampling?

I just wanted to ask which are in your opinion the best available books on bootstrap out there. By this I don't necessarily only mean the one written by its developers. Could you please indicate ...
1
vote
2answers
77 views

What model should I use for this?

Consider a dataset with 30 samples. We have a response $Y$ and 12 potential predictors $X_1, \cdots, X_{12}$. We fit two models. The first model $M_1$ includes only $X_1$ and $X_2$ and has an RSS of ...
0
votes
0answers
31 views

How to evaluate variable contribution to a prediction

I need to produce a logistic regression model that: Gives a ranked list of the most important factors Allows you to break out most important factors for each new observation scored Given that I'm ...
1
vote
1answer
172 views

what if response variable is 'yes or no' in R?

How to analyze above the data to predict the probability that people have disease with a model? Factors thought to influence infection include city, age, and diet. BUT, I don't know how to do ...
4
votes
1answer
109 views

Justifying and choosing a proper scoring rule

Most resources on proper scoring rules mention a number of different scoring rules like log-loss, Brier score or spherical scoring. However, they often don't give much guidance on the differences ...
0
votes
1answer
34 views

What is the correct order of these hierarchical priors?

I'm quite new to Bayesian data analysis, so this is most likely am easy question. I have the following model: a function f has two exponential rate parameters $\lambda_1$ and $\lambda_2$ and for some ...
0
votes
1answer
51 views

Structural equation modelling: model selection

I am currently trying to fit a structural equation model in R with the Lavaan package. I have this model that fits my data pretty good. This model is what I consider the full model, it has all paths ...
1
vote
0answers
27 views

Comparing a model with a latent variable to one without

I would like to test whether my 3 dependent variables all load onto an underlying latent variable or if a latent variable is not necessary to explain the relationship between the IVs and the DVs. In ...
3
votes
2answers
52 views

How to compare predictions from MLE-based regression Vs. predictions from bayesian regression?

Say I have two linear regression models that I want to use for predictions. Linear regression: \begin{equation} \mathbf{y} \sim \mathcal{N}(\mathbf{X^Tb}, \Sigma_y) \end{equation} Bayesian linear ...
0
votes
0answers
47 views

AIC rankings: Why would a global model rank lower than an intercept-only model?

I'm working with some real-world (i.e. potentially messy) data on the nesting ecology of several bird species. I'm attempting to relate the daily survival rate of nests to vegetation characteristics ...
1
vote
1answer
40 views

$R^2$ for mixed models = ICC?

I will be referring here to Nakagawa and Schielzeth (2013). As those authors state, $R^2$ for OLS regression could be defined as follows: $$R^2 = \frac{\sum^n_{i=1}(\bar{y} - ...
0
votes
0answers
11 views

Averaging against mixed-effect model

I have experimental data that contains information about 50 participants who performed a task in five different conditions (different set sizes). The result is the time spent on the task. My data is ...
0
votes
0answers
15 views

With posterior inclusion probability how do I settle on the final predictive model?

After using the spike-and-slab prior to perform Bayesian model selection, I get the posterior distribution of my variables, from which I calculate the inclusion probability for each variable. How do ...
1
vote
0answers
39 views

Selecting an appropriate VAR model

I would like to receive critical comments on an idea explained below. Suppose I have variables $x_1$ through $x_K$, and this is a time series setting. My aim is to forecast variable $x_1$. I know ...
0
votes
0answers
47 views

Advice for feature selection or feature extraction with semi-supervised learning

I am trying to solve a semi-supervised learning problem using LaplacianSVM. However, before applying LapSVM I would like either to perform feature selection or feature extraction. Furthermore, after ...
3
votes
0answers
48 views

Explain log likelihood behaviour

(This question is related to a previous one I made, here) I have a set of 2D observations (measured data) of sample size $N$: $$O = \{(x_1, y_1), (x_2, y_2), ..., (x_N, y_N)\}$$ I also have a model ...
1
vote
1answer
78 views

Multiple Linear Regression Variable Selection

Using all possible subsets we consider the adjusted $R^2$, Akaike's Information Criterion (AIC), corrected AIC ($AIC_c$), and Bayesian Information Criterion. The model with the highest adjusted $R^2), ...
1
vote
0answers
14 views

Heckman's selection / switching model- selection variable as a regressor

I am conducting some analyses on the effects of membership and the membership rank on income. Everyone in the sample can choose to apply to join an organization. The membership is given upon ...
7
votes
2answers
145 views

How to summarize knowledge about importance of variables?

Assume that we have 30 features (inputs) each of which can potentially influence the result (output). We try to use the available ("observed") mapping from features to targets to develop a predictive ...
0
votes
1answer
43 views

Good Literature about Problems with R squared

A question from a newbie. Recently, I was told that R squared or adjusted R squared can not used as a criteria to select a good regression model (model selection) due to, for example, overfitting . I ...
0
votes
2answers
52 views

Multiple linear regression, backward selection : Normality of the residuals?

I need to create a Multiple Linear regression model on those data explaining max03 T9 T12 T15 Ne9 Ne12 Ne15 Vx9 Vx12 Vx15 maxO3v !My data 1 My first intuition was to make a backward selection : ...
0
votes
0answers
42 views

Akaike information criterion for Cox proportional hazard models

I am conducting an analysis of survival data using Cox proportional hazard (CPH) models, to figure out what is the best model to use. The models I am comparing are non-nested. My plan is to compute ...
1
vote
0answers
22 views

Confidence intervals for the Log Loss metric for model comparison?

Quite a few Kaggle competitions have used or are using the Logarithmic Loss metric as the quality measure of a submission. I'm wondering if there are other ways besides N-fold cross-validation to ...
0
votes
0answers
4 views

Modelling for budget allocation experiment

I'm trying to figure out the best way to model the following process: An individual is given $k$ points and asked to fully allocate it between $m$ items on a menu. The items all share upto $n$ ...
0
votes
0answers
41 views

Comparing Two classification models using F1-score

I am trying to compare the results of 2 classifiers trained with SVM using the F1 score. Some papers that I have read and that do this have made me a bit confused. I have trained the 2 classifiers ...
0
votes
0answers
56 views

R: selecting appropriate fit statistics from GBM output

I'm using the gbm.step function in the dismo package in R to evaluate the contribution of three continuous variables on my ...