Model selection is a problem of judging which model from some set performs best. Popular methods include $R^2$, AIC and BIC criteria, test sets, and cross-validation. To some extent, feature selection is a subproblem of model selection.

learn more… | top users | synonyms (1)

2
votes
1answer
25 views

Is AIC a valid criterion for the selection of variance structures in GLS?

In generalized least squares, I’ve specified a weights function that accounts for heterogeneity in residual variance that exist along the range of a covariate. The validation graphs (residuals ...
1
vote
1answer
37 views

What comes first: outlier detection or model selection?

I'm fitting a GLMM (mixed logistic regression) in R. I have five covariates. For model selection, I'm using glmmLasso() (in R) to determine which of the five covariates and their interactions should ...
1
vote
0answers
4 views

Should I run variable selection within MICE for Multiple Imputation?

I have a dataset with around 100 variables, and I plan on getting multiply imputed datasets using the mice package in R. The literature I have read seems to advocate regressing each variable against ...
3
votes
0answers
11 views

PDF/CDF ratio interpretation

What would be the intuitive interpretation of the ratio of probability and cummulative density function during the first stage of Heckman selection model? Is it the likelihood of selection for the ...
0
votes
0answers
2 views

What component of the result should I look at when doing a LLM model fit?

I am running mixed effects models with poisson and negative binomial fits. To asses which of the models are better, what components of the models should I look at? Some popular methods I follow: a) ...
0
votes
0answers
11 views

range of possible values and model complexity

I have a prediction model where the likelihood is a function of two components $(A,B)$ $L \sim \bf{A}^\alpha + \bf{B}^\beta$ , where both $\bf{A}$ and $\bf{B}$ are $n\times n$ matrices ($n$ is the ...
0
votes
0answers
12 views

Cancelling roots in ARMA(1,1) with external regressors

I am trying to find out what cancelling roots would imply for the estimators of my external regressors in my ARMA(1,1) model. Unfortunately however I'm stuck in my final step since I'm insecure about ...
0
votes
0answers
28 views

Help discussing stationarity using correlograms? ARMA/ARIMA modelling

I am currently trying to understand how to use correlograms to examine stationarity and analysis the appropriate models. Please can you advice, below I have included my ACFs and PACFs, and I am trying ...
1
vote
0answers
16 views

Likelihood ratio test seems to show little difference between models with AICc difference of 3

I'm running a multinomial logistic regression analysis of the behavioural responses of deer to camera traps using no reaction, reaction and strong reaction as dependent variables and season, camera ...
0
votes
1answer
40 views

Using Information theory with all possible models to select the best Model

I have a simple data set to find out about the effect of cultivation period length on soil organisms. The main factor of interest is age_class, a categorical variable defining the age of a field under ...
3
votes
0answers
25 views

AIC equivalent to Mallows' Cp and Mallows' Cp unbiased for test MSE

Part 1: The goal is to show that with Gaussian errors and a linear model, Mallows' $C_p$ and $AIC$ are equivalent. Using our definition of Mallows' $C_p$: $$C_p=1/n(RSS+2d\hat\sigma^2)$$ and $AIC$: ...
1
vote
1answer
34 views

Comparing the distribution fits of a bivariate and a univariate model

Suppose I've done an experiment and I have a distribution of observations $x$ that vary between $-\pi$ and $\pi$. Now suppose each $x$ is associated with a second observation $y$ that may or may not ...
0
votes
1answer
33 views

Model-based clustering evaluation with BIC

Let's say I have fitted two models using EM-clustering and they differ in both the number of clusters and are fitted on different subset of features (chosen from the same set of features). Could I ...
0
votes
1answer
13 views

Can one use k-fold cv and holdout analysis together?

I would like to start by saying i have just started using cross-validation, so please bear with me if the questions seems very trivial. I am reviewing someones work where the person has used k-fold ...
3
votes
0answers
33 views

Comparing the performance of two classifiers using cross-validation

Consider the following excerpt (paraphrased, see sec. 4.6.3 for original wording) from Introduction to Data Mining (free chapter) by Pang-Ning Tan, Michael Steinbach, and Vipin Kumar. Suppose we ...
3
votes
2answers
35 views

How to make sure kernel density estimation has a proper data size?

I'm using cross validation with kernel density estimation. In cross validation, the dataset is divided into several fold, which would make the test dataset has smaller size. And I'm wondersing if ...
0
votes
0answers
55 views

Log-likelihood ratio test vs. information criteria for model selection

I am trying to select the model with the best fit among GARCH(1,1), ARMA(2,2) and GJR-GARCH(1,1) models for a time series of log returns. The results from IC (Akaike, Bayesian) and likelihood-ratio ...
0
votes
0answers
38 views

R^2 for mixed effect models (both generalized linear and additive)

I have seen from several discussion threads that there are a few ways of calculating R^2 for LMMs and GLMMs - albeit with a caveat for GLMM being that the existing methods work for only gaussian ...
0
votes
1answer
25 views

Using AICc distributions to assess goodness-of-fit and model selection

I have a couple of ordinary differential equation models that I'm trying to fit to time-dependent biological data ($y_n$). One model is more complex than then other as it has more free parameters. I ...
2
votes
0answers
18 views

GLM with exponential and linear terms (variable exposure)

Suppose I'm counting the number of cars of a particular type that pass by a spot. I drive out to the spot at some time (that may differ depending on the day), sit there counting the cars, and note the ...
1
vote
0answers
19 views

Does a check failing to compare observed and predicted data qualify as a posterior predictive check?

I consider a Gaussian mixture distribution and I want to implement posterior predictive checks for choosing the model with the correct number of mixture components. I know the true number of ...
1
vote
0answers
26 views

Should difference between accuracy of model on training data and testing data be considered for model selection?

Suppose I have two models (Model 1 and Model 2), Where Accuracy of Model 1 on test data is higher than that in Model 2 Difference between accuracy of model on test data and training data is higher ...
0
votes
1answer
34 views

What to check in cross-validation - MAE or MSE?

When using cross-validation for model selection, should one look at MSE or MAE. I know that MSE and MAE are related but which is the more appropriate measure?
0
votes
0answers
12 views

Cross-validation for model selection

When using cross-validation for model selection should does one have to use the same k-folds to train all the models. What is mean is if i have say an OLS model and i want to find out which ...
71
votes
2answers
4k views

How much do we know about p-hacking “in the wild”?

The phrase p-hacking (also: "data dredging", "snooping" or "fishing") refers to various kinds of statistical malpractice in which results become artificially statistically significant. There are many ...
0
votes
1answer
62 views

Time series model selection: AIC vs. out-of-sample SSE and their equivalence

AIC is frequently recommended as criterion to compare models for time series forecasting. See for example this in the context of dynamic regression models: The AIC can be calculated for the final ...
1
vote
0answers
13 views

Statistical models for explanation purposes?

I am familiar with many "black-box" models used for predictive purposes (e.g. random forests, xgboost, neural networks), however these do not explain which variables are most significant in explaining ...
1
vote
0answers
24 views

Model selection in ensembles

I'm trying to build an ensemble for a ML problem where fast prediction time is critical. So I'm interested in keeping my set of level-0 models for the ensemble pruned. Which measures can I use to ...
1
vote
0answers
15 views

Regressor selection using full subset selection and simulation error minimization

I hope my question is in line with the topic of this site. I would like to identify the set of correct regressors form the following equation, which is a kind of Henon map: $ y(k) = 1 - a \big( ...
3
votes
1answer
77 views

Model construction: How to build a meaningful gam model? (generalized additive model)

I have seen there are various questions concerning the interpretation and construction of gams, which seems to illustrate the difficulty for non-statisticians to deal with those. Unfortunately, from ...
1
vote
0answers
50 views

Is it possible that, for the smaller value of the cost parameter, the margin become small, while for the larger cost, the margin become larger?

I have written an R code for soft margin svm using the ipop function of kernlab package. Despite it's working fine, but I still have some doubt whether this code working properly or not. According to ...
2
votes
2answers
67 views

How to reduce the final set of significant variables from logistic model?

I have built a logistic model, which has 40 significant predictors, p value<0.0001. I want to reduce them to say about 10 variables, so that it can be presented to business. How do i go about doing ...
2
votes
0answers
10 views

Higher SIC and lower S.E. of residuals

I used the Schwarz Information Criterion (a.k.a., BIC) and the Akaike Information Criterion (AIC) to select the models for a time series Analysis. AIC got me an ARMA (5,4) and SIC got me ARMA (2,1). ...
3
votes
3answers
107 views

A question about the choice and interpretation of the jumping distribution in Metropolis-Hastings algorithm

In order to implement the MH algorithm you need a proposal density or jumping distribution $q(⋅|⋅)$, from which it is easy to sample. If you want to sample from a distribution $f(⋅)$, the MH algorithm ...
0
votes
1answer
42 views

Interpreting temporal trends and selecting predictors in regression models

Scientific question: I want to know if temperature is changing across time (specifically, if it is increasing or decreasing). Data: My data consists of monthly temp averages across 90 years from a ...
0
votes
1answer
55 views

Bayes Factors for more than 2 hypotheses

Naïve question: I would like to use Bayesian framework for model selection. I have more than 10 models with the same number of parameters (just different assumptions on underlying parameters of the ...
2
votes
2answers
117 views

Does Bayesian Statistics have no concept of statistical hypothesis testing?

I was told that the framework of Bayesian Statistics has no concept of statistical hypothesis testing or confidence intervals. How does this make sense? Bayesian statistics only says that we ...
0
votes
1answer
37 views

How this code of cross-validation work? [closed]

I am new in sklearn and I try to learn how to use cross-validation to choose the best model of an SVM. I found this example How to split the dataset for cross validation, learning curve, and final ...
1
vote
0answers
50 views

mixed-effects models with (g)lmer in R and model selection

Mixed-effects models are wonderful for analyzing data, but it is always a hassle to find the best model, i.e. the model with the lowest AIC, especially when the number of predictor variables is large. ...
5
votes
2answers
73 views

Why information criterion (not adjusted $R^2$) are used to select appropriate lag order in time series model?

In time series models, like ARMA-GARCH, to select appropriate lag or order of the model different information criterion, like AIC, BIC, SIC etc, are used. My question is very simple, why donot we ...
0
votes
0answers
15 views

Correct evaluation/ comparison between undercomplete and overcomplete representations

Suppose I'm performing Unsupervised Feature Learning method to learn a representation of the data that is under-complete (e.g. 100 features) and use another algorithm to learn an over-complete ...
3
votes
0answers
53 views

Model selection: OLS vs TLS

I have two sets of real-valued data and I am interested in their correlation. From my perspective, there appear to be errors both variables, so I am inclined to perform a regression with TLS (Total ...
0
votes
1answer
105 views

VAR/VECM/ARDL optimal lag selection

Question 1: Is it necessary to consider AIC and the BIC criteria when selecting the lag for a VAR, VECM or ARDL model OR can I use something else? Example: Can I pick 12 lags because the model simply ...
0
votes
0answers
34 views

Model Selection - adding categorical covariates to mixed linear models

I have a dataset consisting of genotypes (crop lines) being grown in multi-replication trials across environments. Here is the mixed linear model I've been working with: $$ Y_{ijk} = \mu + G_i + E_j ...
3
votes
3answers
103 views

How to best model interaction effect of two continuous predictor variables?

Consider the following problem: In a logistic regression model, we believe that two continuous predictor variables $X_1$ and $X_2$ impact the probability of event. It is hypothesized that the ...
0
votes
0answers
6 views

Picking a particular model from regsubsets

I ran regsubsets in r from the 'leaps' library. I have gotten some 16 models in their order of which is best according to certain criterion. How do I select, say, model no.14 from this order and run ...
2
votes
0answers
39 views

GLMs with transformed response variable

I wonder if use of generalized linear models (GLMs) with transformed response variables is correct. My particular case: I compared goodness of fit of several GLMs with response variable transformed ...
1
vote
0answers
40 views

model fitting of data to multiple distributions

I have a set of numbers $ X = \{x_1, x_2,\ldots,x_n\}$ and I am interested in finding the most fitting combination of these numbers to multiple exponential distributions. Using predefined rules, I ...