We changed our privacy policy. Read more.

Questions tagged [model-selection]

Model selection is a problem of judging which model from some set performs best. Popular methods include $R^2$, AIC and BIC criteria, test sets, and cross-validation. To some extent, feature selection is a subproblem of model selection.

631 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
8
votes
1answer
374 views

How to train a model when instead of a target we have a range where it is?

Often in machine learning we have a situation when target is numeric (real or integer). Each target comes with an associated input vector. The goal is to learn the mapping from the input vectors to ...
6
votes
0answers
262 views

Reference Request: Information Geometry for Ridge Regression

I am reading the book "regression estimators" by Gruber 2010 where he uses this technique to compare Ridge Regressors, however he concentrates on deriving the mathematical results without ...
6
votes
0answers
145 views

Graphical nominal model

Suppose I have a set of $k$ matrices. $$ \mathbb D = A_1,A_2,...,A_k $$ Each column of $A$ is categorical vector. $$ A = v_1,v_2,...,v_n $$ I want to find the mapping $$ f: A \...
6
votes
0answers
720 views

How do you handle the situation where the residual variance is very high compared to the other variance parameter estimates?

Context An experiment in agronomy whose aim is to investigate the possible effect of a treatment, with 13 possible levels, on the height of trees. Model $ Y_{ijk} = \mu_{\cdot \cdot \cdot} + \...
5
votes
0answers
666 views

Why not use Ridge after Lasso vs relaxed Lasso

Has anyone ever applied a ridge regression on a model subset selected from a cross validated lasso? In other words, take a data set with p features and run lasso, grid searched to find optimal ...
5
votes
0answers
265 views

Why does "mixtools" return the model with highest AIC as the "winner" if lower AIC is better?

Mixtools package is used to fit mixtures of normal/regressions. The package documentation is given here The regmixmodel.sel fits the mixture model for varying ...
5
votes
0answers
429 views

How to compare multivariate forecasting methods?

Let $X$ be a multivariate time series of $N$ variables and $T$ observations. Let split $X$ into two separate datasets : $X_{train}$ : a train set with $N$ variables and $T_{train}$ observations $X_{...
5
votes
1answer
182 views

Match model selection strategies with modelling objectives

I am confused trying to match different model selection strategies with different modelling objectives. (Unfortunately, my confusion is reflected in the length of the post. Please be patient.) Model ...
5
votes
0answers
314 views

When using lmer is a random intercept being estimated more than once if specified in seperate grouping factors?

I know there are a slew of lmer specification questions already floating around. Please let me know if this is a duplicate, or if it is deemed off-topic, and I'll delete it. I am using a forward ...
5
votes
0answers
628 views

AIC with Mantel's tests

Mantel's tests are commonly used to compare genetic distances (say, between a number of individuals) with true or hypothesized landscape distances between those same individuals. For example, “does ...
5
votes
0answers
1k views

What are some useful data visualisation or data mining techniques to investigate horse racing forms?

I have a dataset of 13k horse races from four different tracks with an average of 11 runners per race. In all, there are 26k unique runners in the data. For each race, I know who came first, second, ...
5
votes
1answer
904 views

Model selection between parametric nonparametric methods

I have a real data set ($n=50$). I would like to fit some parametric models to this data set and then compare the maximum log-likelihood values with my spline model which is a nonparametric model. ...
4
votes
0answers
36 views

Is it wrong to compare multiple models on the same test set and choose the best model?

Suppose we split a dataset into 3 parts (train, validation, and test). I know that it's important to make sure the test set doesn't influence our decisions during model selection or hyperparameter ...
4
votes
0answers
411 views

The extrapolation problem: model selection, performance metrics, and improvement

Machine learning models are fit to a response variable within a given range. This leads to weak and sometimes disastrous performance when it comes to instances with an actual response variable outside ...
4
votes
0answers
240 views

Is there a measure of "complexity" for linear/nonlinear model terms?

My apologies if this is grossly misunderstood or mis-worded, but I've been mildly bugged by a question to which I've not found a satisfactory answer. I can't say that I have seen a discussion about ...
4
votes
0answers
48 views

How much of a problem is inference after model selection when few models are manually compared?

tl;dr: I found a better model than the one I first thought of while inspecting the data and performed a few steps of variable selection/model fine-tuning. I assume that this is a (mild) case of ...
4
votes
0answers
4k views

Multivariate ARIMA modelling in R

I am currently using the Marima package for R invented by Henrik Spliid in order to forecast multivariate time series with ARIMA. Overview can be found here: https://cran.r-project.org/web/packages/...
4
votes
0answers
36 views

How does Lindley compare a Bayes factor and a p-value?

I was reading this paper by Dennis Lindley ("Analysis of a Wine Tasting", J. Wine Econ. 2006). Statistically, the paper is a straightforward analysis of a $10\times 11$ two-way table. To test whether ...
4
votes
0answers
238 views

GP: How to select a model for a classification task, based in overall accuracy and log-marginal likelihood?

I have fitted a Gaussian Process (GP) to perform a binary classification task. The dataset is balanced, so I have an equal number of samples with 0/1 label for the training. The covariance function ...
4
votes
0answers
1k views

Maximum lag length when working with daily time series data

When working with (financial) time series data in R, one may use a Vector autoregressive model (VAR). One important issue when working with VARs is determining their lag length. In R, the command <...
4
votes
0answers
244 views

Intuition regarding Bayesian pseudo-priors

One approach to model comparison in a Bayesian framework uses a Bernoulli indicator variable to indicate which of two models is likely to be the "true model". When applying MCMC-based tools for ...
4
votes
1answer
197 views

Resources for building explanatory regression models?

Does anyone have any good resources for building explanatory regression models? Using the distinction between explanatory vs. predictive models described in Shmueli (2010) (text available here), I'm ...
4
votes
0answers
168 views

Variability in LASSO models for predicting rare events

I want to build a model for predicting a rare (ca 10%) event in my dataset of around 300 samples and 15 candidate predictors (of these, I know that five, when looked at individually in the whole ...
4
votes
0answers
222 views

Comparing the performance of two classifiers using cross-validation

Consider the following excerpt (paraphrased, see sec. 4.6.3 for original wording) from Introduction to Data Mining (free chapter) by Pang-Ning Tan, Michael Steinbach, and Vipin Kumar. Suppose we ...
4
votes
0answers
446 views

Generalized logistic model, sometimes with a bump

I am trying to fit a generalized sigmoidal function with a bump which is a Gaussian kernel to some data. My model is of the form $y\sim f(t)+\epsilon$ where $f$ is the function: $$ f(t) = A + \frac{K}{...
4
votes
1answer
742 views

develop minimum adequate model with correlated predictors

Could someone guide me what should my approach be regarding what predictors to include if they are correlated and how to develop my minimum adequate model. For e.g. lets say I have 10 predictors some ...
4
votes
0answers
544 views

GLMM with 2 insignificant variables has lower AIC or BIC compared to same model without those variables...?

Background This post has been heavily edited from its previous version (three months ago). I am investigating habitat selection of 35 territorial wolves over several years of denning seasons (41 ...
4
votes
0answers
1k views

Model diagnostics for a glmmPQL in R mixed-effects model

Several texts (both online and published books) have been reviewed prior to asking this. What diagnostics are accepted as best practise for a generalised linear mixed-effects model fitted in R using ...
4
votes
0answers
5k views

Backward stepwise regression with cross validation in R

I would like to do model selection using backward stepwise procedure and cross validation. https://www.otexts.org/fpp/5/3 I have used stepAIC in ...
4
votes
0answers
232 views

Computing a multi-sample (i.e., pooled) Akaike Information Criterion

I have a set of multivariate time series observations that I am trying to model using VAR processes, using AIC to choose the best model. However, instead of determining the best model order for each ...
4
votes
0answers
2k views

Can I use an automated model selection approach on an lmer object?

I am attempting to use MuMIn to run a model selection analysis on a mixed model fitted using lme4. Because this model is fit ...
4
votes
0answers
278 views

Logistic regression model for analysis of many IVs with a relatively small sample size

I'm trying to determine the influence (direction and relative strength) of certain attributes of incoming students to an academic program on their successful completion of the program. My sample size ...
4
votes
0answers
189 views

Relationship between LASSO T and LARS number of steps k

We can see on the figure (cf Least Angle Regression p30, Efron, Hastie, Johnstone, Tibshirani - link: Least Angle Regression) that there is a direct relationship between: LASSO T absolute norm of $\...
4
votes
0answers
138 views

Theoretical corrections of the training error for time series data

With $y_1, \ldots, y_n$ a real valued time series and $\hat{f} : \mathbb{R} \to \mathbb{R}$ a (least square) estimate of the function $y \mapsto E(Y_i \mid Y_{i-1} = y)$ the training error $$\text{err}...
3
votes
1answer
35 views

Why do we choose the hyperparameters that gives the lowest validation error? Do we assume that it also gives the lowest generalization error?

The usual way of selecting hyperparameters is to tune it on the validation set and select the hyperparameters that gives the lowest validation error (Lets assume the validation sample is large so we ...
3
votes
0answers
35 views

Model complexity vs regularization

How do the complexity of a model and regularization behave with each other? Like we could decrease the degree of a polynomial or add a regularization term. Or both? In other words: Why is there ...
3
votes
0answers
53 views

Looking for an advice regarding finding the best time series model

I have weekly time series data, which looks like follows: The data seems to be non-stationary. Then I took the first difference of the data. Now the data seems to be more stationary. After that, I ...
3
votes
1answer
50 views

Regression methods for different sizes of $n$

I thought about something interesting today. Suppose we have a regression problem where the relationship between the response and the predictor variables is approximately linear. Let $n$ be the ...
3
votes
0answers
21 views

Allowing data to remain unclassified in Bayesian model selection

This is a follow up to a previous question. In my analysis, I am using a Bayesian model selection (using MCMC) to determine which model best fits several data sets. However, because I want to allow my ...
3
votes
0answers
32 views

AIC and model selection with multimodel

For a simple example, I am doing a multi-model regression/likelihood estimation. The data is $(y_1,y_2,x_1,x_2,x_3,x_4,x_5)$. The first model (A) consists with two regressions: $y_1=e^{a_1x_1}+e^{...
3
votes
0answers
112 views

Likelihood ratio tests for quasi- models

I have been playing around with over-dispersion in binomial data and looking into qausi-binomial models as a solution. When comparing binomial models through the change in deviance, I can reproduce ...
3
votes
0answers
68 views

When to disregard AIC as a criterion in model selection

I have the following problem: I'm working on a dataset and it looks completely quadratic. A quadratic regression fits the data really good. However, when using piecewise linear functions I get a lower ...
3
votes
1answer
97 views

Effective search space vs guided search space

In ISLR (Intro to Stat Learning using R by James, Witten, Hastie, Tibs), in the section on Forward Selection on page 208, the footer states: Though forward stepwise selection considers $p(p+1)/2 + ...
3
votes
0answers
2k views

Calculating the AICc and BIC with RSS instead of likelihood

I have found here that that the akaike information criteria, corrected for small sample sizes is: where: And that the likelihood can be replaced with residual sum of squares (RSS) divided by n, the ...
3
votes
0answers
228 views

Disagreement of k-fold cross-validated error with aic, adjusted r-squared etc

I am comparing linear regression models. For the first measure, I compute the aic, adjusted r-squared and standard-error-of-the-regression with the average squared error computed on the validation ...
3
votes
1answer
1k views

VAR lag selection tests: Which one do I choose?

When running varselect in R, I usually get a few different models to choose from based on different statistics. I know of: Akaike information criterion (AIC) ...
3
votes
0answers
439 views

Understanding KS Statistic as Model Selection Tool

As a hobbyist learning about predictive modeling and machine learning, I am having some difficulty finding clarity regarding the KS statistic as a method for model selection. My mentor has been ...
3
votes
0answers
62 views

Bayesian Decision Making (for particular problem)

I've read several papers why p-values should be replaced by Bayes factors and trying to use them. What I have: say, I have matrix of 2000 rows and 1000 columns. In each column I need to make a ...
3
votes
0answers
44 views

While selecting model order for VAR models is it sound to stop increasing when a root outside the unit circle is found?

Basic question I guess. I'm fitting VAR models (and derivatives), and I've tried my hand on model order selection based on regularization but now I'm back to informative criteria (IC). Thing is my ...
3
votes
0answers
108 views

Marginal likelihood for a half-normal posterior?

So I know if we have a normal likelihood $P(\mathbf{y|b}) = \mathcal{N}(\mathbf{y}|\mathbf{Gb}, \mathbf{\Sigma}_y)$ and a normal prior $P(\mathbf{b}|\mathbf{\theta}) = \mathcal{N}(\mathbf{b} | \mu_p, \...

1
2 3 4 5
13