Questions tagged [model-selection]

Model selection is a problem of judging which model from some set performs best. Popular methods include $R^2$, AIC and BIC criteria, test sets, and cross-validation. To some extent, feature selection is a subproblem of model selection.

Filter by
Sorted by
Tagged with
0
votes
0answers
11 views

Akaike Information Criterion and distribution of residuals

TOY EXAMPLE. Suppose we are interested in the relationship between two quantities, $X$ and $Y$. We set $X = 1, 2, 3$ with certainty and measure $Y$. Our measurements are subject to iid measurement ...
0
votes
0answers
18 views

What type of regression analysis to use?

I am trying to use regression analysis in the following situation, but am not quite sure what kind of regression analysis is most appropriate. My data consists of information on different samples. For ...
0
votes
0answers
8 views

Least squares model selection criteria: standard error of point estimator vs variance of non-zero components in corresponding column of design matrix

I am performing ordinary least squares regression, and I would like some help with my model selection. One possible model is the following: The design matrix is denoted as $ \mathbf Z $ and the vector ...
0
votes
1answer
47 views

Lagged values in a Lasso regression

While working on the statistics for my thesis, I became confused while building up my model. I am currently working on a forecasting model with the use of a LASSO regression. The model is build as ...
0
votes
0answers
15 views

What does my ACF and PACF plots indicate for ARIMA order?

I am looking at forecasting first-order differences oil prices using the ARIMA models. However, I struggle to see what p and q orders my ACF and PACF plots would indicate. I have added the act and ...
2
votes
1answer
15 views

Nested Cross Validation: How to do the whole Shebang (Algorithmic Selection, Model Selection, Parameter Tuning, Preprocessing) [closed]

First post! If you don't want to read the background you can skip to the Problem heading below. Background Hello everyone, I'm a Physics student doing physics education research. My professor wants ...
1
vote
2answers
42 views

Randomly sampling parameters for model selection

Suppose I'm fitting a complicated (e.g. neural network) model's parameters $\theta$ to some data $D$, and I'm trying to tune hyperparameters (e.g. number of layers, size of layers) $\eta$. Normally I ...
0
votes
1answer
40 views

model selection for Mixed Poisson model for replicated point pattern analysis

I have fitted a mixed Poisson Point Process Model to Several Point Patterns using the mppm function from the R package spatstat. ...
2
votes
0answers
17 views

Why does model selection (AIC and LOO) outcomes differ between ML and bayesian approaches

I am interested in understanding whether my continuous data (dput code at bottom for reproducibility) are fit better by a linear model (Gaussian distribution) or a gamma distributed model. I ...
3
votes
2answers
43 views

Redundancy of a $\mathbb{E}[\cdot]$ term in AIC (1973)?

In the first line of the original paper for AIC (1973) "Information Theory and an Extension to Maximum Likelihood Principle", there is the following statement: I'm wondering if the underlined red ...
1
vote
1answer
48 views

AIC for logistic regression

On page 231 of The Elements of Statistical Learning AIC is defined as follows in (7.30) Given a set of models $f_\alpha(x)$ indexed by a tuning parameter $\alpha$, denote by $\overline{err}(\...
2
votes
2answers
47 views

Models with low variance but high bias

If we have a classification/regression problem, when would we generally prefer to use families of models with high bias and low variance like multiple regression (logistic regression for ...
0
votes
0answers
13 views

Compare regression and classification models. Which fits better?

I have 2 seperate approaches to a similar problem, one approach is regression focused and the other is classification based. I have the same predictor variable data set but with different outcomes (...
1
vote
0answers
7 views

Generalization Error and Matrix Factorizations?

This is more of a discussion/conceptual idea but: Is the notion of generalization error well-defined for a problem such as matrix factorization? I'm working with tensors/matrices and performing ...
0
votes
0answers
16 views

Which GARCH model to choose if both are showing same AIC/BIC and log-likelihood?

I am doing a GARCH model for returns under different error distributions using the R rugarch package. However, two models, under the generalised error distribution and the skewed generalised error ...
1
vote
0answers
8 views

What is the difference between validation and verification in statistical models?

We assume errors in our models are additive and uncorrelated, this is a big hypothesis but we'll assume that it is true. This means that observations have an inherent error $\epsilon_{o}$. ...
1
vote
0answers
28 views

Choosing some statistical tests

Imagine one big river (let's call it the river A) and a smaller one (river B). The river B is the tributary of the river A. I have some data collected including concentration of some important ...
1
vote
0answers
33 views

Determining order of ARMA model in R

I am working with a time series dataset and wish to do some ARMA modeling in R. I believe that I have detrended and deseasonalized the data correctly and now have a stationary time series. Now the ...
2
votes
1answer
46 views

Can AIC/BIC be used to compare models that differ by transformation of response variable? [duplicate]

I fit two models: one with a log-transformed response variable, and one without. All else is the same between them. Can I use AIC (or BIC or other, similar criteria) to choose which approach best fits ...
1
vote
1answer
40 views

How would a select an ARIMA model based on the ACF and PACF?

I am trying to estimate a model based on the ACF and PACF. I understand the ACF applies to MA and PACF applies to AR. Do I just count the significant lines and order it that way? I have used a Box Cox ...
1
vote
0answers
13 views

Trying to understand AIC and model explanatory power vs. model complexity

I am running a series of gam models using the mgcv package in R. They are attempting to model various intrinsic factors of an organism and it's growth rate. The two primary factors I am looking at are ...
1
vote
0answers
39 views

Model selection for multilevel zero-inflated models and model assumption checks

I am working on research that involves frequency of events (summed during an hour) and duration of events (calculated a seconds during an hour). Both data types have 0s and are positively skewed. The ...
0
votes
0answers
6 views

Compare AIC values first or whether each model meets nominal and scale assumptions?

I am trying to test different hypotheses about what drives variation in the consistency of bird food availability. I surveyed people to see for each season, what was the food availability? They ...
1
vote
0answers
18 views

Comparison of simple linear regression, stepwise, lasso, and ridge

I am a totally beginner of machine learning. Please understand if my question is somehow basic. :) I have a dataset of 25 features related to a rental house and I want to predict the price based on ...
0
votes
1answer
65 views

What regression method should I use?

I have a very high dimensional (p=946, n =123) dataset. For the one's i have checked, the covariates do not really have a relationship with the response so i have ruled out linear regression. Then, i ...
0
votes
1answer
24 views

Model selection when AIC scores are very close

Let's say you have a twice differenced time series. The AIC for two potential models are 2000 and 2000.1. As a rule of thumb you usually choose the one with the lower AIC, but if the model with AIC ...
0
votes
0answers
24 views

Bayesian model averaging when none of the models is well specified

Usually, from what I've read of Bayesian Model Averaging (BMA), a typical assumption is that one of the models is well specified... However, what will happen when all models are misspecified? What ...
1
vote
0answers
17 views

Fitting glmms to data with very unbalanced sampling effort - could subsampling help?

I have a data set comprising measurements of invertebrate species richness from grab samples of seabed sediment collected from a shallow coastal area over a period of 20 years. The sampling effort is ...
0
votes
2answers
52 views

the meaning of likelihood ratio test

I just learned the likelihood ratio test (LRT) method for model selection and worked out some examples. However, I am still a bit confused with the meaning of it. Basically, for a family of model ...
0
votes
1answer
18 views

What is the meaning of model selection and model assessement?

I think in model selection one has to look for the best tuning parameter. But what is the tuning parameter? Is this the k in the k-fold cross validation? And is the model assessment just the ...
2
votes
1answer
63 views

auto.arima vs arima.sim

I noticed auto.arima is frequently giving a different model than simulated with arima.sim, so I tested it crudely: ...
1
vote
0answers
11 views

Multiple Time Series in one Auto_arima code [closed]

I hope you can help me with the following. I am using the Auto_ARIMA function (pmdarima library) to calculate my p, d and q ...
0
votes
0answers
17 views

How if the coefficient constant Insignificant when Estimate ARMA?

I'm doing my thesis about estimate Volatility where I use ARMA for Mean Equation. when I'm trying to estimate the ARMA, I get Insignificant for coefficient constant but the AR and MA both are ...
1
vote
1answer
31 views
1
vote
0answers
11 views

How to choose between models with very small deltaAIC? (model averaging not an option)

My question is regarding selecting the best fitting model using AIC scores when the deltaAIC is very small between best models. I am running PMM's - Phylogeneticly-informed Mixed Models- using the ...
1
vote
1answer
19 views

Is it possible to have two valid, competing models with the exact same AIC (or AICc) result?

Say I am modeling occupancy and these are two of several models in my candidate set: Ψ(covariate A + covariate B + covariate C)p(.) Ψ(covariate A + covariate D + covariate B)p(.) Is it possible for ...
0
votes
0answers
15 views

Model selection based on Jarque-Bera, Shapiro-Wilk and Ljung-Box tests

I am kinda new to econometrics I have six models with Jarque-Bera, Shapiro-Wilk and Ljung-Box results. How can I choose the best model using the tests mentioned above? Thank you
2
votes
1answer
19 views

Overfitting through model selection

I'm asking this question as I found little explanation of this phenomenon otherwhere. I am wondering about how to best deal with overfitting that comes from the model selection itself. Say I want to ...
0
votes
0answers
5 views

Voting ensemble rationale and ensemble weights

what is the rationale for averaging multiple base models' predictions? One base learner might work towards reducing variance, another might work towards reducing bias, etc, but is there a source ...
1
vote
1answer
46 views

Comparing generalized linear mixed models (varying the distribution & link function)

I have some questions on performing mixed models on multi-rater data when residuals are heteroskedastic. I've found some of the info on Cross Validated confusing and quite technical-- would be very ...
0
votes
0answers
19 views

VAR and VECM (lag selection)

I have one data set which include data for GDP,Revenue and Expenditures.So I want to build VECM model in order to see how is interaction between this three variables.All this series are I(1). In order ...
0
votes
1answer
36 views

Relatively fast approximations to the marginal likelihood?

Let $\theta\in{\mathbb R}^d$ be a multidimensional parameters, where $d$ can be large (e.g. $d=100$ or more). What approximations can I use for the marginal likelihood: $$\int f(x\mid \theta)\pi(\...
0
votes
0answers
8 views

Are lower order terms less likely to be removed from the model?

I have a model $$y=x_0+x_1+x_2+x_0x_1+x_1x_2+x_0x_2+x_1x_2x_3$$, which is saturated, I am trying to remove terms which are found to be insignificant. I understand that if, say $x_1$ was removed, then ...
2
votes
2answers
72 views

Inference, Prediction, & Model Fit?

I have a background in statistics (for social science), but I am confused about the ways in which Data Science textbooks (in particular, An Introduction to Statistical Learning and Practical ...
0
votes
1answer
25 views

Optimal Lag Selection Indicates Lag 98

I am trying to identify the optimal lag for my multivariate time series and currently I am getting the optimal AIC at lag 98. I have never seen such large optimal lag is this correct? Note that my ...
0
votes
1answer
24 views

Akaike’s Information Criteria and Bayesian Information Criteria

Akaike’s Information Criteria generally tries to find unknown model that has high dimensional reality. Bayesian Information Criteria comes across only True models. What exactly they mean by "...
2
votes
1answer
54 views

How are inner loop and outer loops used to evaluate and build a machine learning model?

Ok. First off, I want to say that I know that there have been many questions that have been asked regarding cross validation or nested cross validation ( Nested cross validation for model selection ...
0
votes
0answers
20 views

How to choose between these 2 Arima models

I have two Arima models with interventions. Both used automatic procedures to find the interventions. The forecasts diverge a lot, mostly because of the drift term in model 1. How can I ...
0
votes
1answer
24 views

knowing which predictors are significant in a logistic regression model

I am trying to make a logistic regression model based on 5 predictors: 2 of these are categorical and 3 are numerical. The output is simply 1 or 0, and upon performing the Matlab function glmfit(x,y,'...
0
votes
2answers
60 views

Determining order of SARIMA model by ACF/PACF (and dealing with seasonality)

I have a ts that has the average monthly measure of pollutants in the air and i'm trying to use a SARIMA$(p,d,q)(P,D,Q)$ to model it but I'm having trouble determining the order because I'm somehow ...

1
2 3 4 5
32