Questions tagged [model-selection]

Model selection is a problem of judging which model from some set performs best. Popular methods include $R^2$, AIC and BIC criteria, test sets, and cross-validation. To some extent, feature selection is a subproblem of model selection.

Filter by
Sorted by
Tagged with
0
votes
0answers
16 views

Given the following ACF and PACF, what model would you say this is?

I believed it to be an ARIMA(2,1,1) model, but my MA coefficient is estimated to be -1. I differenced the raw data at lag 1 to remove trend.
0
votes
0answers
30 views

Selection of already fitted models

As a result of a certain experiment, I have obtained 3 matrices ($M_{1,2,3}$, size of 5x5 each). I also have 3 stochastic models, which gives me matrices $A_{1,2,3}$ for first,$B_{1,2,3}$ for second ...
2
votes
1answer
20 views

Ideal copula family to evaluate a joint CDF of a process with non-linear dependency?

Say a process exhibits ARCH/GARCH type non-linear dependence and we wish to evaluate its joint CDF. Without relying on a criterion, such as AIC what is the best copula family that fits this framework? ...
0
votes
0answers
178 views

How to Assess Linearity assumption of logit in logistic regression

In Applied Lineare Regression, (Hosmer, Lemeshow, Sturdivant 3rd ed.) Ch. 4, they present "Purposeful Selection." Part of step 5 is to assess the validity of the linearity assumption of the logit vs ...
1
vote
1answer
44 views

Select model in piecewise regression

How could we compare models fitted with piecewise regression that have different breakpoints? I used the code written here. In the example below, the "best" model, i.e. minimizing the residuals is ...
4
votes
2answers
158 views

Is post-selection inference a problem when robust tests are used?

It's pretty well acknowledged that error control via p-values fails when models are selected based on the data rather than decided on a priori. I've always viewed this as an issue of marginal vs ...
1
vote
2answers
50 views

When is it okay to make changes to your model after validating?

Let’s say I’m building a model to predict cancer relapse for a scientific paper. I use my training set to build many models and validate the best one on my test set to get an AUC of 0.65. I then go ...
0
votes
0answers
33 views

Panel unit root tests - How to determine the lag order?

I am using Pesaran's panel unit root test in the presence of cross-sectional dependence. How do I specify the lag order? I find that the results of my tests changes when I change the number of lags. ...
0
votes
0answers
46 views

Akaike Information Criterion and distribution of residuals

TOY EXAMPLE. Suppose we are interested in the relationship between two quantities, $X$ and $Y$. We set $X = 1, 2, 3$ with certainty and measure $Y$. Our measurements are subject to iid measurement ...
0
votes
0answers
20 views

What type of regression analysis to use?

I am trying to use regression analysis in the following situation, but am not quite sure what kind of regression analysis is most appropriate. My data consists of information on different samples. For ...
0
votes
0answers
11 views

Least squares model selection criteria: standard error of point estimator vs variance of non-zero components in corresponding column of design matrix

I am performing ordinary least squares regression, and I would like some help with my model selection. One possible model is the following: The design matrix is denoted as $ \mathbf Z $ and the vector ...
1
vote
1answer
206 views

Lagged values in a Lasso regression

While working on the statistics for my thesis, I became confused while building up my model. I am currently working on a forecasting model with the use of a LASSO regression. The model is build as ...
0
votes
1answer
34 views

What does my ACF and PACF plots indicate for ARIMA order?

I am looking at forecasting first-order differences oil prices using the ARIMA models. However, I struggle to see what p and q orders my ACF and PACF plots would indicate. I have added the act and ...
2
votes
1answer
89 views

Nested Cross Validation: How to do the whole Shebang (Algorithmic Selection, Model Selection, Parameter Tuning, Preprocessing) [closed]

First post! If you don't want to read the background you can skip to the Problem heading below. Background Hello everyone, I'm a Physics student doing physics education research. My professor wants ...
1
vote
2answers
50 views

Randomly sampling parameters for model selection

Suppose I'm fitting a complicated (e.g. neural network) model's parameters $\theta$ to some data $D$, and I'm trying to tune hyperparameters (e.g. number of layers, size of layers) $\eta$. Normally I ...
0
votes
1answer
60 views

model selection for Mixed Poisson model for replicated point pattern analysis

I have fitted a mixed Poisson Point Process Model to Several Point Patterns using the mppm function from the R package spatstat. ...
6
votes
1answer
218 views

Why do model selection (AIC and LOO) outcomes differ between ML and bayesian approaches

I am interested in understanding whether my continuous data (dput code at bottom for reproducibility) are fit better by a linear model (Gaussian distribution) or a gamma distributed model. I ...
5
votes
2answers
64 views

Aikaike's AIC is cross-entropy because of redundancy of $\mathbb{E}[\cdot]$ term?

In the first line of the original paper for AIC (1973) "Information Theory and an Extension to Maximum Likelihood Principle", there is the following statement: I'm wondering if the underlined red ...
2
votes
1answer
356 views

AIC for logistic regression

On page 231 of The Elements of Statistical Learning AIC is defined as follows in (7.30) Given a set of models $f_\alpha(x)$ indexed by a tuning parameter $\alpha$, denote by $\overline{err}(\...
3
votes
2answers
84 views

Models with low variance but high bias

If we have a classification/regression problem, when would we generally prefer to use families of models with high bias and low variance like multiple regression (logistic regression for ...
1
vote
0answers
20 views

Generalization Error and Matrix Factorizations?

This is more of a discussion/conceptual idea but: Is the notion of generalization error well-defined for a problem such as matrix factorization? I'm working with tensors/matrices and performing ...
1
vote
0answers
45 views

Which GARCH model to choose if both are showing same AIC/BIC and log-likelihood?

I am doing a GARCH model for returns under different error distributions using the R rugarch package. However, two models, under the generalised error distribution and the skewed generalised error ...
1
vote
0answers
11 views

What is the difference between validation and verification in statistical models?

We assume errors in our models are additive and uncorrelated, this is a big hypothesis but we'll assume that it is true. This means that observations have an inherent error $\epsilon_{o}$. ...
1
vote
0answers
32 views

Choosing some statistical tests [closed]

Imagine one big river (let's call it the river A) and a smaller one (river B). The river B is the tributary of the river A. I have some data collected including concentration of some important ...
1
vote
0answers
193 views

Determining order of ARMA model in R

I am working with a time series dataset and wish to do some ARMA modeling in R. I believe that I have detrended and deseasonalized the data correctly and now have a stationary time series. Now the ...
2
votes
1answer
523 views

Can AIC/BIC be used to compare models that differ by transformation of response variable? [duplicate]

I fit two models: one with a log-transformed response variable, and one without. All else is the same between them. Can I use AIC (or BIC or other, similar criteria) to choose which approach best fits ...
1
vote
1answer
50 views

How would a select an ARIMA model based on the ACF and PACF?

I am trying to estimate a model based on the ACF and PACF. I understand the ACF applies to MA and PACF applies to AR. Do I just count the significant lines and order it that way? I have used a Box Cox ...
1
vote
0answers
43 views

Trying to understand AIC and model explanatory power vs. model complexity

I am running a series of gam models using the mgcv package in R. They are attempting to model various intrinsic factors of an organism and it's growth rate. The two primary factors I am looking at are ...
1
vote
0answers
127 views

Model selection for multilevel zero-inflated models and model assumption checks

I am working on research that involves frequency of events (summed during an hour) and duration of events (calculated a seconds during an hour). Both data types have 0s and are positively skewed. The ...
1
vote
0answers
65 views

Comparison of simple linear regression, stepwise, lasso, and ridge

I am a totally beginner of machine learning. Please understand if my question is somehow basic. :) I have a dataset of 25 features related to a rental house and I want to predict the price based on ...
0
votes
1answer
71 views

What regression method should I use?

I have a very high dimensional (p=946, n =123) dataset. For the one's i have checked, the covariates do not really have a relationship with the response so i have ruled out linear regression. Then, i ...
0
votes
1answer
88 views

Model selection when AIC scores are very close

Let's say you have a twice differenced time series. The AIC for two potential models are 2000 and 2000.1. As a rule of thumb you usually choose the one with the lower AIC, but if the model with AIC ...
0
votes
0answers
32 views

Bayesian model averaging when none of the models is well specified

Usually, from what I've read of Bayesian Model Averaging (BMA), a typical assumption is that one of the models is well specified... However, what will happen when all models are misspecified? What ...
1
vote
0answers
37 views

Fitting glmms to data with very unbalanced sampling effort - could subsampling help?

I have a data set comprising measurements of invertebrate species richness from grab samples of seabed sediment collected from a shallow coastal area over a period of 20 years. The sampling effort is ...
0
votes
2answers
85 views

the meaning of likelihood ratio test

I just learned the likelihood ratio test (LRT) method for model selection and worked out some examples. However, I am still a bit confused with the meaning of it. Basically, for a family of model ...
0
votes
1answer
23 views

What is the meaning of model selection and model assessement?

I think in model selection one has to look for the best tuning parameter. But what is the tuning parameter? Is this the k in the k-fold cross validation? And is the model assessment just the ...
3
votes
1answer
344 views

auto.arima vs arima.sim

I noticed auto.arima is frequently giving a different model than simulated with arima.sim, so I tested it crudely: ...
1
vote
0answers
18 views

Multiple Time Series in one Auto_arima code [closed]

I hope you can help me with the following. I am using the Auto_ARIMA function (pmdarima library) to calculate my p, d and q ...
1
vote
1answer
38 views
1
vote
0answers
45 views

How to choose between models with very small deltaAIC? (model averaging not an option)

My question is regarding selecting the best fitting model using AIC scores when the deltaAIC is very small between best models. I am running PMM's - Phylogeneticly-informed Mixed Models- using the ...
1
vote
1answer
27 views

Is it possible to have two valid, competing models with the exact same AIC (or AICc) result?

Say I am modeling occupancy and these are two of several models in my candidate set: Ψ(covariate A + covariate B + covariate C)p(.) Ψ(covariate A + covariate D + covariate B)p(.) Is it possible for ...
2
votes
1answer
28 views

Overfitting through model selection

I'm asking this question as I found little explanation of this phenomenon otherwhere. I am wondering about how to best deal with overfitting that comes from the model selection itself. Say I want to ...
1
vote
1answer
124 views

Comparing generalized linear mixed models (varying the distribution & link function)

I have some questions on performing mixed models on multi-rater data when residuals are heteroskedastic. I've found some of the info on Cross Validated confusing and quite technical-- would be very ...
0
votes
1answer
46 views

Relatively fast approximations to the marginal likelihood?

Let $\theta\in{\mathbb R}^d$ be a multidimensional parameters, where $d$ can be large (e.g. $d=100$ or more). What approximations can I use for the marginal likelihood: $$\int f(x\mid \theta)\pi(\...
2
votes
2answers
172 views

Inference, Prediction, & Model Fit?

I have a background in statistics (for social science), but I am confused about the ways in which Data Science textbooks (in particular, An Introduction to Statistical Learning and Practical ...
0
votes
1answer
49 views

Optimal Lag Selection Indicates Lag 98

I am trying to identify the optimal lag for my multivariate time series and currently I am getting the optimal AIC at lag 98. I have never seen such large optimal lag is this correct? Note that my ...
0
votes
1answer
82 views

Akaike’s Information Criteria and Bayesian Information Criteria

Akaike’s Information Criteria generally tries to find unknown model that has high dimensional reality. Bayesian Information Criteria comes across only True models. What exactly they mean by "...
2
votes
1answer
332 views

How are inner loop and outer loops used to evaluate and build a machine learning model?

Ok. First off, I want to say that I know that there have been many questions that have been asked regarding cross validation or nested cross validation ( Nested cross validation for model selection ...
0
votes
1answer
39 views

knowing which predictors are significant in a logistic regression model

I am trying to make a logistic regression model based on 5 predictors: 2 of these are categorical and 3 are numerical. The output is simply 1 or 0, and upon performing the Matlab function glmfit(x,y,'...
0
votes
2answers
307 views

Determining order of SARIMA model by ACF/PACF (and dealing with seasonality)

I have a ts that has the average monthly measure of pollutants in the air and i'm trying to use a SARIMA$(p,d,q)(P,D,Q)$ to model it but I'm having trouble determining the order because I'm somehow ...

1
3 4
5
6 7
35