Questions tagged [model-selection]

Model selection is a problem of judging which model from some set performs best. Popular methods include $R^2$, AIC and BIC criteria, test sets, and cross-validation. To some extent, feature selection is a subproblem of model selection.

Filter by
Sorted by
Tagged with
0
votes
0answers
16 views

How can I compare overall explanatory power across a group of models using different sets of features?

I have several datasets of measured outcomes from different subjects, all measured in the same experimental setup. There are many possible explanatory features that may predict the outcome, but the ...
0
votes
1answer
112 views

ARIMA model selection in R

I have run an ARIMA model on univariate time series data. I have the below statistical results with the lag differencing at 3. I am not sure which of the model to select to forecast. Any help to ...
0
votes
0answers
65 views

Cross validation with Sequential Feature Selection

I am trying to implement a sequential backwards selection algorithm to select features with cross validation. I find this straightforward when it comes to the steps: start with n features remove ...
1
vote
0answers
44 views

Comparing curve fits

I have imaging data where I imaged different instances of the (somewhat) same phenomenon (2 different experimental conditions). I have already settled on the equations I use for curve fitting and ...
5
votes
1answer
532 views

KL-divergence: P||Q vs. Q||P

Assume, that we have several data generating measures $P_{1}, \dots, P_{k}$ and $Q$, all defined on the same probability space. Next, assume, we have the same amount of independently sampled data from ...
1
vote
1answer
56 views

Distribution Selection based on Kolmogorov Smirnov Test

I am trying to model the distribution of some non normal data, to do so i am fitting many different distributions(Student, Pareto...) to the data. When computing the Kolmogorov Smirnov Statistic for ...
0
votes
0answers
10 views

Recommendation tool predicting TOP buyers for a product

I am trying to build a recommendation model for the internal sales team, which would predict the list of TOP buyers who would buy a specific property (we are in a real estate area). I have a dataset ...
0
votes
0answers
25 views

AIC and transformation of the independent variable [duplicate]

I set up different models: always same dependent variable and dataset, only the independent variable changes. All model assumptions are fullfilled. Now i do a model selection with the AIC. I look at ...
13
votes
1answer
282 views

Why cross-validation gives biased estimates of error?

I came across many posts on CrossValidated discussing cross-validation and nested cross-validation as an alternative (e.g. here or here). I don't quite understand why 'ordinary' K-fold cross-...
3
votes
1answer
40 views

Appropriate model for amount of statistical errors in articles

I recently started my PhD and I am currently working on a project about finding statistical reporting errors. Our work is similar to Nuijten et al. (2016) only for economics. So, I have a database ...
8
votes
2answers
272 views

Do we want to move away from significance?

Recently I have found that many statisticians are speaking of moving away from significance. I understand that many studies base their conclusions on p-values, which I agree can be misleading at ...
0
votes
0answers
22 views

Find the minimum of two variables jointly (to select an optimal model)

Suppose that we have a few machine learning models and would like to perform model selection. Let's assume that I have tuples representing ...
2
votes
0answers
72 views

Selecting the correct Gaussian process prior for a regression function

Let $$ y_i = f(x_i) + \varepsilon_i \quad i=1,\ldots,n $$ where the $\varepsilon_i$ are iid $N(0,\sigma^2)$. Consider the Gaussian process priors $\pi_1$ and $\pi_2$: $$ \pi_1: f \sim GP(0,\lambda A) $...
0
votes
0answers
24 views

Determining whether logistic regression with robust variance for repeated measures is appropriate for my data, or which other model type to use

I am doing an analysis to identify independent predictors of a positive drug test result for patients who self-report being on medication in a cohort study (i.e., I am assessing recent medication ...
15
votes
4answers
2k views

Examples of Simpson's Paradox being resolved by choosing the aggregate data

Most of the advice around resolving Simpson's paradox is that you can't decide whether the aggregate data or grouped data is most meaningful without more context. However, most of the examples I've ...
1
vote
1answer
75 views

What is the most sound way to perform variable selection on an lmer() model?

Suppose I have 25 candidate predictors in an lmer model. I want to find out which ones are genuine predictors of the dependent variable. What is the best way to perform variable selection on that lmer ...
0
votes
0answers
115 views

Root Mean Square error or Standard Deviation to focus for Machine Learning model selection?

I have used Linear Regression and Support Vector Machine regressor model to predict the dependent variable. In Linear regression prediction the Root Mean Square error is more but standard deviation is ...
2
votes
0answers
31 views

Error metric to compare ratios derived from a binary prediction task

I'm working on a research problem where a binary classification task ultimately produces a ratio downstream. I would like to understand the best way to quantitatively compare the resultant ratio to ...
2
votes
0answers
117 views

Hannan–Quinn information criterion and Kashyap information criterion (KIC)

As we may know, the capacity of a model to overfit could easily increase by an increase in the complexity of that model (take complexity roughly as a number of parameters). To handle this problem, ...
1
vote
0answers
21 views

If two models have similar predictive power, why should we prefer the one with fewer parameters?

Was thinking a bit about model selection earlier, and I ended up getting hung up on the question: “If two models have similar predictive power, which model should I select?” For example, we often ...
0
votes
0answers
7 views

Comparing functional hypotheses accounting for uncertain interpretation of their predictions

I am interested in using an information-theoretic approach (likely AIC) to compare the explanatory power of several functional hypotheses. As an example, hypothesis H1 predicts significant association ...
0
votes
0answers
21 views

Choosing the best fitting model with AIC and p-value

I have a financial time series, exchange rates. Between ARCH(10) and GARCH(1,1) I would like to see which model fits best my TS. For ARCH I have a p-value smaller than 0.05 and for GARCH p-value is ...
0
votes
0answers
81 views

Can RMSE be greater than standard deviation of noise

I am working on model selection problem for noisy data sets. I am having non-parametric models like SVR, regression splines etc. which have can overfit if the hyperparameters are not tuned properly. I ...
3
votes
1answer
50 views

Regression methods for different sizes of $n$

I thought about something interesting today. Suppose we have a regression problem where the relationship between the response and the predictor variables is approximately linear. Let $n$ be the ...
1
vote
0answers
26 views

What are the appropriate ways of performing model selection? [closed]

I am reading up on model selection and ran into some intresting questions that I would like to understand to build intuition on the topic. My questions were: What are the appropriate ways of ...
3
votes
1answer
64 views

Is it expensive nested Cross-Validation (nested CV)?

I have a dataset, which contains 10 folds. The authors of the paper have created these 10 folds and in each fold there is a training set $D_{tr}$ and a test set $D_{te}$ (obviously, for each fold $D_{...
1
vote
0answers
252 views

How to compare two LASSO models - is there an equivalent to AIC/BIC?

It is often stated online that competing OLS models explaining a common dependent variable y can be compared by calculating an AIC or BIC for each fit, and that the model with the lowest value should ...
1
vote
0answers
17 views

Intrepretation of a term that is insiginficant but which when removed causes a significant increase in deviance

I began with a maximal model which looked something like this: Response ~ Predictor 1 + Predictor 2 + Predictor 3 I used backwards stepwise elimination and likeilhood ratio tests to then try and find ...
0
votes
0answers
16 views

Is is possible to use GMM on a data set where T>N?

I am using annual data on 5 US states over the last 25 years, so N=5 and T=25. Currently, I am using fixed effects to estimate my model which I arrived at after using the Hausman to compare it against ...
1
vote
1answer
80 views

Automatic GAM selection - single smooth and parametric terms

I'm just starting to experiment with the mgcv package in r. My problem is this - I'm modelling the count of a bird survey in space, with a number of different ...
0
votes
0answers
27 views

Descriptive survival analysis: Estimating median lifetime

Having only recently dipped my toes into the world of survival analysis, I think the general approach to my problem is pretty straightforward, but I'd love some sort of validation (perhaps, cross-...
0
votes
1answer
116 views

Question regarding selection variables for a multiple logistic regression analysis, through univariate analysis

I use univariate analysis to select variables for a multiple logistic regression. However, one of the categorical independent variables has a non significant dummy (one of the three). Should I include ...
0
votes
1answer
41 views

Comparing different probabilistic models using the log likelihood of held out data?

I'm reading a paper that compares different probabilistic models using the log likelihood of held-out data. This is just... wrong, correct? There's no meaningful way to compare the LL between two ...
5
votes
1answer
107 views

Model selection for multivariate mixed models

I would like to perform a multivariate mixed model but am a bit confused about model selection for such models. I wonder if I could get some help here. When fitting a univariate mixed model, to avoid ...
0
votes
0answers
30 views

How do I interpret model selection results when the best models are univariate?

I am trying to identify ecological relationships between several predictors and a response variable. The data are from an expensive study of marked animals that resulted in a small sample size. My ...
0
votes
1answer
69 views

Training of a deep Artificial Neural Network

I have few doubts related to training a neural network with more parameters (weights and biases) than number of data points. I know there exists discussion (on this platform) related to training such ...
1
vote
1answer
59 views

References for when a predictor variable did not reach the statistical significance but included in the model when conducting a model selection

I am analyzing a data set to identify a useful predictive model. I used a model selection approach (Burnham & Anderson, 2002) referring to AIC to select the most useful model for prediction. ...
1
vote
0answers
35 views

Should I use the sum of KL divergences for multi-objective model selection?

I have a model implemented in Python with 2 free parameters. I would like to find the parameter values that provide the best fit to empirical data comprising of response times and accuracy of human ...
0
votes
0answers
15 views

How to use auto.arima when attaining different results with a manual model selection?

I have daily sales data for over 2 years. I am having several questions of how to manually define model order and how to do it using auto_arima. Manual The first step I do it to see if my data is ...
1
vote
0answers
24 views

Why is the residual deviance higher for the more complex model?

The residual deviance is higher for the more complex model. Why is this the case? Can I compare the residual deviance of a GLM and a multi-level model in this way or should I use something else ...
2
votes
0answers
61 views

When do expected KL-divergence and expected MSE coincide?

The AIC is an approximately unbiased estimator of the (relative) risk of the Kullback-Leibler loss. I read that If you use AIC to choose among a family of models, AIC (approximately) yields the model ...
1
vote
0answers
62 views

Comparing parametric models with non parametric models using AIC

I still have a doubt related to using AIC to compare parametric models with non parametric models. In my previous question AIC can recommend an overfitting model?, I received some comments/answers ...
5
votes
2answers
1k views

In ML, once we remove a feature, can we safely assume that feature will not be important again?

This is probably a very simple question. Let's say we use some metric to remove features, whether that be AIC, regularization like lasso, variable importance, t-tests, etc... Assuming we use the ...
1
vote
0answers
34 views

Compare two regression models for a given test set in R

I formulated two linear models to fit the data from my dataset. I split the dataset into a training, validation, and test partition. Since I have two models I want to know which one predicts better ...
3
votes
2answers
92 views

choosing a baseline algorithm using a nested cross validation

I am trying to work on a project for school and I am currently trying to select the best model (models) for the classification task. I have a few questions regarding this: What does a baseline model ...
0
votes
0answers
51 views

Reporting metrics performance using nested cross-validation with grid search

I have a small dataset with 45 samples in total (25 from class 1 and 20 from class 2) and a large number of features (200). I would like to use nested cross-validation with grid search in order to do ...
0
votes
0answers
12 views

Causal inference and regression model selection

Measure the effect of X1 on Y, I built different regression model specifications by including different control variables, which model should I choose as the best model? Is there any standard ways to ...
1
vote
1answer
88 views

Does a threshold effect the training or testing fold in cross validation?

I am trying to better understand how changing a threshold affects a cross validation model. So if you trained a random forest model, the default threshold is ...
1
vote
1answer
76 views

Is a nonlinear regression model valid even if it not Homoscedastic?

I have a couple of experimental datasetsa where I am trying to determine if there is, for each data set, a particular linear/nonlinear regression model that correlates both variables. To this end, I ...
0
votes
1answer
41 views

Model selection using nested cross validation

I am working on a school project using remote sensing data, for classification purposes. And I am trying to select the best model (models) for my data. The approach that I adopted is the following: ...

1 2 3
4
5
35