Questions tagged [model-selection]

Model selection is a problem of judging which model from some set performs best. Popular methods include $R^2$, AIC and BIC criteria, test sets, and cross-validation. To some extent, feature selection is a subproblem of model selection.

Filter by
Sorted by
Tagged with
3
votes
1answer
31 views

Repeated Nested Cross validation

I'm aware that nested cross-validation is used for hyperparameter tuning and model selection and that repeated k-fold cross-validation is used to improve the estimated performance of the model. My ...
1
vote
0answers
23 views

Is there a rate of change performance measure for KL-divergence?

In the example figure below, KL-divergence is being used to measure how far the distribution of different parameterizations of Poisson are from an empirical distribution (real data). The minimum of ...
0
votes
1answer
41 views

To compare two KL-divergence scores, does the prior model have to be the same for both?

The KL-divergence compares a theoretical model $p$'s distribution with the empirical model $q$'s distribution, giving a score of $0$ if they, or their information contents, are identical. Say we have ...
0
votes
0answers
12 views

Binary probability scoring: Intuition on why a method might perform better in terms of Brier, log loss but worse in terms of Area under ROC/PR curve?

I'm trying to compare two methods. I have surface knowledge about these scorers, so I've noticed that scorers in which method A performs better are both proper scoring rule, while B performs better in ...
2
votes
0answers
18 views

How can be information criteria used with non-independent samples?

I'm puzzled by the following. Many, if not all, of the Information Criteria (AIC, WAIC, LOOIC, PSIS, ...) rely on the independence of the samples -- that we can remove a part, that one point is ...
3
votes
1answer
50 views

Can I compare lmer models with different fixed effects using anova

I know that this question sounds familiar to some other, but I believe the responses were not clear in those and were focused on REML models. I would like to know if it is sensible to compare 2 or ...
0
votes
1answer
32 views

Best practise for model selection when building predictive models?

What is the best practise when it comes to choosing how many models to evaluate when building a predictive model? It seems there are countless possibilities so I'm not sure how one chooses where to ...
0
votes
0answers
18 views

Is the model averaging mathematically justified?

Let's say that there is a full Poisson family GLM which contains a set of 4 explanatory variables with 140 observations. This model tests the effect of environmental parameters which potentially may ...
1
vote
0answers
15 views

Seasonal ARIMA lag differencing p-value not significant

I am using the below data to forecast using seasonal ARIMA model. I see at d=1 the p-value is not significant. But still ...
2
votes
0answers
15 views

Model generated 2 different results, which is the best SARIMA model?

I got 2 different forecasted results using different orders using SARIMA model. I am unable to choose the best model out of the two below. One have very low AIC but the SR1 co-efficient is close to 1 ...
1
vote
1answer
23 views

Can I still use an overfitted model with high test accuracy?

Below is the training statistics output from training a Keras/TF model. You can see val_accuracy peaks at Epoch 4 with 0.6633. After that accuracy(train) continues to go up but val_accuracy becomes ...
0
votes
0answers
15 views

How choose smoothing function for Generative Additive Models? (GAMs)

I would like to know which strategies are used in practice to choose the correct smoothing function for each features in the GAMs, both for classification and regression tasks. I thought of plotting ...
0
votes
0answers
15 views

Picking optimal lag values and intervals - multivariate time series

I'm working on my first project using time series: I have the weekly stocked amount of a product and I have to predict if it will go up or down (binary), looking for seasonality, I started trying this:...
4
votes
1answer
80 views

The Ambiguity in Schwarz Information Criterion Definition

Suppose there are 100 countries, $i = 1, 2, ..., 100$. Let $b_i$ be the median birth weight of all new born boys in country #i in 2019. Let $g_i$ be the median birth weight of all new born girls in ...
12
votes
5answers
1k views

When we plot data and then use nonlinear transformations in a regression model are we data-snooping?

I've been reading up on data snooping, and how it can mean the in-sample error does not provide a good approximation of the out-of-sample error. Suppose we are given a data set $(x_1,y_1),(x_2,y_2),......
1
vote
1answer
69 views

Differences between formulas for AIC and BIC

I have a question regarding the information criteria AIC and BIC: I found different formulas for the AIC/BIC, the common ones including the likelihood $\mathcal{L}$ are $$AIC = 2K - 2 ln(\mathcal{L})\...
1
vote
0answers
9 views

Updating an unsupervised model but retaining similarity

In my example I am using topic modelling (specifically a version of LDA) although I think avenues for exploring this could relate to other unsupervised techniques like clustering. I train a model and ...
0
votes
0answers
14 views

Why are 'Mean squared error' or 'Squared correlation coefficient' only calculated for Epsilon SVR and Nu SVR?

Why does libsvm only calculate 'Mean squared error' or 'Squared correlation coefficient' the for SVM types of Epsilon-SVR and nu-SVR? What is the reason these aren't appropriate for C-SVC or nu-SVC? ...
0
votes
0answers
11 views

How could I find the best model with the function dredge when there is no big difference in the AICc?

How could i find the best model when all the values of AICc are so similar?
0
votes
0answers
10 views

How can I compare overall explanatory power across a group of models using different sets of features?

I have several datasets of measured outcomes from different subjects, all measured in the same experimental setup. There are many possible explanatory features that may predict the outcome, but the ...
0
votes
1answer
40 views

ARIMA model selection in R

I have run an ARIMA model on univariate time series data. I have the below statistical results with the lag differencing at 3. I am not sure which of the model to select to forecast. Any help to ...
0
votes
0answers
40 views

Cross validation with Sequential Feature Selection

I am trying to implement a sequential backwards selection algorithm to select features with cross validation. I find this straightforward when it comes to the steps: start with n features remove ...
1
vote
0answers
34 views

Comparing curve fits

I have imaging data where I imaged different instances of the (somewhat) same phenomenon (2 different experimental conditions). I have already settled on the equations I use for curve fitting and ...
2
votes
1answer
36 views

KL-divergence: P||Q vs. Q||P

Assume, that we have several data generating measures $P_{1}, \dots, P_{k}$ and $Q$, all defined on the same probability space. Next, assume, we have the same amount of independently sampled data from ...
1
vote
1answer
35 views

Distribution Selection based on Kolmogorov Smirnov Test

I am trying to model the distribution of some non normal data, to do so i am fitting many different distributions(Student, Pareto...) to the data. When computing the Kolmogorov Smirnov Statistic for ...
0
votes
0answers
10 views

Recommendation tool predicting TOP buyers for a product

I am trying to build a recommendation model for the internal sales team, which would predict the list of TOP buyers who would buy a specific property (we are in a real estate area). I have a dataset ...
0
votes
0answers
23 views

AIC and transformation of the independent variable [duplicate]

I set up different models: always same dependent variable and dataset, only the independent variable changes. All model assumptions are fullfilled. Now i do a model selection with the AIC. I look at ...
9
votes
1answer
104 views

Why cross-validation gives biased estimates of error?

I came across many posts on CrossValidated discussing cross-validation and nested cross-validation as an alternative (e.g. here or here). I don't quite understand why 'ordinary' K-fold cross-...
3
votes
1answer
37 views

Appropriate model for amount of statistical errors in articles

I recently started my PhD and I am currently working on a project about finding statistical reporting errors. Our work is similar to Nuijten et al. (2016) only for economics. So, I have a database ...
8
votes
2answers
241 views

Do we want to move away from significance?

Recently I have found that many statisticians are speaking of moving away from significance. I understand that many studies base their conclusions on p-values, which I agree can be misleading at ...
0
votes
0answers
20 views

Find the minimum of two variables jointly (to select an optimal model)

Suppose that we have a few machine learning models and would like to perform model selection. Let's assume that I have tuples representing ...
2
votes
0answers
37 views

Selecting the correct Gaussian process prior for a regression function

Let $$ y_i = f(x_i) + \varepsilon_i \quad i=1,\ldots,n $$ where the $\varepsilon_i$ are iid $N(0,\sigma^2)$. Consider the Gaussian process priors $\pi_1$ and $\pi_2$: $$ \pi_1: f \sim GP(0,\lambda A) $...
0
votes
0answers
21 views

Determining whether logistic regression with robust variance for repeated measures is appropriate for my data, or which other model type to use

I am doing an analysis to identify independent predictors of a positive drug test result for patients who self-report being on medication in a cohort study (i.e., I am assessing recent medication ...
12
votes
4answers
1k views

Examples of Simpson's Paradox being resolved by choosing the aggregate data

Most of the advice around resolving Simpson's paradox is that you can't decide whether the aggregate data or grouped data is most meaningful without more context. However, most of the examples I've ...
1
vote
1answer
32 views

What is the most sound way to perform variable selection on an lmer() model?

Suppose I have 25 candidate predictors in an lmer model. I want to find out which ones are genuine predictors of the dependent variable. What is the best way to perform variable selection on that lmer ...
0
votes
0answers
32 views

Root Mean Square error or Standard Deviation to focus for Machine Learning model selection?

I have used Linear Regression and Support Vector Machine regressor model to predict the dependent variable. In Linear regression prediction the Root Mean Square error is more but standard deviation is ...
2
votes
0answers
17 views

Error metric to compare ratios derived from a binary prediction task

I'm working on a research problem where a binary classification task ultimately produces a ratio downstream. I would like to understand the best way to quantitatively compare the resultant ratio to ...
0
votes
0answers
19 views

Hannan–Quinn information criterion and Kashyap information criterion (KIC)

As we may know, the capacity of a model to overfit could easily increase by an increase in the complexity of that model (take complexity roughly as a number of parameters). To handle this problem, ...
1
vote
0answers
19 views

If two models have similar predictive power, why should we prefer the one with fewer parameters?

Was thinking a bit about model selection earlier, and I ended up getting hung up on the question: “If two models have similar predictive power, which model should I select?” For example, we often ...
0
votes
0answers
7 views

Comparing functional hypotheses accounting for uncertain interpretation of their predictions

I am interested in using an information-theoretic approach (likely AIC) to compare the explanatory power of several functional hypotheses. As an example, hypothesis H1 predicts significant association ...
0
votes
0answers
16 views

Choosing the best fitting model with AIC and p-value

I have a financial time series, exchange rates. Between ARCH(10) and GARCH(1,1) I would like to see which model fits best my TS. For ARCH I have a p-value smaller than 0.05 and for GARCH p-value is ...
0
votes
0answers
37 views

Can RMSE be greater than standard deviation of noise

I am working on model selection problem for noisy data sets. I am having non-parametric models like SVR, regression splines etc. which have can overfit if the hyperparameters are not tuned properly. I ...
2
votes
1answer
41 views

Regression methods for different sizes of $n$

I thought about something interesting today. Suppose we have a regression problem where the relationship between the response and the predictor variables is approximately linear. Let $n$ be the ...
1
vote
0answers
23 views

What are the appropriate ways of performing model selection? [closed]

I am reading up on model selection and ran into some intresting questions that I would like to understand to build intuition on the topic. My questions were: What are the appropriate ways of ...
3
votes
1answer
45 views

Is it expensive nested Cross-Validation (nested CV)?

I have a dataset, which contains 10 folds. The authors of the paper have created these 10 folds and in each fold there is a training set $D_{tr}$ and a test set $D_{te}$ (obviously, for each fold $D_{...
1
vote
0answers
65 views

How to compare two LASSO models - is there an equivalent to AIC/BIC?

It is often stated online that competing OLS models explaining a common dependent variable y can be compared by calculating an AIC or BIC for each fit, and that the model with the lowest value should ...
1
vote
0answers
17 views

Intrepretation of a term that is insiginficant but which when removed causes a significant increase in deviance

I began with a maximal model which looked something like this: Response ~ Predictor 1 + Predictor 2 + Predictor 3 I used backwards stepwise elimination and likeilhood ratio tests to then try and find ...
0
votes
0answers
12 views

Is is possible to use GMM on a data set where T>N?

I am using annual data on 5 US states over the last 25 years, so N=5 and T=25. Currently, I am using fixed effects to estimate my model which I arrived at after using the Hausman to compare it against ...
1
vote
1answer
47 views

Automatic GAM selection - single smooth and parametric terms

I'm just starting to experiment with the mgcv package in r. My problem is this - I'm modelling the count of a bird survey in space, with a number of different ...
0
votes
0answers
21 views

Descriptive survival analysis: Estimating median lifetime

Having only recently dipped my toes into the world of survival analysis, I think the general approach to my problem is pretty straightforward, but I'd love some sort of validation (perhaps, cross-...

1
2 3 4 5
33