All Questions
Tagged with predictive-models model-selection
100 questions
4
votes
1
answer
149
views
Choosing Between Intercept-Only and AR-NN Models: Justified to not use the model with the lowest RMSE/MAE?
I have created two autoregressive models for forecasting: a basic intercept-only model and an AR-NN (autoregressive neural network) model. Both models show similar performance based on recursive one-...
- 363
1
vote
0
answers
17
views
Model choice based on test/train/validation split [duplicate]
My question is very simple, but no matter where I look it up, it seems that I get another answer.
Take a simple classification task. Let's say I trained a kNN, LDA and logistic regression on it for ...
- 677
3
votes
2
answers
538
views
Variable selection in logistic regression [duplicate]
So I'm trying to make a multivariate logistic regression model in R studio. I'm not sure how to go about this. What seemed to make sense to me was to model every predictor against the response ...
- 126
0
votes
0
answers
24
views
How should I approach statistical model development from rubric-based data?
Background:
I am currently working in a role where I work in Assessment and Selection of right-fit applicants for teaching roles at a partner organisation. We presently use a rubric with a few ...
- 101
0
votes
0
answers
20
views
Was approaching this as a classification problem a mistake and should I have to use regression instead?
So I am training a model to predict baseball plate appearance outcomes, which I have been modelling as a single multi-class output problem, namely because single, mutually exclusive outcomes is what ...
1
vote
0
answers
129
views
Model calibration in overfitted models
Why in Shrinkage, due to an overfitted prediction model, do we tend to overestimate risk for "high risk" subjects and to underestimate risk for "low risk" subjects ?
Intuitively I ...
- 45
10
votes
4
answers
4k
views
Is it required to train the model in entire data after cross validation?
I have a model trained as follows.
...
- 223
2
votes
1
answer
204
views
Model Selection vs. Ensemble Learning
Is model selection just a specific kind of ensemble learning, where ensemble learning is loosely defined as "combining multiple models in some capacity to hopefully get an improved model"?
...
3
votes
3
answers
2k
views
Calculate AIC for both linear and non-linear models
I have data made of vectors $\textbf{x}$ and $\textbf{y}$. I want to predict $\textbf{y}$ with $\textbf{x}$ and a set of hyperparameters $a_{1, ..., 3}$ to be fitted with a linear and a nonlinear ...
- 593
0
votes
0
answers
513
views
The order of SMOTE, Feature selection, Model selection?
Please teach me if I am wrong.
The appropriate order should be:
SMOTE
Feature selection (e.g., by using a wrapper method)
Model selection (e.g., by selecting the model with highest AUC)
Then ...
0
votes
1
answer
434
views
Forcing covariates to always be part of a Lasso model
I want to use a Lasso to predict outcomes for different policy scenarios. At the optimal degree of regularization obtained by cross-validation, one important variable in whose impact I'm interested in ...
- 1
0
votes
0
answers
321
views
Paths to optimal K for GAM model selection
Let's say I have 10 different model combinations to compare via AIC for one year. There are 3 years of data, roughly 200-400 observations each year. For covariates, 2-3 of 5 appear to require tweaking ...
3
votes
0
answers
202
views
Why does it matter if we use an oracle estimator?
I read this question while studying adaptive LASSO, and while I think I have a decent understanding of the oracle property in theory, I am confused about what it means to use an oracle vs. non-oracle ...
- 685
2
votes
1
answer
52
views
Should one use the usual splitting (Learning/Validation/Test) when using cross-validation?
Say you want to tune several parameters of your model using $N$ data. What you usually do is splitting your $N$ data into 3 sets:
learning set: used to build your model;
validation set: used to ...
- 341
0
votes
0
answers
20
views
How can compare suggestion models with different performances?
I have 4 class binary classification models. That models identify which class a particular students is suitable for.
For example, we have user 1 and 4 classes ...
- 101
1
vote
0
answers
149
views
Nested Cross validation with two settings on KNN
I am trying to perform model selection and evaluation using a 5-fold (internal) CV for the iris data. The things that I performed so far.
Partitioned ...
- 259
0
votes
0
answers
106
views
Picking a suitable performance metric when comparing the same model but using different sets of training data (Causal inference model)
I am comparing the same models prediction accuracy (Causal Impact) using different control variables as predictors and looking for a metric to decide which set of controls to use. Reading into AIC and ...
- 1
3
votes
1
answer
175
views
Best practise for model selection when building predictive models?
What is the best practise when it comes to choosing how many models to evaluate when building a predictive model? It seems there are countless possibilities so I'm not sure how one chooses where to ...
- 893
2
votes
1
answer
315
views
Nested Cross Validation: How to do the whole Shebang (Algorithmic Selection, Model Selection, Parameter Tuning, Preprocessing) [closed]
First post! If you don't want to read the background you can skip to the Problem heading below.
Background
Hello everyone, I'm a Physics student doing physics education research. My professor wants ...
- 21
1
vote
0
answers
99
views
Comparison of simple linear regression, stepwise, lasso, and ridge
I am a totally beginner of machine learning. Please understand if my question is somehow basic. :)
I have a dataset of 25 features related to a rental house and I want to predict the price based on ...
1
vote
1
answer
67
views
R: Help: model selection when summary(lm) shows significant effect BUT anova(model2, model3) does NOT?
All is in the title, but here are the details:
...
- 11
4
votes
2
answers
1k
views
Inference, Prediction, & Model Fit?
I have a background in statistics (for social science), but I am confused about the ways in which Data Science textbooks (in particular, An Introduction to Statistical Learning and Practical ...
- 71
6
votes
1
answer
250
views
Recurring problem with retrospective data collection study designs I'm seeing
I've noticed a lot of medical research that I am involved in goes as follows:
Collect data on 300-1000 patients, including all sorts of baseline characteristics such as BMI, age, gender and then ...
- 2,331
1
vote
0
answers
105
views
How to pick the model that minimizes the mean absolute error when the amount of observations is small
I am given a data set with 1 target variable and 12 features for only 18 observations. My goal is to build a model that has the smallest expected prediction error.
I am allowed to use simple methods ...
2
votes
1
answer
80
views
Picking a model: Diagnostics or Model Strengths
I am building a lot of models and want to pick one to use for predicting. I am using linear regression, elastic net, and partial least squares regression.
I know my data is highly correlated and that ...
- 23
0
votes
0
answers
76
views
Is my model selection procedure problematic for inference?
I'm not sure if this is "step-wise" model selection, but here is what I'm doing
Decide a handful of models through exploratory data analysis.
Fit the models to the data, and calculate their AIC.
Pick ...
- 1,817
0
votes
0
answers
34
views
Is it useful to compute R Squared for regression trees? [duplicate]
I have a regression tree and want to validate the peformance. The first measure I have is the mse to find which model is the best. After that I want to check if the model peforms better then an ...
1
vote
1
answer
65
views
How to give more importance to one variable in a logistic regresion model? [closed]
I'm adjusting a logistic regression model for prediction, but if the person interested says: All variables are important for me, but especially X2 is more important.
How I give that variable more ...
- 11
3
votes
2
answers
418
views
What's the real purpose of cross validation?
As for cross evaluation (CV), I have two questions to ask:
1) CV has nothing to do with parameter selection, but only model evaluation? Specifically, which model?
2) In k-fold CV, what's the final ...
- 131
0
votes
1
answer
39
views
Can overfit happen in spite of validation and what to do with it?
Let's consider a standard situation where we need to find a predictive model.
We train all the available model using a training data set.
We validate all the trained model using a validation data ...
- 724
0
votes
1
answer
31
views
How does train-validation-test procedure deals with the sampling error of the accuracy measure?
Let's consider a standard model selection procedure:
We have N different untrained models (for example linear regression, neural network, decision tree and so on).
We use a data set A to train each ...
- 724
1
vote
0
answers
29
views
Are there two motivations for Bayesian information criteria?
Are there two motivations for all these Bayesian information criteria? I am only aware of the motivation of "expected out-of-sample prediction score."
Let the in-sample data be $y$ and the parameter ...
- 21.5k
0
votes
1
answer
107
views
cross-validation analysis not diagnostic
I'm using k-fold cross-validation analysis for model selection, however, it does not appear to favor any particular model. There are several variants of the models and two of them are nested within (...
1
vote
0
answers
472
views
Regression with a high number of 0's for target variable. How do I approach this?
I have a dataset where the probability of an event happening is very low (15%-20%). When the event happens, there's a dollar amount attached to it. The distribution is very right skewed, ranging from -...
- 121
1
vote
2
answers
160
views
How well does my logistic regression model fit?
I performed a logistic regression to my dataset which has 6 variables. I got output from R as the following:
I used the step() ...
- 21
6
votes
2
answers
559
views
In a linear regression, should I include independent variables that is already known to be predictive of the dependent variable?
I own an online shop and I'm trying to find the factors that would predict the profitability of a merchandise. To do this, I ran a linear regression with profit as the dependent variable. For the ...
- 161
0
votes
1
answer
330
views
How to know which forecasting tool to use in R?
I have around 5 years of data points that show how sales revenue a business made each month it looks a little like this:
Jan 13: $101,323.51
Feb 13: $125,021.44
...
Jun 18: $431,032....
3
votes
1
answer
951
views
LASSO: Difference in selecting tuning parameter for variable selection and prediction purposes
I'm reading Kirkland et al. "LASSO Tuning Parameter Selection" (2015) regarding methods for selecting the tuning Parameter in LASSO regressions. I'm a bit confused about the following Statements.
"...
- 515
1
vote
0
answers
31
views
Model (and algorithm) selection when variance is high
I work on a classification task in which the uncertainty regarding some groups of entities is pretty high, i.e. there is little information on these entities in the data. I am trying to select a model ...
- 241
2
votes
1
answer
4k
views
Which one should I consider: AIC or AICc
I want to run geographically weighted regression (GWR) with 5 independent variables (the first one is of my direct interest, while four are confounders). I ran various models (with fixed bandwidth [BW]...
- 21
2
votes
0
answers
34
views
Where is studying the Bispectrum useful?
In cosmology it is well known that studying the bispectrum of the large scale structure of the universe is a powerful way to distinguish different models of cosmic initial conditions. I had assumed ...
- 131
4
votes
2
answers
225
views
Common variable transformations
I want to predict a variable $Y$ given a set of variables $X_i$. To account for nonlinearity, my $X_i$ are put in several quantile dummies, so that I prefer transforming my $y$.
My $Y$ variable are ...
- 1,197
2
votes
1
answer
239
views
Theory question: How to use Mean Absolute Error properly in a log scaled linear regression
First of all, I had a look here and in a couple of other questions: I couldn't find what I am looking for.
So my question is purely theoretical (although I have an example by my hands).
Suppose I ...
- 2,286
3
votes
1
answer
2k
views
Will ROC curve for a model always be symmetric if we have enough training data?
ROC curve usually looks like the following figure:
If we have enough data, could we safely assume that ROC curve for a model will always be symmetric around the line y = 100 - x? If not, is there any ...
- 439
1
vote
0
answers
2k
views
Faster alternative of forward/backward model selection for big datasets
I want to perform a forward/backward selection to build a predictive model. My data set is very large though and if I include all variables in the selection process it is way too slow. Therefore I am ...
- 1,228
2
votes
0
answers
289
views
How to predict the probability of a discrete choice problem?
I am looking for a discrete choice model (e.g. a logit) to describe the reaction of a pedestrian when a car is arriving. Depending on the severity of the conflict and other geometrical parameters, he ...
1
vote
1
answer
205
views
How to select between models with different data sizes?
I have 1200 data points and I'm building 6 different time series models. One with 1200 data, another model with 1100, 1000,..., 700 and then I would choose the best model but I don't know how to ...
- 125
4
votes
2
answers
3k
views
Mixed effects model output - no difference in AIC values
In our study we are looking at the change in the numbers of acoustically tagged fish detections with respect to tidal state (ebbing, flowing), light period (dawn, day, dusk, night) and month (February,...
- 43
3
votes
0
answers
109
views
model selection using cross validation
I was wondering about model selection problem. To be more specific, how to split the data and use cross validation. So let's imagine situation:
We want to create some predictive model on data set D. ...
- 91
0
votes
0
answers
53
views
What is a fair way to compare models after model selection procedure?
The question is phrased a bit awkwardly probably, but let me explain it in more detail.
I am using feedforward neural nets with one hidden layer to model a variable of interest. I have a set of ...
- 13