Questions tagged [predictive-models]
Predictive models are statistical models whose primary purpose is to predict other observations of a system optimally, as opposed to models whose purpose is to test a particular hypothesis or explain a phenomenon mechanistically. As such, predictive models place less emphasis on interpretability and more emphasis on performance.
515 questions
104
votes
8
answers
41k
views
When is unbalanced data really a problem in Machine Learning?
We already had multiple questions about unbalanced data when using logistic regression, SVM, decision trees, bagging and a number of other similar questions, what makes it a very popular topic! ...
- 141k
37
votes
2
answers
37k
views
Interpretation of simple predictions to odds ratios in logistic regression
I'm somewhat new to using logistic regression, and a bit confused by a discrepancy between my interpretations of the following values which I thought would be the same:
exponentiated beta values
...
- 867
135
votes
6
answers
218k
views
Difference between confidence intervals and prediction intervals
For a prediction interval in linear regression you still use $\hat{E}[Y|x] = \hat{\beta_0}+\hat{\beta}_{1}x$ to generate the interval. You also use this to generate a confidence interval of $E[Y|x_0]$....
- 1,545
292
votes
7
answers
459k
views
What does AUC stand for and what is it?
Searched high and low and have not been able to find out what AUC, as in related to prediction, stands for or means.
- 3,279
64
votes
4
answers
10k
views
What is the root cause of the class imbalance problem?
I've been thinking a lot about the "class imbalance problem" in machine/statistical learning lately, and am drawing ever deeper into a feeling that I just don't understand what is going on.
...
- 36.3k
23
votes
3
answers
11k
views
Obtaining a formula for prediction limits in a linear model (i.e.: prediction intervals)
Let's take the following example:
set.seed(342)
x1 <- runif(100)
x2 <- runif(100)
y <- x1+x2 + 2*x1*x2 + rnorm(100)
fit <- lm(y~x1*x2)
This creates a ...
- 21.9k
22
votes
1
answer
11k
views
Ridge\Lasso -- Standardization of dummy indicators
Say I have a data set with say 5000 rows and about 150 columns (5000 samples, 150 predictors/features) and I'm interested in a applying a ridge or lasso regression. (Let us assume using a logit link ...
- 1,410
7
votes
3
answers
2k
views
$R^2$ on out-sample data set
The conventional definition of $R^2$ is:
$R^2 = 1-SSE/SST$, where SSE denotes sum of squared errors and SST is total sum of squares ($n\times variance$, n being number of sample points in train set).
...
- 317
69
votes
3
answers
36k
views
Variables are often adjusted (e.g. standardised) before making a model - when is this a good idea, and when is it a bad one?
In what circumstances would you want to, or not want to scale or standardize a variable prior to model fitting? And what are the advantages / disadvantages of scaling a variable?
- 6,318
40
votes
2
answers
19k
views
Is there any algorithm combining classification and regression?
I'm wondering if there's any algorithm could do classification and regression at the same time. For example, I'd like to let the algorithm learn a classifier, and at the same time within each label, ...
- 581
56
votes
5
answers
56k
views
Prediction in Cox regression
I am doing a multivariate Cox regression, I have my significant independent variables and beta values. The model fits to my data very well.
Now, I would like to use my model and predict the survival ...
- 563
36
votes
5
answers
16k
views
What problem does oversampling, undersampling, and SMOTE solve?
In a recent, well recieved, question, Tim asks when is unbalanced data really a problem in Machine Learning? The premise of the question is that there is a lot of machine learning literature ...
- 36.3k
85
votes
6
answers
13k
views
Variable selection for predictive modeling really needed in 2016?
This question has been asked on CV some yrs ago, it seems worth a repost in light of 1) order of magnitude better computing technology (e.g. parallel computing, HPC etc) and 2) newer techniques, e.g. [...
- 3,392
33
votes
2
answers
10k
views
Why are p-values misleading after performing a stepwise selection?
Let's consider for example a linear regression model. I heard that, in data mining, after performing a stepwise selection based on the AIC criterion, it is misleading to look at the p-values to test ...
- 2,157
80
votes
16
answers
41k
views
Practical thoughts on explanatory vs. predictive modeling
Back in April, I attended a talk at the UMD (University of Maryland) Math Department Statistics group seminar series called "To Explain or To Predict?". The talk was given by Prof. Galit ...
19
votes
2
answers
4k
views
Example when using accuracy as an outcome measure will lead to a wrong conclusion
I am looking into various performance measures for predictive models. A lot was written about problems of using accuracy, instead of something more continuous to evaluate model performance. Frank ...
- 7,769
7
votes
1
answer
2k
views
Difference between splines from different packages (mgcv, rms etc.)
I recently came across the mgcv package and the great potentiality of GAM.
One - maybe naive - question is what is the overall difference (if there is any which is significant) with the gam() function ...
- 706
67
votes
5
answers
50k
views
Is adjusting p-values in a multiple regression for multiple comparisons a good idea?
Lets assume you are a social science researcher/econometrician trying to find relevant predictors of demand for a service. You have 2 outcome/dependent variables describing the demand (using the ...
- 773
32
votes
3
answers
22k
views
Cross-validation or bootstrapping to evaluate classification performance?
What is the most appropriate sampling method to evaluate the performance of a classifier on a particular data set and compare it with other classifiers? Cross-validation seems to be standard practice, ...
- 423
11
votes
1
answer
575
views
How competitive is stepwise regression when it comes to pure prediction?
When we want to do inference on parameters or nested models, stepwise variable selection causes a number of problems, discussed by Frank Harrell and others.
However, if we validate the stepwise model-...
- 67k
17
votes
2
answers
5k
views
Prediction and Tolerance Intervals
I have a couple of questions for prediction and tolerance intervals.
Let's agree on the definition of the tolerance intervals first: We are given a confidence level, say 90%, the percentage of the ...
14
votes
1
answer
5k
views
Determine accuracy of model which estimates probability of event
I'm modelling an event with two outcomes, a and b. I have created a model which estimates the probability that either a or b will happen (i.e. the model will calculate that a will happen with 40% ...
- 243
12
votes
3
answers
2k
views
T-consistency vs. P-consistency
Francis Diebold has a blog post "Causality and T-Consistency vs. Correlation and P-Consistency" where he presents the notion of P-consistency, or presistency:
Consider a standard linear regression ...
- 69.5k
6
votes
4
answers
5k
views
What's wrong to fit periodic data with polynomials?
Suppose we have toy daily temperate data and we want to fit a model.
A reasonable thing to do is fitting a periodic model with Fourier basis
$$
f(x)=\beta_0+\beta_1 \cos(2\pi x/24)+\beta_2 \sin(2\pi ...
- 37.3k
18
votes
2
answers
1k
views
How would econometricians answer the objections and recommendations raised by Chen and Pearl (2013)?
In their article, Chen and Pearl (2013), critically examined 6 econometric textbooks, among these the textbooks written by Wooldridge (2009) {the introductory book}, and Stock & Watson (2011). ...
- 4,792
47
votes
3
answers
24k
views
whether to rescale indicator / binary / dummy predictors for LASSO
For the LASSO (and other model selecting procedures) it is crucial to rescale the predictors. The general recommendation I follow is simply to use a 0 mean, 1 standard deviation normalization for ...
- 987
20
votes
2
answers
16k
views
Why P>0.5 cutoff is not "optimal" for logistic regression?
PREFACE: I don't care about the merits of using a cutoff or not, or how one should choose a cutoff. My question is purely mathematical and due to curiosity.
Logistic regression models the posterior ...
- 345
15
votes
1
answer
9k
views
Goodness-of-fit test in Logistic regression; which 'fit' do we want to test?
I am referring to the question and its answers: How to compare (probability) predictive ability of models developed from logistic regression? by @Clark Chong and answers/comments by @Frank Harrell. ...
user83346
12
votes
1
answer
27k
views
MAPE vs R-squared in regression models
Usually regression models are evaluated using $R^2$. I understand this metric can be misleading too at times but as far as I understand the first parameter we look at is $R^2$.
There is another ...
- 230
70
votes
6
answers
19k
views
Standard errors for lasso prediction using R
I'm trying to use a LASSO model for prediction, and I need to estimate standard errors. Surely someone has already written a package to do this. But as far as I can see, none of the packages on CRAN ...
- 58.3k
46
votes
4
answers
10k
views
Variance of $K$-fold cross-validation estimates as $f(K)$: what is the role of "stability"?
TL,DR: It appears that, contrary to oft-repeated advice, leave-one-out cross validation (LOO-CV) -- that is, $K$-fold CV with $K$ (the number of folds) equal to $N$ (the number of training ...
- 12.9k
45
votes
1
answer
23k
views
Manually calculated $R^2$ doesn't match up with randomForest() $R^2$ for testing new data
I know this is a fairly specific R question, but I may be thinking about proportion variance explained, $R^2$, incorrectly. Here goes.
I'm trying to use the ...
- 4,323
32
votes
5
answers
5k
views
How can you account for COVID-19 in your models?
How are you dealing with the coronavirus "event" in your machine learning models?
Let's say you used to predict the number of sales each month. The virus affected your results last year and ...
- 837
135
votes
4
answers
60k
views
Differences between cross validation and bootstrapping to estimate the prediction error
I would like your thoughts about the differences between cross validation and bootstrapping to estimate the prediction error.
Does one work better for small dataset sizes or large datasets?
- 1,571
9
votes
2
answers
2k
views
What is the relationship between minimizing prediction error versus parameter estimation error?
With the advent of statistical learning techniques, people are talking a lot about prediction error, while in classical statistics, one is focusing on parameter estimation error. What is the ...
- 3,184
72
votes
9
answers
16k
views
How can I help ensure testing data does not leak into training data?
Suppose we have someone building a predictive model, but that someone is not necessarily well-versed in proper statistical or machine learning principles. Maybe we are helping that person as they are ...
- 4,791
38
votes
2
answers
6k
views
Is this the state of art regression methodology?
I've been following Kaggle competitions for a long time and I come to realize that many winning strategies involve using at least one of the "big threes": bagging, boosting and stacking.
For ...
- 555
8
votes
1
answer
360
views
How much does an inability to predict an apparent anomaly mean that we lack something in the feature space to distinguish it from business as usual?
I have read a number of questions where the crux is a lamentation that a rare outcome is unable to be predicted by a regression model of some kind. While I understand the desire to be able to reliaby ...
- 67k
43
votes
1
answer
52k
views
Relative variable importance for Boosting
I'm looking for an explanation of how relative variable importance is computed in Gradient Boosted Trees that is not overly general/simplistic like:
The measures are based on the number of times a ...
- 6,217
6
votes
2
answers
4k
views
How to compare (probability) predictive ability of models developed from logistic regression?
I know some well-known measures are $c$ statistic, Kolmogorov-Smirnov $D$ statistic. However, as far as I know, those statistics take into account only of the rank order of the observations, and is ...
- 601
6
votes
1
answer
6k
views
Prediction based on bayesian model
I have created a bayesian model that estimates 6 parameters using rjags from R. Now i want to do some predictions based on new data in R. Can anyone help me with an example.
...
5
votes
2
answers
1k
views
How to predict the next number in a series while having additional series of data that might affect it?
Let's say we want to predict the price of Big Mac for the year 2020. We have 2 indexes that we think might make an influence to Big Mac price determination.
...
- 153
70
votes
10
answers
66k
views
What is the difference between prediction and inference?
I'm reading through "An Introduction to Statistical Learning" . In chapter 2, they discuss the reason for estimating a function $f$.
2.1.1 Why Estimate $f$?
There are two main reasons we ...
- 801
45
votes
1
answer
49k
views
When and how to use standardized explanatory variables in linear regression
I have 2 simple questions about linear regression:
When is it advised to standardize the explanatory variables?
Once estimation is carried out with standardized values, how can one predict with new ...
- 2,061
20
votes
4
answers
16k
views
Determining best fitting curve fitting function out of linear, exponential, and logarithmic functions
Context:
From a question on Mathematics Stack Exchange (Can I build a program), someone has a set of $x-y$ points, and wants to fit a curve to it, linear, exponential or logarithmic.
The usual ...
- 2,049
20
votes
2
answers
829
views
Minimizing bias in explanatory modeling, why? (Galit Shmueli's "To Explain or to Predict")
This question references Galit Shmueli's paper "To Explain or to Predict".
Specifically, in section 1.5, "Explaining and Prediction are Different", Professor Shmueli writes:
In explanatory ...
- 36.3k
10
votes
6
answers
53k
views
Logistic regression is predicting all 1, and no 0
I am running an analysis on the probability of loan default using logistic regression and random forests.
When I use logistic regression, the prediction is always all '1' (which means good loan). ...
- 233
3
votes
0
answers
1k
views
Regression: Causation vs Prediction vs Description
In my experience it seems me that the interpretation about regression, its meaning and its scope, are debatable and great confusion exist about those things. It seems me that confusions are not go ...
- 5,779
2
votes
2
answers
2k
views
More Statistical Way to Average N Predictions
I've run a RandomForestRegressor (Scikit Ensemble) over N loops, each time changing the random seed and therefore changing the train test split. This way I've N sets of predictions (M predictions for ...
- 151
34
votes
2
answers
29k
views
What is the difference between Maximum Likelihood Estimation & Gradient Descent?
What are the pro & cons of both the methods?
- 5,683