318

Abbreviations AUC = Area Under the Curve. AUROC = Area Under the Receiver Operating Characteristic curve. AUC is used most of the time to mean AUROC, which is a bad practice since as Marc Claesen pointed out AUC is ambiguous (could be any curve) while AUROC is not. Interpreting the AUROC The AUROC has several equivalent interpretations: The expectation ...


81

Imagine your job is to forecast the number of Americans that will die from various causes next year. A reasonable place to start your analysis might be the National Vital Statistics Data final death data for 2014. The assumption is that 2017 might look roughly like 2014. You'll find that approximately 2,626,000 Americans died in 2014: 614,000 died of heart ...


60

Although I'm a bit late to the party, but here's my 5 cents. @FranckDernoncourt (+1) already mentioned possible interpretations of AUC ROC, and my favorite one is the first on his list (I use different wording, but it's the same): the AUC of a classifier is equal to the probability that the classifier will rank a randomly chosen positive example higher ...


60

All the answers so far provided are helpful, but they aren't very statistically precise, so I'll take a shot at that. At the same time, I'm going to give a general answer rather than focusing on this election. The first thing to keep in mind when we're trying to answer questions about real-world events like Clinton winning the election, as opposed to made-...


50

It seems self-evident to me that $$ \exp(\beta_0 + \beta_1x) \neq\frac{\exp(\beta_0 + \beta_1x)}{1+\exp(\beta_0 + \beta_1x)} $$ unless $\exp(\beta_0 + \beta_1x)=0$. So, I'm less clear about what the confusion might be. What I can say is that the left hand side (LHS) of the (not) equals sign is the odds of being undernourished, whereas the RHS is the ...


49

The confusion matrix is a way of tabulating the number of misclassifications, i.e., the number of predicted classes which ended up in a wrong classification bin based on the true classes. While sklearn.metrics.confusion_matrix provides a numeric matrix, I find it more useful to generate a 'report' using the following: import pandas as pd y_true = pd.Series(...


48

Kyung et al. (2010), "Penalized regression, standard errors, & Bayesian lassos", Bayesian Analysis , 5, 2, suggest that there might not be a consensus on a statistically valid method of calculating standard errors for the lasso predictions. Tibshirani seems to agree (slide 43) that standard errors are still an unresolved issue.


45

Important considerations are not included in any of these discussions. The procedures discussed above invite inappropriate thresholding and utilize improper accuracy scoring rules (proportions) that are optimized by choosing the wrong features and giving them the wrong weights. Dichotomization of continuous predictions flies in the face of optimal decision ...


37

There have been rumors for years that Google uses all available features in building its predictive algorithms. To date however, no disclaimers, explanations or white papers have emerged that clarify and/or dispute this rumor. Not even their published patents help in the understanding. As a result, no one external to Google knows what they are doing, to the ...


34

You are right to ask this question. In general, when a proper accuracy scoring rule is used (e.g., mean squared prediction error), ridge regression will outperform lasso. Lasso spends some of the information trying to find the "right" predictors and it's not even great at doing that in many cases. Relative performance of the two will depend on the ...


32

Following Cox model, the estimated hazard for individual $i$ with covariate vector $x_i$ has the form $$\hat{h}_i(t) = \hat{h}_0(t) \exp(x_i' \hat{\beta}),$$ where $\hat{\beta}$ is found by maximising the partial likelihood, while $\hat{h}_0$ follows from the Nelson-Aalen estimator, $$ \hat{h}_0(t_i) = \frac{d_i}{\sum_{j:t_j \geq t_i} \exp(x_j' \hat{\beta})} ...


32

When statisticians want to predict a binary outcome (Hillary wins vs Hillary does not win), they imagine that the universe is tossing an imaginary coin - Heads, Hillary wins; tails, she loses. To some statisticians, the coin represents their degree of belief in the outcome; to others, the coin represents what might happen if we reran the election under the ...


30

Inference: Given a set of data you want to infer how the output is generated as a function of the data. Prediction: Given a new measurement, you want to use an existing data set to build a model that reliably chooses the correct identifier from a set of outcomes. Inference: You want to find out what the effect of Age, Passenger Class and, Gender has on ...


28

There has been some work on adapting deep learning methods for sequential data. A lot of this work has focused on developing "modules" which can be stacked in a way analogous to stacking restricted boltzmann machines (RBMs) or autoencoders to form a deep neural network. I'll highlight a few below: Conditional RBMs: Probably one of the most successful ...


26

When statisticians say this they are not referring to the margin of victory or the share of the vote. They are running a large number of simulations of the election and counting what percentage of the vote each candidate gains. For many robust presidential models they have forecasts for each state. Some are close and if the race is run multiple times, both ...


26

Problems with the chart: It implies refugees are more likely than other groups of people to commit acts of terror. Why not frame it in terms of migrants in general? And what about acts of terror committed by a country's own citizens? How does it define a refugee? The comparative groups don't make sense. If we are going to look a killings why not compare it ...


25

Formula for weighted Pearson correlation can be easily found on the web, StackOverflow, and Wikipedia and is implemented in several R packages e.g. psych, or weights and in Python's statsmodels package. It is calculated like regular correlation but with using weighted means, $$ m_X = \frac{\sum_i w_i x_i}{\sum_i w_i}, ~~~~ m_Y = \frac{\sum_i w_i y_i}{\sum_i ...


24

The difference is not the mathematical expression, but rather what you are measuring. Mean squared error measures the expected squared distance between an estimator and the true underlying parameter: $$\text{MSE}(\hat{\theta}) = E\left[(\hat{\theta} - \theta)^2\right].$$ It is thus a measurement of the quality of an estimator. The mean squared prediction ...


24

Recall that the functional form of logistic regression is $$ f(x) = \frac{1}{1 + e^{-(\beta_0 + \beta_1 x_1 + \cdots + \beta_k x_k)}} $$ This is what is returned by predict_proba. The term inside the exponential $$ d(x) = \beta_0 + \beta_1 x_1 + \cdots + \beta_k x_k $$ is what is returned by decision_function. The "hyperplane" referred to in the ...


23

It's sort of an optical illusion: the eye looks at the graph, and sees that the red and blue graphs are right next to each. The problem is that they are right next to each other horizontally, but what matters is the vertical distance. The eye most easily see the distance between the curves in the two-dimensional space of the Cartesian graph, but what matters ...


21

As @Glen mentions you have to use a stat_smooth method which supports extrapolations, which loess does not. lm does however. What you need to do is use the fullrange parameter of stat_smooth and expand the x-axis to include the range you want to predict over. I don't have your data, but here's an example using the mtcars dataset: ggplot(mtcars,aes(x=disp,y=...


21

On a related note, which may be helpful, Tibshirani and colleagues have proposed a significance test for the lasso. The paper is available, and titled "A significance test for the lasso". A free version of the paper can be found here


21

This question and excellent exchange was the impetus for creating the predictInterval function in the merTools package. bootMer is the way to go, but for some problems it is not feasible computationally to generate bootstrapped refits of the whole model (in cases where the model is large). In those cases, predictInterval is designed to use the arm::sim ...


20

From version 1.8.8 of mgcv predict.gam has gained an exclude argument which allows for the zeroing out of terms in the model, including random effects, when predicting, without the dummy trick that was suggested previously. predict.gam and predict.bam now accept an 'exclude' argument allowing terms (e.g. random effects) to be zeroed for prediction. For ...


20

Statistical modeling is different from machine learning. For example, a linear regression is both a statistical model and a machine learning model. So if you compare a linear regression to a random forest, you’re just comparing a simpler machine learning model to a more complicated one. You’re not comparing a statistical model to a machine learning model. ...


19

The term you're searching for is 'extrapolation'. The problem is that no matter how much data you have, and how many intermediate levels you have between your endpoints on disk size (i.e., between 5 and 30), it is always possible that there is some degree of curvature in the true underlying function, that you simply don't have the power to detect. As a ...


18

This chart is definitely incomplete without at least the following information: how "terrorism" is defined for these purposes, how "refugee" is defined for these purposes, what time-span this data covers, and which people are included--for instance, does the lighting strike data include people who live in nursing homes and never go outside? Presumably (...


17

If you look at the help for predict.lme you will see that it has a level argument that determines which level to make the predictions at. The default is the highest or innermost which means that if you don't specify the level then it is trying to predict at the subject level. If you specify level=0 as part of your first predict call (without subject) then ...


17

The difference is the pre-processing. predict.train automatically centers and scales the new data (since you asked for that) while predict.randomForest takes whatever it is given. Since the tree splits are based on the processed values, the predictions will be off. Max


Only top voted, non community-wiki answers of a minimum length are eligible