Skip to main content
53 votes
Accepted

What is the difference between decision_function, predict_proba, and predict function for logistic regression problem?

Recall that the functional form of logistic regression is $$ f(x) = \frac{1}{1 + e^{-(\beta_0 + \beta_1 x_1 + \cdots + \beta_k x_k)}} $$ This is what is returned by ...
Matthew Drury's user avatar
50 votes
Accepted

One-hot vs dummy encoding in Scikit-learn

Scikit-learn's linear regression model allows users to disable intercept. So for one-hot encoding, should I always set fit_intercept=False? For dummy encoding, fit_intercept should always be set to ...
Matthew Drury's user avatar
40 votes

How to compute the standard errors of a logistic regression's coefficients

The standard errors of the model coefficients are the square roots of the diagonal entries of the covariance matrix. Consider the following: Design matrix: $\textbf{X = }\begin{bmatrix} 1 & x_{...
j_sack's user avatar
  • 401
39 votes
Accepted

Ensemble of different kinds of regressors using scikit-learn (or any other python framework)

Actually, scikit-learn does provide such a functionality, though it might be a bit tricky to implement. Here is a complete working example of such an average ...
constt's user avatar
  • 506
37 votes

Mean absolute percentage error (MAPE) in Scikit-learn

here is an updated version: ...
Antonín Hoskovec's user avatar
34 votes
Accepted

The reason of superiority of Limited-memory BFGS over ADAM solver

There are a lot of reasons that this could be the case. Off the top of my head I can think of one plausible cause, but without knowing more about the problem it is difficult to suggest that it is the ...
David Kozak's user avatar
  • 1,832
33 votes
Accepted

PCA in numpy and sklearn produces different results

The difference is because decomposition.PCA does not standardize your variables before doing PCA, whereas in your manual computation you call ...
amoeba's user avatar
  • 106k
32 votes

Multilabel classification metrics on scikit

The subset accuracy is indeed a harsh metric. To get a sense of how good or bad 0.29 is, some idea: look at how many labels you have an average for each sample look at the inter-annotator agreement, ...
Franck Dernoncourt's user avatar
32 votes
Accepted

Nystroem Method for Kernel Approximation

Let's derive the Nyström approximation in a way that should make the answers to your questions clearer. The key assumption in Nyström is that the kernel function is of rank $m$. (Really we assume that ...
Danica's user avatar
  • 25k
30 votes
Accepted

XGBoost vs Python Sklearn gradient boosted trees

You are correct, XGBoost ('eXtreme Gradient Boosting') and sklearn's GradientBoost are fundamentally the same as they are both gradient boosting implementations. However, there are very significant ...
K88's user avatar
  • 444
30 votes
Accepted

Why uppercase for $X$ and lowercase for $y$?

The question about why $X$ and $y$ are popular choices in mathematical notions has been answered in the History of Science and Mathematics SE website: Why are X and Y commonly used as mathematical ...
usεr11852's user avatar
  • 44.7k
29 votes
Accepted

Scikit correct way to calibrate classifiers with CalibratedClassifierCV

There are two things mentioned in the CalibratedClassifierCV docs that hint towards the ways it can be used: base_estimator: If cv=prefit, the classifier must have been fit already on data. cv:...
Pintas's user avatar
  • 406
27 votes
Accepted

Difference between selecting features based on "F regression" and based on $R^2$ values?

TL:DR There won't be a difference if F-regression just computes the F statistic and pick the best features. There might be a difference in the ranking, assuming <...
Winks's user avatar
  • 3,721
27 votes

Scikit correct way to calibrate classifiers with CalibratedClassifierCV

I am interested in this question as well and wanted to add some experiments to better understand CalibratedClassifierCV (CCCV). As has already been said, there are two ways to use it. ...
Sam Weisenthal's user avatar
26 votes

How does one interpret SVM feature weights?

I am trying to interpret the variable weights given by fitting a linear SVM. A good way to understand how the weights are calculated and how to interpret them in the case of linear SVM is to perform ...
Xavier Bourret Sicotte's user avatar
26 votes
Accepted

Why not use the "normal equations" to find simple least squares coefficients?

For the problem $Ax \approx b$, forming the Normal equations squares the condition number of $A$ by forming $A^TA$. Roughly speaking $log_{10}(cond)$ is the number of digits you lose in your ...
Mark L. Stone's user avatar
25 votes
Accepted

Scikit-learn Normalization mode (L1 vs L2 & Max)

The options lead to different normalizations. if $x$ is the vector of covariates of length $n$, and say that the normalized vector is $y = x / z$ then the three options denote what to use for $z$: L1:...
Sven's user avatar
  • 1,121
25 votes
Accepted

Difference between scikit-learn implementations of PCA and TruncatedSVD

PCA and TruncatedSVD scikit-learn implementations seem to be exactly the same algorithm. No: PCA is (truncated) SVD on centered data (by per-feature mean substraction). If the data is already ...
ogrisel's user avatar
  • 3,769
23 votes
Accepted

SelectKBest - Feature Selection - Python - SciKit Learn

No, SelectKBest works differently. It takes as a parameter a score function, which must be applicable to a pair ($X$, $y$). The score function must return an array ...
user43451's user avatar
  • 566
23 votes
Accepted

Adjusted Rand Index vs Adjusted Mutual Information

Short answer Use ARI when the ground truth clustering has large equal sized clusters Use AMI when the ground truth clustering is unbalanced and there exist small clusters Longer answer I worked on ...
Simone's user avatar
  • 7,118
22 votes

How to systematically remove collinear variables (pandas columns) in Python?

Thanks SpanishBoy - It is a good piece of code. @ilanman: This checks VIF values and then drops variables whose VIF is more than 5. By "performance", I think he means run time. The above ...
Prashant's user avatar
  • 321
22 votes

Difference between selecting features based on "F regression" and based on $R^2$ values?

I spent some time looking through the Scikit source code in order to understand what f_regression does, and I would like to post my observations here. The original ...
user43451's user avatar
  • 566
22 votes
Accepted

What happens when bootstrapping isn't used in sklearn.RandomForestClassifier?

My question is this: is a random forest even still random if bootstrapping is turned off? Yes, it's still random. Without bootstrapping, all of the data is used to fit the model, so there is not ...
Sycorax's user avatar
  • 92.2k
21 votes
Accepted

Why does statsmodels.api.OLS over-report the r-squared value?

This is not technically an error in statsmodels, rather it is because statsmodels.OLS does not add the intercept/constant term ...
Jake Westfall's user avatar
21 votes
Accepted

Why can't scikit-learn SVM solve two concentric circles?

Let's start with warnings: All the preprocessing should be done using training set's fitted values: ...
gunes's user avatar
  • 57.6k
21 votes

Why do we use Linear Models when tree based models often work better than linear models?

Many excellent answers already. I would add a few more aspects. You do not say what you mean by "outperform", as in "tree models often outperform linear models". You presumably ...
Stephan Kolassa's user avatar
20 votes

PCA in numpy and sklearn produces different results

Here is a nice implementation with discussion and explanation of PCA in python. This implementation leads to the same result as the scikit PCA. This is another indicator that your PCA is wrong. <...
Nikolas Rieble's user avatar
19 votes
Accepted

Normalizing vs Scaling before PCA

Scaling (what I would call centering and scaling) is very important for PCA because of the way that the principal components are calculated. PCA is solved via the Singular Value Decomposition, which ...
John Madden's user avatar
  • 4,445
19 votes
Accepted

One hot encoding of a binary feature when using XGBoost

It's true that you're not missing information when you use only $k-1$ categories. In linear models, we are all familiar with the dummy variable trap and the relationship between a model with $k-1$ ...
Sycorax's user avatar
  • 92.2k
18 votes
Accepted

How to obtain optimal hyperparameters after nested cross validation?

Overview As @RockTheStar correctly concluded in the commentaries, the nested cross-validation is used only to access the model performance estimate. Dissociated from that, to find the best ...
Firebug's user avatar
  • 19.4k

Only top scored, non community-wiki answers of a minimum length are eligible