54

Your clue to figuring this out should be that the parameter estimates from the scikit-learn estimation are uniformly smaller in magnitude than the statsmodels counterpart. This might lead you to believe that scikit-learn applies some kind of parameter regularization. You can confirm this by reading the scikit-learn documentation. There is no way to switch ...


47

Scikit-learn's linear regression model allows users to disable intercept. So for one-hot encoding, should I always set fit_intercept=False? For dummy encoding, fit_intercept should always be set to True? I do not see any "warning" on the website. For an unregularized linear model with one-hot encoding, yes, you need to set the intercept to be false or else ...


43

Recall that the functional form of logistic regression is $$ f(x) = \frac{1}{1 + e^{-(\beta_0 + \beta_1 x_1 + \cdots + \beta_k x_k)}} $$ This is what is returned by predict_proba. The term inside the exponential $$ d(x) = \beta_0 + \beta_1 x_1 + \cdots + \beta_k x_k $$ is what is returned by decision_function. The "hyperplane" referred to in the ...


41

I'm not sure what you want to do in step 7a. As I understand it right now, it doesn't make sense to me. Here's how I understand your description: in step 7, you want to compare the hold-out performance with the results of a cross validation embracing steps 4 - 6. (so yes, that would be a nested setup). The main points why I don't think this comparison ...


39

Theory Polynomial regression is a special case of linear regression. With the main idea of how do you select your features. Looking at the multivariate regression with 2 variables: x1 and x2. Linear regression will look like this: y = a1 * x1 + a2 * x2. Now you want to have a polynomial regression (let's make 2 degree polynomial). We will create a few ...


39

Actually, scikit-learn does provide such a functionality, though it might be a bit tricky to implement. Here is a complete working example of such an average regressor built on top of three models. First of all, let's import all the required packages: from sklearn.base import TransformerMixin from sklearn.datasets import make_regression from sklearn.pipeline ...


36

here is an updated version: import numpy as np def mean_absolute_percentage_error(y_true, y_pred): y_true, y_pred = np.array(y_true), np.array(y_pred) return np.mean(np.abs((y_true - y_pred) / y_true)) * 100


36

The f1-score gives you the harmonic mean of precision and recall. The scores corresponding to every class will tell you the accuracy of the classifier in classifying the data points in that particular class compared to all other classes. The support is the number of samples of the true response that lie in that class. You can find documentation on both ...


31

The standard errors of the model coefficients are the square roots of the diagonal entries of the covariance matrix. Consider the following: Design matrix: $\textbf{X = }\begin{bmatrix} 1 & x_{1,1} & \ldots & x_{1,p} \\ 1 & x_{2,1} & \ldots & x_{2,p} \\ \vdots & \vdots & \ddots & \vdots \\ 1 & x_{n,1} & \ldots &...


30

The subset accuracy is indeed a harsh metric. To get a sense of how good or bad 0.29 is, some idea: look at how many labels you have an average for each sample look at the inter-annotator agreement, if available (if not, try yourself to see what subset accuracy the obtained when you are the classifier) think whether topic are well defined look at how many ...


28

The difference is because decomposition.PCA does not standardize your variables before doing PCA, whereas in your manual computation you call StandardScaler to do the standardization. Hence, you are observing this difference: PCA on correlation or covariance? If you replace pca.fit_transform(x) with x_std = StandardScaler().fit_transform(x) pca....


27

You are correct, XGBoost ('eXtreme Gradient Boosting') and sklearn's GradientBoost are fundamentally the same as they are both gradient boosting implementations. However, there are very significant differences under the hood in a practical sense. XGBoost is a lot faster (see http://machinelearningmastery.com/gentle-introduction-xgboost-applied-machine-...


25

TL:DR There won't be a difference if F-regression just computes the F statistic and pick the best features. There might be a difference in the ranking, assuming F-regression does the following: Start with a constant model, $M_0$ Try all models $M_1$ consisting of just one feature and pick the best according to the F statistic Try all models $M_2$ ...


25

There are two things mentioned in the CalibratedClassifierCV docs that hint towards the ways it can be used: base_estimator: If cv=prefit, the classifier must have been fit already on data. cv: If “prefit” is passed, it is assumed that base_estimator has been fitted already and all data is used for calibration. I may obviously be interpreting this ...


24

Short answer is: YES. Average Precision is a single number used to summarise a Precision-Recall curve: You can approximate the integral (area under the curve) with: Please take a look at this link for a good explanation.


24

Let's derive the Nyström approximation in a way that should make the answers to your questions clearer. The key assumption in Nyström is that the kernel function is of rank $m$. (Really we assume that it's approximately of rank $m$, but for simplicity let's just pretend it's exactly rank $m$ for now.) That means that any kernel matrix is going to have rank ...


24

For the problem $Ax \approx b$, forming the Normal equations squares the condition number of $A$ by forming $A^TA$. Roughly speaking $log_{10}(cond)$ is the number of digits you lose in your calculation if everything is done well. And this doesn't really have anything to do with forming the inverse of $A^TA$. No matter how $A^TAx = A^Tb$ is solved, you've ...


22

There are a lot of reasons that this could be the case. Off the top of my head I can think of one plausible cause, but without knowing more about the problem it is difficult to suggest that it is the one. An L-BFGS solver is a true quasi-Newton method in that it estimates the curvature of the parameter space via an approximation of the Hessian. So if your ...


21

I am trying to interpret the variable weights given by fitting a linear SVM. A good way to understand how the weights are calculated and how to interpret them in the case of linear SVM is to perform the calculations by hand on a very simple example. Example Consider the following dataset which is linearly separable import numpy as np X = np.array([[3,4],...


21

Nested cross-validation is used to avoid optimistically biased estimates of performance that result from using the same cross-validation to set the values of the hyper-parameters of the model (e.g. the regularisation parameter, $C$, and kernel parameters of an SVM) and performance estimation. I wrote a paper on this topic after being rather alarmed by the ...


21

First in terms of usage. You can get the prediction in statsmodels in a very similar way as in scikit-learn, except that we use the results instance returned by fit predictions = results.predict(X_test) Given the predictions, we can calculate statistics that are based on the prediction error prediction_error = y_test - predictions There is a separate ...


21

I spent some time looking through the Scikit source code in order to understand what f_regression does, and I would like to post my observations here. The original question was: Q: Does SelectKBest(f_regression, k = 4) produce the same result as using LinearRegression(fit_intercept=True) and choosing the first 4 features with the highest scores? The ...


21

PCA and TruncatedSVD scikit-learn implementations seem to be exactly the same algorithm. No: PCA is (truncated) SVD on centered data (by per-feature mean substraction). If the data is already centered, those two classes will do the same. In practice TruncatedSVD is useful on large sparse datasets which cannot be centered without making the memory usage ...


21

The question about why $X$ and $y$ are popular choices in mathematical notions has been answered in the History of Science and Mathematics SE website: Why are X and Y commonly used as mathematical placeholders? (In short: cause Descartes said so!) In terms of Linear Algebra, it is extremely common to use capital Latin letters for matrices (e.g. design ...


20

Update: Thanks to this discussion, scikit-learn was updated and works correctly now. Its LDA source code can be found here. The original issue was due to a minor bug (see this github discussion) and my answer was actually not pointing at it correctly (apologies for any confusion caused). As all of that does not matter anymore (bug is fixed), I edited my ...


20

Thanks SpanishBoy - It is a good piece of code. @ilanman: This checks VIF values and then drops variables whose VIF is more than 5. By "performance", I think he means run time. The above code took me about 3 hours to run on about 300 variables, 5000 rows. By the way, I have modified it to remove some extra loops. Also, i've made it a bit cleaner and return ...


20

There are two observations needed to understand this implementation. The first is that pred is not a probability, it is a log odds. The second is a standard algebraic manipulation of the binomial deviance that goes like this. Let $P$ be the log odds, what sklearn calls pred. Then the definition of the binomial deviance of an observation is (up to a ...


20

The rule of thumb is: Use ARI when the ground truth clustering has large equal sized clusters Usa AMI when the ground truth clustering is unbalanced and there exist small clusters I worked on this topic. Reference: Adjusting for Chance Clustering Comparison Measures


20

This is not technically an error in statsmodels, rather it is because statsmodels.OLS does not add the intercept/constant term to the right-hand-side of the regression equation by default -- you have to explicitly add it. In contrast, sklearn (and the vast majority of other regression programs) add the constant/intercept term by default unless it is ...


19

PCA is a dimension reduction tool, not a classifier. In Scikit-Learn, all classifiers and estimators have a predict method which PCA does not. You need to fit a classifier on the PCA-transformed data. Scikit-Learn has many classifiers. Here is an example of using a decision tree on PCA-transformed data. I chose the decision tree classifier as it works well ...


Only top voted, non community-wiki answers of a minimum length are eligible