15
As Demetri suggested, we need to add penalty='none' for the code to give expected results.
The revised code is as follows:
from sklearn.linear_model import LogisticRegression
def fit_predict(X, y, fit_intercept=True):
model = LogisticRegression(fit_intercept=fit_intercept, penalty='none')
model.fit(X.reshape(-1, 1), y)
print(f'model coefficients: {...
10
Easy visualization is a huge point in favor of using polynomial regression for illustration. (Note that both "illustration" and "demonstration", etymologically, have to do with showing pictures!) It also helps that the degree of the polynomial controls the amount of overfitting, and that polynomial regression allows looking at bona fide ...
10
I will add my own answer to this question in order to shine some light on why a penalty is added by default. I'm also posting for posterity as you are not the first person to get caught by this and you won't be the last.
Back in 2019, Zachary Lipton discovered sklearn applies the penalty by default too and this sparked a very intense debate on twitter and ...
8
No. A GLM is characterized by its link function and its target distribution. TweedieRegressor assumes a Tweedie distribution for the latter, which do not include the Bernoulli distribution needed for logistic regression.
8
Logistic regression corresponds to a Binomial distribution, a member of the exponential family, so in that sense it is nested within the Exponential Dispersion class of models. The $\mathrm{Tweedie}(\mu, \sigma^2)$ family specifically is also contained within $\mathrm{ED}$, but imposes the mean-variance relationship
\begin{align*}
\mu &= \mathbf{E}[Y]\\
\...
7
By default, sklearn logistic regression is penalized. From the documentation:
penalty{‘l1’, ‘l2’, ‘elasticnet’, ‘none’}, default=’l2’
Specify the norm of the penalty:
'none': no penalty is added;
'l2': add a L2 penalty term and it is the default choice;
'l1': add a L1 penalty term;
'elasticnet': both L1 and L2 penalty terms are added.
The penalty is ...
6
A more colloquial answer to this would be that the polynomial regression, the higher the order, has more flexibility to learn but also the tendency to memorize every data point in the training set, but with unneccessary and unpredictable wriggly waves, that makes forecasting nearly impossible as algorithms can get stuck in one area and extrapolation is quite ...
5
There is also a difference in how attribute explained_variance_ is calculated.
Let the data matrix $\mathbf X$ be of $n \times p$ size, where $n$ is the number of samples and $p$ is the number of variables. And $\mathbf{X}_c$ is the centered data matrix, i.e. column means have been subtracted and are now equal to zero in this matrix. Assume that we are ...
5
I am going to explain the case of Lasso, you can apply the same logic to ElasticNet.
How is the duality gap defined in the case of Lasso (/ElasticNet)?
The duality gap is the difference between a solution of the primal problem and a solution of the dual problem as said here.
The primal problem is the following:
$$ \min_{w \in \mathbb{R}^{n}} \frac{1}{2} ||...
5
What is the meaning of isotropic gaussian blobs , which are generated by sklearn.datasets.makeblobs?
The make_blobs() function draws samples from a special Gaussian mixture model. A general Gaussian mixture model with $k$ clusters has a density of the form
$$
p(x) = \sum_{i=1}^k \pi_i \mathcal{N}(\mu_i, \Sigma_i)
$$
where $\pi_i \ge 0$ are the weights of each cluster with $\sum_{i=1}^k \pi_i = 1$, $\mu_i$ are the cluster centers, and $\Sigma_i$ are the ...
4
Let me copy and paste a warning message from sklearn permutation importance page
Warning: Features that are deemed of low importance for a bad model (low cross-validation score) could be very important for a good model. Therefore it is always important to evaluate the predictive power of a model using a held-out set (or better with cross-validation) prior ...
4
I can only answer this question in a confirmatory way.
Normally, many people in machine learning have huge datasets with lots of features and rows or only a few features from Kaggle with a moderate amount of rows.
What is common to most people regardless of the dataset is that they do not derive a hypothesis or work out material. They see it as exploratory ...
4
I would recommend on preparing a pipeline for your users. The pipeline will scale, impute, rebalance data and it is even possible to programm it with different parameters or several estimators from a family. They have to do the same transformation, but with the pipeline you can be sure that it will be done in the same way for everyone. You should also make ...
4
Don't ask about the algorithm: focus on solving your problem.
The peak-finding solution I posted at https://stats.stackexchange.com/a/428083/919 will help you analyze the situation and decide how many peaks to identify using its mode trace plot:
Either four or five peaks looks like a reasonable number to use: this one shows the best locations of five peaks, ...
3
This looks fine:
You have completely separated the training and hyperparameter tuning sets from the calibration dataset.
You are tuning hyperparameters based on roc_auc, which is mostly invariant to calibration. If you had been tuning based on some other score (one that isn't based solely on rank-ordering, and especially one that depends on a decision ...
3
While gower distance hasn't been fully implemented into scikit-learn as a ready-to-use metric, we are lucky that many of the clustering-related functions (e.g., NearestNeighbor, DBSCAN) can take precomputed distance matrices instead of the raw data. To do this, you just need to specify metric = "precomputed" in the argument's for DBSCAN (see ...
3
I totally agree with @Matthew Drury and @Cameron Bieganek's analysis of perfect collinearity and degree of freedom.
However, I want to argue here that we do not need to avoid perfect collinearity if we are using methods such as gradient descends as our optimizer.
The reason why we might want to avoid perfect collinearity is when we are using linear ...
3
Scikit learn - GMM log-likelihood. Why use Cholesky's precision matrix instead of covariance matrix?
Working with Cholesky factors is cheaper because determinants of triangular matrices are given by the product of the elements on the diagonal. This massively reduces the computation time for the determinants because we effectiely get to skip the computations which are needed to compute the determinant in the general case.
Working with Cholesky factors also ...
3
What it is saying is that if you optimize your parameters based on the test set (that you didn't train on) performance, you might be simply overfitting on your test set, not reflecting the real performance of the model in real life situation. That's why a validation set is used.
The performance of your model should be approximately the same on both of these ...
3
Under the hood, BernoulliNB binarizes the features based on a numeric threshold (default 0.0) given by self.binarize (passed into the constructor), so the non-zero features would be treated as identical. So, as you mention, BernoulliNB is indeed using binary predictors even if your data is not binarized initially. The binarization operation is performed near ...
3
TL;DR -- The code in the question maximizes model misfit because it uses "minimize" in conjunction with neg_mean_squared_error. The correct usage is "maximize" in conjunction with "neg_mean_squared_error". We know this because the sklearn documentation says so, and also because we can demonstrate it with a simple script.
In ...
3
I’m not sure what is problem you’re trying to solve here. The prediction intervals are a feature, not a bug. The Gaussian process is a nonparametric, data-based model. The intervals are thin for regions where it saw enough of the data to be certain and broad for low-data regions, where it’s uncertain. The width of the interval tells you when the predictions ...
3
Yes, out-of-bag error is an estimate of the error rate (which is 1 - accuracy) that this training approach has for new data from the same distribution. This estimate is based on the predictions that you get for each data point by using only averaging those trees, for which the record was not in the training data.
If you have a low number of trees the OOB ...
3
PRESS is an out-of-sample measure. Even when you have an intercept, out-of-sample $R^2<0$ is possible. It means that you would have been better off predicting the mean of $y$ every time, regardless of your predictors/features. In other words, you have a poor model that is outperformed by a naïve model.
You say that this is unexpected performance. You're ...
3
Dave has addressed your specific question and I suggest you accept that answer. The point about "unexpected performance" in machine learning is quite on target.
What follows is more to get you headed toward better ways to handle your particular data set.
First, don't do a test/train split with so few cases. Build you model on the entire data set, ...
3
With time-series data, where you can expect auto-correlation in the data you should not split the data randomly to train and test set, but you should rather split it on time so you train on past values to predict future. Scikit-learn has the TimeSeriesSplit functionality for this.
The shuffle parameter is needed to prevent non-random assignment to to train ...
2
Actually, we can use cosine similarity in knn via sklearn.
The source code is here.
This works for me:
model = NearestNeighbors(n_neighbors=n_neighbor,
metric='cosine',
algorithm='brute',
n_jobs=-1)
model.fit(user_item_matrix_sparse)
2
As with any data problem, you should make sure your variables are correctly coded, e.g. as an unordered or ordered factor.
Any good decision tree algorithm then will treat the gill-color variable as an unordered factor. E.g., function ctree (R package partykit) finds the best grouping for unordered factors in a different way than for ordered factors, ...
2
This is possible without discretizing your counts or changing the form of your model to something with less natural assumptions (e.g. Gaussian).
The likelihood for a multinomial distribution can be expressed the way you've written it, but it can also be written differently to allow for nonnegative real counts.
$$p(\mathbf {x} \mid C_{k})
=\frac
{
\left(
...
2
By construction, "default" PCA (in scikit-learn or otherwise) always returns uncorrelated components.
Whitening also ensures that the different components of PCA have unit variance; this can be useful to improve the predictive accuracy of some algorithms.
To summarise:
whitening = decorrelation (e.g. PCA) + normalisation
But PCA can be performed ...
Only top voted, non community-wiki answers of a minimum length are eligible