Xavier Bourret Sicotte
  • Member for 3 years, 11 months
  • Last seen this week
Understanding Lasso Regression's sparsity geometrically
Accepted answer
10 votes

Each circle around your point $\beta$ is actually an isoline in the 3rd dimension, i.e. upwards, and every points on such a line have the same value for the loss function. You could draw infinitely ...

View answer
Simulating ratio of two independent normal variables
1 votes

An exact expression for the ratio of two correlated normal random variables was given by Hinkley in 1969. The notation is taken from the stackexchange post For $Z = \frac{X}{Y}$ with $$ \begin{...

View answer
How to create partial dependency plot for logistic regression in Python sklearn
Accepted answer
1 votes

You could use PDPBox which is compatible with all Sklearn algorithms https://github.com/SauceCat/PDPbox

View answer
How we can avoid making L2 regularization causing the model to learn a moderate weight for some non-informative features.?
2 votes

A few ideas (out of many possible approaches) Use PCA to reduce the dimensionality (i.e. keep top 10 PCA dimensions) Use some form of feature selection to remove the non-informative features Use a ...

View answer
how to obtain the 'formula' of a ML classification model
3 votes

My question is, how do I extract the 'equation' out from the model It all depends on the model you are using. Some models do not have a straighrforward formula you can extract, and are often called ...

View answer
Choosing optimal alpha in elastic net logistic regression
Accepted answer
11 votes

Clarifying what is meant by $\alpha$ and Elastic Net parameters Different terminology and parameters are used by different packages, but the meaning is generally the same: The R package Glmnet uses ...

View answer
How will one determine a classifier to be of high bias or high variance?
2 votes

I presume you are interested in the intrinsic quality of an algorithm. This is a non trivial question and the topic of active research. Bounds on the bias and variance of an algorithm can be proven ...

View answer
Machine Learning and Missing Data: Impute, and If So When?
2 votes

The inclination of some collaborators is to go with the complete case type analysis, where only subjects with full data are used, but this makes me slightly nervous, as I feel like those missing data ...

View answer
How do I use weight vector of SVM and logistic regression for feature importance?
Accepted answer
2 votes

1) Assuming you have properly pre-processed your data then I would consider the absolute value of the weight. Negative value just means that it has a negative impact on the outcome, but a large ...

View answer
How to choose between t-test or non-parametric test e.g. Wilcoxon in small samples
3 votes

Simulating the difference of means of Gamma populations Comparing the t-test and the Mann Whitney test Summary of results When the variance of the two populations is the same, the Mann Whitney test ...

View answer
What causes lasso to be unstable for feature selection?
7 votes

Comments from Daniel J. McDonald Assistant professor at Indiana University Bloomington, author of the two papers mentioned in the original response from Xavier Bourret Sicotte. Your explanation ...

View answer
What causes lasso to be unstable for feature selection?
Accepted answer
14 votes

UPDATE See this second post for McDonald's feedback on my answer where the notion of risk consistency is related to stability. 1) Uniqueness vs Stability Your question is difficult to answer ...

View answer
How to plot High Dimensional supervised K-means on a 2D plot chart
2 votes

2D graph visualization T-SNE is obviously the first thing to come to mind, but you could perhaps push the exercise further by applying 2D graph visualization. Here is a good example from Sklearn ...

View answer
Unstable Prediction Probabilities
1 votes

A first answer Providing a definitive answer wil be difficult without knowing more about your data, understanding the models and chosen hyperparameters, and performing various checks on the results. ...

View answer
How to do k-fold cross validation with classifiers?
2 votes

This model is not a regressor. It is a classifier, with a variable reward/penalty moreover. So I cannot calculate the MSE of validation folds, there is no MSE here. How can I apply k-fold validation ...

View answer
k-fold crossvalidation: how does MSE go with k?
3 votes

I would argue that your claims, and diagram, are incorrect generalizations and can be misleading. Definitions and terminology Based on the terminology and definitions from Cross Validation and ...

View answer
Logistic Regression is a nonlinear regression problem?
Accepted answer
8 votes

Recall that the Logistic regression model is a non linear transformation of $\beta^Tx$ Probability of $(Y = 1)$: $p = \frac{e^{\alpha + \beta_1x_1 + \beta_2 x_2}}{1 + e^{ \alpha + \beta_1x_1 + \...

View answer
Algorithms to model non-linear relationship between two vectors
Accepted answer
5 votes

I suppose the answer depends on what you are trying to accomplish: Predicting future values Performing statistical inference Other ? My answer is that you can use any of the many regression models ...

View answer
Feature filtering with LASSO and cross validation
Accepted answer
2 votes

LASSO is unstable for feature selection The combination of LASSO and Cross Validation isn't always unstable for feature selection. Factors leading to instability are: More features than data points: ...

View answer
Cross Validation and Confidence Interval of the True Error
4 votes

Background Consider the following definitions: A training set $D = {z_1,...,z_n}$ with $Z_i \in Z$ independently sampled from an unknown distribution $P$. An algorithm $A$ which maps a data set to ...

View answer
Different ways to produce a confidence interval for odds ratio from logistic regression
4 votes

A comparison of confidence intervals methods on an example from ISL The book "Introduction to Statistical Learning" by Tibshirani, James, Hastie provides an example on page 267 of confidence ...

View answer
How does lasso regularization select the "less important" features?
5 votes

how does lasso method find which features are redundant to shrink their coefficients to zero? The Lasso method on its own does not find which features to shrink. Instead, it is a combination of Lasso ...

View answer
Lasso features selection through Crossvalidation
Accepted answer
1 votes

Clarifying your approach the 0 lasso coefs changed What value of $\lambda$ (the Lasso parameter) did you use and how did you determine it ? Your approach seems confusing: Have you performed k-...

View answer
How to forecast time-series data in Python, using many small samples rather than one large one
Accepted answer
1 votes

If your data meets the required assumptions you can perform linear regression and use the model to predict points - see here for an example Testing Hypothesis with Time series and Location Data If the ...

View answer
Testing Hypothesis with Time series and Location Data
3 votes

While Ert's and Ben's answers are excellent, they rely on the assumption that you understand how to perform statistical modelling on time series, which is not a trivial topic. Entire books are written ...

View answer
Locally Weighted Linear Regression implementation in either R or Python
Accepted answer
5 votes

Is there anyway to implement Locally Weighted Linear Regression without these problems? (preferably in Python) Yes, you can use Alexandre Gramfort's implementation - available on his Github page. (...

View answer
Feature selection in repeated cross-validation
Accepted answer
4 votes

I am doing repeated cross-validation/holdout, because the results vary a LOT depending on the splits. To me it seems that your model (which you don't talk about but probably should) is unstable. ...

View answer
Why does k-fold cross validation generate an MSE estimator that has higher bias, but lower variance then leave-one-out cross-validation?
4 votes

There has been much debate, confusion and contradiction on this topic, both on stats.stackexchange and in scientific literature. A useful paper is the 2004 study by Bengio & Grandvalet which ...

View answer
Find a weak learner in Boosting
2 votes

Why find the weak learner by the formula in the blue box? You define a class of weak learners as part of your boosting algorithm. For example you can perform boosting on decision trees, on SVM, ...

View answer
What is the way to plot a learning curve for k fold cross-validation
Accepted answer
3 votes

When using k fold cross validation method, to plot the learning curve, would training error be the misclassification error on DataTrain and cross-validation error be the misclassification error using ...

View answer