New answers tagged

0

Boruta, based on RF, doesn't care about scale. It permutes a column several times and looks at the distribution of permuted importance vs. the non-permuted. It only labels as reject those columns that are more likely in the distribution than not. Things that are questionable are retained, but have an estimated importance value. Boruta looks at ...


0

There are cases each feature doesn't correlate with the target variable but the combination of features is strongly correlated with the target variable. In the extreme, there is the famous XOR problem. Let's say there are two features and one target variable. Two features and the target variable are binary taking True or False. And the target variable is ...


0

There's a python library for feature selection TextFeatureSelection. This library provides discriminatory power in the form of score for each word token, bigram, trigram etc. Those who are aware of feature selection methods in machine learning, it is based on wrapper method and provides ML engineers required tools to improve the classification accuracy in ...


0

This would be awesome! In fact it would be even better, if the firm would know how the weights would have been generated (for example if the weights might be biased due to ridge regression/lasso). This knowledge could be added in various ML models as some kind of "pre-knowledge" for example in a Baysian Framework. Or just be used as a guidline in model ...


9

You might be asking too much of Lasso. Even in the first simulation, Example 1 in the original paper (available from here), Lasso only chose the "correct" model about 1/4 of the time. And that was with 3 out of 8 predictors having non-zero coefficients, a high signal-to-noise ratio, and more observations (20 per simulation) than predictors. In Example 3 of ...


0

LOG is the geometric mean of $|y|$: $$ \left(\prod_{i=1}^n |y_i|\right)^{\frac{1}{n}} = \exp\left(\frac{1}{n}\sum_{i=1}^n \log|y_i| \right) $$ This builds in a strong assumption that $y_i \neq 0\forall i$ because $\log(0)$ is not a real number.


1

This seems like a sensible approach and has been used in other areas of changepoint detection to pick the penalty. See Lavielle (2005) for an intuitive approach (for changepoints rather than random forests) and the recent pre-print which uses the broken stick model. Romano et. al (2020+) https://arxiv.org/abs/2005.01379 Lavielle (2005) https://rmgsc.cr....


0

Answers here advises against variable selection, but the problem is real ... and still done. One idea that should be tried out more in practice is blind analyses, as discussed in this nature paper Blind analysis: Hide results to seek the truth. This idea has been mentioned in another post at this site, Multiple comparison and secondary research. The idea ...


1

Basic Idea of Boruta Algorithm Perform shuffling of predictors' values and join them with the original predictors and then build random forest on the merged dataset. Then make comparison of original variables with the randomised variables to measure variable importance. Only variables having higher importance than that of the randomised variables are ...


0

There is no substitute for good variables that can help classify who will succeed or likely drop out in their first year. My observations is that student who are better prepared for their first year college courses do better. This means taking recommended courses even if this results in a drop in their overall GPA (which is likely a good variable reflecting ...


1

Extracting keywords as binary features were the state of the art for a very long time. Most implementations of decision trees/forests can deal with a pretty large set of features. You should also consider weighting the features with TF-IDF scores. If speed is really critical, there are toolkits for linear models that can easily deal with a large number of ...


0

Dimensionality reduction is used to reduce the number of dimensions of your data. This is achieved by transforming your data into such form that has smaller dimension (less columns), but preserves some of the main characteristics of the data. This is different from feature selection, i.e. selecting some features (columns), while dropping other columns from ...


0

Visualize data in a 2D or 3D space is a good application of dimension reduction algorithms, but the algorithms can have other advantages. Intuitively, many real world data contain "redundant" information, and people want to remove them and have a cleaner view on data, and build a simpler model. For example, some real world data may record one people's ...


2

Preserving the most important features of the data is indeed the point of dimensionality reduction. Simplifying data allows you to plot them nicely in a 2D or 3D space, but you also have other possible applications: PCA allows to identify strongly correlated observations, and hence to reduce redundancy in data; it saves computation time while running a ...


0

Yes, it may very well make sense to do feature selection after PCA. PCA often yields a small number of components that explain a large fraction of variance in the data. Discarding the low-variance components and keeping only the high-variance components for future analysis can offer good a good balance w.r.t. the bias-variance tradeoff. In fact, PCA is often ...


0

There is no one-size-fits-all solution for this. The threshold could be judged by the researcher based on the association between the variables. For the high correlation issue, you could basically test the collinearity of the variables to decide whether to keep or drop variables (features). You could check Farrar-Glauber test (F-G test) for ...


3

The selection of the penalty value in penalized likelihood typically makes use of some form of information criteria such as AIC, BIC etc. Then the optimal value for the penalty term is where the information criteria minimizes. See the example below in R where a linear model is created with 45 zero coefficients and 5 non-zero ones and 500 random samples are ...


1

The information criterion (AIC, BIC, etc.) are equivalent to an adjusted RSquare measure simply penalizing the Goodness-of-fit of the whole model for the number of variables used. They won't give you information at the variable level. Given that I would eliminate the information criterion to evaluate specific variables. But, I would keep those to ...


1

It is most likely not a good idea. If you have many coefficients that are not very useful, i.e low T statistics, but adding up 50 of them might give you something huge... which just doesn't make sense. T-statistic doesn't take into account the explained variance. Worst scenario, one of one of your categories end up in a sweet spot, it has low number of ...


1

If you want to search over the number of features to retain, then you need some sort of cross-validation, and since as you point out this needs to be done inside the training set of the main model fit, this will require nested cross-validation. If that's not a computational problem for you, then sklearn makes this pretty simple. pipe = Pipeline(steps=[('...


Top 50 recent answers are included