# Tag Info

-1

It may be an excellent idea to run an OLS regression after LASSO. This is simply to double check that your LASSO variable selection made sense. Very often when you rerun the model using OLS regression you uncover that many of the variables selected by LASSO are nowhere near being statistically significant and/or have the wrong sign. And, that may invite ...

0

First, I would not double up on variable selection methods. This does not make much sense. I would instead use the best method given your data and model framework, and go with that. Second, there are numerous reasons to be very cautious regarding using LASSO as your preferred variable selection method. There is a simple reason why not using LASSO for ...

0

There is a simple reason why not using LASSO for variable selection. It just does not work as well as advertised. This is due to its fitting algorithm that includes a penalty factor that penalizes the model against higher regression coefficients. It seems like a good idea, as people think it always reduces model overfitting, and improves predictions (on ...

3

Always perform feature selection inside the cross-validation loop on the training data only. The alternative, selecting features first and then splitting into folds, will lead to biased results. If you do that, you are using both the training and the test data to find features that are associated with your target variable - you are using the test data to ...

0

Your question isn't very clear about what exactly you wanted to do. That said, f_classif does ANOVA for feature selection for you instead of chi-squared. R package FSelector provides chi squared feature selection with function chi.squared. Here is the example from the document: https://rdrr.io/cran/FSelector/man/chi.squared.html

4

This is a standard least squares problem and, as such, is solved by finding parameters that minimize the sum of squared residuals. A general approach is first to fit the model $$E[Y] = \beta_0 + \beta_1 x_1 + \cdots + \beta_k x_k.$$ Replace the $y_i$ by their residuals $$e_i = y_i - (\hat\beta_0 + \hat\beta_1 x_1 + \cdots + \hat\beta_k x_k).$$ The ...

0

A possible mistake that you're doing is not displaying the effect of $x_{k+1}$ after the effect of all other $x_i$. For this, rather than directly plotting $y$ against $x_{k+1}$ you should Partial Regression Plots (aka added variable plots; Wiki, or here). The three steps are (from Wiki): Computing the residuals of regressing the response variable against ...

3

Sounds like regularization models could be of use to you (Lasso especially, since your problem seems to be centered around picking variables, as opposed to Ridge and Elastic net). However, these can be difficult to set up. They are included in Stata 15 if you have access to that. In R, it's often a bit more handywork but hopefully someone can chime in with ...

17

As mentioned by @DemetriPananos, theoretical justification would be the best approach, especially if your goal is inference. That is, with expert knowledge of the actual data generation process, you can look at the causal paths between the variables and from there you can select the variables which are important, which are confounders and which are mediators....

4

Theoretical justification beyond all else. Aside from that, LASSO or similar penalized methods would be my next suggestion.

0

I am certainly not an expert in cancer research, but have read that genomic markers (also known as genetic markers) have been studied with respect to their relationship to various diseases. To unmask the true strength of such genetic markers, I suspect, one might want to control for exposure to agents likely associated with cancer especially relating to ...

3

One problem with dumping all of your predictors into the model is the invitation to extreme collinearity, which will inflate your standard errors and likely make your results uninterpretable. Judea Pearl has pointed to a second problem, if your inference is aimed at modeling causal relationships. In trying to "control for everything" by including all ...

3

Thanks for the data. Although this is just your graph with a smoother added, I suggest that -- beyond eyeballing the data -- the graph isn't supportive of any idea that there is an underlying sigmoid. There is nothing magic about the bandwidth used, and I tried lower and higher values with the same conclusion. Naturally I can't exclude the possibility that ...

0

If you are targeting on getting better accuracy instead of interpretability, DO NOT do feature selection. Use all features and add regularization. In general, more features means more information, in an extreme case, the feature has nothing to do with the prediction target, the fit will get the coefficient to be 0 automatically. So, more feature will not ...

0

Partially answered in comments: It seems reasonable; however, you might want to at least contemplate the possibility that the coefficient may be other-than-one (perhaps it may be that the response is not-proportional to numViews). One way to investigate that if you considered it a serious possibility would be to also have it as a predictor (as ...

0

Looking separately at predictors and omitting predictors likely to be related to outcome are not generally good ideas in Cox survival modeling. Seriously consider a different approach. Even in linear regression, omitting a predictor that is correlated both with the included predictors and with outcome leads to omitted-variable bias in the regression ...

2

This has been answered but here are a few more tips. When creating a model, you must always be conscious of the bias/variance trade off curves of the model. If a model has many features, it will predict with high variance, causing less accurate results. Too few features, and the model will have high bias, causing the model too often predict near the same ...

4

This is an instructive encounter with Hughes phenomenon. Naïvely, one would think that the more information one has the better one can model a system and make predictions. However, this prejudice ignores the so-called curse of dimensionality. Suppose for convenience that each feature (or variable) can only take on a finite number of values. In order to ...

5

Adding too many predictors will lead to overfitting. Always. Take a look at our overfitting tag. Don't just throw predictors into your model. (Cross-validation and regularization help somewhat, but they will not prevent all overfitting.) Also: Why is accuracy not the best measure for assessing classification models?

0

In linear regression, multi-collinearity problem leads to unreliable coefficients and corresponding p-values. Therefore if p-values are used to do step-wise variable selection/elimination, then you have to remedy multi-collinearity first before taking the decision to select or eliminate a variable.

0

Create a smaller matrix, with syntetic dependence between class and features. (i.e. class=quantile (feature1+2*feature2+3*feature3+..) Generate syntetic features which have similar distibution to your data. Run all Univariate selection methods you want to check which one works the best on syntetic data, and use it on real data.

Top 50 recent answers are included