79

Since RF can handle non-linearity but can't provide coefficients, would it be wise to use Random Forest to gather the most important Features and then plug those features into a Multiple Linear Regression model in order to explain their signs? I interpret OP's one-sentence question to mean that OP wishes to understand the desirability of the following ...


29

There are several issues to address. $R^2$ measures by themselves never measure goodness of fit; they measure mainly predictive discrimination. Goodness of fit only comes from comparing $R^2$ with the $R^2$ from a richer model The Hosmer-Lemeshow test is for overall calibration error, not for any particular lack of fit such as quadratic effects. It does ...


29

The resources you consider to be "wasted" are, in fact, information gains provided by logistic regression. You started out with the wrong premise. Logistic regression is not a classifier. It is a probability/risk estimator. Unlike SVM, it allows for and expects "close calls". It will lead to optimum decision making because it does not try to trick the ...


24

None of those proposed methods have been shown by simulation studies to work. Spend your efforts formulating a complete model and then fit it. Univariate screening is a terrible approach to model formulation, and the other components of stepwise variable selection you hope to use should likewise be avoided. This has been discussed at length on this site. ...


19

You will almost always get a better model after refitting on the whole sample. But as others have said you have no validation. This is a fundamental flaw in the data splitting approach. Not only is data splitting a lost opportunity to directly model sample differences in an overall model, but it is unstable unless your whole sample is perhaps larger than ...


17

Many people believe that you should use some strategy like starting with the most highly associated variable, and then adding additional variables in turn until one is not significant. However, there is no logic that compels this approach. Moreover, this is a kind of 'greedy' variable selection / search strategy (cf., my answer here: Algorithms for ...


16

To me the distinction is that with hypothesis testing one is considering contrasts of model parameters and is not entertaining the thought of changing the model. For example, in ANOVA, people are smart enough not to convert a 4 degree of freedom $F$-test to a 3 d.f. $F$-test when comparing 5 groups and finding that two of the groups have similar means. ...


16

Discriminant analysis assumes a multivariate normal distribution because what we usually consider to be predictors are really a multivariate dependent variable, and the grouping variable is considered to be a predictor. This means that categorical variables that are to be treated as predictors in the sense you wish are not handled well. This is one reason ...


15

The answer by @Sycorax is fantastic. In addition to those fully described aspects of the problem related to model fit, there is another reason not to pursue a multi-step process such as running random forests, lasso, or elastic net to "learn" which features to feed to traditional regression. Ordinary regression would not know about the penalization that ...


14

Both OLS and quantile regression (QR) are estimation techniques for estimating the coefficient vector $\beta$ in a linear regression model $$ y = X\beta + \varepsilon $$ (for the case of QR see Koenker (1978), p. 33, second paragraph). For certain error distributions (e.g. those with heavy tails), the QR estimator $\hat\beta_{QR}$ is more efficient than ...


14

I'll try to give you an intuitive understanding with minimal emphasis on the mathematics. The main problem with observational data and analyses that stem from it is confounding. Confounding occurs when a variable affects not only the treatment assigned but also the outcomes. When a randomized experiment is performed, subjects are randomized to ...


14

I'm not meaning to use models close to the data collecting process but rather doing continuous Bayesian monitoring of posterior probabilities, which require no penalty for multiplicity. Instead of computing an arbitrary target sample size I'd prefer to compute a maximum possible sample size (for budget approval) and otherwise to stop "when we get the answer"...


13

It is not appropriate to do feature screening and then to feed surviving features into a method that does not understand how much data torture was done previously. It is better to use a method that can handle all potential features (e.g., elastic net). Others' suggestions about using data reduction are also excellent ideas.


13

This is perhaps more of a comment than an answer, but I am not allowed to comment. This comment is meant to be complementary to the existing answer and comments. Nonlinear regression (least squares) model is generally taken to mean that the model is nonlinear in the parameters (nonlinear in at least one parameter anyway). As exemplified in Ekaba Bisong's ...


13

Almost any approach that does some form of model selection and then does further analyses as if no model selection had previously happened typically has poor proporties. Unless there are compelling theoretical arguments backed up by evidence from e.g. extensive simulation studies for realistic sample sizes and feature versus sample size ratios to show that ...


13

One way to look at this issue is that goodness of fit is training error and predictive accuracy is test error. ("Predictive power" is not a very precise term.) That is, goodness of fit is how well a model can "predict" data points you've already used to estimate its parameters, whereas predictive accuracy is how well a model can predict new data points, for ...


12

A part of this answer that I've learned since asking is that not binning and binning seeks to answer two slightly different questions - What is the incremental change in the data? and What is the difference between the lowest and the highest?. Not binning says "this is a quantification of the trend seen in the data" and binning says "I don't have enough ...


12

You are right, the loadings can help you here. They can be used to compute the correlation between the variables and the principal components. Moreover, the sum of the squared loadings of one variable over all principal components is equal to 1. Hence, the squared loadings tell you the proportion of variance of one variable explained by one principal ...


12

The over-arching issue is to decide for what types of problems linearity is to be expected, otherwise allow relationships to be nonlinear as the sample size allows. Most processes in biology, social sciences, and other fields are nonlinear. The only situations where I expect linear relationships are: Newtonian mechanics Prediction of $Y$ from $Y$ measured ...


12

I have the naive thought that linear regression is suitable only when one suspects that there are linear functional relationships between explanatory variables and the response variable. But not many real-world applications would seem to meet this criterion. This is not a correct understanding of what is "linear" in "linear regression". It is not the ...


11

I think that the accepted answer can be dangerously misleading (-1). There are at least four different questions mixed together in the OP. I will consider them one after another. Q1. How much of the variance of a given PC is explained by a given original variable? How much of the variance of a given original variable is explained by a given PC? These two ...


11

The following uses the R rms package using ordinary least squares modeling, and models the nonlinear effect smoothly using a restricted cubic spline with 4 knots at default knot locations. This generates one linear component and 2 nonlinear components for a total of 3 parameters per treatment group. require(rms) dd <- datadist(mydata); options(datadist='...


11

Well, since your model is linear, with the expected mpg equal to the linear predictor, you can read mpg straight off the linear predictor scale. For each variable, you find its value on the relevant scale. For example, imagine we wanted to find a predicted mpg for a car with wt=4, am=1, qsec=18: which gives a predicted mpg of about 18.94. Substituting ...


11

As I describe in detail in my book Regression Modeling Strategies (2nd edition available 2015-09-04, e-book available now), the process of attempting to transform variables before modeling is frought with problems, one of the most important being the distortion of type I error and confidence intervals. Categorization causes even more severe problems, ...


11

Logistic regression does NOT assume a linear relationship between the dependent and independent variables. It does assume a linear relationship between the log odds of the dependent variable and the independent variables (This is mainly an issue with continuous independent variables.) There is a test called the Box-Tidwell that you can use for this. The ...


11

A straightforward way to calibrate Cox survival models is to use the calibrate function provided by the rms package in R, as in the page that you linked. This package provides a cph method for Cox models that is designed to work with the calibration and validation methods that it provides for several types of regression models. Quoting from the manual page: ...


11

This is a very good question. When the number of candidate predictors $p$ is more than the effective sample size $n$, and one does not place any restrictions on the regression coefficients (e.g., one is not using shrinkage, a.k.a. penalized maximum likelihood estimation or regularization), the situation is hopeless. I say that for several reasons including ...


10

A first approach is to use PCA in order to reduce the dimensionality of the dataset. Try to retain ~97% of the total variance, this may help out quite a bit. Another option is to use something like stochastic gradient descent, this can be a much faster algorithm and able to fit into R's memory. EDIT: One problem with R is that you can only use your RAM ...


10

Your conclusion is correct. Think of two aspects: Statistical power to detect an effect. Unless the power is very high, one can miss even large real effects. Reliability: having a high probability of finding the right (true) features. There are at least 4 major considerations: Is the method reproducible by you using the same dataset? Is the method ...


10

Leave it in. The data are incapable of really telling you which model is "better" unless you use AIC in a highly structured way (e.g. on a pre-specified large group of variables), and removing insignificant variables invalidates the estimate of $\sigma^2$ and all $P$-values, standard errors, and confidence limits in addition to invalidating the formula for ...


Only top voted, non community-wiki answers of a minimum length are eligible