Questions tagged [regression-strategies]

Regression Modeling Strategies

Filter by
Sorted by
Tagged with
98
votes
8answers
43k views

What is the benefit of breaking up a continuous predictor variable?

I'm wondering what the value is in taking a continuous predictor variable and breaking it up (e.g., into quintiles), before using it in a model. It seems to me that by binning the variable we lose ...
56
votes
4answers
24k views

Can a random forest be used for feature selection in multiple linear regression?

Since RF can handle non-linearity but can't provide coefficients, would it be wise to use random forest to gather the most important features and then plug those features into a multiple linear ...
18
votes
5answers
13k views

Can I ignore coefficients for non-significant levels of factors in a linear model?

After seeking clarification about linear model coefficients over here I have a follow up question concerning non-signficant (high p value) for coefficients of factor levels. Example: If my linear ...
13
votes
3answers
14k views

Model Selection: Logistic Regression

Suppose we have $n$ covariates $x_1, \dots, x_n$ and a binary outcome variable $y$. Some of these covariates are categorical with multiple levels. Others are continuous. How would you choose the "best"...
14
votes
1answer
7k views

Goodness-of-fit test in Logistic regression; which 'fit' do we want to test?

I am referring to the question and its answers: How to compare (probability) predictive ability of models developed from logistic regression? by @Clark Chong and answers/comments by @Frank Harrell. ...
12
votes
3answers
23k views

Should I remove non-significant variables from my regression model

I have run a multiple linear regression using stepwise regression to select the best model, however the best model returned has a non-significant variable. When I remove this the AIC value goes up ...
30
votes
2answers
4k views

Should final (production ready) model be trained on complete data or just on training set?

Suppose I trained several models on training set, choose best one using cross validation set and measured performance on test set. So now I have one final best model. Should I retrain it on my all ...
2
votes
2answers
4k views

Which model should I use to fit my data ? ordinal and non-ordinal, not normal and not homoscedastic

Here is the kind of data I have: I have two predictor variables: 1) discrete non-ordinal --> c('a','b','c') 2) discrete ordinal --> c(10,100,200,500) Response variable: Proportion of TRUE over a ...
6
votes
2answers
3k views

How to compare (probability) predictive ability of models developed from logistic regression?

I know some well-known measures are $c$ statistic, Kolmogorov-Smirnov $D$ statistic. However, as far as I know, those statistics take into account only of the rank order of the observations, and is ...
1
vote
2answers
5k views

Regression with categorical predictors - use only some dummy variables [duplicate]

I am working on a regression and I have a factor variable "Marital Status" Marital status has 5 levels: Single, Married, Divored, Widowed, Other (don't ask me what constitutes someone being an 'other'...
3
votes
0answers
6k views

Family in GLM - how to choose the right one?

When modeling data sampled in the field, I often come across the problem of determining the Family of the dependent variable for GLM (or GLMM). An example: in an ecological study, I have ~ 60 patches. ...
27
votes
3answers
43k views

Evaluating logistic regression and interpretation of Hosmer-Lemeshow Goodness of Fit

As we all know, there are 2 methods to evaluate the logistic regression model and they are testing very different things Predictive power: Get a statistic that measures how well you can predict the ...
14
votes
1answer
14k views

Logistic Regression with regression splines in R

I have been developing a logistic regression model based on retrospective data from a national trauma database of head injury in the UK. The key outcome is 30 day mortality (denoted as "Survive&...
21
votes
4answers
45k views

How should I check the assumption of linearity to the logit for the continuous independent variables in logistic regression analysis?

I am confused with the assumption of linearity to the logit for continuous predictor variables in logistic regression analysis. Do we need to check for the linear relationship while screening for ...
21
votes
2answers
856 views

Does LASSO suffer from the same problems stepwise regression does?

Stepwise algorithmic variable-selection methods tend to select for models which bias more or less every estimate in regression models ($\beta$s and their SEs, p-values, F statistics, etc.), and are ...
4
votes
1answer
4k views

Which variable relative importance method to use?

Following is a plot from relaimpo package of R which shows relative importance of predictor variables for regression of mpg on all other variables in mtcars dataset. The relative importance is ...
14
votes
3answers
14k views

Testing nonlinearity in logistic regression (or other forms of regression)

One of the assumption of logistic regression is the linearity in the logit. So once I got my model up and running I test for nonlinearity using Box-Tidwell test. One of my continuous predictors (X) ...
7
votes
1answer
7k views

Interpretation of calibration curve

I have a step-wise derived binary logistic regression model. I have used the calibrate(, bw=200, bw=TRUE) function in the rms ...
6
votes
1answer
3k views

Relative importance of variables in Cox regression

I've understood that relative importance of predictors is a tricky question. Suggested methods range from very complex models to very simple variable transformations. I've understood that the ...
9
votes
3answers
2k views

How to reduce predictors the right way for a logistic regression model

So I have been reading some books (or parts of them) on modeling (F. Harrell's "Regression Modeling Strategies" among others), since my current situation right now is that I need to do a logistic ...
18
votes
3answers
5k views

Model building and selection using Hosmer et al. 2013. Applied Logistic Regression in R

This is my first post on StackExchange, but I have been using it as a resource for quite a while, I will do my best to use the appropriate format and make the appropriate edits. Also, this is a multi-...
34
votes
5answers
45k views

Overfitting a logistic regression model

Is it possible to overfit a logistic regression model? I saw a video saying that if my area under the ROC curve is higher than 95%, then its very likely to be over fitted, but is it possible to ...
16
votes
4answers
20k views

Which variables explain which PCA components, and vice versa?

Using this data: head(USArrests) nrow(USArrests) I can do a PCA as thus: plot(USArrests) otherPCA <- princomp(USArrests) ...
18
votes
2answers
21k views

Can we use categorical independent variable in discriminant analysis?

In discriminant analysis, the dependent variable is categorical, but can I use a categorical variable (e.g residential status: rural, urban) along with some other continuous variable as independent ...
11
votes
3answers
20k views

Ranking features in logistic regression

I used Logistic Regression. I have six features, I want to know the important features in this classifier that influence the result more than other features. I used Information Gain but it seems that ...
3
votes
3answers
3k views

What problems do non-normality in predictor variables cause for a multiple regression analysis?

I am talking about a situation in which I have several continuous predictor variables predicting a continuous outcome. One of the predictors has a very non-normal distribution and has some wild ...
9
votes
5answers
12k views

Treating ordinal variables as continuous for regression problems

In the social sciences I have encountered that it is common to treat ordinal variables as continuous, for example variables originating from rating or Likert scales (strongly disagree, disagree, agree,...
1
vote
2answers
11k views

Logistic regression - how good is my model? [duplicate]

I am a beginner in ML so apologize in advance if this sounds silly. I did a logistic regression on a real data set and I am having problems measuring how well my model fits. I still don't understand ...
2
votes
1answer
388 views

Problems with zero values when testing for linearity of the logit

From Field's 'Discovering Statistics using SPSS' I read that you should test for linearity of the logit when using a logistic regression. The process explained in the book covers the natural log ...
2
votes
1answer
4k views

Test overfitting of logistic regression with limited volume

I have a set of samples with two labels red and black. I can build a logistic regression model to predict the label colour. Once a model is built, I would like to test whether it is overfitting or not....
13
votes
2answers
7k views

When is logistic regression suitable?

I'm currently teaching myself how to do classification, and specifically I'm looking at three methods: support vector machines, neural networks, and logistic regression. What I am trying to understand ...
17
votes
4answers
8k views

Why does propensity score matching work for causal inference?

Propensity score matching is used for make causal inferences in observational studies (see the Rosenbaum / Rubin paper). What's the simple intuition behind why it works? In other words, why if we ...
11
votes
1answer
1k views

How to interpret variables that are excluded from or included in the lasso model?

I got from other posts that one cannot attribute 'importance' or 'significance' to predictor variables that enter a lasso model because calculating those variables' p-values or standard deviations is ...
12
votes
3answers
8k views

In general, does normalization mean to normalize the samples or features?

I'm just getting into machine learning, and I have seen two conflicting practices for normalization. To be concrete, let's suppose that we have a $n \times d$ matrix containing our training data, ...
9
votes
1answer
5k views

How to make calibration plot for survival data without binning data?

To make a calibration plot for survival probabilities estimated from a Cox model, one can divide the estimated risk into groups, calculate the average risk within a group, and then compare this to the ...
9
votes
4answers
4k views

Adjust for everything you have in propensity score?

I have a methodological question, and therefore no sample dataset is attached. I'm planning to do a propensity score adjusted Cox regression that aims to examine whether a certain drug will reduce ...
11
votes
3answers
4k views

GLM with continuous data piled up at zero

I am trying to run a model to estimate how well catastrophic illnesses such as TB, AIDS etc affect spending on hospitalization. I have "per hospitalization cost" as the dependent variable and various ...
11
votes
1answer
2k views

Why is feature selection important, for classification tasks?

I'm learning about feature selection. I can see why it would be important and useful, for model-building. But let's focus on supervised learning (classification) tasks. Why is feature selection ...
2
votes
2answers
6k views

Minimum number of observations needed for penalized regression?

I'm wondering what should be the minimum sample size to perform ridge, lasso or elastic net regression. I have a binomial outcome that I want to relate with a set of features (18 features in some ...
6
votes
1answer
2k views

Reporting the effect of a predictor in a logistic regression fitted with a restricted cubic spline

I have been playing around with using restricted cubic splines using the RMS package. Output below. ...
2
votes
1answer
903 views

Is it reasonable to drop an interaction term?

I'm regressing a model $Y = X_1 + X_2 + X_1X_2$ and the result turns out that none of them are significant. However, if I drop the interaction term, $X_1$ becomes significant. Is it ok to drop the ...
8
votes
6answers
3k views

Dealing with non-normal distribution in "big" datasets, when do we throw out the CLT?

Apologies from the go as this question comes from an absolute newbie and will definitely not satisfy a lot of the detail required. Hence, your guidance in providing you the right information to allow ...
3
votes
2answers
2k views

Cox PH model: managing continuous variables and linearity assumption

In an epidemiological study, I'm using martingale plot to assess the linearity of continuous variables. Here are the Martingale Residuals (from Null Model) using R's ...
24
votes
2answers
2k views

Bayesian thinking about overfitting

I've devoted much time to development of methods and software for validating predictive models in the traditional frequentist statistical domain. In putting more Bayesian ideas into practice and ...
9
votes
1answer
2k views

How to estimate a calibration curve with bootstrap (R)

Question: I have fitted a probabilistic model (bayesian network) for modeling a binary outcome variable. I would like to create a high-resolution calibration plot (e.g. spline) corrected for ...
9
votes
5answers
11k views

Logistic Regression on Big Data

I have a data set of around 5000 features. For that data I first used Chi Square test for feature selection; after that, I got around 1500 variables which showed significance relationship with the ...
13
votes
3answers
15k views

Why should one do a WOE transformation of categorical predictors in logistic regression?

When is weight of evidence (WOE) transformation of categorical variables useful? The example can be seen in WOE transformation (So for a response $y$, & a categorical predictor with $k$ ...
8
votes
3answers
5k views

Internal validation via bootstrap: What ROC curve to present?

I am using the bootstrap approach for internal validation of a multivariate model built with either standard logistic regression OR elastic net. The procedure I use is as follows: 1) build model ...
10
votes
3answers
2k views

What are criteria and decision making for non-linearity in statistical models?

I hope that the following general question makes sense. Please keep in mind that for the purposes of this particular question I'm not interested in theoretical (subject domain) reasons for introducing ...
6
votes
4answers
3k views

Linear regression - is a model "useless" if $R^2$ is very small?

Given a complex output which depends on many underlying factors, I am given 3 explanatory variables and about 10K data points and the task to assess their impact on the output. The OLS model is very ...