# Questions tagged [stepwise-regression]

Stepwise regression (often called forward or backward regression) involves fitting a regression model and adding or removing predictors based on $t$ statistics, $R^2$ or information criteria to arrive in a *stepwise* manner at a final model. This tag can also be used for forward selection, backward elimination & best subsets variable selection strategies.

302 questions
Filter by
Sorted by
Tagged with
1 vote
24 views

### effect sizes from stepwise regression

I'm new to stepwise regression and I've been asked to conduct one for my boss. In doing so, they also asked for the effect sizes from each predictor in the model. Disregarding any debates around ...
1 vote
29 views

### PCA, stepAIC, and negative binomial regression

I have some output data (around 800 data points) that very nicely fits a negative binomial distribution. I checked using fitdistr() in R and it is a very good fit. Given this, my plan was to use ...
1 vote
12 views

### What is the difference between the criteria argument and the k= argument in step() in R?

What exactly is difference between the 'criteria' argument for the step() function in r thats described in the documentation here: https://search.r-project.org/CRAN/refmans/emdi/html/step.html versus ...
18 views

### Why odds ratios cannot be higher than the stepwise betas?

I am performing a logistic regression analysis after performing a stepwise regression. I would like to know why the odds ratios of the logistic regression cannot be higher than the stepwise betas.
• 419
30 views

### Reporting Stepwise Regression

G'day. For my research i calculated a backward stepwise regression for my first time using ...
1 vote
22 views

### forward stepwise AIC approach

I am learning about performing stepwise model selection by AIC and I have 2 questions here: 1.Regarding to stepwise AIC, what is contribution (effectiveness) of the number of parameters in the model ...
• 51
1 vote
34 views

### Stepwise model selection by AIC

I am learning about performing stepwise model selection by AIC and having some questions: What is the regularization parameter for step-AIC? In what way is forward step-AIC an evolution of univariate ...
• 51
55 views

### How does lmStepAIC work in caret when using cross-validation?

In caret package in R , one can train linear models with stepwise selection based on AIC, using this function : ...
• 253
59 views

### In Variable selection method, should we include all the variables?

I've been working on a class project. While building a model, We've selected a few initial predictors (8 predictors out of 20) based on Business Knowledge. Next, we wanted to choose predictors based ...
131 views

### Should stepwise regressions also be avoided for exploratory (hypothesis generating) modelling?

In a recent paper, Andrew Tredennick and colleagues (2021) suggested to use the drop1() function in R for exploratory modelling (that is to generate new hypotheses ...
• 507
1 vote
89 views

### Does my predictor in my multiple regression have too many variables?

So I am trying to work out what is the best predictor of a) awareness over environmental issues, b) concern over environmental issues and c) pro-environmental behaviour from a set of sociodemographics ...
1 vote
162 views

### Can I use coefficients of ridge regression as feature importance values?

If I have normalized all predictors, can I use coefficients of ridge regression as feature importance values? Normalization in my case means: $(x - u) / s$ where $u$ is the mean and $s$ is the ...
• 13
1 vote
21 views

### PCA to identify patterns in the data, forced to a particular variable?

Dataset: I have a hyperspectral dataset that consists 250 wavelength bands (x1,...x250) and corresponding reflectance measurements (y) for each band. Plotting X vs Y yields a spectral profile. I have ...
1 vote
39 views

### What am I missing to perform forward stepwise regression when p > n?

I keep running into warnings in RStudio when I use subsets where p > n. ISLR 6.4.3 mentions that forward stepwise can be useful for high dimensional data, which I'm trying to just test out for ...
79 views

### How can I do Stepwise Method if only independent variables are given? [closed]

This is the data set How can I determine if this 4 groups differed significantly in their performance
145 views

### How to run backward elimination in $R$ with both categorical data and numeric data? [closed]

Usually in backward elimination, we start with the full model of all covariates, check the $p$-value of $t$-statistic for each covariate (which is compared between the full model and the model minus ...
• 245
49 views

### Best-Subset Regression based on BIC versus Forward Selection based on AIC

I am trying to get a better grasp of BIC and AIC scores. I know BIC has a harsher penalty than AIC regarding model size (it prefers smaller, less complex models). Suppose there is a situation where I ...
51 views

### colinearity in stepwise regression

I would be very grateful if you can be so kind to clarify what may be wrong in the following steps and interpretation of the validity conditions of the stepwise regression. I am afraid there is a ...
95 views

### Regression on principal components (number of modes, evaluation, predictive value)

I seek advice on the use of regression on principal components. I have a large dataset of anatomical shapes with point correspondence, on which I have performed PCA to investigate the main modes of ...
• 60
25 views

### Interpreting beta estimates of categorical variable in stepwise forward selection in r

I am recruiting individuals based on their spends in a large retail store. 60 features are considered altogether and many are categorical type with many levels. State is one such variable with 27 ...
• 203
106 views

### Is there a good replacement of sum of squared deviations that do not tend to split on edges?

I build a predictive model (regression) on a dataset that has just one real-valued feature and one real-valued target. To make it even simpler I want to find just a step function (decision tree with ...
• 355
53 views

### What is the name of this algorithm for choosing which features to include in a neural network?

Once I read about one kind of neural network (for classification or feature selection) for a supervised training where you start with all inputs, then you proceed with a training step and randomly (or ...
• 143
74 views

### leaps for stepwise selection: issues with factor variables

I'm trying to use the forward stepwise selection from the package leaps in order to choose the best predictors for a logistic regression. This is the structure of my database ...
• 1
399 views

### (Why) Are stepwise regression coefficients biased?

Issues with stepwise regression are known to statisticians. It yields R-squared values that are badly biased to be high. The F and chi-squared tests quoted next to each variable on the printout do ...
• 35.4k
159 views

### panel data model and variable selection?

I am looking for a reference to perform variable selection with panel data. I have 9 variables and want to test which variables to keep or remove. Someone recommended step-wise analysis, but I cannot ...
77 views

### Regression in R - dummy variables

Hey I want to build a model in R and one of my idependent variable is categorical (it takes 10 different values). I change the type of this variable from "char" to factor and build a model ...
• 117
90 views

### Reducing a linear regression (OLS) model by dropping non-significant coefficients

Would it be proper for me to reduce a model by iterating though the coefficients and dropping the ones with high p-values and then refitting and doing this again until all coefficients are significant?...
• 441
182 views

### Does the column ordering matter in the stepwise algorithms used by R?

Suppose I have a large data set with variables $x_1, x_2, \ldots, x_p$ to predict response $y$ where $p$ is very large (however $n >> p$). I would like to perform forward stepwise regression on ...
• 175
53 views

### Forward selection metrics

When using forward selection for multiple linear regression, I've seen several metrics: (1) Using MSE - at each step, try adding each variable one at a time, see which variable reduces the MSE the ...
44 views

### Is it correct to combine any feature selection (backward/forward/stepwise) with regularization in logistic regression?

I use stepwise regression for exclude "worst" features (based on p-value) and after try to build model with L2 regularization on selected features. Emperically, this model is better that ...
1 vote
98 views

### Reference request: accounting for AIC stepwise model selection with bootstrapped standard errors

The use of the AIC for model selection via comparison over many models is well-known and rightly maligned. But I read an interesting passage from Hastie et. al.'s classic The Elements of Statistical ...
• 1,143
174 views

### On the test of the fixed effects parameters in glmer (stepwise selection)

I'm using a logistic mixed-effect model with random intercept and I want to perform a stepwise selection of the parameters of the fixed-effects. Is there any way to perform s stepwise selection using ...
• 135
208 views

### How to choose first variable to do forward selection with (using p)?

I have four variables, and I'm supposed to forward select (using their p values) them at the 5% level. Normally you'd probably start with the variable that has the lowest p value, but I have some ...
1 vote
144 views

### Different selection criterion (not AIC) for stepwise model selection in R?

I'm trying to use the step() function in R for variable selection of a linear model. My model looks like this: ...
• 15
1 vote
105 views

### How does forward stepwise regression select variables

Suppose there are $p = 3$ variables total and suppose the forward stepwise procedure selects the third variable. The forward stepwise procedure will assign it a positive coefficient if and only if the ...
• 1,653
85 views

### Stepwise AIC - on which model is the forward- and backward step performed

After I have read on different websites different claims, I hope that someone could may help me with the following question. :) I refere to stepwise AIC w.r.t regression models. In fact, I wonder on ...
35 views

### Can we extract the non-significant variables by using stepwise algorithm?

In my logistic regression model, I want to remove all the significant variables and keep only the non-significant ones. Because I want to have only the variables that are not distinguished between « ...
• 135
35 views

### Fitting a model after step with missing values

If I initially fit a linear model using only complete cases, and I then perform stepwise regression to obtain a nested model, which no longer includes some of the variables which had missingness. ...
• 566
1 vote
751 views

### Multiple linear regression: p-value=0.25 pre-filter variable selection

I have used many times in a multiple logistic regression the criteria of p-value=0.25 like pre-filter variable selection using bivariate logistic regression , then I use a MANUAL stepwise (backward) ...
1 vote
107 views

### How to find the best model when there are lots of possible interactions

I have a dataset with 6 variables such as price, country (this is a categorical variable with 7 levels), Rating_A, Rating_B, Rating_C, and Rating_D. However, if I fit the GLM like ...
• 11
122 views

### Is overfitting an issue if all I care about is training error

I am working on a project where we perform non-response adjustment by weighting survey respondents by their probability of response. In order to do this, we need to estimate each respondents ...
• 1,408
2k views

### Is Fig 3.6 in Elements of Statistical Learning correct?

Here is the figure from the textbook: It shows a decreasing relationship between subset size $k$ and mean squared error (MSE) of the true parameters, $\beta$ and the estimates $\hat{\beta}(k)$. ...
• 398
111 views

### How to report findings from regression model chosen using stepwise AIC

I want to test the effect of several explanatory variables on an independent. I am using stepwise AIC to chose the best model. Given that I have found the 'best model' how would you recommend I report ...
• 1,143
403 views

### Are repeated cross validation, forward feature selection, and LASSO compatible?

I am working on building a predictive model on medical images based on global images features extraction (texture). As I only have 51 patients, I want to build a simple model with cross-validation. ...
• 13
66 views

### Stepwise Regression for VAR or VECM Models

I'm doing an analysis of integrated price series for two different types of crude oil. These crudes are priced off a differential to a main type ("marker"). I built an exogenous variable ...
99 views

### How to deal with predictors which are not significant, although r-squared is significant?

I did factor analysis and found three factors. To examine if which factors significantly affected a certain dependent variable, I added all three factors to a regression model. The correlation ...
• 21
49 views

### R: Multiple Linear Regression - Prediction model with 4 Indenpendent Continuous Variables

We are tasked to build a linear model to predict the current of the river based on river width, river depth, distance to the ocean and bank height in order to understand the variation in stream ...
• 31
129 views

### Significance of epidemiological confounders in a generalized linear model

I am identifying risk factors for children snoring among several predictors with generalized linear model. With backward selection, age and sex do not appear to be significant so I removed them from ...
413 views

### Backward stepwise selection stopping rule in an ordinal logistic regression context when a model's ranking ability is of importance

First off, I am aware that there are some problems with stepwise regression as for instance described here ;) I am saying this to avoid that the discussion goes in the direction of stepwise being an ...
• 153