# Questions tagged [stepwise-regression]

Stepwise regression (often called forward or backward regression) involves fitting a regression model and adding or removing predictors based on $t$ statistics, $R^2$ or information criteria to arrive in a *stepwise* manner at a final model. This tag can also be used for forward selection, backward elimination & best subsets variable selection strategies.

301 questions
Filter by
Sorted by
Tagged with
11 views

### forward stepwise AIC approach

I am learning about performing stepwise model selection by AIC and I have 2 questions here: 1.Regarding to stepwise AIC, what is contribution (effectiveness) of the number of parameters in the model ...
• 41
1 vote
23 views

### Stepwise model selection by AIC

I am learning about performing stepwise model selection by AIC and having some questions: What is the regularization parameter for step-AIC? In what way is forward step-AIC an evolution of univariate ...
• 41
25 views

### Manual variable selection in GLMM

I'm modeling my data using GLMM with 1 random factor and 10 variables that are of interest. Instead of using automatic selection, I started with the full model (including all variables except for the ...
15 views

### How does lmStepAIC work in caret when using cross-validation?

In caret package in R , one can train linear models with stepwise selection based on AIC, using this function : ...
• 253
25 views

### In Variable selection method, should we include all the variables?

I've been working on a class project. While building a model, We've selected a few initial predictors (8 predictors out of 20) based on Business Knowledge. Next, we wanted to choose predictors based ...
70 views

### Should stepwise regressions or overfitting also be avoided for exploratory (hypothesis generating) modelling?

In a recent paper, Andrew Tredennick and colleagues (2021) suggested to use the drop1() function in R for exploratory modelling (that is to generate new hypotheses ...
• 383
1 vote
63 views

### Does my predictor in my multiple regression have too many variables?

So I am trying to work out what is the best predictor of a) awareness over environmental issues, b) concern over environmental issues and c) pro-environmental behaviour from a set of sociodemographics ...
1 vote
99 views

### Can I use coefficients of ridge regression as feature importance values?

If I have normalized all predictors, can I use coefficients of ridge regression as feature importance values? Normalization in my case means: $(x - u) / s$ where $u$ is the mean and $s$ is the ...
• 13
1 vote
15 views

### PCA to identify patterns in the data, forced to a particular variable?

Dataset: I have a hyperspectral dataset that consists 250 wavelength bands (x1,...x250) and corresponding reflectance measurements (y) for each band. Plotting X vs Y yields a spectral profile. I have ...
1 vote
37 views

### What am I missing to perform forward stepwise regression when p > n?

I keep running into warnings in RStudio when I use subsets where p > n. ISLR 6.4.3 mentions that forward stepwise can be useful for high dimensional data, which I'm trying to just test out for ...
77 views

### How can I do Stepwise Method if only independent variables are given? [closed]

This is the data set How can I determine if this 4 groups differed significantly in their performance
67 views

### How to run backward elimination in $R$ with both categorical data and numeric data? [closed]

Usually in backward elimination, we start with the full model of all covariates, check the $p$-value of $t$-statistic for each covariate (which is compared between the full model and the model minus ...
• 143
40 views

### Best-Subset Regression based on BIC versus Forward Selection based on AIC

I am trying to get a better grasp of BIC and AIC scores. I know BIC has a harsher penalty than AIC regarding model size (it prefers smaller, less complex models). Suppose there is a situation where I ...
41 views

### colinearity in stepwise regression

I would be very grateful if you can be so kind to clarify what may be wrong in the following steps and interpretation of the validity conditions of the stepwise regression. I am afraid there is a ...
69 views

### Regression on principal components (number of modes, evaluation, predictive value)

I seek advice on the use of regression on principal components. I have a large dataset of anatomical shapes with point correspondence, on which I have performed PCA to investigate the main modes of ...
• 60
24 views

### Interpreting beta estimates of categorical variable in stepwise forward selection in r

I am recruiting individuals based on their spends in a large retail store. 60 features are considered altogether and many are categorical type with many levels. State is one such variable with 27 ...
• 203
98 views

### Is there a good replacement of sum of squared deviations that do not tend to split on edges?

I build a predictive model (regression) on a dataset that has just one real-valued feature and one real-valued target. To make it even simpler I want to find just a step function (decision tree with ...
• 463
46 views

### What is the name of this algorithm for choosing which features to include in a neural network?

Once I read about one kind of neural network (for classification or feature selection) for a supervised training where you start with all inputs, then you proceed with a training step and randomly (or ...
• 143
37 views

### leaps for stepwise selection: issues with factor variables

I'm trying to use the forward stepwise selection from the package leaps in order to choose the best predictors for a logistic regression. This is the structure of my database ...
• 1
28 views

### Regarding stepwise and hierarchical regression (regression model building)

I am currently analysing a dataset and I was wondering if it is wrong to use a stepwise regression (to reduce the number of predictors) followed by a hierarchical regression (add demographics to the ...
20 views

### feature selection by considering a reference variable before clustering

I want to cluster 39 homes based on 22 features like their build type, city, etc; However, after clustering, the project's main purpose is to analyze the energy consumption of each cluster. So the ...
287 views

### (Why) Are stepwise regression coefficients biased?

Issues with stepwise regression are known to statisticians. It yields R-squared values that are badly biased to be high. The F and chi-squared tests quoted next to each variable on the printout do ...
• 31.1k
26 views

### panel data model and variable selection?

I am looking for a reference to perform variable selection with panel data. I have 9 variables and want to test which variables to keep or remove. Someone recommended step-wise analysis, but I cannot ...
63 views

### Regression in R - dummy variables

Hey I want to build a model in R and one of my idependent variable is categorical (it takes 10 different values). I change the type of this variable from "char" to factor and build a model ...
• 117
76 views

### Reducing a linear regression (OLS) model by dropping non-significant coefficients

Would it be proper for me to reduce a model by iterating though the coefficients and dropping the ones with high p-values and then refitting and doing this again until all coefficients are significant?...
• 441
132 views

### Does the column ordering matter in the stepwise algorithms used by R?

Suppose I have a large data set with variables $x_1, x_2, \ldots, x_p$ to predict response $y$ where $p$ is very large (however $n >> p$). I would like to perform forward stepwise regression on ...
• 175
35 views

### Forward selection metrics

When using forward selection for multiple linear regression, I've seen several metrics: (1) Using MSE - at each step, try adding each variable one at a time, see which variable reduces the MSE the ...
35 views

### Is it correct to combine any feature selection (backward/forward/stepwise) with regularization in logistic regression?

I use stepwise regression for exclude "worst" features (based on p-value) and after try to build model with L2 regularization on selected features. Emperically, this model is better that ...
1 vote
79 views

### Reference request: accounting for AIC stepwise model selection with bootstrapped standard errors

The use of the AIC for model selection via comparison over many models is well-known and rightly maligned. But I read an interesting passage from Hastie et. al.'s classic The Elements of Statistical ...
• 1,033
151 views

### On the test of the fixed effects parameters in glmer (stepwise selection)

I'm using a logistic mixed-effect model with random intercept and I want to perform a stepwise selection of the parameters of the fixed-effects. Is there any way to perform s stepwise selection using ...
• 135
145 views

### How to choose first variable to do forward selection with (using p)?

I have four variables, and I'm supposed to forward select (using their p values) them at the 5% level. Normally you'd probably start with the variable that has the lowest p value, but I have some ...
1 vote
109 views

### Different selection criterion (not AIC) for stepwise model selection in R?

I'm trying to use the step() function in R for variable selection of a linear model. My model looks like this: ...
• 15
1 vote
104 views

### How does forward stepwise regression select variables

Suppose there are $p = 3$ variables total and suppose the forward stepwise procedure selects the third variable. The forward stepwise procedure will assign it a positive coefficient if and only if the ...
• 1,613
56 views

### Stepwise AIC - on which model is the forward- and backward step performed

After I have read on different websites different claims, I hope that someone could may help me with the following question. :) I refere to stepwise AIC w.r.t regression models. In fact, I wonder on ...
33 views

### Can we extract the non-significant variables by using stepwise algorithm?

In my logistic regression model, I want to remove all the significant variables and keep only the non-significant ones. Because I want to have only the variables that are not distinguished between « ...
• 135
19 views

### Fitting a model after step with missing values

If I initially fit a linear model using only complete cases, and I then perform stepwise regression to obtain a nested model, which no longer includes some of the variables which had missingness. ...
• 556
1 vote
519 views

### Multiple linear regression: p-value=0.25 pre-filter variable selection

I have used many times in a multiple logistic regression the criteria of p-value=0.25 like pre-filter variable selection using bivariate logistic regression , then I use a MANUAL stepwise (backward) ...
1 vote
65 views

### How to find the best model when there are lots of possible interactions

I have a dataset with 6 variables such as price, country (this is a categorical variable with 7 levels), Rating_A, Rating_B, Rating_C, and Rating_D. However, if I fit the GLM like ...
• 11
111 views

### Is overfitting an issue if all I care about is training error

I am working on a project where we perform non-response adjustment by weighting survey respondents by their probability of response. In order to do this, we need to estimate each respondents ...
• 1,408
2k views

### Is Fig 3.6 in Elements of Statistical Learning correct?

Here is the figure from the textbook: It shows a decreasing relationship between subset size $k$ and mean squared error (MSE) of the true parameters, $\beta$ and the estimates $\hat{\beta}(k)$. ...
• 388
98 views

### How to report findings from regression model chosen using stepwise AIC

I want to test the effect of several explanatory variables on an independent. I am using stepwise AIC to chose the best model. Given that I have found the 'best model' how would you recommend I report ...
• 1,113
354 views

### Are repeated cross validation, forward feature selection, and LASSO compatible?

I am working on building a predictive model on medical images based on global images features extraction (texture). As I only have 51 patients, I want to build a simple model with cross-validation. ...
• 13
51 views

### Stepwise Regression for VAR or VECM Models

I'm doing an analysis of integrated price series for two different types of crude oil. These crudes are priced off a differential to a main type ("marker"). I built an exogenous variable ...
65 views

### How to deal with predictors which are not significant, although r-squared is significant?

I did factor analysis and found three factors. To examine if which factors significantly affected a certain dependent variable, I added all three factors to a regression model. The correlation ...
• 21
47 views

### R: Multiple Linear Regression - Prediction model with 4 Indenpendent Continuous Variables

We are tasked to build a linear model to predict the current of the river based on river width, river depth, distance to the ocean and bank height in order to understand the variation in stream ...
• 31
125 views

### Significance of epidemiological confounders in a generalized linear model

I am identifying risk factors for children snoring among several predictors with generalized linear model. With backward selection, age and sex do not appear to be significant so I removed them from ...
298 views

### Backward stepwise selection stopping rule in an ordinal logistic regression context when a model's ranking ability is of importance

First off, I am aware that there are some problems with stepwise regression as for instance described here ;) I am saying this to avoid that the discussion goes in the direction of stepwise being an ...
• 153
2k views

### Forward and backward stepwise regression (AIC) for negative binomial regression (with real data)

I am doing some count data analysis. The data is in this link. Column A is the count data, and other columns are the independent variables. At first I used Poisson regression to analyze it: ...
• 23
184 views

### Understanding 'aggressiveness' of lasso, forward stepwise selection and best subset selection in Hastie, T., Tibshirani, R. & Tibshirani, R.J. (2017)

Hastie et al. (2017) explain how the above mentioned methods perform depending on the signal-to-noise ratio (SNR) with their varying 'aggressiveness'. Now I don't understand why the different methods ...
• 115