Questions tagged [stepwise-regression]

Stepwise regression (often called forward or backward regression) involves fitting a regression model and adding or removing predictors based on $t$ statistics, $R^2$ or information criteria to arrive in a *stepwise* manner at a final model. This tag can also be used for forward selection, backward elimination & best subsets variable selection strategies.

Filter by
Sorted by
Tagged with
1 vote
1 answer
24 views

effect sizes from stepwise regression

I'm new to stepwise regression and I've been asked to conduct one for my boss. In doing so, they also asked for the effect sizes from each predictor in the model. Disregarding any debates around ...
1 vote
0 answers
29 views

PCA, stepAIC, and negative binomial regression

I have some output data (around 800 data points) that very nicely fits a negative binomial distribution. I checked using fitdistr() in R and it is a very good fit. Given this, my plan was to use ...
1 vote
0 answers
12 views

What is the difference between the criteria argument and the k= argument in step() in R?

What exactly is difference between the 'criteria' argument for the step() function in r thats described in the documentation here: https://search.r-project.org/CRAN/refmans/emdi/html/step.html versus ...
0 votes
0 answers
18 views

Why odds ratios cannot be higher than the stepwise betas?

I am performing a logistic regression analysis after performing a stepwise regression. I would like to know why the odds ratios of the logistic regression cannot be higher than the stepwise betas.
0 votes
0 answers
30 views

Reporting Stepwise Regression

G'day. For my research i calculated a backward stepwise regression for my first time using ...
1 vote
0 answers
22 views

forward stepwise AIC approach

I am learning about performing stepwise model selection by AIC and I have 2 questions here: 1.Regarding to stepwise AIC, what is contribution (effectiveness) of the number of parameters in the model ...
  • 51
1 vote
1 answer
34 views

Stepwise model selection by AIC

I am learning about performing stepwise model selection by AIC and having some questions: What is the regularization parameter for step-AIC? In what way is forward step-AIC an evolution of univariate ...
  • 51
0 votes
0 answers
55 views

How does lmStepAIC work in caret when using cross-validation?

In caret package in R , one can train linear models with stepwise selection based on AIC, using this function : ...
  • 253
0 votes
0 answers
59 views

In Variable selection method, should we include all the variables?

I've been working on a class project. While building a model, We've selected a few initial predictors (8 predictors out of 20) based on Business Knowledge. Next, we wanted to choose predictors based ...
6 votes
3 answers
131 views

Should stepwise regressions also be avoided for exploratory (hypothesis generating) modelling?

In a recent paper, Andrew Tredennick and colleagues (2021) suggested to use the drop1() function in R for exploratory modelling (that is to generate new hypotheses ...
  • 507
1 vote
1 answer
89 views

Does my predictor in my multiple regression have too many variables?

So I am trying to work out what is the best predictor of a) awareness over environmental issues, b) concern over environmental issues and c) pro-environmental behaviour from a set of sociodemographics ...
1 vote
0 answers
162 views

Can I use coefficients of ridge regression as feature importance values?

If I have normalized all predictors, can I use coefficients of ridge regression as feature importance values? Normalization in my case means: $(x - u) / s$ where $u$ is the mean and $s$ is the ...
  • 13
1 vote
0 answers
21 views

PCA to identify patterns in the data, forced to a particular variable?

Dataset: I have a hyperspectral dataset that consists 250 wavelength bands (x1,...x250) and corresponding reflectance measurements (y) for each band. Plotting X vs Y yields a spectral profile. I have ...
1 vote
0 answers
39 views

What am I missing to perform forward stepwise regression when p > n?

I keep running into warnings in RStudio when I use subsets where p > n. ISLR 6.4.3 mentions that forward stepwise can be useful for high dimensional data, which I'm trying to just test out for ...
0 votes
3 answers
79 views

How can I do Stepwise Method if only independent variables are given? [closed]

This is the data set How can I determine if this 4 groups differed significantly in their performance
2 votes
1 answer
145 views

How to run backward elimination in $R$ with both categorical data and numeric data? [closed]

Usually in backward elimination, we start with the full model of all covariates, check the $p$-value of $t$-statistic for each covariate (which is compared between the full model and the model minus ...
  • 245
0 votes
1 answer
49 views

Best-Subset Regression based on BIC versus Forward Selection based on AIC

I am trying to get a better grasp of BIC and AIC scores. I know BIC has a harsher penalty than AIC regarding model size (it prefers smaller, less complex models). Suppose there is a situation where I ...
  • 1
0 votes
0 answers
51 views

colinearity in stepwise regression

I would be very grateful if you can be so kind to clarify what may be wrong in the following steps and interpretation of the validity conditions of the stepwise regression. I am afraid there is a ...
  • 1
0 votes
1 answer
95 views

Regression on principal components (number of modes, evaluation, predictive value)

I seek advice on the use of regression on principal components. I have a large dataset of anatomical shapes with point correspondence, on which I have performed PCA to investigate the main modes of ...
  • 60
0 votes
0 answers
25 views

Interpreting beta estimates of categorical variable in stepwise forward selection in r

I am recruiting individuals based on their spends in a large retail store. 60 features are considered altogether and many are categorical type with many levels. State is one such variable with 27 ...
  • 203
0 votes
1 answer
106 views

Is there a good replacement of sum of squared deviations that do not tend to split on edges?

I build a predictive model (regression) on a dataset that has just one real-valued feature and one real-valued target. To make it even simpler I want to find just a step function (decision tree with ...
  • 355
2 votes
2 answers
53 views

What is the name of this algorithm for choosing which features to include in a neural network?

Once I read about one kind of neural network (for classification or feature selection) for a supervised training where you start with all inputs, then you proceed with a training step and randomly (or ...
  • 143
0 votes
0 answers
74 views

leaps for stepwise selection: issues with factor variables

I'm trying to use the forward stepwise selection from the package leaps in order to choose the best predictors for a logistic regression. This is the structure of my database ...
  • 1
5 votes
1 answer
399 views

(Why) Are stepwise regression coefficients biased?

Issues with stepwise regression are known to statisticians. It yields R-squared values that are badly biased to be high. The F and chi-squared tests quoted next to each variable on the printout do ...
  • 35.4k
0 votes
0 answers
159 views

panel data model and variable selection?

I am looking for a reference to perform variable selection with panel data. I have 9 variables and want to test which variables to keep or remove. Someone recommended step-wise analysis, but I cannot ...
  • 1
0 votes
1 answer
77 views

Regression in R - dummy variables

Hey I want to build a model in R and one of my idependent variable is categorical (it takes 10 different values). I change the type of this variable from "char" to factor and build a model ...
  • 117
0 votes
1 answer
90 views

Reducing a linear regression (OLS) model by dropping non-significant coefficients

Would it be proper for me to reduce a model by iterating though the coefficients and dropping the ones with high p-values and then refitting and doing this again until all coefficients are significant?...
  • 441
0 votes
1 answer
182 views

Does the column ordering matter in the stepwise algorithms used by R?

Suppose I have a large data set with variables $x_1, x_2, \ldots, x_p$ to predict response $y$ where $p$ is very large (however $n >> p$). I would like to perform forward stepwise regression on ...
  • 175
0 votes
0 answers
53 views

Forward selection metrics

When using forward selection for multiple linear regression, I've seen several metrics: (1) Using MSE - at each step, try adding each variable one at a time, see which variable reduces the MSE the ...
2 votes
0 answers
44 views

Is it correct to combine any feature selection (backward/forward/stepwise) with regularization in logistic regression?

I use stepwise regression for exclude "worst" features (based on p-value) and after try to build model with L2 regularization on selected features. Emperically, this model is better that ...
1 vote
1 answer
98 views

Reference request: accounting for AIC stepwise model selection with bootstrapped standard errors

The use of the AIC for model selection via comparison over many models is well-known and rightly maligned. But I read an interesting passage from Hastie et. al.'s classic The Elements of Statistical ...
  • 1,143
4 votes
1 answer
174 views

On the test of the fixed effects parameters in glmer (stepwise selection)

I'm using a logistic mixed-effect model with random intercept and I want to perform a stepwise selection of the parameters of the fixed-effects. Is there any way to perform s stepwise selection using ...
  • 135
0 votes
1 answer
208 views

How to choose first variable to do forward selection with (using p)?

I have four variables, and I'm supposed to forward select (using their p values) them at the 5% level. Normally you'd probably start with the variable that has the lowest p value, but I have some ...
1 vote
1 answer
144 views

Different selection criterion (not AIC) for stepwise model selection in R?

I'm trying to use the step() function in R for variable selection of a linear model. My model looks like this: ...
1 vote
1 answer
105 views

How does forward stepwise regression select variables

Suppose there are $p = 3$ variables total and suppose the forward stepwise procedure selects the third variable. The forward stepwise procedure will assign it a positive coefficient if and only if the ...
  • 1,653
0 votes
0 answers
85 views

Stepwise AIC - on which model is the forward- and backward step performed

After I have read on different websites different claims, I hope that someone could may help me with the following question. :) I refere to stepwise AIC w.r.t regression models. In fact, I wonder on ...
0 votes
1 answer
35 views

Can we extract the non-significant variables by using stepwise algorithm?

In my logistic regression model, I want to remove all the significant variables and keep only the non-significant ones. Because I want to have only the variables that are not distinguished between « ...
  • 135
0 votes
0 answers
35 views

Fitting a model after step with missing values

If I initially fit a linear model using only complete cases, and I then perform stepwise regression to obtain a nested model, which no longer includes some of the variables which had missingness. ...
  • 566
1 vote
1 answer
751 views

Multiple linear regression: p-value=0.25 pre-filter variable selection

I have used many times in a multiple logistic regression the criteria of p-value=0.25 like pre-filter variable selection using bivariate logistic regression , then I use a MANUAL stepwise (backward) ...
1 vote
0 answers
107 views

How to find the best model when there are lots of possible interactions

I have a dataset with 6 variables such as price, country (this is a categorical variable with 7 levels), Rating_A, Rating_B, Rating_C, and Rating_D. However, if I fit the GLM like ...
  • 11
5 votes
1 answer
122 views

Is overfitting an issue if all I care about is training error

I am working on a project where we perform non-response adjustment by weighting survey respondents by their probability of response. In order to do this, we need to estimate each respondents ...
  • 1,408
18 votes
4 answers
2k views

Is Fig 3.6 in Elements of Statistical Learning correct?

Here is the figure from the textbook: It shows a decreasing relationship between subset size $k$ and mean squared error (MSE) of the true parameters, $\beta$ and the estimates $\hat{\beta}(k)$. ...
0 votes
0 answers
111 views

How to report findings from regression model chosen using stepwise AIC

I want to test the effect of several explanatory variables on an independent. I am using stepwise AIC to chose the best model. Given that I have found the 'best model' how would you recommend I report ...
0 votes
2 answers
403 views

Are repeated cross validation, forward feature selection, and LASSO compatible?

I am working on building a predictive model on medical images based on global images features extraction (texture). As I only have 51 patients, I want to build a simple model with cross-validation. ...
  • 13
0 votes
0 answers
66 views

Stepwise Regression for VAR or VECM Models

I'm doing an analysis of integrated price series for two different types of crude oil. These crudes are priced off a differential to a main type ("marker"). I built an exogenous variable ...
2 votes
1 answer
99 views

How to deal with predictors which are not significant, although r-squared is significant?

I did factor analysis and found three factors. To examine if which factors significantly affected a certain dependent variable, I added all three factors to a regression model. The correlation ...
  • 21
3 votes
2 answers
49 views

R: Multiple Linear Regression - Prediction model with 4 Indenpendent Continuous Variables

We are tasked to build a linear model to predict the current of the river based on river width, river depth, distance to the ocean and bank height in order to understand the variation in stream ...
  • 31
3 votes
2 answers
129 views

Significance of epidemiological confounders in a generalized linear model

I am identifying risk factors for children snoring among several predictors with generalized linear model. With backward selection, age and sex do not appear to be significant so I removed them from ...
0 votes
0 answers
413 views

Backward stepwise selection stopping rule in an ordinal logistic regression context when a model's ranking ability is of importance

First off, I am aware that there are some problems with stepwise regression as for instance described here ;) I am saying this to avoid that the discussion goes in the direction of stepwise being an ...
  • 153
2 votes
1 answer
3k views

Forward and backward stepwise regression (AIC) for negative binomial regression (with real data)

I am doing some count data analysis. The data is in this link. Column A is the count data, and other columns are the independent variables. At first I used Poisson regression to analyze it: ...
  • 23

1
2 3 4 5
7