Episode #125 of the Stack Overflow podcast is here. We talk Tilde Club and mechanical keyboards. Listen now

Questions tagged [stepwise-regression]

Stepwise regression (often called forward or backward regression) involves fitting a regression model and adding or removing predictors based on $t$ statistics, $R^2$ or information criteria to arrive in a *stepwise* manner at a final model. This tag can also be used for forward selection, backward elimination & best subsets variable selection strategies.

Filter by
Sorted by
Tagged with
0
votes
1answer
29 views

Regression analysis not showing the first level of treatments

I have my data that looks like this: data: or in R: ...
1
vote
0answers
27 views

How does stepwise ARIMA work?

How does a stepwise ARIMA model work? I understand how ARIMA works but i didn't find any good material to understand about stepwise ARIMA. Any leads will be helpful. Thanks
0
votes
1answer
28 views

Feature selection (backward elimination) in polynomial regression

I have a polynomial multiple (univariate) regression with 2nd degree (for example) as below. Question. When I execute backward elimination to select features, should I remove features from the ...
0
votes
1answer
45 views

When should you check if assumptions are met when using stepwise selection?

Suppose I want to find a linear model with Gaussian error for a given data set. (The data set contains insurance claims and the end goal is to predict claim cost from claim features.) Also, suppose ...
0
votes
0answers
31 views

My predictors are all categorical variables but the dependent is numerical, how to eliminate dummies?

My predictors are all categorical values but the dependent is numerical. How can I eliminate dummies if I use a linear regression model? The values are tough to solve with backward elimination; ...
0
votes
1answer
43 views

In stepwise regression, how to interpret non-significant variables? [duplicate]

I have more than 15 IVs such as age, gender, education, first language, technology proficiency, health condition, etc, and one of my DVs is health literacy level, which is measured through a standard ...
1
vote
0answers
84 views

How exactly does Bidirectional (Stepwise) elimination works?

I read about Forward selection and Backward elimination algorithms while learning to build machine learning models. I think I'm not quite clear on next approach, which was bidirectional elimination(...
0
votes
0answers
10 views

Test Error on various types of Regression

I'm testing a dataset for various types of regression, comparing test error for each one to the Mean Prediction Error, that I found at the beginning. Unfortunately I don't have any experience in this ...
6
votes
0answers
116 views

Understanding equation used by Hastie et al

I am trying to recreate FIGURE 3.6 from Elements of Statistical Learning. The only information about the figure is included in the caption. I am not clear on what the equation on the Y-axis means ...
0
votes
1answer
109 views

forward selection with mixed model using lmer [closed]

I am running a mixed model in R and would like to perform forward selection using the step function. However, when I set the direction to forward ...
6
votes
0answers
216 views

Recreating figure from Elements of Statistical Learning [closed]

I am trying to recreate FIGURE 3.6 from Elements of Statistical Learning. The only information about the figure is included in the caption. To recreate the forward stepwise line my process is as ...
18
votes
2answers
326 views

Does LASSO suffer from the same problems stepwise regression does?

Stepwise algorithmic variable-selection methods tend to select for models which bias more or less every estimate in regression models ($\beta$s and their SEs, p-values, F statistics, etc.), and are ...
1
vote
0answers
27 views

Equivalence of variable selection criteria in forward stepwise regression

Say we have already selected $x_1$ through $x_k$. To select the next variable, forward stepwise regression either: a. picks the variable that when added to the already selected variables, gives ...
0
votes
0answers
19 views

How to choose the independent variables in a GLMM without performing stepwise selection? With a global model? How to decide then?

I am trying to conduct an inferential binomial GLMM with a large dataset and many independent variables. I was attempting to do a stepwise AIC selection but keep reading it is a bad idea. However, ...
3
votes
1answer
83 views

Stepwise regression limitations avoided on bootstrap/independent datasets?

One of the prime objections to best-subset and stepwise regression techniques (forward selection and/or backward elimination) is that multiple hypothesis tests are conducted on the same dataset, ...
0
votes
0answers
26 views

Time series novel

I've exhaustively attempted to find a proper way to analyse a dataset. Despite finding several piece of information of what could be done, I kindly ask for suggestions of could be done, mainly in R. ...
0
votes
0answers
9 views

Is there a good plug and play method for online/iterative regression in scikit-learn?

I have an independent variable with 19 dimensions(19 features) and I need to perform stepwise regression. I need to perform iterative regression because the target value I am predicting becomes ...
2
votes
1answer
28 views

Linear Regression in stats model OLS

After removing the insignificant variables(p-value >.05), I fitted the OLS model again. I found there are still many variables which had p-value < .05 earlier have p-value > .05 now. Do I need to ...
1
vote
1answer
36 views

Backward and forward selection finds insignificant predictors

I have a set of possible predictors for a binary outcome. In order to obtain the best model, I start from the zero model, and do a stepwise selection (in R) in order to obtain the best predictors. The ...
0
votes
1answer
22 views

Confusing stepwise regression process

According to the algorithm for the backward stepwise selection from the book ISLR which is shown below: says that we need to choose the model among the $k$ models by having a smallest RSS or highest $...
0
votes
0answers
24 views

does stepwise regression only work when there are a few explanatory variables with a significant correlation with the dependent variable?

I understand that stepwise regression is computationally intensive in general but is it only "suitable" in cases where you can ignore several variables from the model due to statistical insignificance,...
0
votes
3answers
146 views

Lasso Regression as Variable Selection

Suppose we are initially given $p$ predictor variables. In lasso regression, we want to find estimates of the coefficients $\beta_1, \dots, \beta_p$ that minimize $\text{RSS}+ \lambda \sum_{j=1}^{p} |\...
1
vote
1answer
44 views

Should I report the pseudo $R^2$ value for full or final logistic regression model after removing NA's & running stepwise selection?

I'm working with a logistic regression model in r. model <- glm(response~., family="binomial", data) and I'm using ...
1
vote
0answers
34 views

Stepwise logistic regression: What exactly is meant by eliminating features based on contribution?

I would like to know how my program is selecting and removing features during stepwise regression. I'm using R's caret package which in turn I think is using stepAIC from the MASS package. I was ...
0
votes
0answers
42 views

Equivalence between Stepwise Regression and Lasso

A while ago I had learned of a theoretical result that suggested there was a correspondence with Forward/Backward/Stepwise regression and LASSO/RIDGE regression in terms of the coefficient of ...
1
vote
1answer
344 views

How to fit a stepwise regression with ARIMA errors using Arima function in R?

I am fitting a regression model with ARIMA errors in R using the Arima function from the ...
1
vote
1answer
116 views

feature importance using forward selection

In the following article the author has correctly mentioned that the "petal" is more important than "sepal" in case of iris data: https://towardsdatascience.com/feature-importance-and-forward-...
0
votes
0answers
40 views

Step-wise Multiple Regression or ANCOVA

I have an assignment that gives a dataset and a choice of 3 tests: Step-wise Multiple Regression, ANCOVA and Log-Linear Analysis. The dataset consists of ...
17
votes
1answer
708 views

Stepwise AIC - Does there exist controversy surrounding this topic?

I've read countless posts on this site that are incredibly against the use of stepwise selection of variables using any sort of criterion whether it be p-values based, AIC, BIC, etc. I understand why ...
1
vote
2answers
59 views

Significant main effects lost during ANCOVA due to interaction terms. Is type III the way to go?

I have some experimental data which I am analysing using step wise multiple regression (ANCOVA) in R using the step function. The response data (wp) is the leaf ...
0
votes
1answer
402 views

Backward elimination in a multinomial logistic regression model?

Following this UCLA article, I have fit a multinomial logistic regression model in R (say that Group is a factor with levels ...
4
votes
2answers
1k views

How to do stepwise regression with a binary dependent variable?

I want to use stepwise regression to reduce the number of variables. My dependent variable is a dummy variable (Fraud=1, None fraud=0) and I have 25 predictive variables. How can I do this?
1
vote
1answer
407 views

forward model selection on multivariate polynomial regression with high dimension data

I am trying to fit the best multivariate polynomial on a dataset using stepAIC(). My problem is that I have more variables (p=3003) than observations (n=500), so ...
0
votes
1answer
63 views

Why does R step backwards regression drop variable with lowest AIC?

I'm running a backwards selection process in R using the step() function and it seems to be dropping variables based on lowest AIC associated with that variable. Is ...
3
votes
1answer
57 views

Efficiently add a new predictor to an estimated multiple linear regression model

I want to use forward selection to choose predictors in a multiple linear regression model. If you have a regression with N predictors and want to add another predictor, is there a way to update the ...
1
vote
1answer
62 views

Clarification-Forward stepwise regression

I'm learning about forward stepwise and there are some things which are not so clear: If I have $p$ predictors, is it true that forward stepwise does $p$ iterations? If I add the predictors in each ...
1
vote
0answers
42 views

How do you determine which “direction” you should go in a stepwise regression?

I realize that you can both go forward or backward, or even in both directions, however I'm finding it a little confusing when one is more appropriate than the other? Can somebody explain to me or ...
1
vote
2answers
152 views

Metrics for selecting a logistic regression model

I have been running logistic regressions using forward, backward and 'both direction' stepwise procedures to guide the selection of the variables included in the model. I have been using AIC as a ...
0
votes
1answer
63 views

Step-wise Regression with only Categorical Predictors

Suppose we are assessing the impact three factors, each with two levels, have on some response $Y$. Let's call the factors $A$ with levels $\{a_1, a_2\}$, $B$ with levels $\{b_1, b_2\}$and $C$ with ...
1
vote
1answer
542 views

Beta regression (betareg) with caret and train [closed]

I have a dataset with a dependent in range (0,1) and numerical/categorical predictors. Chiefly to streamline the code and easily accomplish cross validation (feature selection/model fitting), I would ...
0
votes
0answers
92 views

model selection and model comparison

I have a question regarding to model comparison using multcomp for model comparison. Suppose I have a linear model y~x1+x2+x3, and there are three levels in x1, say x1_l1, x1_l2 and x1_l3. I would ...
0
votes
0answers
697 views

Backward elimination for a non-linear multivariate regression

I'm trying to determine what would be a good model for my problem. I am not a statistician and use some words colloquially - please excuse my lack of knowledge. I'll illustrate the problem with the <...
1
vote
1answer
253 views

Hypothesis Testing on coefficients in two subsets of data after Stepwise Regression

Is it a reasonable approach to run a hypothesis test to test whether the coefficients of a variable in two regressions on two different subsets of the same population are different if you have used ...
0
votes
1answer
134 views

Variable selection in Hierarchical Linear Modelling HLM through nlme lme()

Background of my question:- In Linear Regression through R we can mention the direction="both"/"forward"/"backward" in step(lm()) function to tell R for choosing the best set of variables based on AIC....
0
votes
0answers
85 views

In using backward elimination procedure how to control for type I error?

Use backward elimination procedure to decide which predictor variables can be dropped from the regression model. Control the type I error at = . 10 at each stage In using backward ...
1
vote
1answer
474 views

Why does forward stepwise selection reduce the AUC of a classifier to values < 0.500?

I've recently been benchmarking different methods for feature selection, and found a weird issue when using forward stepwise regression. Specifically, when I train a sparse logistic regression model ...
1
vote
1answer
1k views

Stepwise regression based on F-statistic in R [closed]

I know that the stepAIC function in R allows us to perform stepwise regression but I was wondering if there's any option (or other function) to perform a F-...
3
votes
1answer
5k views

Forward or backward sequential feature selection?

I was trying to carry out feature selection on a dataset using sequential feature selection. The dataset contains more than 5000 observations (rows) and 22 features (columns). Now I see that there are ...
3
votes
2answers
3k views

What is stepwise linear regression? [duplicate]

I am reading about 'interaction effects on linear regression' here and came across 'stepwise linear regression'. There are originally 5 predictors in the model. This means to say that by using the ...
0
votes
0answers
33 views

Stepwise not returning expected results in R

Why would step() output different outcomes when a better fit can be produced? I have two datasets model that should have the same relationships with a set of ...