What's the difference between kernel and stepwise logistic regression? I am confused by different terms of logistic regression. What are the differences between stepwise, kernel, forward, and backward?
 A: Kernel logistic regression (KLR) refers to using logistic regression into a kernel induced feature space, as is more common in support vector machines (SVM). By working in a feature space, kernel methods (including KLR) can be used to model nonlinear relationships. KLR is less popular than SVM because it's much slower to train models and typically does not offer better predictive performance.
Stepwise (logistic) regression refers to an implicit feature selection mechanism baked into the regression modeling process. Stepwise procedures are a controversial issue: the feature selection works well in some situations but with statistical issues that need to be addressed (poor estimates of model uncertainty). Because of this, stepwise regression is sometimes used as an example of data dredging.
Forward, backward and bidirectional elimination or different approaches used in stepwise regression.
A: All you need to know about stepwise is that it's ambiguous, inefficient, and a bad approach to feature selection. You begin by defining a selection criterion (oft neglected in the discussion of stepwise), but we can safely assume that AIC is the best choice. AIC is a likelihood / complexity tradeoff that is used to evaluate competing models of vastly different applications (when standard likelihood testing would not be appropriate). Stepwise regression successively adds (forward) or subtracts (backward) model terms iteratively (huge risk for local minima compared to best subsets regression) until an apparently optimal AIC is obtained, and that is called the target model.
As a note, none of these terms are specific to logistic regression. It's just that logistic regression is especially well-suited to estimating a binary classifier and its associated uncertainty.
