# Choosing between transformations in logistic regression

In linear regression, the transformations of explanatory variables is done to have maximum correlation with the dependent variable.

What is the best measure of choosing between multiple transformations in logistic regression as dependent variable is binary and not continuous?

The end goal is to maximize the lift (predictive power) of the model.

## 3 Answers

The optimality criterion used by logistic regression (and many other methods) is the likelihood function. It is used to estimate $\beta$ including multiple $\beta$ representing one $X$ to achieve quadratic, cubic, and piecewise polynomial (spline) fits. It can also be used to choose from among competing transformations of $X$ but the act of choosing will not be reflected in the information matrix, so the resulting variance of $X\hat{\beta}$ will be too small, making confidence intervals not have the stated coverage probability. If you make transformation estimation an explicit goal of model fitting (and regression splines are excellent ways to do this) you will preserve all aspects of statistical inference. Depending on the sample size, a restricted (linear in both tails) cubic spline with 4 knots, requiring 3 parameters, can be a good choice.

• Thanks for the response. Following are my concerns: 1) Will a univariately chosen transformation be the same if I was to choose one multivariately? To me there is no reason to believe that the univariately chosen will be the best in combination with other transformed variables. 2) I do not prefer using splines because of possibility of over-fitting and poor performance in validation sets. I was thinking of using Box-Cox transformation for explanatory variables and finding the best transformation with optimal value of ${\lambda}$. Does this make sense? Any thoughts? Commented Jul 15, 2013 at 9:33
• No, that doesn't resonate. Box-Cox is used for continuous univariate $Y$, and many users of Box-Cox do not know know to penalize for uncertainty in $\lamba$ nor that Box-Cox makes a strong assumption about the measurement origin (zero). Splines do not overfit any more than having too many predictors, and you can control the amount of fitting with the number of knots and with shrinkage (penalization; see the R rms lrm function for quadratic penalization). As you said, it is best to estimate transformations in an adjusted rather than a univariate fashion. Commented Jul 15, 2013 at 11:46
1. No, in linear models the transformation is not (or ought not) be done to have maximum correlation with the dependent variable. It should be done to either a) Meet model assumptions about the residuals or b) Have a more sensible explanatory variable; that is, one that makes sense, substantively. As @Andy points out, this may not be sufficient. But, in that case, I'd then look for an alternate method of regression (see below) rather than take some weird transformation. E.g. a model such as $Y = b_0 + b_1x_1^{.21} + b_2x_2^{.73}$ is going to be a mess to explain.

2. In logistic regression (at least, in dichotomous logistic) there are fewer assumptions (and none about the residuals, as far as I know), so only b) applies.

Even for linear models, I'd favor using b). And then, if the assumptions aren't met, using some other form of regression (could be robust regression, could be a spline model, could be polynomials).

• The statement have a more sensible explanatory variable is quite ambiguous and should be expanded upon. I would typically take it to mean a transformation that allows easier interpretation of the regression coefficients, but that is obviously not in and of itself sufficient (for either OLS or Logistic regression). Commented Jul 15, 2013 at 0:53
• As I said in my post, predictive power is of primary concern. Having sensible explanatory variables is desireable but not a priority. Therefore, if $Y = b_0 + b_1x_1^{.21} + b_2x_2^{.73}$ gives me better lift then is acceptable at this stage. The question is how to choose the best set of transformations to give the maximum lift. Commented Jul 15, 2013 at 9:15

With generalized linear modeling the mathematical measure that is minimized is called the "deviance" (-2*log-likelihood). There are several sorts of residuals that can be developed. The "deviance residuals" are the individual terms in a modestly complex expression. I think these a most understandable when applied to categorical variables. For a categorical variable using logistic regression these are just the differences between the log-odds(model) and log-odds(data), but for continuous variables they are somewhat more complex. Deviance residuals are what are minimized in the iterative process. See this description at the UCLA website for some nice plots of deviance residuals.

It looks to me that analysis of "lift" is done on the scale of probabilities, rather than on the log-odds or odds scale or likelihoods. I see that Frank Harrell has offered some advice and any perceived dispute between Frank and I should be resolved by massive weighting of Frank's opinion. (My advice would be to buy Frank's RMS book.) I'm surprised he didn't offer advice to consider penalized methods and that he didn't issue a caution against over-fitting. I would think that choosing a transformation simply because it maximized "lift" would be akin to choosing models that maximized "accuracy". I know he doesn't endorse that strategy.