Effect of feature re-weighting on logistic regression

Question

I've been recently working on the following problem:

Let $F = \{F_1, F_2,F_3\}$ denote a set of feature sets. For example, $F_1$ is comprised of 100 actual features. Before training a logistic regression classifier, I re-weight the individual feature sets (re-scaled between 0 and 1) as follows:

$$F_x = w_x * F_x,$$

i.e., by multiplying each feature set with a weight between 0 and 1.

My question is the following:

Assuming I get (for the aforementioned example) weights: $\mathcal{W} = \{w_1 = 0.2, w_2 = 0.9,w_3 = 0.000004\}$ (the index corresponds to an individual feature set from $F$).

Can I interpret this in the lines of: "The second feature set (2) contributes the most to the learning" etc.?

With other words: How sensitive is logistic regression to such changes in feature values.

Thanks!

Answer 1

This depends on whether you are talking about logistic regression or penalized (e.g. LASSO, ridge or elastic net) logistic regression.

• In standard logistic regression re-scaling features in any way doesn't really make any difference to the maximum likelihood estimate (obviously, coefficients will correspondingly end up being re-scaled), unless you multiply the feature by 0 (which just destroys all information). I.e. if before re-scaling the log-odds for an event increase by 0.5 for every year of age and you multiply age by 0.1, then the new coefficient will be 5 (i.e. for every 10 years of age the log-odds go up by 5). However, re-scaling coefficients can cause or fix numerical issues in fitting a logistic regression model.
• In penalized logistic regression, re-scaling features matters, because (as explained above) the absolute size of the maximum likelihood estimates changes and so features that were re-scaled with small factors get penalized more and those re-scaled with large factors get penalized less. Here, "penalized more" means that the regression coefficients will be shrunk from the maximum likelihood estimate relatively more towards zero than for the coefficients for other features.
• The second bullet is why, if one has no particular reason to favor some features over others, it is common to standardize (i.e. center on zero and fix SD to 1) features before applying penalized logistic regression. The kind of approach you describe can be applied thereafter to do something like expressing a prior belief about which coefficients are more likely to be truly further away from zero. If done without standardizing first, I'm not really sure why I would do it.

Answer 2

Current experiments suggest LR is indeed susceptible to such feature re-weightings.

Further, there is the trivial case when a given $w = 0$, as here no signal is present.