# Linear regression feature selection equivalent for a classification problem?

I have the following:

• Label (y): a boolean flag indicating something is good or bad
• Features (X): lower-level features that are believed to be correlated with the boolean flag. Some of them are continuous numbers and some are categorical.

Using these data, I need find a "magic formula" which calculates the "goodness score" of the sample based on the set of features (X). Despite I already have label and features for each sample, I need to figure out a numerical score which I could use to rank the samples by.

How should I go about designing/implementing this? If the label was a continuous number, I could simply use linear regression with L1 regularization, get coefficients, and multiply each feature by the weights to calculate the score. However, given the label is boolean, this becomes a classificiation problem and I'm wondering if there's a similar model I could use to suit my needs.

I know ensemble methods from sklearn provide feature_importances_, but I'm not sure if it's meant to be used to reconstruct a score by multiplying features. It does a great job telling me how much each feature is contributing to the decision process, but I'm not sure if it's meant to be used for my purpose.

Any help would be greatly appreciated!

## 1 Answer

What you want is called logistic regression! :)

Elaborating: logistic regression maps a vector of inputs to a number between 0 and 1. It is designed in such a way that "0" and "1" are extreme, limit cases which do not actually appear in prediction. This would not be the case with a linear map, which would need to be clipped at these values.

Although the prediction gives a continuous number between 0 and 1, the fitting requires just binary values as labels.

Logistic regression has a probabilistic interpretation. Being trained on a set of inputs and "0 or 1" labels, it then predicts, for a new input, the EXPECTED VALUE of the output label. The expected value is a number between 0 and 1. It also turns out to be equal to the probability of the label being 1! Logistic regression is fitted via maximum likelihood estimation, which is theoretically sound. When you want a general continuous "score" for a label, the expected value of the label is a very good place to start!

• Thanks, @DavidMoseler. Logistic regression definitely seems promising, but I have a follow-up question. For every single sample, I have the set of features (X) as well as the boolean label (y). The purpose of creating and training a ML classifier is not to actually predict the label, but to figure out a "goodness score" based on the set of features. Can output from a logistic regression model be used for this exact purpose? – Brian C Jan 23 '19 at 22:41
• I think one other option is create a random forest classifier, train it using same label and features, get feature_importances_ vector, and use it as weight for each feature to create a linear formula. Do you think this approach will also work? What would be the pros and cons of each method? – Brian C Jan 23 '19 at 22:42
• I clarified the original problem statement a bit since it wasn't super-clear when I re-read it. Could you take a look and see if it aligns with how you understood it? – Brian C Jan 23 '19 at 23:41
• That is exactly what logistic regression does. "Prediction" is basically a name to "spill out an output". In a strict prediction task, what you would do would be to take the continous output from logistic regression and round it to the nearest integer. So, when you got a score of "0.6", you would say it is 1, and when ou got a score of "0.2" you would predict it as 0. For your task, where you just want the score, all you have to do is to not make this final rounding. – David Moseler Jan 24 '19 at 0:09