# Can we use Standard Deviation for feature selection?

I am working on the House Price Prediction dataset on Kaggle and am trying to identify the good features for our price predictions. For numerical variables, I have gone with a high correlation with the prices (our target variable) to give me a good subset to choose my features from. But am confused about how to get a good subset of categorical features to be used in the same way.

One way I was thinking of is by using the standard deviation of the median prices for each group in a categorical feature. Let me elaborate. Suppose I have a feature 'House quality' which has three unique values in it - 'Good', 'Okay', and 'Bad'. First I group the data based on this feature and then get the prices for each group. This way I'll have the prices for every record with 'House quality' as 'Good', 'Okay', and 'Bad'. Then I get the median prices value for each group, leaving me with three values in our example. Then I calculate the standard deviation for these three values to give me the standard deviation of median of price per group (please let me know if there's a better name for this).

My logic behind using this is that the features which would have low standard deviation values won't have much difference in the median prices for every group, signaling that each group has a similar(-ish) price distribution. Such a price distribution won't teach much to our model which makes it a bad feature to train our model on.

PS - I have taken into account the features which have a high number of unique values and a low number of unique values just in case. I'll be performing the above stuff on these two types of features separately if the number of values in a series might affect its standard deviation (please lemme know if it doesn't). Each feature in these two types of features has the same number of unique values.

I am also looking for a way to normalize this value I get as without normalization I can't define a proper method of deciding "how high of this value is adequate". For this, I am thinking of dividing the standard deviation of those three values, by the mean of those three values. Is this a good idea?

• Yes, this makes sense; if you had used mean instead of median it would be an ANOVA; median makes sense in this context given the nature of price data. Commented Aug 26, 2022 at 13:34
• @JohnMadden I don't know how ANOVA works but am glad that I got a method like that. But how about the normalization bit? Any ideas on that?
– Zero
Commented Aug 26, 2022 at 14:24
• Ah, well that's the kind of thing knowing how ANOVA works might help with ;) But maybe there's another way: what about just plotting the "median-variance score" of all your features from biggest to smallest, and seeing if there's a clear cutoff? Commented Aug 26, 2022 at 16:41