How to choose the correct dataset transformation

Question

I'm doing a project using the California Housing Price dataset from Kaggle. The objectetive of the project is to implement from scratch the Ridge Regression algorithm, apply it the to the prediction of the label medianHouseValue in the dataset and to study the dependence of the cross-validated risk estimate on the parameter alpha.

I started by exploring the dataset and i obtained several versions of it after some transformation but i don't know if i done the things in the right way. First i replaced the five values ISLAND in "ocean_proximity" with NEAR OCEAN. Then i filled the 207 NaN values in "total_bedrooms" with the median of the feature. From this point a tried several options.

Version 1

I've removed the outliers by using the IRQ method

Q1 = data.quantile(0.25)
Q3 = data.quantile(0.75)
IQR = Q3 - Q1
data = data[~((data < (Q1 - 1.5 * IQR)) | (data > (Q3 + 1.5 * IQR))).any(axis=1)]

Then i normalized the dataset using z-scoring (subrtacting the mean, divide per standard deviation) and apply one-hot enconder to "ocean_proximity" to obtain dummy features.

Version 2

Same pipeline as before, but between the removal of the outliers and normalization step i've generated new features with

data1['rooms_per_household']=data1['total_rooms']/data1['households']
data1['population_per_household']=data1['population']/data1['households']
data1['bedrooms_per_room']=data1['total_bedrooms']/data1['total_rooms']

Version 3

I studied the skeweness of the columns and decided wich is the best transformation between log, sqrt and reciprocal to apply to each column to reduce the skewness. I ended up with

features_to_log = ['rooms_per_household', 'population_per_household']
features_to_sqrt = ['total_rooms', 'total_bedrooms', 'population', 'households', 'median_income', 'median_house_value']
features_to_inv =  ['longitude', 'bedrooms_per_room', 'latitude']
for i in features_to_log:
    data_no_skw[i] = np.log(data_no_skw[i])
for i in features_to_sqrt:
    data_no_skw[i] = np.sqrt(data_no_skw[i])
for i in features_to_inv:
    data_no_skw[i] = np.reciprocal(data_no_skw[i])

I dont know it's correct to appliy different transformation to the same dataset in this way, let me know. After that i normalized and applied one-hot encoder.

Version 4

Same pipeline as 1, but i first one-hot encoded, then normalized the entire dataset (with the encoded features) and after removed the outliers. It removed a much number of rows: instead of having 17621 i get 13475 rows!

Version 5

Same pipeline as 2, but i first one-hot encoded, then normalized also the encoded features and after all removed the outliers

Version 6

Same pipeline as 3, but i first one-hot encoded, then normalized also the encoded features and after all removed the outliers

Evaluation

I then evaluated the performance of the different datasets by computing the quadratic loss with 5-fold cross validation using Ridge Regression with alpha=0.001. I got that going from version 1 to version 3 the expected loss of the predictor goes down, so add features and then trasnfrom them help the process to fit the data in a better way. Is adviced to add a step of feature selection?

I also noticed that swapping from first normalizing and then get dummies to get dummies and then normalize reduce the loss of about 0.02 in all three cases. Is correct to standardize also a categorical value enconded?

Versions 4-5-6 have a significantly lower risk estimate, but i don't know if it's exact to remove outliers at the end of the process. Notice how the risk goes up with all the transoformation.

Below a table with the results obtained

Version 1-2-3	Version 1-2-3 norm. after	Version 4-5-6
0.442540	0.420820	0.229163
0.438836	0.417703	0.213106
0.403884	0.381606	0.257296

I also tried to remove the outliers in each step by discarding rows with absolute value > 3 after normalization step but endeed up with poorer results.

What is the most correct approach among all that i tried to clean and prepare a dataset? Should i have to try to add some steps like dropping most correlated features? Note that the objective of the project is not to obtain the best possible predictor but study the dependence of the risk estimate on the parameter alpha, so it's not necessary to obtain a "state of the art" cleaned dataset but anyway i want to figure out which is the most correct way to process it.

Answer 1

I'll comment every step before comming to a conclusion.

Version 1

Generally, depends on the data and the field you work in. Normal mean-std:mean+std or IQR method is ok for detecing outliers. There is also called an appraoch namely Cook distance, which detects outliers, simply spoken, which increase R² the most when removing them. Also you should keep in mind what are you trying to predict, when talking about outliers.

When you say: I want a model to predict regular house prices in california, that means really regular. So its clear, that you have to kill the luxury ones, and the 'bad holes' because you want to build a model, that can do a certain task, predicting regular prices. There is no harm in letting the cook distance approach pre check on your data. But all in all. yeah. outlier detection as you did it is okay. You want a model, which can predict of a regular house, not to predict a price of every house!

Version 2

When you create new features from other features, as long as they are not a linear combination of other features (this would result in a matrix where the new feature has no new information) and you have the aspect of multicollinearity in mind, this is OK. However, in your case you created new variables which are all dependent of total rooms.

Thus, although you seem to have 3 different variables in there, they are all dependent from total rooms, thus a part of the effect (total rooms) -> price is in every variable. While this can lead to a good prediction it is to the dependency of total rooms. I would try to circumvent that and work with clear features. I'm not a big fan of feature engineering of relative variables e.g. clickrates on abnners or variables which are just a new clauclation of another. it would distort or overfit your model, and creates problems with multicollinearity (which most ML people do'nt bother with, but we should ..)

Version 3

I wouldn't do that, the interpretation of your regression then becomes quite messy, as every feature has its own transformation, for what ? For the sake of accuracy? Although accuracy might be a proper scoring rule for a regression (prediction task). We won't achieve it all cost, and this would distort/destroy our interpretability.

Use one transformation for all features, if it helps all variables in general a bit, that is sufficient. Having not so good values in skewness is not a real problem for regression they are a little robust to it. I wouldnt bother about that, if you wanna do it, just use one transf. e.g. the log, so that you can back original values with the inverse function exp

Version 4/5/6 Deals mostly with normalization AFTER one-hot-encoding, it is somehow similar to dummy encoding:

https://stats.stackexchange.com/questions/224051/one-hot-vs-dummy-encoding-in-scikit-learn

If you know how to deal with your nominal variables, you should go to this link, as it can be also useful to think about with on one-hot-ecnoding/dummy

On the normalization of dummies before regression here with lasso, the regu-reg.:

https://stats.stackexchange.com/questions/69568/whether-to-rescale-indicator-binary-dummy-predictors-for-lasso

In summary: normalizing you must (Yoda), when you want fair play for all your predictors, but you leave them as 0/1 if you want interpretability in your later model.

However, some might argue that dummies have the same scale like continous predictors AFTER normalization (mean 0 std 1), so this part is up to you, and sure with your method above this would kill more outliers. try to not normalize dummies as an alternative. It is not completely wrong as you saw/read, but you need argumentation.

Outliers removal imho is to be done, before standardizing, because this is imho a confirmatory way of looking at data, just as the part with multicollinearity, which you should check, before running your model: see here: you can delete features on the fly while running your model and detect multicollinearity e.g. with the VIF, but these modeles will never be so good as models where you clear critical correlated variables upfront. See also here from me: https://stats.stackexchange.com/questions/511929/how-to-understand-and-interpret-multicollinearity-in-regression-models

I would try to find a model which has less complexity ,which is problematic as ridge regression wont drop features from the data set, but it does not hurt if you get 40% accuracy but with e.g. 3 features instead of 45% with 5 features. So i wouldnt bother about a 'pipeline' that deals with so much preprocessing and outlier detection in so many ways.

To sum up:

• (read this after i wrote my monologue) your data has over 20000 rows, do not impute the 207 NaN, just kill them, if your model would be highly dependent on these 200, they would also be outliers. So you have enough data for modeling.
• Make your outlier detection on your target (the house prices). You want a model to predict regular house prices in california, no luxury villa, no crap hole. Your model will have some shortcomings, but its could for its regular task
• make your one-hot encoding
• scale your input data (but not the dummies, but its up to you, as we have seen above)
• check for possible multicollinearity, calculate VIFs maybe upfront
• do not use predictors that are in some way dependent of each other like the total rooms. there is also something called supression effects or other things like multicollinearity that could distort your model
• transform in log, and do not mix up transformations
• have a look at sklearn pipelining, the real pipelining! https://scikitlearn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html Maybe you ant to try different alpha parameters? that would be the way to do it.

If you have any questions left, do not hesitate to ask me.