I'm doing a project using the California Housing Price dataset from Kaggle. The objectetive of the project is to implement from scratch the Ridge Regression algorithm, apply it the to the prediction of the label medianHouseValue in the dataset and to study the dependence of the cross-validated risk estimate on the parameter alpha.
I started by exploring the dataset and i obtained several versions of it after some transformation but i don't know if i done the things in the right way. First i replaced the five values ISLAND in "ocean_proximity" with NEAR OCEAN. Then i filled the 207 NaN values in "total_bedrooms" with the median of the feature. From this point a tried several options.
I've removed the outliers by using the IRQ method
Q1 = data.quantile(0.25) Q3 = data.quantile(0.75) IQR = Q3 - Q1 data = data[~((data < (Q1 - 1.5 * IQR)) | (data > (Q3 + 1.5 * IQR))).any(axis=1)]
Then i normalized the dataset using z-scoring (subrtacting the mean, divide per standard deviation) and apply one-hot enconder to "ocean_proximity" to obtain dummy features.
Same pipeline as before, but between the removal of the outliers and normalization step i've generated new features with
data1['rooms_per_household']=data1['total_rooms']/data1['households'] data1['population_per_household']=data1['population']/data1['households'] data1['bedrooms_per_room']=data1['total_bedrooms']/data1['total_rooms']
I studied the skeweness of the columns and decided wich is the best transformation between log, sqrt and reciprocal to apply to each column to reduce the skewness. I ended up with
features_to_log = ['rooms_per_household', 'population_per_household'] features_to_sqrt = ['total_rooms', 'total_bedrooms', 'population', 'households', 'median_income', 'median_house_value'] features_to_inv = ['longitude', 'bedrooms_per_room', 'latitude'] for i in features_to_log: data_no_skw[i] = np.log(data_no_skw[i]) for i in features_to_sqrt: data_no_skw[i] = np.sqrt(data_no_skw[i]) for i in features_to_inv: data_no_skw[i] = np.reciprocal(data_no_skw[i])
I dont know it's correct to appliy different transformation to the same dataset in this way, let me know. After that i normalized and applied one-hot encoder.
Same pipeline as 1, but i first one-hot encoded, then normalized the entire dataset (with the encoded features) and after removed the outliers. It removed a much number of rows: instead of having 17621 i get 13475 rows!
Same pipeline as 2, but i first one-hot encoded, then normalized also the encoded features and after all removed the outliers
Same pipeline as 3, but i first one-hot encoded, then normalized also the encoded features and after all removed the outliers
I then evaluated the performance of the different datasets by computing the quadratic loss with 5-fold cross validation using Ridge Regression with alpha=0.001. I got that going from version 1 to version 3 the expected loss of the predictor goes down, so add features and then trasnfrom them help the process to fit the data in a better way. Is adviced to add a step of feature selection?
I also noticed that swapping from first normalizing and then get dummies to get dummies and then normalize reduce the loss of about 0.02 in all three cases. Is correct to standardize also a categorical value enconded?
Versions 4-5-6 have a significantly lower risk estimate, but i don't know if it's exact to remove outliers at the end of the process. Notice how the risk goes up with all the transoformation.
Below a table with the results obtained
|Version 1-2-3||Version 1-2-3 norm. after||Version 4-5-6|
I also tried to remove the outliers in each step by discarding rows with absolute value > 3 after normalization step but endeed up with poorer results.
What is the most correct approach among all that i tried to clean and prepare a dataset? Should i have to try to add some steps like dropping most correlated features? Note that the objective of the project is not to obtain the best possible predictor but study the dependence of the risk estimate on the parameter alpha, so it's not necessary to obtain a "state of the art" cleaned dataset but anyway i want to figure out which is the most correct way to process it.