# Selection of random forest regression models based on r2_score

I'm making a regression model which predicts the concentration of air pollutant. It consists of the following features: Features

Things that I have done so far :

1. Assigned mean values to the missing values or 'Nan'

2. Took care of categorical features using labelEncoder and oneHotEncoder

3. Split the data into train and test set using train_test_split
4. Features selection using random forest with n_estimators=1000 and removing features having a low value of feature_importances from both train and test set
5. Used random forest for actual prediction. Used kFold with negMeanSquaredError as scoring metric and got mean value=-2522. Also, got r2_score=0.69
6. Applied GridSearch to get optimal value of hyperparameters and using these parameters and got r2_score=0.60 and mean value around -3000

Questions:

1) Which model is better? How can I compare these regression models in general?

2) How can I further improve my model's performance?

MyCode:

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

#To count the number of non NaN values in each column
dataset.count()

#Independent and dependent columns
X = dataset.iloc[:, [1,2,6,7,8,9,10,11,12]].values
y = dataset.iloc[:, 5].values

#Taking care of Missing Values
from sklearn.preprocessing import Imputer
imputer = Imputer(missing_values='NaN', strategy='mean', axis=0)
yTemp = y.reshape(-1,1)
yTemp = imputer.fit_transform(yTemp)
y = yTemp.flatten()

#Taking care of categorical variables
from sklearn.preprocessing import LabelEncoder, OneHotEncoder
labelEncoder = LabelEncoder()
X[:, 0] = labelEncoder.fit_transform(X[:,0])
X[:, 5] = labelEncoder.fit_transform(X[:,5])
oneHotEncoder = OneHotEncoder(categorical_features=[0])
X = oneHotEncoder.fit_transform(X).toarray()

#Deleting column from the year to avoid dummy variable trap
X = np.delete(X, 0, 1)

#Encoding month
oneHotEncoder = OneHotEncoder(categorical_features=[4])
X = oneHotEncoder.fit_transform(X).toarray()

#Deleting column avoid dummy variable trap
X = np.delete(X, 0, 1)

#Encoding wind direction
oneHotEncoder = OneHotEncoder(categorical_features=[18])
X = oneHotEncoder.fit_transform(X).toarray()

#Deleting column avoid dummy variable trap
X = np.delete(X, 0, 1)

#Train test split
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33,
random_state=42)

#Feature Selection
from sklearn.ensemble import RandomForestRegressor
clf = RandomForestRegressor(n_estimators=1000, random_state=0, n_jobs=-1)
clf.fit(X_train, y_train)

for feature in clf.feature_importances_:
print(feature)

#deleting less important features
X = np.delete(X, 23, 1)
X_test = np.delete(X_test, 23, 1)
X_train = np.delete(X_train, 23, 1)

X = np.delete(X, 22, 1)
X_test = np.delete(X_test, 22, 1)
X_train = np.delete(X_train, 22, 1)

#Fitting the regressor
from sklearn.ensemble import RandomForestRegressor
clf = RandomForestRegressor(n_estimators=1000, random_state=0, n_jobs=-1)
clf.fit(X_train, y_train)

ans = clf.predict(X_test)

#Cheking the performance of algorithm using kFoldCrossValidation
from sklearn.model_selection import cross_val_score
scores = cross_val_score(clf, X_train, y_train, scoring='neg_mean_squared_error', cv=10,n_jobs=-1)
scores.mean()

#Computing r2_score
from sklearn.metrics import r2_score
r2_score(y_test, ans)

#Applying gridSearch
from sklearn.model_selection import GridSearchCV
parameters = [{'n_estimators': [800, 1200], 'max_depth': [None, 5, 8],
'min_samples_split':[10,15,100], 'min_samples_leaf':[1, 2],
'max_features':["log2", "sqrt"]
}]
gridSearch = GridSearchCV(estimator = clf, param_grid=parameters, scoring= 'neg_mean_squared_error', cv=10, n_jobs=-1)
gridSearch = gridSearch.fit(X_train, y_train)

bestParams = gridSearch.best_params_
acc = gridSearch.best_score_

#Applying best_params to Regressor
from sklearn.ensemble import RandomForestRegressor
clf = RandomForestRegressor(n_estimators=1200, max_depth=None, max_features="log2",
min_samples_split=10, min_samples_leaf=1,
random_state=0, n_jobs=-1)
clf.fit(X_train, y_train)

ansAfterOpt = clf.predict(X_test)

from sklearn import model_selection
scoresAfterOpt = model_selection.cross_val_score(clf, X_train, y_train, scoring='neg_mean_squared_error', cv=10,n_jobs=-1)

scoresAfterOpt.mean()

r2_score(y_test, ansAfterOpt)

• What's your evaluation metric? What are you trying to optimise for? Mean squared error? If so, then check what MSE is for each model and the one with the lowest MSE will be best. Besides, the number of estimators you use given the number of features is crazy high, so you probably overfit a lot. – CephasW Jul 4 '18 at 14:38
• Thanks for helping me out. I tried using the same regressor with esitmators=50, 100 and1000. Both the training error and test error didn't reduce significantly after n=100. So, I guess n_estimators=100 might be the optimal value. This is how we check for overfitting, right? If the training loss decreases while test loss increases, then we say the model is overfitted. Also, if the performance is comparable then choose the less complicated one. Sorry if I'm asking silly questions. I'm a newbie. Thanks again! – Abhishek Das Jul 5 '18 at 5:01