Improve Gradient Boosting Regressor

Question

In this my code I am trying to use GradientBoostingRegressor

-I am confused because I am getting a large Root Mean Squared Error but I am not sure how to evaluate if it is too high.

-How can get on RMSE less than zero?

Any advice, please?

 The mean squared error (MSE) on test set: 5.1529 
 The root mean squared error (RMSE) on test set: 2.2700

import math
import pandas as pd
import csv
import numpy as np
import math
from sklearn.ensemble  import GradientBoostingRegressor,RandomForestRegressor
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import mean_squared_error ,mean_absolute_error
from sklearn.model_selection import train_test_split
from sklearn import metrics
from sklearn import datasets

df = datasets.load_boston()
X_train, X_test, y_train, y_test = train_test_split(df.data, df.target, random_state=42, test_size=0.1)
sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)
gr=GradientBoostingRegressor (n_estimators=1000,
           max_depth= 3,
           min_samples_split= 5,
           learning_rate=0.01,
           loss ='ls')

gr.fit(X_train ,y_train )
Y_Pred = gr.predict(X_test )
# Create the mean squared error
mse = mean_squared_error(y_test, gr.predict(X_test))
print("The mean squared error (MSE) on test set: {:.4f}".format(mse))
rmse = math.sqrt(mse)
print("The root mean squared error (RMSE) on test set: {:.4f}".format(rmse))

Answer 1

1. RMSE is non-negative, so it obviously doesn't make sense to ask for a negative one.

2. Your response is in 1000 USD, so the RMSE is also in this unit. Using the empirical rule, you can say that approximately, 2/3 of the predictions are within 2200 USD of the observed values.

3. If you prefer a relative performance measure, you can study by how much better the MSE of your model is compared to the average. This is called R-squared and can easily be obtained using the score() method.

4. Hyperparameter selection might leads to a better model.

5. There is no need to scale the input of a tree-based model.

6. The Boston data is not ideal to learn ML. It is small, boring, and racist. Here is an alternative: https://www.openml.org/d/43093 It contains 14,000 houses recently sold in Miami.