# How to reduce RMS error value in regression analysis & predictions - feature engineering, model selection

There's this dataset containing the metadata of Twitch's top 1,000 streamers of 2020. You can have the details here.

I am currently participating in a challenge to predict the values for Followers gained, by creating and training the model using the remaining features from the dataset. The kernel objective is to get the lowest RMSE (Root-Mean Squared Error) metric value from the model's predictions.

Until now, I have made numerous attempts to lower down the RMSE loss value as much as possible. My current lowest achievement is around 101,000, which I got by augmenting the dataset and training a DNN model with 7 hidden-layers.
Yet, I am trying to lower the RMSE error value to 5-digits from 6. I've tried the removal of outliers, data augmentation, feature engineering, polynomial regression, trained DNN models with more than 5 hidden-layers on average; created and trained multiple models and stacked them in order to make a final prediction (and I was heard from the community that using a stacked model is one of the keys to achieve regression predictions resulting in low error metrics.)

All the results from my models have not surpassed the threshold of 6-digits of RMSE error. Feature engineering was conducted to contain only the new variables with high correlation with the target prediction value. Nearly all hyperparameters of the models created using the Tensorflow library were adjusted to show the best performance. And yet, RMSE values doesn't seem to show reduction in its value.

Here are some of the codes I wrote explaining the procedure of feature engineering and model creation & training. These did not resulted in a lower RMSE value than around 101,000. But instead, it resulted in a higher value, nearly 110,000.

[DNN Model]

inputs = Input(shape=(7))

x1 = Dense(430, activation='selu', kernel_initializer=keras.initializers.RandomUniform())(inputs)
d1 = Dropout(0.9)(x1)

x2 = Dense(430, activation='selu', kernel_initializer=keras.initializers.RandomUniform())(d1)
d2 = Dropout(0.8)(x2)

x3 = Dense(256, activation='selu', kernel_initializer=keras.initializers.RandomUniform())(d2)
d3 = Dropout(0.7)(x3)

x4 = Dense(256, activation='selu', kernel_initializer=keras.initializers.RandomUniform())(d3)
d4 = Dropout(0.6)(x4)

x5 = Dense(128, activation='selu', kernel_initializer=keras.initializers.RandomUniform())(d4)
d5 = Dropout(0.7)(x5)

x6 = Dense(128, activation='selu', kernel_initializer=keras.initializers.RandomUniform())(d5)
d6 = Dropout(0.9)(x6)

x7 = Dense(32, activation='selu', kernel_initializer=keras.initializers.RandomUniform())(d6)
d7 = Dropout(0.9)(x7)

x8 = Dense(32, activation='selu', kernel_initializer=keras.initializers.RandomUniform())(d7)
d8 = Dropout(0.9)(x8)

outputs = Dense(1)(d8)

model = keras.Model(inputs=inputs, outputs=outputs)

def rmse(y_true, y_pred):
return K.sqrt(mse(y_true, y_pred))

model.compile(
)


[XGBRegressor]

from xgboost import XGBRegressor

# Model generation and training
xgb_model = XGBRegressor(objective='reg:linear',
n_estimators=5000,
max_depth=15,
eta=0.001,
subsample=0.8,
colsample_bytree=0.8,
eval_metric='rmse')

xgb_model.fit(X_train, y_train, eval_set=[(X_val, y_val)], verbose=0)

# Make predictions
train_pred = xgb_model.predict(X_train)
test_pred = xgb_model.predict(X_test)

# Train set performance
xgb_train_evs = explained_variance_score(y_train, train_pred)
xgb_train_rmse = rmse(y_train, train_pred)

# Test set performance
xgb_test_evs = explained_variance_score(y_test, test_pred)
xgb_test_rmse = rmse(y_test, test_pred)

# Output results
xgb_results = f"""
XGBoost Train EVS: {xgb_train_evs}
XGBoost Train RMSE: {xgb_train_rmse}

XGBoost Test EVS: {xgb_test_evs}
XGBoost Test RMSE: {xgb_test_rmse}
"""

print(xgb_results)


[LGBRegressor]

from lightgbm import LGBMRegressor as lgb

# Model generation and training
lgb_model = lgb(boosting_type='gbdt', objective='regression',
num_leaves=150, learning_rate=0.001, n_estimators=10**4)
lgb_model.fit(X_train, y_train)

# Make predictions
train_pred = lgb_model.predict(X_train)
test_pred = lgb_model.predict(X_test)

# Train set performance
lgb_train_evs = explained_variance_score(y_train, train_pred)
lgb_train_rmse = rmse(y_train, train_pred)

# Test set performance
lgb_test_evs = explained_variance_score(y_test, test_pred)
lgb_test_rmse = rmse(y_test, test_pred)

# Output results
lgb_results = f"""
LightGBM Train EVS: {lgb_train_evs}
LightGBM Train RMSE: {lgb_train_rmse}

LightGBM Test EVS: {lgb_test_evs}
LightGBM Test RMSE: {lgb_test_rmse}
"""

print(lgb_results)


[RandomForestRegressor]

from sklearn.ensemble import RandomForestRegressor

# Model generation and training
forest = RandomForestRegressor(n_estimators=350, verbose=1)
forest.fit(X_train, y_train)

# Make predictions
train_pred = forest.predict(X_train)
test_pred = forest.predict(X_test)

# Train set performance
rf_train_evs = explained_variance_score(y_train, train_pred)
rf_train_rmse = rmse(y_train, train_pred)

# Test set performance
rf_test_evs = explained_variance_score(y_test, test_pred)
rf_test_rmse = rmse(y_test, test_pred)

# Output results
rf_results = f"""
Random Forests Train EVS: {rf_train_evs}
Random Forests Train RMSE: {rf_train_rmse}

Random Forests Test EVS: {rf_test_evs}
Random Forests Test RMSE: {rf_test_rmse}
"""

print(rf_results)


[Stacked Models]

from sklearn.ensemble import StackingRegressor
from sklearn.linear_model import LinearRegression

# Models to use
estimators = [
('XGBRegressor', xgb_model),
('LGBMRegressor', lgb_model),
('RFRegressor', forest)
]

# Build Stacked Model
stack_model = StackingRegressor(
estimators=estimators, final_estimator=LinearRegression()
)

# Train Stacked Model
stack_model.fit(X_train, y_train)

# Make Predictions
sm_train_pred = stack_model.predict(X_train)
sm_test_pred = stack_model.predict(X_test)

# Train Set Performance
sm_train_evs = explained_variance_score(y_train, sm_train_pred)
sm_train_rmse = rmse(y_train, sm_train_pred)

# Test Set Performance
sm_test_evs = explained_variance_score(y_test, sm_test_pred)
sm_test_rmse = rmse(y_test, sm_test_pred)

# Output results
sm_results = f"""
Stacked Model Train EVS: {sm_train_evs}
Stacked Model Train RMSE: {sm_train_rmse}

Stacked Model Test EVS: {sm_test_evs}
Stacked Model Test RMSE: {sm_test_rmse}
"""

print(sm_results)


[Model Predictions - in this case, the final prediction is calculated as the mean value between the DNN model's and the Stacked model's predictions.]

dnn_predictions = model.predict(test.values)
dnn_predictions = dnn_predictions.transpose()[0]

stacked_predictions = model.predict(test.values)
stacked_predictions = stacked_predictions.transpose()[0]



Apparently, using the dataset created after feature engineering tends to result predictions with a higher error value. I'm looking for a reason why, and the opportunities of enhancement. How can an optimal feature engineering be performed in the case of this dataset? You can access the .csv file of the original dataset from the shared link above. Also, what is the most recommended model structure in regression tasks like this? I guess the more complex a model becomes, the harder it is to make predictions due to overfitting.

I agree with Franks answer that you have effectively overfit in your approach.

An important point: You are only using training and testing. By searching for the approach that performs best on testing, you are fitting to the test data.

Any time you use a chunk of data to search through a possible space (whether that’s find best coefficients for a regression or choosing between RNN and CNN) you are by definition fitting to that data.

A more robust approach would be to do a 3 part split: Training, Validation and Testing.

• Training is used to fit a particular model.
• Validation is used to search through the space of models.
• Testing is used to check that your favourite model from validation stills performs well OOS.
• Well put, and I'll just add that if you have a unified pre-specified modeling process you can get an unbiased estimate of future model performance without splitting the data at all, using 100 repeats of 10-fold cross-validation or using the optimism bootstrap. You have to repeat all modeling steps for each resample. Sep 25 '21 at 13:01

You have fallen into the common trap of tweaking an analysis until you get a satisfactory result. This process is guaranteed to overstate the value of the predictive instrument, or to make your predictions apply only conditionally. For example, removal of outliers results in a model that is conditional on outliers never appearing in future data to which you want to apply your model.

I suggest choosing a flexible technique that is robust and that uses smart penalization (e.g., penalizes complex parts of the model more than simple parts such as top-layer additive signals) and sticking with the result. If you want to try a few methods (but not tweaking them) then model averaging may be worthwhile. If you had a huge test sample (e.g., n=50,000) you could play with the data more aggressively to optimize performance on that sample, then apply the result to yet another large test sample for unbiasedly estimating model performance.