2
$\begingroup$

I have a random forest model to predict MLB player's fantasy points. I have the MSE and R^2 score, but I would like to know the accuracy of each individual prediction as opposed to the accuracy of the entire model. Is there a way to do this?

Here is the code

import pandas as pd
import numpy as np
import json
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import train_test_split
from sklearn.metrics import r2_score, mean_squared_error
from sklearn.preprocessing import StandardScaler

training_data = pd.read_csv('/Users/aus10/Desktop/MLB_Data/ML/Data/Training_Data/Batter_FPTS.csv')
df_model = training_data.copy()
scaler = StandardScaler()

features = [['Line', 'Total', 'Team_Total', 'Percent_Difference_TT' ]]
for feature in features:
    df_model[feature] = scaler.fit_transform(df_model[feature])

test_data = pd.read_csv('/Users/aus10/Desktop/MLB_Data/ML/Data/Test_Data/Test_Data_Batter_FPTS.csv')
X = training_data.iloc[:,1:5]  #independent columns
y = training_data.iloc[:,-1]   #target column 

results = []

# fit final model
model = RandomForestRegressor(n_estimators=1000, random_state=4)
model.fit(X, y)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=4)

y_train_pred = model.predict(X_train)
y_test_pred = model.predict(X_test)

model.fit(X_train, y_train)

y_pred = model.predict(X_test)

print('MSE train: %.3f, test: %.3f' % (
    round(mean_squared_error(y_train, y_train_pred),2),
    round(mean_squared_error(y_test, y_test_pred),2)
))

print('R^2 train: %.3f, test: %.3f' % (r2_score(y_train, y_train_pred), r2_score(y_test, y_test_pred)))

# define one new data instance

index = 0
count = 0

while count < len(test_data):
    team = test_data.loc[index].at['Team']
    money_line = test_data.loc[index].at['Line']
    total = test_data.loc[index].at['Total']
    team_total = test_data.loc[index].at['Team_Total']
    perc_dif = test_data.loc[index].at['Percent_Difference_TT']

    Xnew = [[ money_line, total, team_total, perc_dif ]]
    # make a prediction
    ynew = model.predict(Xnew)
    # show the inputs and predicted outputs
    results.append(
        {
            'Team': team,
            'Projected_FPTS': (round(ynew[0],2))
        })
    index += 1
    count += 1
    
sorted_results = sorted(results, key=lambda k: k['Projected_FPTS'], reverse=True)

df = pd.DataFrame(sorted_results, columns=[
    'Team', 'Projected_FPTS'])
writer = pd.ExcelWriter('/Users/aus10/Desktop/MLB_Data/ML/Results/Projected_Batter_FPTS.xlsx', engine='xlsxwriter') # pylint: disable=abstract-class-instantiated
df.to_excel(writer, sheet_name='Sheet1', index=False)
df.style.set_properties(**{'text-align': 'center'})
pd.set_option('display.max_colwidth', 100)
pd.set_option('display.width', 1000)
writer.save()

I would like to be able to return an object like

{
   'Team': team,
   'Projected_FPTS': (round(ynew[0],2))
   'Projection_Accuracy': # accuracy or probability of prediction here

}
$\endgroup$
3
  • 1
    $\begingroup$ What do you mean by "accuracy of an individual prediction"? $\endgroup$
    – Sycorax
    Sep 13, 2020 at 4:12
  • 1
    $\begingroup$ Can you add some information on the problem you're approaching? From your question and the last snippet I assume you are making predictions per team, so by "accuracy of an individual prediction" you mean something like the average performance of your model when making predictions for a given team ? $\endgroup$
    – deemel
    Sep 13, 2020 at 7:36
  • $\begingroup$ So when you use RandomForestClassifier you can use predict_proba() which will give you the probability of the classification. I want to do the same thing while also getting a predicted score. I want each new prediction to have a probability of that score? If that's possible $\endgroup$ Sep 13, 2020 at 11:54

2 Answers 2

3
$\begingroup$

RandomForestClassifier is a fundamentally different model than RandomForestRegression. The key difference is that the regression model is predicting some continuous value (e.g. a predicted profit/loss, or an estimated height, or something similar). The RandomForestClassifier predicted probability is the estimated probability that the sample belongs to each class, where a class can be something like "cat", "dog", "fish". The connection between "class" and "classifier" is important, and what distinguishes the model from RandomForestRegression.

Likewise, of a RandomForestClassifier is defined as the proportion of correctly predicted class. Your model doesn't have classes, it has some kind of continuously-valued target, so it does not make sense to talk about accuracy in the sense of a RandomForestClassifier.

$\endgroup$
1
  • $\begingroup$ Awesome thanks for the explanation that helps! $\endgroup$ Sep 14, 2020 at 18:19
2
$\begingroup$

Sycorax is right to point out that “accuracy” has a specific, technical meaning in machine learning (and it is a surprisingly bad performance metric). However, if you mean “accuracy” more colloquially, what you mean makes sense. You want to see how good each particular measurement is.

You have the true observation, and you have the predicted observation.

Subtract one from the other. Since you are using MSE and $R^2$ (watch out for using $R^2$ in a nonlinear model, however), you seem to be interested in some kind of squares error, so square the difference between the true and predicted value.

$$L(y_i, \hat{y}_i)=(y_i-\hat{y}_i)^2$$

If you sum over $i$ and divide by the number of observations, that is the MSE.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.