Getting the probability or accuracy from each prediction in random forest? I have a random forest model to predict MLB player's fantasy points. I have the MSE and R^2 score, but I would like to know the accuracy of each individual prediction as opposed to the accuracy of the entire model. Is there a way to do this?
Here is the code
import pandas as pd
import numpy as np
import json
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import train_test_split
from sklearn.metrics import r2_score, mean_squared_error
from sklearn.preprocessing import StandardScaler

training_data = pd.read_csv('/Users/aus10/Desktop/MLB_Data/ML/Data/Training_Data/Batter_FPTS.csv')
df_model = training_data.copy()
scaler = StandardScaler()

features = [['Line', 'Total', 'Team_Total', 'Percent_Difference_TT' ]]
for feature in features:
    df_model[feature] = scaler.fit_transform(df_model[feature])

test_data = pd.read_csv('/Users/aus10/Desktop/MLB_Data/ML/Data/Test_Data/Test_Data_Batter_FPTS.csv')
X = training_data.iloc[:,1:5]  #independent columns
y = training_data.iloc[:,-1]   #target column 

results = []

# fit final model
model = RandomForestRegressor(n_estimators=1000, random_state=4)
model.fit(X, y)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=4)

y_train_pred = model.predict(X_train)
y_test_pred = model.predict(X_test)

model.fit(X_train, y_train)

y_pred = model.predict(X_test)

print('MSE train: %.3f, test: %.3f' % (
    round(mean_squared_error(y_train, y_train_pred),2),
    round(mean_squared_error(y_test, y_test_pred),2)
))

print('R^2 train: %.3f, test: %.3f' % (r2_score(y_train, y_train_pred), r2_score(y_test, y_test_pred)))

# define one new data instance

index = 0
count = 0

while count < len(test_data):
    team = test_data.loc[index].at['Team']
    money_line = test_data.loc[index].at['Line']
    total = test_data.loc[index].at['Total']
    team_total = test_data.loc[index].at['Team_Total']
    perc_dif = test_data.loc[index].at['Percent_Difference_TT']

    Xnew = [[ money_line, total, team_total, perc_dif ]]
    # make a prediction
    ynew = model.predict(Xnew)
    # show the inputs and predicted outputs
    results.append(
        {
            'Team': team,
            'Projected_FPTS': (round(ynew[0],2))
        })
    index += 1
    count += 1
    
sorted_results = sorted(results, key=lambda k: k['Projected_FPTS'], reverse=True)

df = pd.DataFrame(sorted_results, columns=[
    'Team', 'Projected_FPTS'])
writer = pd.ExcelWriter('/Users/aus10/Desktop/MLB_Data/ML/Results/Projected_Batter_FPTS.xlsx', engine='xlsxwriter') # pylint: disable=abstract-class-instantiated
df.to_excel(writer, sheet_name='Sheet1', index=False)
df.style.set_properties(**{'text-align': 'center'})
pd.set_option('display.max_colwidth', 100)
pd.set_option('display.width', 1000)
writer.save()

I would like to be able to return an object like
{
   'Team': team,
   'Projected_FPTS': (round(ynew[0],2))
   'Projection_Accuracy': # accuracy or probability of prediction here

}

 A: RandomForestClassifier is a fundamentally different model than RandomForestRegression. The key difference is that the regression model is predicting some continuous value (e.g. a predicted profit/loss, or an estimated height, or something similar). The RandomForestClassifier predicted probability is the estimated probability that the sample belongs to each class, where a class can be something like "cat", "dog", "fish". The connection between "class" and "classifier" is important, and what distinguishes the model from RandomForestRegression.
Likewise, accuracy of a RandomForestClassifier is defined as the proportion of correctly predicted class. Your model doesn't have classes, it has some kind of continuously-valued target, so it does not make sense to talk about accuracy in the sense of a RandomForestClassifier.
A: Sycorax is right to point out that “accuracy” has a specific, technical meaning in machine learning (and it is a surprisingly bad performance metric). However, if you mean “accuracy” more colloquially, what you mean makes sense. You want to see how good each particular measurement is.
You have the true observation, and you have the predicted observation.
Subtract one from the other. Since you are using MSE and $R^2$ (watch out for using $R^2$ in a nonlinear model, however), you seem to be interested in some kind of squares error, so square the difference between the true and predicted value.
$$L(y_i, \hat{y}_i)=(y_i-\hat{y}_i)^2$$
If you sum over $i$ and divide by the number of observations, that is the MSE.
