Random forest that aggregates by taking the maximum over the trees instead of taking the average

Question

I want to make a Random forest that aggregates by taking the maximum over the decision trees instead of taking the average.

By default Sklearn is taking the average, and I couldn't find how to change this to taking the maximum.

Does anyone know how to do this?

Answer 1

Yes, you can do it like this

import numpy as np
import pandas as pd
from sklearn.ensemble import RandomForestRegressor as rf

# generate some data
x = pd.DataFrame({'a' : [1,2,4], 'b' : [1,3,6]})
y = np.array([1,2,3])

# fit random forest
clf = rf(n_estimators = 10)
clf.fit(x, y)

# get predictions from each tree
estimates = {}
for e in clf.estimators_:
    estimates[e] = e.predict(x)

# take treewise maximum
pd.DataFrame(estimates).T.max()

Here is the same thing written as a function. This is probably not a very efficient way of doing it though.

def predict_using_max(clf, x):
    
    # clf  : fitted random forest object
    # x    : data frame to be predicted
    
    estimates = {}
    for e in clf.estimators_:
        estimates[e] = e.predict(x)
        
    return pd.DataFrame(estimates).T.max()

Edit: to make this into a class and override the predict method, you can do this (now in a classification context):

from sklearn.ensemble import RandomForestClassifier

class MyRandomForest(RandomForestClassifier):
    
    def predict(self, x):
        estimates = {}
        for e in self.estimators_:
            estimates[e] = e.predict_proba(x)[:, 1]

        return pd.DataFrame(estimates).T.max()

For example:

x = pd.DataFrame({'a' :  np.random.normal(0, 1, 20), 'b' : np.random.normal(0, 1, 20)})
y = np.array(np.random.choice([0,1], 20, replace=True))

clf = MyRandomForest(n_estimators = 10, max_depth=3)
clf.fit(x, y)

clf.predict(x)