# Optimal feature selection for MAPE criteria with RandomForest cross-validation

I am trying to optimize my set of features against random forest cross-validation using MAPE criteria.

I tried forward selection with Univariate linear regression test (f_regression in sklearn), I calculate MAPE for each set of variables selected by SelectKBest:

for i in range(1,len(X.columns)):
selektor = SelectKBest(f_regression, k = i)
clf = RandomForestRegressor(n_estimators=10, max_depth=None)
pred = clf.fit(X, y).predict(T)
MAPE = mean(abs(pred-y_real)/y_real)


I also tries backward selection with the RandomForest feature_importance attribute. I start with full set and in each iteration I remove the least important feature, and I calculate MAPE. I remove then, the feature with the least important MAPE:

while X:
clf = RandomForestRegressor(n_estimators=n, max_depth=None)
pred = clf.fit(X, y).predict(T)
imp = dict(zip(list(X.columns.values), clf.feature_importances_))
todrop = min(imp.iteritems(), key=itemgetter(1))[0]


First technique returns decent results but it is not optimized because I select K best features with liner regression score criteria. Second technique returns noisy results (MAPE doesn't converge).

I am looking for a technique for feature selection to use in sklearn that will measure MAPE directly. The size of my full set is 150 features and 80,000 observations.

Thank you in advance for any suggestion...

• You need to take absolute value of your denominator as well in your MAPE computation. May 18, 2014 at 8:44
• @fgnu y_real is always positive May 18, 2014 at 8:45
• That is not evident at all from your question. Code hygiene might be a good reason to do it anyway. May 18, 2014 at 8:47
• Also, if that is indeed the case, you might be better off fitting a model to the logarithms of your response. May 18, 2014 at 8:49
• @fgnu duly noted May 18, 2014 at 8:49