1
$\begingroup$

I have 5 categorical variables. I binarize them into buckets and then assign 1 for the matched and 0 for others.

Next, I run a RandomForestRegressor using scikit-learn. But, my accuracy is only 18% on the test dataset. I don't know what to improve upon ? Should we use Random Forest where the predictor variables are categorical in nature?

categorical = [u'vendor_id', u'part_id', u'ship_to_location_id',
       u'bill_to_location_id', u'carrier_number']

    df_x.iloc[0].tolist
    Out[652]: 
    <bound method Series.tolist of 
vendor_id_435835                           1
vendor_id_437307                           0
vendor_id_422290                           0
vendor_id_421933                           0
vendor_id_425392                           0
vendor_id_421725                           0
vendor_id_421961                           0
vendor_id_437323                           0

The output variable is integer. It is the lag time for each vendor, part_id, source country,destination country and carrier chosen (air, ship, road etc). There is a relationship for sure, but how to effectively predict the delivery lag days which is my output variable. Sample values are as below:

df_y
Out[655]: 
0         4
1         1
2         1
3         9
4         1
5        58
6         3
7         7
8         5
9         5
10        7

To check the values, wrote a block of code. As pointed out, accuracy_metrics is only capturing the exact matches.

count = 0
for i,val in enumerate(pred):
    if ((val + val * 0.1) == actual[i]):
        count += 1
    elif ((val - val * 0.1) == actual[i]):
        count += 1
    elif val == actual[i]:
        count += 1
    else:
        print ("actual:",actual[i],"predicted:",val)

EDIT 2: @Matthew Drury pointed out the MSE would be a better score to track.

import sklearn.metrics as sm
print ("MSE RandomForest:",sm.mean_squared_error(actual,pred))

Using this the RandomForest gave a very small score for MSE.

$\endgroup$
  • $\begingroup$ What do you mean by "binarize into buckets"? Please describe the raw data (both the predictors and response) in terms of their distributions. It might be a good idea to plot PDFs or PMFs of each variable (including the response) and add them to your question. $\endgroup$ – Josh May 7 '17 at 17:36
0
$\begingroup$

You say the response variable is an integer measuring the number of days for something to happen, but then measure the performance of your model using the accuracy.

Consequently, you are only counting a success if your model gets the answer exactly right. Is this really what you want? Is your model's prediction really worthless if you predict 10 lag days, but the reality is 9? Using accuracy, you are not crediting your model at all for being close, only when it is exactly, completely, and totally correct.

I would instead evaluate your model based on some measure of the difference between your prediction, and the true value. The average squared error, or average absolute error are two common choices.

$\endgroup$
  • $\begingroup$ Ahh..you are so right. I just now wrote a logic to printout the values. Editing my question to add the block. I am going to check the MSE for both RF and XGBoost. Let me come back with the nos. $\endgroup$ – ForeverLearner May 8 '17 at 6:12
  • $\begingroup$ Thanks for clearing my doubt. The RF performed better than XGBoost in terms of MSE. $\endgroup$ – ForeverLearner May 9 '17 at 5:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.