Hi I am new to machine learning. I just created my first working RandomForest classification ml model. It works amazingly well no error and accuracy is 100%. I have used Apache Spark MLlib to implement this algorithm. Other machine learning experts around say 100% accuracy is like dream we never get 100% accuracy is it true? I have trained randomforest classification algo with 95 decision trees and 15 depth of tree. I am using gini impurity and feature strategy as sqrt. I have cross validated my model with test data response values it matches 100%. I have two response values Actionalble/NonActionable. I told my senior I will test model with more data set of real time to see its truthfulness. Please guide. Thanks in advance.
$\begingroup$
$\endgroup$
11
-
$\begingroup$ Did you train the model with nested cross validation? Selecting hyperparameters at the inner CV and then evaluating out-of-sample performance at the outer CV step? $\endgroup$– Sycorax ♦Commented Jun 5, 2015 at 17:59
-
$\begingroup$ Hi thanks for the response how do we do nested cross validation I have 80 % training data and 20% test data I did testing and error calculation on 20% data using model I created. Sorry I am very new to machine learning so please bear with my basic questions. $\endgroup$– unk1102Commented Jun 5, 2015 at 18:16
-
$\begingroup$ It's just like regular CV, except it has 2 steps. CV the whole data set. Take all but 1 fold, and CV partition that data. Then use the inner data to CV select hyperparameters. Then train a model on all the inner data, and test the selected model on the holdout set. Repeat for all outer holdout sets. $\endgroup$– Sycorax ♦Commented Jun 5, 2015 at 18:19
-
$\begingroup$ Any pointers/links please with R code sample etc to do what you just explained. Though I am using Spark MLlib Java but I will understand R code. $\endgroup$– unk1102Commented Jun 5, 2015 at 18:24
-
1$\begingroup$ No, you haven't. You only "get" one out of sample estimate per CV layer. You've used that OOS estimate to select the model hyperparameters. You'll need another batch of out-of-sample data to estimate performance for the selected hyperparameters in any reasonable fashion. $\endgroup$– Sycorax ♦Commented Jun 6, 2015 at 19:09
|
Show 6 more comments
1 Answer
$\begingroup$
$\endgroup$
Highly probable you have a "label leakage" in some of the features (feature has 100% correlation with label). E.g. if you have a data like this:
\begin{array} {|r|r|} \hline Feature & Label \\ \hline 1.0 & Actionalble \\ 0.0 & NonActionable \\ \hline \end{array}
Then model can always predict correct label by checking the value of the feature.
