What I'd like to do is optimize the class weights of a Random Forest Classifier (using python and the sklearn library) for multiclass classification, in which different misclassification errors have different costs. In other words, I want the Decision Tree to take into account a cost matrix while it is finding the best splits in the data. My question, however, can be extended to any sort of 'Grid Search' approach in ML in which a predefined parameter space is interrogated systematically and the performance of different parameter combinations generated through X-fold CV.
As I understand it, defining different class weights will modify how the Decision Tree algorithm will recognize the best split (as measured by Gini or Entropy, for e.g.) at a given point. As was reported here, one can have a Decision Tree indirectly take into account a cost matrix when splitting the data by setting the class weights.
So I attempted to use
scipy.optimize.minimize to find the weight vector which minimizes the misclassification error (as determined by a defined cost matrix).
During the minimization process, different weight vectors are generated by
minimize and tested using X-fold CV, which produces a misclassification error score to be passed back to
The problem is that a RF classifier with the same parameter settings (i.e. weight vector) will produce slightly different CV scores each time it is refitted. This slight variation is enough, I believe, to throw off the minimization of misclassification error using the weight vector. This same phenomenon occurs for regular Decision Trees as well. (Importantly, the X and y data in addition to the defined train/test samples are exactly the same as well).
This is an issue which extends beyond the particular question I am asking, to any situation in which one wishes to interrogate a parameter space via a 'grid search' approach coupled with CV: if the exact same parameter space can produce different CV scores across multiple runs, how can we say that one particular parameter combination is the best?
Any comments/thoughts about the specific question I am asking or the overall question pertaining to Grid Searching are welcome.