Reducing Bias from a Random Forest - Feature Importance

I'm currently looking to show which of three variables is more important in classifying something as True or False. Everyone agrees that all three variables are important, but not all agreeing on what is more important. The three variables are correlated and I've created a spearman correlation matrix.

I believe that Feature 0 has more significance on determining its classification compared to Feature 2. To test this I was told to create a random forest which I did using python. If I used my own personal opinion to classify each row before running the Random Forest (Ex: if Feature 0 is below "X", then mark that data point as "Busy") then that would be purposely adding bias to my answer.

To try to remove any bias from my answer, I decided to randomly assign the value of True or False to whether the data point is "Busy" or not using the following python code.

random_busy = np.random.rand(len(test_data)) < 0.5
random_busy = np.where(random_busy == True, 1,0)
test_data['Is Hour Busy'] = random_busy


I ran the Random Forest multiple times (n= 1,000 trees) for the dataset for whether I randomly chose 10%, 25%, 50%, 65%, and 75% of the data points and no matter what random % I chose to test, the answers were all within 0.5% of each other and I've included some of the data from my tests below.

Before my analysis using the Random Forest, I expected on average Feature 0 to have the greatest importance, Feature 1 having the next, and Feature 2 having the least. However I was surprised to see Feature 2 being the 2nd most important. My data is collected from three different locations. To investigate further, I ran the Random Forest on each location separately and the data shown in the chart is when the split between Busy/Non-Busy is 50%.

These results are interesting. Some locations had Feature 0 being much more important than others and one location had all three being about the same. But when you average those importance its almost exactly the same from when all data point locations were calculated within the same Random Forest. From what has been shown, am I correct to say that on average Feature 0 has greater importance to classification than the other features based on the data? I am concerned about the correctness of my method to remove bias by making my target variable randomly generated.

Bonus: I really enjoy creative visuals and I made this image of one of the trees in the random forest. I created it using GraphViz that I discovered from this Towards Data Science article from Will Koehrsen.

• I think it's important to point out that, while this is a well crafted question, the crucial phrase "importance of a variable" is left undefined. This isn't like inferential statistics where you are trying to estimate some parameter that you really believe exists in the universe, variable importance is not a well defined feature of our world. It can mean whatever you want it to mean. If you take it as "the quantity measured by this function on this implementation of random forest" then you have plenty of proof. More than that, you need a more rigorous definition. – Matthew Drury Jan 3 at 2:26
• @MatthewDrury Basically I had a three-step idea I wanted to write a white paper for. First step is explaining in-depth of how the three features interact with one another and how Feature 2 can be dependent on many variables that can skew classification if its used as the primary metric for classification. Second step is showing multiple real life examples where prioritizing Feature 2 produces false positive errors. Third step was to explore a large amount of data from three locations and use a some type of statistical method to show that Feature 0 has a strong influence on the data. – hoobs52 Jan 3 at 8:26
• @MatthewDrury I say "strong influence on the data" this because Feature 2 has an upper bound while Feature 0&1 have none. Feature 2 can hit its upper bound while Feature 0 & 1 are still increasing. There are real world examples where Feature 2 stays near constant at its upper bound (~2% variation) for 10+ hours a day over many days while Features 0&1 vary between 300-900% from its min-max for specific examples.Due to that, it shouldn't be considered as the prime variable for classification and I wanted to show some math that this situation can be seen in many data points over many locations – hoobs52 Jan 3 at 8:39
• Since no rigorous definition of classification exists because not everyone can agree, I tried using PCA to avoid having a target variable, but I have since learned that wouldn't help my purpose. While I have a good background in mathematics but my schooling definitely lacked in statistics outside of the most basic linear regression problems. I was told to go technical into Step 3 if possible so I've tried my best to do that but I am uncertain whether the ambiguity of the Random Forest feature selection would derail the point I am trying to make with the white paper. – hoobs52 Jan 3 at 9:15