43
$\begingroup$

I am kind of new to random forest so I am still struggling with some basic concepts.
In linear regression, we assume independent observations, constant variance…

  • What are the basic assumptions/hypothesis we make, when we use random forest?
  • What are the key differences between random forest and naive bayes in terms of model assumptions?
$\endgroup$
33
$\begingroup$

Thanks for a very good question! I will try to give my intuition behind it.

In order to understand this, remember the "ingredients" of random forest classifier (there are some modifications, but this is the general pipeline):

  1. At each step of building individual tree we find the best split of data
  2. While building a tree we use not the whole dataset, but bootstrap sample
  3. We aggregate the individual tree outputs by averaging (actually 2 and 3 means together more general bagging procedure).

Assume first point. It is not always possible to find the best split. For example in the following dataset each split will give exactly one misclassified object. Example of the dataset with no best split

And I think that exactly this point can be confusing: indeed, the behaviour of the individual split is somehow similar to the behaviour of Naive Bayes classifier: if the variables are dependent - there is no better split for Decision Trees and Naive Bayes classifier also fails (just to remind: independent variables is the main assumption that we make in Naive Bayes classifier; all other assumptions come from the probabilistic model that we choose).

But here comes the great advantage of decision trees: we take any split and continue splitting further. And for the following splits we will find a perfect separation (in red). Example of the decision boundary

And as we have no probabilistic model, but just binary split, we don't need to make any assumption at all.

That was about Decision Tree, but it also applies for Random Forest. The difference is that for Random Forest we use Bootstrap Aggregation. It has no model underneath, and the only assumption that it relies is that sampling is representative. But this is usually a common assumption. For example, if one class consist of two components and in our dataset one component is represented by 100 samples, and another component is represented by 1 sample - probably most individual decision trees will see only the first component and Random Forest will misclassify the second one. Example of weakly represented second component

Hope it will give some further understanding.

$\endgroup$
10
$\begingroup$

In one 2010 paper the authors documented that random forest models unreliably estimated the importance of variables when variables were multicolinear across multi-dimensional statistical space. I usually check for this before running random forest models.

http://www.esajournals.org/doi/abs/10.1890/08-0879.1

$\endgroup$
  • 3
    $\begingroup$ You believe the conclusions of "Quantifying Bufo boreas connectivity in Yellowstone National Park with landscape genetics" in Ecology authored by Colorado State authors over Berkeley authors in Machine Learning on the topic of machine learning algorithms? $\endgroup$ – Hack-R Nov 3 '15 at 16:07
  • 8
    $\begingroup$ I don't think they are at odds with one another. Breiman didn't investigate this 'special case' of multicolinearity across multi-dimensional space. Also, people at Colorado State can be smart too- and these guys are. $\endgroup$ – Mina Jun 1 '16 at 19:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.