On the existence of rule of thumb for machine learning algorithms

Question

I want to know if there are conditions about the minimum number of observations to have (the relation between the number of variables and the number of presence and absence records) in order to use the following machine learning algorithms: CART, RFs, and BRI (I will be very thankful if there are any references).

Answer 1

There are a few rules of thumb found in Frank Harrell's Regression Modelling Strategies. If $p$ is the number of variables in the model, then the rule of thumb is to use between $p=m/10$ and $p=m/20$ variables. here $m$ is the "limiting sample size" according to Frank. When the outcome is continuous then the $m$ is the total number of observations. When the outcome is binary, $m = \min(n_1, n_0)$ where $n_1, n_0$ are the counts cases where the outcome was 1 and 0 respectively.

Concretely, if you wanted to do regression with a continuous outcome and 1000 observations, you could roughly accommodate 50 and 100 variables. If you had a binary outcome and there were only 30 observations of the outcome, you could do at most 3 variables (though some work on this has shown that for logistic regression, this rule of thumb is not as informative as we like. See here).

The unfortunate truth is that this requirements are probably on the low end especially for CART. CART is a low bias but extremely high variance estimator. Random Forest implements a number of things in order to combat this variability while preserving the unbiasedness, but I've not seen investigations into how many variables can be accommodated as a function of the sample size. My intuition is that Random Forest can handle many many variables since each tree in the forest randomly subsamples variables at each node, but I have no evidence for this.