In a decision tree, when we search for an optimal split, we usually minimize root mean square deviation (RMSE). In addition to that we might forbid splits that give too small leaves (for example a leaf cannot contain less that 100 data points).
I wonder if there is a less "rigid" and more statistically driven approach. For example can we replace RMSE by another formula that contains leaves sizes (
For example, we might "dislike" splits with small leaves but still accept them, if the corresponding RMSE is very good. So, in other words, are there smooth analytical penalties on leaves sizes.