I'm a user of ctree function from partykit package in R. I always wondered for which purpose we want to use Monte Carlo to compute the distribution of test statistics? The literature suggest that it goes for large samples. I checked vignettes for partykit and coin but I didn't get clear answer. Are there any different conditions we might use it? I noticed that when the MC is used, ctree returns more variables for a given tree. Why?
1 Answer
The default method for obtaining the p-values in conditional inference trees is by using the asymptotic permutation distribution (which is normally or chi-squared distributed, depending on the type of test). Alternatively, you can approximate the exact finite-sample permutation distribution by drawing a sufficiently large number of Monte Carlo samples. The two approaches will typically be similar - increasingly so for larger sample sizes. In small(er) samples there might be a bit of a difference with the approximated finite-sample distribution sometimes leading to slightly more powerful tests. Hence, in your case I suspect that the sample size is small to moderate and you get a few more significant splits in the tree.
-
$\begingroup$ Thank you, that's highly informative. It's very nice that someone like you gives the answer. In my study I have 2500 observations for 120 variables. I understand that if I use permutation tests I shouldn't be worried about the distribution of the response. Is there any difference beetween Monte Carlo and Bonferroni when using non-normal data? With non-normal data is it safe to assume higher expectations towards nonmonotonic relations between the variables and switch to maximally selected statistics instead of linear? $\endgroup$– TomMay 10, 2020 at 11:02
-
1$\begingroup$ Yes, asymptotics should kick in earlier for approximately normal data - but they also work otherwise. Maximally selected statistics often work better for nonmonotonic relationships. $\endgroup$ May 10, 2020 at 21:08