Real world model training in R: how to get instant feedback?

Question

I want to train a model. I can just randomly choose method (e.g. random forest), put whole dataset, wait a few hours, check accuracy, plot every possible curve (like accuracy vs train size) and see from the chart that i could have chosen different number of trees or different mtry or just to stop earlier because error rate doesn't seem to decrease.

If I could see all those curves being plotted during the training, I could make the process shorter or at least estimate the duration.

So my question is: how it's being done in the real world? Does R provide some way of on-the-fly plotting, or maybe we have to program it by ourselves? Like train only using some subset, plot, train again using a bit bigger subset and plot again? Or maybe there is a way to train the model incrementally? Then we don't have to train each time new model from scratch.

Answer 1

To the best of my knowledge there is no package or option in R that does what you ask for, so you would have to code something from null. However, I would not advice that, since manually written functions are really slow in R. Most packages are implemented in C or Fortran, with R functions only as wrappers. Another problem I have run into with R, is that many interesting packages are written by people who are not programmers first. Many pakcages, for this reason and others, scale horrible and the by the time the computation is done the question is no longer relevant.

For more speed, there are a couple of tricks.

1) Pre-allocate RAM, remove redudant data and avoid copying things.

2) Compile functions (if you write your own)

3) Use a parallel package (such as foreach, usefull for RandomForest)

4) Only do calculations that you really need, for instance do you need usual std. errors, when the data is very likely heteroskedastic. Do you need residuals, or predictions?

Tip 4, might seem a bit silly... But if you do something for 100million times, it is going to take a while. You might also consider moving from R, to some other software which is multithreaded by default.

All this said, I would follow @gung, use a (random) subset of your data and see where that takes you. The (weak/strong) law of large numbers is quite powerfull, and a good sample really allows you infer things about the entire dataset.