I have implemented different cross validation algorithms, like CV and GCV, and they are working perfectly. I have to run it on an arm processor, and the time of calculation is very high. Since the cross validation checks for each lamdba , and chooses the minimum RMS of it.I was hoping if there is any algorithm,which chooses optimal lambda without checking all the lambda values.
$\begingroup$
$\endgroup$
2
-
3$\begingroup$ If there were a sound way to choose the optimal lambda without CV, we'd do it. But CV is the best that we've got at the moment. But perhaps you could explore the lambda space more efficiently with a global optimization algorithm. $\endgroup$– Sycorax ♦Commented Nov 10, 2015 at 16:07
-
$\begingroup$ Have you tried Randomized Search CV? (Randomizedsearchcv in Python). It's like GCV, but it gets random combinations of the parameters and it's faster. $\endgroup$– NumbermindCommented Apr 19, 2021 at 9:31
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
3
It depends. You can always speed up cross validation using a coarser grid or reducing the number of fields. But they will harm performance.
What are you cross-validating ? Sometimes, dropping generality of your code can bring you huge speed ups. Call $n$ the number of folds you are using and $l$ the size of the parameter grid.
If you are working with KNN you can store the list of 100 closest neighbours and cross validate $k$ (for $k\in[1,100]$), the number of neighbours to select from this first list. Now you just have to train $n$ different "pre-KNN" and all the other models are evaluated instantly. The naive method would have used $nl$ trainings.
An impressive work has been done regarding the selection of $C$ with SVMs : SVM path, but I don't see many people using (or implementing) it. Likewise, LASSO, ridge and elastic net regressions have algorithms calibrating the entire path without changing the complexity.
If you are working with multiple trees based methods, like random forests or gradient boosting machines and are selecting the number of trees, you can save the trees and add a bunch of them to each model at every step (sklearn enables to extract the trees from a random forest as a list you can then slice)
Edit
After reading various questions about speeding up cross validation I ended up writing this blog article detailing existing (specialized or general) methods to achieve speed ups.
answered Nov 10, 2015 at 17:56
-
$\begingroup$ Anecdotally, every time I try to use the SVM path package in R, it crashes my session. Could be why people don't use it. ;-) $\endgroup$– Sycorax ♦Commented Nov 10, 2015 at 17:59
-
$\begingroup$ Yes I also have some incredible memory requirements with it - even with really small data sets. I wish there were more implementations of this idea though... $\endgroup$ Commented Nov 10, 2015 at 18:00
-
$\begingroup$ old question but has anyone seen anything to do with adversarial "time ordering" of the data? So instead of random sampled CV, literally do either one-step or microbatch predict/train as in online learning and try to find the worst possible ordering? $\endgroup$ Commented Apr 18, 2021 at 15:17