improve performance in a regression methodology

Question

I'm performing my own nonparametric regression in matlab and I'm wondering to switch one nuance of my methodology:

Methodology:

• split Training/test set (i.e. 80/20 %)

• search the best gamma hyperparameter in the training set: I have one loop to do a gridsearch with crossvalidation(leave-one out) along different values of gamma. Inside every loop I perform the most expensive part "the minimization problem" with CVX (sdpt3 solver), after that every forecast is stored in a matrix with their respectiv (parameter, MAE value)

• the previous step repeated in the test set.

Once I have both matrix results with the MAE and gamma I analyze results.

So, I have to wait hours to get results, i noticed that there are two facts to improve:

• the loop system to find the best parameter.
• the minimization problem with cvx package(SDPT3 solver the only that works as I want)

Any suggestions?

Answer 1

I would reduce the complexity by

• instead of leave-one-out CV use K-fold CV. This should reduce the execution time by a factor of N/K (where N is the number of samples and K the number of folds).
• Avoid doing an exhaustive (grid) search in the hyperparameter use random search, gradient search, genetic algorithms, actually any global optimization method would work although it wont find the optimal solution (just a good one).
• Parallelise your code.

I'm sure that the code from cvx toolbox is quite optimised already, and at the end of the day convex problems are solved quite rapidly. So the best strategy in your case is to try to solve fewer number of convex problems.