A question from a newbie: I played around with parameter tuning (grid, random search) in R (caret, xgboost) and my observation is as follows: in terms of accuracy gains I was able to get 3 - 7% but not more. And that's the limit. Meaning that higher accuracy gains could be reached with new features as well as other models/ensembles. Am I wrong?
If someone could point me to a tutorial for parameter tuning exploration, would be nice.
The addition of features or the introduction of tranformations can significantly increase your model's performance.
Now, given that your dataset is fixed and you want to exploit all the power of your model I would suggest Bayesian Optimization. Bayesian Optimization is a state-of-the-art sequential design strategy for global optimization of black-box functions. Assuming that close input values will have close outputs, we model the observed accuracy values using a flexible Gaussian Process prior and try find the input that maximizes the accuracy. Hence, it performs significantly better than line-search or random search.
For a more detailed introduction, you can read: http://papers.nips.cc/paper/4522-practical-bayesian-optimization-of-machine-learning-algorithms.
Now, you could use Spearmint executing your R code through a very simple python wrapper which takes as input the parameters and just returns the accuracy https://github.com/HIPS/Spearmint. Another great package that I like is PyBO https://github.com/mwhoffman/pybo/.
