Do certain machine learning algorithms have higher variance in their predictions than others or have parameters to adjust this? Right now I'm using a regularized linear model for time series prediction on very noisy data and it's resulting in very conservative predictions out of sample i.e. predicting close to the mean value for most observations. I'd like to have a model that has high variance in it's predictions and either hits or misses rather than hits in the middle every time. Are there certain models I should try or tips I could try?
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
3
Yes,Certain machine learning algorithms have higher variance. A good rule of thumb is the more parameters a model has the higher the variance of that model.
Trees have high variance, you reduce the variance by ensembles i.e random forests.
Neural networks also have higher variance, if you don't use regularization or drop out layers.
You can also map your data into a higher dimensional space using polynomials as well.
answered Dec 8, 2016 at 4:01
-
$\begingroup$ I thought the "variance" of these models refers to the variance of their predictions when they are applied to different training sets. This is related to the variance of their out of sample predictions? $\endgroup$– Brian LCommented Dec 8, 2016 at 17:04
-
1$\begingroup$ @BrianL Having high variance (due to different training sets) usually also implies having high variance for "out of sample predictions". (by sample I assume you mean training data?). The two sort of go hand in hand, because a more complex model will make two samples which might be very close in space have very different predictions as opposed to a model that has a simple model and two close samples will almost always be given the same prediction. $\endgroup$ Commented Dec 8, 2016 at 20:28
-
$\begingroup$ Got it. It just seems like there's a difference between the variance of a model on different training sets vs variance of predictions. For example a model could have high variance in its predictions (predicts high and low values often), but its predictions are still consistent when applied to different training sets i.e. it has low variance when comparing training sets. My understanding of bias-variance tradeoff was that it referred to the latter situation of the variance of a model on different training sets. But I guess these two go hand in hand. $\endgroup$– Brian LCommented Dec 8, 2016 at 22:21