The first one is quantile regression and variants thereof. The idea is to use a target function other than the MSE, one that is minimized by the quantile you are looking for. Then a straightforward linear regression is run, but using this quantile target function. We have a quantile-regression tag. One consequence is that you will get a different model for each target probability (and you may run into inconsistencies - the model for a 90% target may output a larger quantile prediction than the model for a 95% target). In principle, you may be able to feed these target functions into other minimizers and get a quantile version of non-regression methods.
Roger Koenker is pretty much the quantile regression guru. His 2005 textbook is called Quantile Regression, he coauthored an eponymous introductory article with Hallock (Koenker & Hallock, 2001, Journal of Economic Perspectives), and he wrote the
quantregpackage for R.The alternative is to use a tool that can output a full predictive density, then take the appropriate quantile of this predicted density. For instance, straightforward linear regression via ordinary least squares will output an expectation point prediction, but also give you a predictive density based on the $t$ distribution and an estimate of parameter and residual uncertainty. You can therefore use the [prediction-interval] from such a model for your quantile. (Be careful not to use the confidence-interval - this is a different thing, and the two are very often confused.)
The advantage is that your quantiles will certainly not be inconsistent, since you will derive the 90% and the 95% quantile prediction from the same predictive density. The disadvantage is that you need full predictive densities. SVM, in particular, does not output these, so even if you found it to perform well in other applications, it may not be the best tool for quantile predictions.
The first one is quantile regression and variants thereof. The idea is to use a target function other than the MSE, one that is minimized by the quantile you are looking for. Then a straightforward linear regression is run, but using this quantile target function. We have a quantile-regression tag. One consequence is that you will get a different model for each target probability (and you may run into inconsistencies - the model for a 90% target may output a larger quantile prediction than the model for a 95% target). In principle, you may be able to feed these target functions into other minimizers and get a quantile version of non-regression methods.
Roger Koenker is pretty much the quantile regression guru. His 2005 textbook is called Quantile Regression, he coauthored an eponymous introductory article with Hallock (Koenker & Hallock, 2001, Journal of Economic Perspectives), and he wrote the
quantregpackage for R.The alternative is to use a tool that can output a full predictive density, then take the appropriate quantile of this predicted density. For instance, straightforward linear regression via ordinary least squares will output an expectation point prediction, but also give you a predictive density based on the $t$ distribution and an estimate of parameter and residual uncertainty. You can therefore use the [prediction-interval] from such a model for your quantile. (Be careful not to use the confidence-interval - this is a different thing, and the two are very often confused.)
The advantage is that your quantiles will certainly not be inconsistent, since you will derive the 90% and the 95% quantile prediction from the same predictive density. The disadvantage is that you need full predictive densities. SVM, in particular, does not output these, so even if you found it to perform well in other applications, it may not be the best tool for quantile predictions.
The first one is quantile regression and variants thereof. The idea is to use a target function other than the MSE, one that is minimized by the quantile you are looking for. Then a straightforward linear regression is run, but using this quantile target function. We have a quantile-regression tag. One consequence is that you will get a different model for each target probability (and you may run into inconsistencies - the model for a 90% target may output a larger quantile prediction than the model for a 95% target). In principle, you may be able to feed these target functions into other minimizers and get a quantile version of non-regression methods.
Roger Koenker is pretty much the quantile regression guru. His 2005 textbook is called Quantile Regression, he coauthored an introductory article with Hallock (Koenker & Hallock, 2001, Journal of Economic Perspectives), and he wrote the
quantregpackage for R.The alternative is to use a tool that can output a full predictive density, then take the appropriate quantile of this predicted density. For instance, straightforward linear regression via ordinary least squares will output an expectation point prediction, but also give you a predictive density based on the $t$ distribution and an estimate of parameter and residual uncertainty. You can therefore use the [prediction-interval] from such a model for your quantile. (Be careful not to use the confidence-interval - this is a different thing, and the two are very often confused.)
The advantage is that your quantiles will certainly not be inconsistent, since you will derive the 90% and the 95% quantile prediction from the same predictive density. The disadvantage is that you need full predictive densities. SVM, in particular, does not output these, so even if you found it to perform well in other applications, it may not be the best tool for quantile predictions.
- The first one is quantile regression and variants thereof. The idea is to use a target function other than the MSE, one that is minimized by the quantile you are looking for. Then a straightforward linear regression is run, but using this quantile target function. We have a quantile-regression tag. One consequence is that you will get a different model for each target probability (and you may run into inconsistencies - the model for a 90% target may output a larger quantile prediction than the model for a 95% target). In principle, you may be able to feed these target functions into other minimizers and get a quantile version of non-regression methods.
The first one is quantile regression and variants thereof. The idea is to use a target function other than the MSE, one that is minimized by the quantile you are looking for. Then a straightforward linear regression is run, but using this quantile target function. We have a quantile-regression tag. One consequence is that you will get a different model for each target probability (and you may run into inconsistencies - the model for a 90% target may output a larger quantile prediction than the model for a 95% target). In principle, you may be able to feed these target functions into other minimizers and get a quantile version of non-regression methods.
Roger Koenker is pretty much the quantile regression guru. His 2005 textbook is called Quantile Regression, he coauthored an eponymous introductory article with Hallock (Koenker & Hallock, 2001, Journal of Economic Perspectives), and he wrote the
quantregpackage for R. The alternative is to use a tool that can output a full predictive density, then take the appropriate quantile of this predicted density. For instance, straightforward linear regression via ordinary least squares will output an expectation point prediction, but also give you a predictive density based on the $t$ distribution and an estimate of parameter and residual uncertainty. You can therefore use the [prediction-interval] from such a model for your quantile. (Be careful not to use the confidence-interval - this is a different thing, and the two are very often confused.)
The advantage is that your quantiles will certainly not be inconsistent, since you will derive the 90% and the 95% quantile prediction from the same predictive density. The disadvantage is that you need full predictive densities. SVM, in particular, does not output these, so even if you found it to perform well in other applications, it may not be the best tool for quantile predictions.
Roger Koenker is pretty much the quantile regression guru. His 2005 textbook is called Quantile Regression, he coauthored an eponymous introductory article with Hallock (Koenker & Hallock, 2001, Journal of Economic Perspectives), and he wrote the quantreg package for R.
- The alternative is to use a tool that can output a full predictive density, then take the appropriate quantile of this predicted density. For instance, straightforward linear regression via ordinary least squares will output an expectation point prediction, but also give you a predictive density based on the $t$ distribution and an estimate of parameter and residual uncertainty. You can therefore use the [prediction-interval] from such a model for your quantile. (Be careful not to use the confidence-interval - this is a different thing, and the two are very often confused.)
The advantage is that your quantiles will certainly not be inconsistent, since you will derive the 90% and the 95% quantile prediction from the same predictive density. The disadvantage is that you need full predictive densities. SVM, in particular, does not output these, so even if you found it to perform well in other applications, it may not be the best tool for quantile predictions.
- The first one is quantile regression and variants thereof. The idea is to use a target function other than the MSE, one that is minimized by the quantile you are looking for. Then a straightforward linear regression is run, but using this quantile target function. We have a quantile-regression tag. One consequence is that you will get a different model for each target probability (and you may run into inconsistencies - the model for a 90% target may output a larger quantile prediction than the model for a 95% target). In principle, you may be able to feed these target functions into other minimizers and get a quantile version of non-regression methods.
Roger Koenker is pretty much the quantile regression guru. His 2005 textbook is called Quantile Regression, he coauthored an eponymous introductory article with Hallock (Koenker & Hallock, 2001, Journal of Economic Perspectives), and he wrote the quantreg package for R.
- The alternative is to use a tool that can output a full predictive density, then take the appropriate quantile of this predicted density. For instance, straightforward linear regression via ordinary least squares will output an expectation point prediction, but also give you a predictive density based on the $t$ distribution and an estimate of parameter and residual uncertainty. You can therefore use the [prediction-interval] from such a model for your quantile. (Be careful not to use the confidence-interval - this is a different thing, and the two are very often confused.)
The advantage is that your quantiles will certainly not be inconsistent, since you will derive the 90% and the 95% quantile prediction from the same predictive density. The disadvantage is that you need full predictive densities. SVM, in particular, does not output these, so even if you found it to perform well in other applications, it may not be the best tool for quantile predictions.
The first one is quantile regression and variants thereof. The idea is to use a target function other than the MSE, one that is minimized by the quantile you are looking for. Then a straightforward linear regression is run, but using this quantile target function. We have a quantile-regression tag. One consequence is that you will get a different model for each target probability (and you may run into inconsistencies - the model for a 90% target may output a larger quantile prediction than the model for a 95% target). In principle, you may be able to feed these target functions into other minimizers and get a quantile version of non-regression methods.
Roger Koenker is pretty much the quantile regression guru. His 2005 textbook is called Quantile Regression, he coauthored an eponymous introductory article with Hallock (Koenker & Hallock, 2001, Journal of Economic Perspectives), and he wrote the
quantregpackage for R.The alternative is to use a tool that can output a full predictive density, then take the appropriate quantile of this predicted density. For instance, straightforward linear regression via ordinary least squares will output an expectation point prediction, but also give you a predictive density based on the $t$ distribution and an estimate of parameter and residual uncertainty. You can therefore use the [prediction-interval] from such a model for your quantile. (Be careful not to use the confidence-interval - this is a different thing, and the two are very often confused.)
The advantage is that your quantiles will certainly not be inconsistent, since you will derive the 90% and the 95% quantile prediction from the same predictive density. The disadvantage is that you need full predictive densities. SVM, in particular, does not output these, so even if you found it to perform well in other applications, it may not be the best tool for quantile predictions.
What you are doing is quantile prediction (or "forecasting", in a time series context, but the differences to the non-time-series case are irrelevant here): you are looking for a single number for each instance (a "point prediction") that will be larger than the actual outcome with a prespecified probability, e.g., 90%, 95% or 99%.
Thus, you are trying to predict a quantile of the (unknown) probability distribution of a future outcome. Since your target probabilities are typically larger than 50%, you are mainly interested in upper quantiles, or the right tail of the distribution.
Note that this unknown distribution may depend on covariates, which makes life a bit harder.
Now, the problem is that most common statistical or machine learning algorithms try to fit and predict a central tendency of this unknown distribution - often the expected value, but frequently the median. (The two may be different if the distribution is asymmetric.) They do this by minimizing an objective function that has its minimum at the expectation, e.g., the mean squared error (MSE). An additional problem is that this is rarely or never explicitly stated, so people are left wondering what to do in case they are looking for a quantile, as you are.
As far as I understand, your approach is two-pronged:
- You penalize under-predictions more strongly. Unfortunately, as long as the predictions aim at the expected value, there is no connection between a "good" penalization scheme and getting at the quantile you actually want.
- You take the point prediction your SVM (or whatever tool) outputs and multiply it by a factor. Again, this won't work very well, because in general, there is no simple multiplicative relationship between the expectation (which your tool outputs) and high quantiles.
There are two standard ways of getting at quantiles.
- The first one is quantile regression and variants thereof. The idea is to use a target function other than the MSE, one that is minimized by the quantile you are looking for. Then a straightforward linear regression is run, but using this quantile target function. We have a quantile-regression tag. One consequence is that you will get a different model for each target probability (and you may run into inconsistencies - the model for a 90% target may output a larger quantile prediction than the model for a 95% target). In principle, you may be able to feed these target functions into other minimizers and get a quantile version of non-regression methods.
Roger Koenker is pretty much the quantile regression guru. His 2005 textbook is called Quantile Regression, he coauthored an eponymous introductory article with Hallock (Koenker & Hallock, 2001, Journal of Economic Perspectives), and he wrote the quantreg package for R.
- The alternative is to use a tool that can output a full predictive density, then take the appropriate quantile of this predicted density. For instance, straightforward linear regression via ordinary least squares will output an expectation point prediction, but also give you a predictive density based on the $t$ distribution and an estimate of parameter and residual uncertainty. You can therefore use the [prediction-interval] from such a model for your quantile. (Be careful not to use the confidence-interval - this is a different thing, and the two are very often confused.)
The advantage is that your quantiles will certainly not be inconsistent, since you will derive the 90% and the 95% quantile prediction from the same predictive density. The disadvantage is that you need full predictive densities. SVM, in particular, does not output these, so even if you found it to perform well in other applications, it may not be the best tool for quantile predictions.
Finally, you may be interested in Gneiting (2011, IJF), who discusses objective functions that are optimized by quantiles (see above), as well as ways of actually evaluating quantile predictions.