I am doing a quantile regression in R with some data and then i want to test the accuracy of the coefficients on a another data set (hold out set). But i am not sure how to go about measuring the accuracy on the hold out set in R.
Is there any ideas?
You could possibly convert this into a logistic regression and use the deviance from that. For example if you fit the quantile regression to find the 75th percentile then you are predicting that 75 percent of values will be below the predicted value. For each data point in the holdout set find its predicted 75th percentile value and code it as a "success" if the actual value is less than the predicted percentile and a failure if it is greater. Then compute the deviance for a logistic model with all predictions being 0.75 (you can do this with the R glm function fitting only an offset).
If you have predicted multiple quantiles than you would need to work out the deviance for the multinomial, e.g. if you calculate the 25th and 75th percentiles then each point will fall into 1 of 3 categories: below 25th, between 25th and 75th, and above 75th. The probabilities (under a proper fit) of the 3 groups would be 0.25, 0.5, and 0.25. I believe the deviance in this case is just the product of the given probabilities for each data point (if a data point is between then you use 0.5, below (or above) use 0.25) and multiply those all together (or sum their logs for the log deviance). Smaller deviances would be a better fit, larger deviances are a worse fit.
