Given a learning algorithm that selects and trains quantile models, how do we evaluate it?

One idea is to - use the algorithm to train a model on a synthetic dataset

  • with labels drawn from an analytical distribution (with known PDF), parameterized by the features
  • and attempt to measure how the predicted quantile values differ from the ones computed from the analytical distribution.
  • while varying the {number of examples shown, training duration, etc}

Would this work?


  • $\begingroup$ Sounds reasonable. $\endgroup$ – JustGettinStarted Jul 19 at 14:28
  • $\begingroup$ In general, you could try to show that the estimator is consistent and try to figure out its convergence rate. $\endgroup$ – Akababa Jul 27 at 21:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.