1
$\begingroup$

I have a collection of datasets with (x,y) - pairs where the value of y is conditional on x. Neither the distribution of x nor of y is known, n is ranging from 1000 to 10000. Here is the scatterplot for one of the sets.

example scatterplot of x,y

Now I want to calculate the percentiles of y conditional on x, so basically a function mapping x to the e.g. 80th percentile.

My thoughts on this subject:

  • To control the variance the percentiles should be calculated on a minimum number of values (e.g. 100, does a rule of the thumb exists for this ?)
  • This lead to the idea to discretize x with equal-width-bins, but a smoother curve is more desirable
  • Which lead to kernel-functions as used in density estimation, but the formula is not directly applicable to percentile estimation (as far as I see)
  • I had a look at Bayesian Networks, but I do not know the distribution of x or y and I am reluctant to make any assumptions here.

At this point I was pretty sure that a standard solution must exist for this kind of problem, for example when it comes to plotting certain properties of y given x (both continuous). So ... what is the common approach here ? Does this problem / method have a name ?

$\endgroup$

1 Answer 1

2
$\begingroup$

You could look at quantile regression. In its simplest form your 80th percentile will change linearly with $x$, but there are various ways to make that more flexible. The simplest of those would be to add splines or polynomials of $x$, but I am sure there are also more formal non(less)-parametric estimators in that family.

$\endgroup$
2
  • $\begingroup$ Sometime one cannot see the forest for the trees. Thank you for lightning a candle and getting this wanderer back on track :) $\endgroup$
    – steffen
    May 20, 2016 at 8:02
  • $\begingroup$ +1 At least as a starting point, quantile regression makes a lot of sense. $\endgroup$ May 20, 2016 at 8:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.