1
$\begingroup$

Sorry if this type of question is not kosher. I'm new around here, so please forgive me.

Anyway, I have a dataset that describes the probability that users will like certain articles from my corpus (with 5 articles chosen randomly and graphed below. Along the x axis are individual users, and along the y-axis their score. The users are sorted by score.)

I'm looking to fit a distribution to each line, but am unsure of which one to choose that would likely best these lines. I was thinking of using an exponential distribution? But there's a lot of information that exponential probably would not capture here. Does anyone have a good idea?

EDIT: I will update my title to include the name of a reasonable distribution so that this question will be more searchable and others would possibly learn from it in the future.

enter image description here

$\endgroup$
2
$\begingroup$

To many statisticians, information like this would usually be thought of as an empirical CDF:

enter image description here

with the scores on the x-axis, and the proportion of people scoring it less than or equal to a given x-value on the y-axis (that is, your x-axis ranks become scaled to proportions of people scoring no more than the person with that rank, and then becomes the new y-axis).

But it's sometimes difficult to identify good approximating distributions from the ECDF; it might be easier to look initially at some other displays (histograms with plenty of bins, kernel density estimates, Q-Q plots or other distributional plots.

You don't say what the range of possible scores is, but I presume they're bounded. If so, a scaled beta distribution might be a reasonable starting model:

enter image description here

As you see, with appropriate choices for the 4 parameters (I didn't try to fit these, just tried some values roughly in the right region), they can look quite similar to at least 4 of the 5 distributions you have there. However, they're probably not suitable in general. Mixture distributions might get you further along.

On the other hand you have a lot of data, so the empirical distribution probably has all you need to answer all kinds of questions about the data.

$\endgroup$
  • $\begingroup$ thank you so much @Glen_b. I'm so sorry I wasn't more specific, but I'm mainly looking for a parameterized model, which empirical cdf doesn't look to be? I'll start with Beta distribution. I'll keep the question open for a bit longer and then accept your answer. Thanks again! $\endgroup$ – Alex Spangher Sep 13 '14 at 1:01
  • $\begingroup$ @Glen_b I've been turning my head sideways for about 5 minutes trying to see how Alex's plot is an ECDF, and I'm not seeing it. The x axis is user ID's and the y axis looks like a logit transform (except sorted backwards). It seems like "probability of like" is a decreasing function of user ID. I think the resemblance to an ECDF is incidental. $\endgroup$ – shadowtalker Sep 13 '14 at 1:26
  • 1
    $\begingroup$ @ssdecontrol Unless I missed something, the numbers on OP's x-axis appears to be (effectively) ranks rather than IDs. If you take all the individual scores in a sample in order, then a plot of their scores against the values $1,2,..., n$ is (up to interchanging y and x, possible axis flipping and some scaling, possibly including a known shift) the same as the ECDF. The rank of a user corresponds to the number of users who score at least as extreme as they do, so a user's ranking can be scaled to the proportion $\leq$ than that user's score. $\endgroup$ – Glen_b -Reinstate Monica Sep 13 '14 at 1:37
  • $\begingroup$ Alex - what do you need from a parameterized model you can't get from the sample? Incidentally a beta won't fit your green one there. Looks like a second mode near 0.8 $\endgroup$ – Glen_b -Reinstate Monica Sep 13 '14 at 1:41
  • $\begingroup$ @Glen_b Got it. With regard to the OP wanting a parameterized distribution, interpreting CDFs isn't that easy. I think a kernel density might be more natural. Maybe a triangular one to be more "empirical." And as regards bounds on the scores, I still suspect the scores are unbounded transformations of the probabilities, although that is up to the OP to clarify. $\endgroup$ – shadowtalker Sep 13 '14 at 2:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.