1
$\begingroup$

I hope you are very well. I have a big dataset (~9 million registries) and I have 2 variables $X$=purchase amount and $Y$=frequency of purchase. I would like to know what distribution should I use for each variable and fitting its distribution with R. For example, for $X$ I'm testing with package 'mixtools' but the results are not really good. I want to create a simulation model, that simulates that variables. First of all, I'm selecting appropriate probability distributions (and associated parameters) to describe the behavior of each uncertain input variable. Thanks.

UPDATE: I have replaced the pdf with a histogram for each variable.

$\endgroup$
6
  • 1
    $\begingroup$ Y looks geometric. Is there a relationship between the two? $\endgroup$ Commented Jul 12, 2013 at 22:18
  • $\begingroup$ Yes. The top image is the purchase amount (X), the bottom image is the frequency of purchase (Y). $\endgroup$
    – MSS
    Commented Jul 12, 2013 at 22:22
  • 5
    $\begingroup$ Could you indicate why you are fitting distributions to these data and what you hope to accomplish with the result? $\endgroup$
    – whuber
    Commented Jul 13, 2013 at 2:32
  • $\begingroup$ I'm going to bet that $Y$ is discrete (times per day) and your automatically-chosen bandwidth is too narrow to be useful. A simple histogram or table would be better in this case. $\endgroup$
    – Hong Ooi
    Commented Jul 13, 2013 at 14:34
  • 1
    $\begingroup$ I hope you are doing well too. $\endgroup$
    – wolfies
    Commented Jul 13, 2013 at 15:38

1 Answer 1

5
$\begingroup$

I'd say you don't need a density estimate at all. With millions of data points, you have a sample that's large enough that it can serve as your simulation model. Simply select data points at random (with replacement) to carry out your simulations.

$\endgroup$
2
  • 1
    $\begingroup$ Great answer! No need to bother with a guess at a named distribution just because it has a name. @MSV since your variables are related, sample from them together. Don't pick a random X and a random Y, pick a random (X, Y) pair. $\endgroup$ Commented Jul 14, 2013 at 17:50
  • 1
    $\begingroup$ +1 However, MSV, while for very large data sets this is probably the approach I'd suggest (taking into account @shujaa's clarification), one must understand that this approach nevertheless contains assumptions (such as independence of (x,y) pairs from other pairs, that the future will be like a random sample of the past). By the way, your histograms in your question should really have more bins. $\endgroup$
    – Glen_b
    Commented Jul 15, 2013 at 0:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.