Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 5 down vote favorite
3

I am a novice in statistics so please correct me if I am doing something fundamentally wrong. After wrestling for a long time with R in trying to fit my data to a good distribution, I figured out that it fits the Cauchy distribution with the following parameters: 

   location      scale  
  37.029894   18.678936 
 ( 3.405665) ( 2.779136)

The data was from a survey where 100 people were asked how many friends they talked to over a period of 20 days and I am trying to see if it fits a known distribution. I generated the QQ-plot with the reference line and it looks like the image given below. From what I have been reading on the web, if the points fall close to the reference line then it is a good evidence that the data comes from this distribution.

So, is this a good evidence to say that the distribution is Cauchy or do I need to run any more tests? If so, can someone tell me the physical interpretation of this result? I mean, I read that if the data falls into a Cauchy distribution, then it will not have a mean and standard deviation but can someone help me understand this in plain English? If it does not have a mean then from what I understand, I cannot sample from this distribution. What is one supposed to infer about the population based on this result? Or should I be looking at other models?

UPDATE: What am I trying to achieve? I am trying to evaluate how much time it takes for some arbitrary piece of information to propagate for a population of size X. As this depends on the communication patterns of people, what I was trying to do was to build a model that could use the information from the 100 people I surveyed to give me patterns for the X number where X could be 500 or 1000.

QQ-Plot

alt text

Density Distribution of my data

alt text

Cauchy Distribution

alt text

QQ-Plot when trying to fit a Normal distribution to my data alt text

UPDATE:

From all the suggestions, I think I now understand why this cannot be a Cauchy distribution. Thanks to everyone. @HairyBeast suggested that I look at a negative binomial distribution so I plotted the following as well:

QQ-Plot when Negative Binomial Distribution was used

alt text

Negative Binomial Distribution

alt text

share|improve this question
2  
This question seems directly relevant. See my post for data vis tips to compare your data to other known distributions in base R. – Chase Nov 28 '10 at 13:18
@Chase: +1 Actually yes :) I think I missed that one. I'll do that rightaway. Thanks a lot. – Legend Nov 28 '10 at 20:31
2  
@Legend You can also try a rootogram (don't know if it overlaps with @Chase's response on SO). Now, I don't understand why you want to try and fit every discrete distribution to your data. Either you have a priori knowledge or hypothesis about the law of your outcomes, or you don't. In the former case, you might want to explain why the observed data don't fit the model. In the latter case, you're left with exploratory data analysis (and, potentially, non-parametric density estimates, mixture models, etc.) – chl♦ Nov 28 '10 at 21:12
1  
@Legend 'Scenario' means that you already have some hypothesis, no? It's difficult to answer your question because you seek to fit the 'best' model (in the sense of goodness-of-fit) to your data, but it is not necessary the 'correct' model. After all, your data may be subjected to measurement error or any other sources of error. Finally, you can still work with your observed sample and use bootstrap to simulate new samples. – chl♦ Nov 28 '10 at 21:31
1  
@Legend Bootstrap is useful to estimate, based on an observed sample, the variability of an estimator when you don't know (or don't want to assume) its law. But in your case, given the context you added, I would suggest to update your question so that people can have a better idea of what you really intend to do with (which is beyond simple distribution fitting, apparently). – chl♦ Nov 29 '10 at 6:01
show 5 more comments

2 Answers

up vote 11 down vote accepted

First off, your response variable is discrete. The Cauchy distribution is continuous. Second, your response variable is non-negative. The Cauchy distribution with the parameters you specified puts about 1/5 of its mass on negative values. Whatever you have been reading about the QQ norm plot is false. Points falling close to the line is evidence of normality, not evidence in favor of being Cauchy distributed (EDIT: Disregard these last 2 sentences; a QQ Cauchy plot - not a QQ norm plot - was used, which is fine.) The Poisson distribution, used for modeling count data, is inappropriate since the variance is much larger than the mean. The Binomial distribution is also inappropriate since theoretically, your response variable has no upper bound. I'd look into the negative binomial distribution.

As a final note, your data does not necessarily have to come from a well known, "named" distribution. It may have come from a mixture of distributions, or may have a "true" distribution whose mass function is not a nice transformation of x to P(X=x). Don't try too hard to "force" a distribution to the data.

share|improve this answer
(+1) Nice points, especially the latest. – chl♦ Nov 28 '10 at 11:57
+1 for the suggestions. I updated my post with a negative binomial distribution as well. It looks like it will serve its purpose except that the third bar is not as expected. As for your final point, I heard that if the data does not come from any known distributions, I can use something like a kernel density estimation. Would you suggest this? If so, can you kindly give me a very short example on how to do this for discrete data using R? Would I still be looking at QQ-plots to verify my model? – Legend Nov 28 '10 at 20:33
I updated my question to explain more about what I am trying to achieve. – Legend Nov 29 '10 at 6:59
up vote 6 down vote

Agree with HairyBeast (+1) that Cauchy is not appropriate here (it's symmetric for one thing) and that negative binomial might well be better.

Disagree about QQ-plot though. You can do a QQ-plot for any distribution, not just normal. What you say about interpretation of a QQ-plot is correct, but note that 2 of your points lie very far indeed from the straight line.

On the Cauchy's lack of moments: this doesn't affect sampling. Once you know the parameters of the distribution sampling from it is easy (as the quantile function has a closed form) and the lack of moments is irrelevant. But the fact that the Cauchy distribution doesn't even have a mean does indicate that it's inappropriate here, as clearly it is meaningful to ask what's the expected number of friends with whom a person has a conversation in a 20-day period.

share|improve this answer
You are right about the QQ plot being applicable to any distribution. I read the question to fast and (for whatever reason) assumed it was a QQ norm plot. One minor note: be careful about concluding a distribution from QQ plots. For example, data that has a t distribution with 20 df will still give you nice QQ norm plots. – HairyBeast Nov 28 '10 at 15:25
+1 for the explanation as to why Cauchy does not make sense in this case. That would have been my next question if it were right :) If you get some time, could you kindly take a look at my comment above? In short, because my data need not come from a specific distribution, my readings yesterday revealed that a kernel density estimation technique can be used but am not really sure if this is the right approach and if it is, how one goes about doing it. – Legend Nov 28 '10 at 20:35
I updated my question to explain more about what I am trying to achieve. – Legend Nov 29 '10 at 6:59

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.