8
$\begingroup$

I have 2 exponentially distributed datasets and I want to be sure that they are from different distributions. Unfortunately a necessary error in the detection of the data forces me to discard all data under a certain threshold. In each set I have about 3000 data points and plotting the data makes me think that the lambda value is different. Fitting also yields different values for lambda.

How can I be sure that both datasets originate from a different distribution?

Here a plot of how the sets look like (Note that all values under lifetime=3sec have to be discarded):

UPDATE: The above distributions are in both cases normalised over N just for comparing them better in a graph because the total number of data points N is different.

UPDATE2: After truncation I have about 150 lifetime values for the red dataset and 350 for the blue dataset. Turns out that 3000 was exaggerated (I am sorry).

UPDATE3: Thank you for bearing with me. Here is the raw data:

http://pastebin.com/raw.php?i=UaGZS0im

http://pastebin.com/raw.php?i=enjyW1uC

So far I fitted an exponential function to both datasets and compared the slopes. Since any normalisation should not change the slope of the data different slopes should imply different underlying exponential distributions (My experience with statistical analysis is very limited).

The values under the threshold are discarded because the measurement detects many events too often in that regime.

UPDATE4: I just realised that my problem is much more complicated than I thought. I have actually left censored (I do not know the beginning of some events) and right censored (don't know the end of some events) data AND I have to discard all lifetimes under 3s (truncation). Is there any way to incorporate all of that into one analysis? So far I found some help on how to work with censored data (survival analysis) but what should I do with the truncation?

$\endgroup$
  • $\begingroup$ Since the plot seems to show densities, how are they estimated? $\endgroup$ – Yves Jan 23 '15 at 15:53
  • $\begingroup$ These sound like censored data. You lose information and potentially bias the results by discarding those below the threshold. Instead, you need to report what that threshold is (3 seconds, I see) and provide the counts of discarded values within each group. You should also explain any quantified data shown below the threshold (such as the brown point at the bottom left). Labeling the vertical axis as "density" is mysterious: surely your original data are just lifetimes and this plot gives counts within narrow lifetime classes? $\endgroup$ – whuber Jan 23 '15 at 16:16
  • $\begingroup$ If you want to compare the two conditional distributions (above the threshold) you can use a F-test for the comparision of the means as described by @Glen_b in stats.stackexchange.com/a/76695/10479 $\endgroup$ – Yves Jan 23 '15 at 16:45
  • $\begingroup$ Please: remove the normalization so we can see the actual data (the normalization destroys essential information) and tell us how many values you typically throw away in the data. $\endgroup$ – whuber Jan 23 '15 at 18:48
  • $\begingroup$ I added the raw data to the question. In the first dataset I have to discard 50% in the second about 30%. $\endgroup$ – MaxJ Jan 24 '15 at 12:15
5
$\begingroup$

Exponentially distributed lifetimes are an especially simple case for survival analysis. Analyzing them is often the first example worked to get students started before moving to more complicated situations. In addition, survival analysis is naturally suited to censored data. In short, I suggest you use survival analysis with a grouping indicator for the two distributions as a treatment effect. You could use a parametric model (e.g., the Weibull model, as the exponential is a special case of the Weibull), or you could use non-parametric methods, such as the log rank test, if you prefer.

$\endgroup$
  • $\begingroup$ Does this also apply for truncated data? I think my problem is a bit different from censoring or? $\endgroup$ – MaxJ Jan 24 '15 at 15:22
  • $\begingroup$ @user3683367, truncation is different from censoring. These would test for differences in the distributions above the detection limit. $\endgroup$ – gung - Reinstate Monica Jan 24 '15 at 15:43
  • $\begingroup$ I got rid of the truncation (new exp. setup) and just used left and right censored data in MATLABS statistical toolbox. I used the Weibull function with left and right censored data. The fitted means are different and their error bounds do not overlap. How can I calculate a probability that my distributions are actually the same? $\endgroup$ – MaxJ Jan 25 '15 at 13:54
  • $\begingroup$ @user3683367, you can't calculate the probability that the distributions are the same. That probability is either $1$ or $0$, & you don't know which. Instead, you can calculate the probability of getting 2 groups this divergent if they came from the same distribution; that's the p-value. As far as how to get MATLAB to give that to you, I don't know--I haven't used MATLAB in a long time, but a p-value on the treatment contrast should come with standard model output. $\endgroup$ – gung - Reinstate Monica Jan 25 '15 at 15:03
2
$\begingroup$

You are interested in the following test: $H_0: \lambda_1 = \lambda_2$ where $\lambda_i$ is the single parameter that uniquely identifies the exponential distribution you are dealing with. Since $\lambda$ also corresponds to the mean of this distribution you are essentially interested in testing the difference of means in these two distributions.

Since you have a large sample size, to test this we may appeal to the central limit theorem which tells us the following:

Central Limit Theorem: suppose $X_1, X_2, ...X_n$ is a sequence of i.i.d. random variables with $E[X_i] = \mu \text{ and } Var[X_i] = \sigma^2 < \infty$. Then as $n$ approaches infinity, the random variable $\sqrt{n}(\bar{X} − \mu)$ converge in distribution to a normal $N(0, σ^2)$ distribution.

In other words, your sample means for each of the two groups are approximately normally distributed. Since you don't know the true value of $\sigma^2$, you may perform a perform a t-test for a difference of means.

$\endgroup$
  • 1
    $\begingroup$ Because the exponential distribution is so skewed, a little analysis is needed to justify applying the CLT. If indeed these data were not censored, it turns out that $n=3000$ would be more than enough to make the normal approximation a good one. (Proof: the cgf of the mean of $n$ iid exponentials is $\psi(t)=t^2/(2 n) + i t^3/(3n^2) + O(t^4)$, implying the skewness is $2\sqrt{n}$, which is small for $n=3000$.) But this could be a serious error if a substantial proportion of either dataset had been discarded. $\endgroup$ – whuber Jan 23 '15 at 16:26
  • $\begingroup$ I edited my question. Can I really apply the CLT for left censored data with N=100-300? $\endgroup$ – MaxJ Jan 23 '15 at 18:44
  • $\begingroup$ Correct me if I'm wrong, but I believe Whuber's point is that if the exponential distribution is truncated than it's no longer a pure exponential distribution. A crux of my argument is that the mean of the exponential distribution uniquely identifies it. If this new truncated distribution is no longer uniquely identified by it's mean than my argument may fail. Whether or not the data is truncated, you can still do a difference of means t-test. If the means are different then the distributions are different. $\endgroup$ – TrynnaDoStat Jan 23 '15 at 19:22
  • $\begingroup$ If Whuber's point is that the CLT doesn't apply then I'll have to disagree with him. Whether or not your data comes from a pure or truncated exponential distribution, you still have iid sample from the same distribution with some mean $\mu$ and some finite variance $\sigma^2$. $\endgroup$ – TrynnaDoStat Jan 23 '15 at 19:23
  • 2
    $\begingroup$ My point is not that the CLT is inapplicable: it is that in order to apply it you must check that the asymptotic approximation is a good one. The CLT says absolutely nothing about the distribution of the mean of any particular finite number of iid, finite-variance variables. And please do not confuse truncation with censoring: the problem created by censoring is that some of the data are not even numbers (they are intervals)--so a fortiori the CLT cannot (directly) apply in that situation. $\endgroup$ – whuber Jan 23 '15 at 20:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.