1
$\begingroup$

I am looking at data set that has four groups. In each group, the data is mostly, 99+% of time, composed of zeros, but, when it is not zero it can be any float number (e.g., 0.01 to 921.2, with most values being under 10). Once I examine dataset 1, I want to examine other datasets that also have 4 groups and similar sparseness in the data. Sometimes the n or number of observations in a group can be as low as 10 or as high as, say, 20,000.

I want to calculate a point estimate and confidence intervals (CI) around that estimate for each group so that I can quickly determine whether group 1 is say, worse than group 2.

My question: is it appropriate to calculate the CI using mean and standard error (stdev / sqrt(n) ) with such a sparse data set? Any advice would be appreciated!!

$\endgroup$
2
$\begingroup$

I want to calculate a point estimate and confidence intervals (CI) around that estimate for each group so that I can quickly determine whether group 1 is say, worse than group 2.

A point estimate isn't necessarily a problem; you can estimate a mean by a mean, though the extreme skewness may be an issue (e.g. a mean may not be representative of either the bulk of zeros nor the mean of the non-zero data)

You might consider modelling it as a Bernoulli 0/not-0 and then find a model for the not-0 cases.

My question: is it appropriate to calculate the CI using mean and standard error (stdev / sqrt(n) ) with such a sparse data set?

The $s/\sqrt{n}$ formula is still a standard error, but a multiple of it may nor be much help as in interval for the mean.

In really large samples (large enough to have say thousands of non-zero observations), that might be a useful approach, but since the sample size can be small this may have some issues as well.

$\endgroup$
  • $\begingroup$ I like the idea of modeling the non-zero's separately! So, to be clear, you think that using the st. error for an interval for the mean is problematic? Do you have thoughts on an alternative? Thanks for you help! $\endgroup$ – captain_ahab Dec 9 '14 at 0:31
  • $\begingroup$ Please describe HOW you will use the standard error for a mean to obtain a CI, and maybe I can tell you whether or not it's problematic. I do plan to come back with some additional information but it will take a while to generate it all. $\endgroup$ – Glen_b Dec 9 '14 at 0:32
  • $\begingroup$ Thanks Glen. I was simply planning on finding the CI around the mean by adding/subtracting 1.96*st.error (at least in cases where n is large enough). My only concern is that the data is extremely skewed and sparse. $\endgroup$ – captain_ahab Dec 9 '14 at 3:06
  • $\begingroup$ Thanks for clarifying. How do you decide when n is large enough? $\endgroup$ – Glen_b Dec 9 '14 at 3:54
  • $\begingroup$ Based on the CLT, a heuristic of at least 30 $\endgroup$ – captain_ahab Dec 9 '14 at 19:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.