1
$\begingroup$

I have the following dataset which appears to be normally distributed but is truncated at 0. enter image description here

If I ignore the values which are 0 I get an even better distribituion.

enter image description here

I would like to conclude that some percentage of the values are within $k\sigma$ of $\mu$ but the problem is that $\overline{X}$ (dashed line) does not match with the centre of my distribution which seems problematic.

In conclusion how should I calculate $\overline{X}$ and $S(X)$ and how/can I uses these estimate $\mu$ and $\sigma$.

Also after I have estimated $\mu$ and $\sigma$ is it valid to say $\Pr(X < \mu + 1.65\sigma) = 0.95$ like if I had a non-truncated normal distribution?

EDIT

The dataset consists of the maximum time derivative values in a set of images (video). All the data points should have a zero derivative because nothing moves in theses videos (just an empty scene) but there is noise to due changes in lighting (clouds), camera vibration, wind, etc.

Some images samples can be found HERE. The original images are on the left and their derivatives are right. Theses samples represent two classes. Ones with movement (top) and ones without (bottom). The class I'm interested in describing is the second class (bottom) this is what is plotted above.

$\endgroup$
14
  • 3
    $\begingroup$ This doesn't seem like truncated normal. You wouldn't have the spike at 0 in it. This is more like left censoring. $\endgroup$ – Aksakal Dec 8 '14 at 2:01
  • $\begingroup$ @Aksakal Technically all the data points should be zero but due to noise in my experiment I get the distribution above. I'm not sure if this helps at all. Is it valid to remove the correct values (0s) and only examine those which are incorrect (>0)? $\endgroup$ – en4bz Dec 8 '14 at 3:11
  • $\begingroup$ Removing zeros doesn't sound right. Can you provide more info on the subject of measurements? What and how are you measuring? $\endgroup$ – Aksakal Dec 8 '14 at 3:14
  • $\begingroup$ @Aksakal See my edit. $\endgroup$ – en4bz Dec 8 '14 at 3:32
  • $\begingroup$ I'm not familiar with this field. Can't derivative be negative? Did you look at extreme value distributions? I think it's quite unlikely for the distribution of maximums to be Gaussian $\endgroup$ – Aksakal Dec 8 '14 at 3:36
3
$\begingroup$

OK fine, I'll wade in... :-)

I have the following dataset which appears to be normally distributed but is truncated at 0.

As we've seen, this is not obviously true. But let's leave the identity of the generating distribution aside for a moment.

If I ignore the values which are 0 I get an even better distribituion.

Well, you get a sample without a spike at 0. But it seems to me that 0 is a case where the system you are measuring gets the right answer. As you note: it 'should' be zero, so the rest is just the shape of some function of the noise.

I would like to conclude that some percentage of the values are within kσ of μ

Right. So, if I may creatively reconstruct a bit, the inference task is as follows:

Given a large sample of values, find $a$ and $b$ such that $x$ percent of values will fall between $a$ and $b$.

For concreteness let's set $x=0.95$

The task is made trickier by the fact you don't have an analytic form for the generating distribution. However, If the underlying distribution is smooth, the sample is random, etc. then we have some guarantee that statistics you compute from the sample you have will converge on their true values as the sample size increases. (Obviously there are regularity conditions here which I'm ignoring for exposition.)

In particular, two such statistics are the 0.025, 0.975 quantiles. You might then solve the task by setting using the sample to estimate them and then setting $a$ to the first and $b$ to the second. We might call this the non-parametric approach.

You have approached the problem in a parametric manner by first searching for a density that your histogram looks like and then trying to use analytic results about the integrals of that density to determine which $a$ and $b$ would bound 95% of the probability.

I speculate that you thought about it as follows (apologies if I've missed your line of thought): You're familiar with some results for the Normal distribution which depend only on knowing its mean and variance and wonder how to apply them here. As you note, applying this distributional assumption is made problematic by the sample's apparent spike at zero and lack of negative values.

That's what you're responding to when you say that

the problem is that X⎯⎯⎯ (dashed line) does not match with the centre of my distribution which seems problematic.

This isn't quite true - the mean is a measure of central tendency - so I guess what you're thinking is problematic is that the mode and the mean don't seem to line up as you're expecting. This is a natural consequence of the skew, and should suggest that a Normal assumption might actually not be the one to reach for.

Nevertheless, you wonder

how should I calculate X⎯⎯⎯ and S(X) and how/can I uses these estimate μ and σ.

but I think the answer might be to stop wondering how to calculate these quantities. They would be useful if your distribution were Normal, but if it isn't, they may not be. Consequently your next question

after I have estimated μ and σ is it valid to say Pr(X<μ+1.65σ)=0.95 like if I had a non-truncated normal distribution?

will not necessarily help solve your actual inference task either, though it's an interesting question in itself.

The answer in general is 'not quite', e.g. 95% of the probability in a unit Normal distribution is below 1.65 but only about 92% of it is below μ+1.65σ if it's truncated at zero (when μ becomes about 0.8 and σ about 0.6). Also, as a minor point, your question asks about a range around the mean whereas the expression above is one-tailed.

Fortunately, it seems to me, you have enough data to get $a$ and $b$ from the data directly. But if you do pursue the 'find a distribution and use its analytical results' route then, since these are maxima you are plotting, you probably want to follow @Aksakal's advice and fit an extreme value distribution.

This leaves the question of the spike at zero. But what you want to do about that depends on the use you want to put your desired interval. Perhaps you want to explicitly model it, but perhaps not.

$\endgroup$
1
$\begingroup$

I agree with the commenters that this doesn't seem to be truncated normal distribution. However, if you have have some complicated distribution, that is hard to deal analytically, what you can do is to employ Monte Carlo methods, that were created exactly for this kind of problems. So the idea would be to use a sampler to draw multiple samples from the distribution that would approximate the distribution of your data, and then draw conclusions on your data using those simulated samples. This will enable you to compute mean, variance and other quantities of the distribution.

EDIT:

However, as conjugateprior and whuber this does not seem to be an efficient approach here because of your sample size.

$\endgroup$
6
  • $\begingroup$ Since the OP already has a rather large sample of the data and wants to characterise it, resampling or sampling from an approximation would seem to be a bit superfluous. $\endgroup$ – conjugateprior Dec 8 '14 at 12:43
  • $\begingroup$ There is not information on sample size in the question. On another hand, this problem doesn't seem to have a clear analytical solution. MC was designed for this kind of problems. So it is a solution. I don't insist that it is a best one. $\endgroup$ – Tim Dec 8 '14 at 12:47
  • $\begingroup$ @conjugate Tim's advice looks useful (+1). However, Tim, if you look closely at the graphics in the question you will conclude that the OP has around 15,000 statistics (each of which summarizes, presumably, quite a few observations), so conjugateprior's implicit suggestion that one ought to resample from the data does look like a good one. Otherwise, you leave the OP somewhat in the lurch, because he will need to find some "approximate" distribution, but evidently part of his problem is that he doesn't know how to find a good approximation! $\endgroup$ – whuber Dec 8 '14 at 16:46
  • 1
    $\begingroup$ @whuber You're right: I didn't notice the small letters on Y axis. Now I agree. $\endgroup$ – Tim Dec 8 '14 at 16:51
  • $\begingroup$ @conjugateprior, I think simulation makes a sense to determine the underlying distribution of which OP is obtaining max. That in turn may help in understanding what would the distribution of max. $\endgroup$ – Aksakal Dec 8 '14 at 20:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.