2
$\begingroup$

I have a bunch of data where each observation represents an error $\in [0,1]$ (computed error between a variable and it's ground truth).

Extra info: These are the results of the difference between a computed shortest path and the shortest path given by a known heuristic. Since no path can be shortest than the shortest, the differences will always be positive.

Edit: My data set is available here: data, and can be loaded with:

study <- subset(heuristic_export, select = c(Case, Heuristic, Error))    
chebyshev <- subset(study, Heuristic == "CHEBYSHEV", select = c(Error))

The variable of interest is chebyshev$Error.

You can visualize the distribution in the following histogram:enter image description here

Same visualization with more bins (50): enter image description here

At first I naively tried to transform the data into a normal distribution (without any degree of success), then I tough that the data would follow a exponential distribution and tried a bunch of fit tests. But after some thought and investigation it makes sense that my errors follow a truncated normal distribution, a segment of a normal distribution between [0,1].

If that's the case, how can I for example test for the population mean?!

E.g. $H_0\!:\ \mu = 0.1,\quad H_{\rm alt}\!:\ \mu <0.1$

Or even, how can I compute the sample mean? Any recommendations?

Any help would be appreciated.

EDIT: Is the CLT applicable for this problem?! And if so, can I apply a simple hypothesis test on the resulting dataset?! Data is cheap.

EDIT2: I'm following gung's advice, and tried fitting the data to both log normal and gamma. From the following plots, my interpretation is that the data better fits a gamma distribution. I have no experience with mixture models, or even the gamma distribution, so help, examples or resources would be appreciated regarding understanding the distribution, hypothesis testing or probability calculation for mixture models. Here are the plots:

Log Normal fit: Log Normal

Gamma fit: enter image description here

$\endgroup$
  • $\begingroup$ It appears that when you computed these errors, you ignored the signs - so in reality you have absolute errors. I suggest keeping the signs, then you'll possibly have a more understandable distribution. Plus, then the location (median or mean) of the error distribution quantifies bias. $\endgroup$ – rvl Sep 27 '14 at 12:55
  • $\begingroup$ I didn't ignore the signs.. these are the results of the difference between a computed shortest path and the shortest path given by a known heuristic. Since no path can be shortest than the shortest, the differences will always be positive. $\endgroup$ – Ramalho Sep 27 '14 at 12:57
  • 1
    $\begingroup$ That doesn't look all that much like a truncated normal to me. You should use more bins in your histogram. Why would they be restricted to be smaller than 1? How was the calculation done? $\endgroup$ – Glen_b Sep 27 '14 at 12:57
  • 1
    $\begingroup$ (ground truth - heuristic)/(ground truth) $\endgroup$ – Ramalho Sep 27 '14 at 13:02
4
$\begingroup$

With enough data, you can use an ordinary $t$ method to compute a CI for the mean, with the CLT as a justification for doing so. However, with the distribution you show, the mean does not seem very meaningful (sorry for the pun).

For example -- It seems like there are a lot of zeros. The probability of a nonzero error seems more descriptive than the mean. The probability of an error greater than some meaningful threshold seems more descriptive. You can estimate either of these, and obtain a confidence interval, using standard methods for estimating proportions.

$\endgroup$
  • $\begingroup$ Thanks for the feedback. I'm kind of new to statistics, so I have some questions regarding your answer. Why do you suggest a t-method? Isn't the t-method only prefered when n<30? Why not use the standard error method (SE = sd(sample)/sqrt(n)?! When should I use one in detriment of the other? You've got a point, the probability of an error is more meaningful. Thank you. $\endgroup$ – Ramalho Sep 28 '14 at 0:17
  • $\begingroup$ I meant, why use a t statistic instead of a z statistic. $\endgroup$ – Ramalho Sep 28 '14 at 0:34
  • 1
    $\begingroup$ A lot of intro stat books have that $n>30$ rule, but it is silly because as the df grows, the $t$ distribution just evolves into the $z$. For large $n$, you don't really need the $t$, but it doesn't hurt to use it. $\endgroup$ – rvl Sep 28 '14 at 3:53
2
$\begingroup$

There are a number of issues to be cleared up here.

  1. First, I don't think you used the formula listed in your comment. You state that the heuristic cannot be shorter than the ground truth, thus the heuristic must be larger and all values calculated by that formula would be negative. You have only positive values, so I suspect you used (heuristic - ground truth)/(ground truth) instead.

    For what it's worth, note that this formula yields the difference as a proportion of the magnitude of the ground truth value. If the length of the shortest path is always the same, you could just use the numerator. If the shortest paths differ substantially, this formula implies that $11-10$ is the same as $110-100$, be sure you want that.

  2. You say that the errors are bounded by $0$ and $1$, but that doesn't appear to be the case. Since the heuristic cannot be shorter than the shortest path, the minimum must be $0$. However, there is no apparent reason that the maximum must be $1$. It is true that the calculation is the difference as a proportion of the shortest path, but a number can be more than 100% of another number.

  3. You state that you have a truncated normal, but you don't on both counts. Even if the data generating process were normal, your sample would be censored not truncated. Truncated means your sample would be biased in the sense that the negative values would not exist in the sample at all, whereas you have $0$'s. Censored would mean that the $0$'s had somehow replaced the negative values.

    In addition, it makes no sense at all to think of your data as being normal since you state that it is logically not possible for the heuristic value to be shorter than the ground truth.

    If it were possible to have negative values, but which were replaced with $0$'s by some censoring process, you could estimate and test for the mean with a Tobit regression model. In R you would do something like the following:

    library(censReg)       # you need to load this package
    set.seed(3124)         # this makes the example exactly reproducible
    y = rnorm(500)         # the data generating process is normal
    y = ifelse(y>0, y, 0)  # this is the censoring process
    
    summary(censReg(y~1))  # this fits the Tobit model
    # ...
    # Observations:
    #          Total  Left-censored     Uncensored Right-censored 
    #            500            246            254              0 
    # 
    # Coefficients:
    #             Estimate Std. error t value Pr(> t)
    # (Intercept)  0.01956    0.05224   0.374   0.708
    # logSigma    -0.04437    0.04936  -0.899   0.369
    # ...
    
  4. As I say, your data cannot be normal; we might wonder what other distribution would be appropriate. Unfortunately, distributions that are bounded at both the bottom and top (such as Beta) won't work because your data have no necessary upper bound. Distributions such as Gamma or the lognormal might come to mind, but they cannot have $0$'s. Although it is worth bearing in mind that there is no reason to believe that real data has to come from, or fit, any named distribution, it may be worth thinking of your data as a mixture where some proportion are strictly $0$'s and the remainder come from one of those distributions.

    To characterize such data, you could estimate the proportion of $0$'s as @rvl discusses. Then you could fit the non-$0$ data to one of those distributions. (You do have to potentially worry that some actual non-$0$ data were rounded down to appear as $0$'s, but this may still work well enough for your purposes.) Here is how you could do it in R:

    set.seed(86)
    library(fitdistrplus)
    x = c(rep(0, 250), rgamma(250, shape=1.5, rate=0.5))
    
    binom.test(sum(x==0), length(x))
    # ...
    # number of successes = 250, number of trials = 500, p-value = 1
    # alternative hypothesis: true probability of success is not equal to 0.5
    # 95 percent confidence interval:
    #  0.4552856 0.5447144
    # sample estimates:
    # probability of success 
    #                    0.5 
    summary(fitdist(x[x!=0], "gamma"))
    # Fitting of the distribution ' gamma ' by maximum likelihood 
    # Parameters : 
    #        estimate Std. Error
    # shape 1.5719570 0.12844233
    # rate  0.5477509 0.05259777
    # Loglikelihood:  -499.9608   AIC:  1003.922   BIC:  1010.965 
    # ...
    summary(fitdist(x[x!=0], "lnorm"))
    # Fitting of the distribution ' lnorm ' by maximum likelihood 
    # Parameters : 
    #          estimate Std. Error
    # meanlog 0.7035092 0.05937440
    # sdlog   0.9387916 0.04198382
    # Loglikelihood:  -514.8215   AIC:  1033.643   BIC:  1040.686 
    # ...
    
$\endgroup$
  • $\begingroup$ You are correct, the metric used was: (heuristic - ground truth)/(ground truth), thanks for the correction. The points on which the ground truth and heuristics are being calculated are being randomly generated. So yes, I just want the error proportion for each concrete case. Otherwise I would be penalizing cases where points where further apart. Regarding the last part, I have no experience on mixing models, how do I analyze the data? E.G. how do I calculate P(x<0.01), is it P(x==0) (on proportion)+p(x<0.01) (on gamma), we have to branch the calculation? Thank you. $\endgroup$ – Ramalho Sep 28 '14 at 10:53
  • $\begingroup$ @Ramalho, the calculations are demonstrated in my answer. Because I am stipulating that this might be a mixture of pure 0's & (say) Gamma, to separate them you just need x==0 or x!=0. Neither Gamma nor the lognormal can have 0's, so that's it. As I said, you would have the possibility that some numbers are so small they get rounded to 0's, though. $\endgroup$ – gung - Reinstate Monica Sep 28 '14 at 15:50
  • $\begingroup$ @Ramalho, are you using R or different software? $\endgroup$ – gung - Reinstate Monica Sep 28 '14 at 15:57
  • $\begingroup$ I did understand the motivation behind modelling the distribution as a mixture, gamma's support doens't include 0. I also understand your code, you'r separating the zeros from the rest of the dataset and fitting a gamma/lnorm distribution to the non zero data. What I don't know how to model is: how do I calculate probabilities with a mixture of models?! My knowledge on statistics it's a little bit skewed toward normal distributions. Resources or examples on how to handle mixture distributions would be highly appreciated. Also, I plotted the gamma fit and it kind of suits the data. Will update. $\endgroup$ – Ramalho Sep 28 '14 at 19:01
  • $\begingroup$ @Ramalho, what software are you using? I don't quite follow your question. How to calculate the probabilities is in the code I posted; binom.test(sum(x==0), length(x)) gives you the proportion of 0's, a CI & a test against .5 (you could put in a more meaningful value w/ the p= argument). If you are using R, you can get the documentation for a function by typing eg ?binom.test at the command prompt. $\endgroup$ – gung - Reinstate Monica Sep 29 '14 at 2:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.