3
$\begingroup$

I don't have any background in statistics, so maybe I may say things that are incorrect.

I have to model the production of waste deposition at each container of a set of containers and I have access to the values of the fill-up rates for certain intervals of time. I need to generate future values based on the historical past values.

First I thought that a Normal Distribution could be adopted, but since there's no negative waste deposition rates I thought on using the Log-Normal Distribution as it starts at the zero value.

Imagine that I have the following ten values: 5; 0; 9; 2; 6; 4; 1; 0; 5; 3

My question is: How can I obtain the parameters μ and σ that are needed to model the probability distribution?

I see that an approach is to use: enter image description here

But then, since I have values of xk that are 0, I have ln (0) which is a problem that I don't know how to work around.

From the wikipedia page I see also that when the individual values x1,x2,...,xn are not available I can calculate the parameters using the following formulas: enter image description here

So, from my sample, $\bar{x}$ = 3.5 and $\hat{\sigma}$ = 2.8771. So, can I say that μ and σ are equal to 0.9946 and to 0.7185, respectively?

If yes, then what bugs me the most is that I have the individual values but some of them happen to be 0. How could it be that I could use those first formulas only if the values are different than 0?

Any help would be great! Thank you in advance!

$\endgroup$
9
  • 3
    $\begingroup$ The support of the lognormal distribution is $(0, +\infty)$. The fact that you have values of $0$ thus immediately rules out the lognormal distribution as a suitable model for these data. What exactly do these fill-up rates mean (e.g. what does a value of $5$ mean)? $\endgroup$ Nov 29, 2019 at 13:54
  • $\begingroup$ A value of 5 means that for a specific day a container was filled up by 5%. A value of 0 represents that no waste was deposited during that day. If there are 0 values but no negative ones, what suitable models could be used to characterize this? $\endgroup$
    – Talochas
    Nov 29, 2019 at 14:17
  • $\begingroup$ So it's always a percentage between 0% and 100%? Am I understanding this correctly? $\endgroup$ Nov 29, 2019 at 14:24
  • $\begingroup$ Yes. It could get overfilled, but let's neglect that $\endgroup$
    – Talochas
    Nov 29, 2019 at 14:31
  • 3
    $\begingroup$ @COOLSerdash given we know that there are exact 0's, the beta itself would be unsuitable for the same reason the lognormal is. A 0- & 1- inflated beta might be a better choice. $\endgroup$
    – Glen_b
    Nov 30, 2019 at 3:27

2 Answers 2

2
$\begingroup$

Because your data appear to be rounded to the nearest integer, "0" really means some value in the interval $[0, 1/2]$ and "$k$" for $k\gt 0$ means some value in the interval $[k-1/2, k+1/2].$ Let's denote that interval as $[k^-,k^+],$ even for $k=0.$

Writing $\Phi$ for the Normal distribution, the likelihood that the parameters of $\log X$ are $(\mu, \sigma)$ for data $x_1, x_2, \ldots, x_n$ therefore is

$$\mathcal L = \prod_{i=1}^n \left(\Phi(\log x_i^+;\mu,\sigma) - \Phi(\log x_i^-;\mu,\sigma)\right).$$

Here, when $x_i=0$ and (therefore) $x_i^-=0,$ take $\log 0 = -\infty$ for the sake of computing. Since $\Phi(-\infty;\mu, \sigma) = \lim_{x\to-\infty}\Phi(x;\mu,\sigma)= 0$ there is no problem.

This is readily maximized numerically to give a Maximum Likelihood estimate for the parameters. For the data in the question, the estimates are $\hat\mu = 0.7652$ and $\hat\sigma = 1.2155.$ These are close to crude estimates $\hat\mu = 0.7613,$ $\hat\sigma = 0.1.2856$ obtained by using the midpoints of the intervals: that is, replacing $[0,1/2]$ by $1/4$ and replacing $[k-1/2, k+1/2]$ by $k$ throughout and estimating $\mu$ as the mean logarithm and $\sigma$ as the standard deviation of the logarithms. But there is a difference in the estimates of $\sigma,$ suggesting something might be gained by the more careful calculation.

(In response to a long-standing answer in this thread, replacing the zeros by $0.00001$ gives $\hat\sigma = 5.4,$ which is grossly wrong. The reason is predictable: the logarithms of $0.00001$ are so far from the data that they exaggerate the value of $\sigma.$)

The following R code does the work. It minimizes $-\log \mathcal L$ as a function of $\mu$ and $\log \sigma$ (thereby avoiding the usual numerical problems and requiring no constraints).

With the estimated parameters in hand, you can generate additional random values easily: use them to generate Normally distributed values and then exponentiate and round them.

#
# Negative log likelihood of theta = (mu, log sigma).
#
lambda <- function(theta) {
  mu <- theta[1]
  sigma <- exp(theta[2])
  -sum(log(pnorm(log(x + 1/2), mu, sigma) - pnorm(log(pmax(0, x - 1/2)), mu, sigma)))
}
#
# The data (nonnegative integers).
#
x <- c(5,0,9,2,6,4,1,0,5,3)
#
# Use the midpoints for an initial estimate.
#
y <- log(pmax(1/4, x))
theta <- c(mean(y), log(sd(y)))
#
# Maximize the likelihood.
#
b <- nlm(lambda, theta)
#
# Extract the estimates.
#
m <- b$estimate[1]
s <- exp(b$estimate[2])
$\endgroup$
-2
$\begingroup$

Pad it with an arbitrary small number, e.g. 0.00001; basically your minimum precision. It will yield a highly negative value of the logarithm, but that's fine.

Assuming the production was continuous in time, you can never actually measure a point where it's exactly zero in reality anyway, it's asymptotic too, so it's not entirely unprincipled.

$\endgroup$
2
  • 4
    $\begingroup$ The "yield a highly negative value" looks like it's going to bias any estimation procedure strongly, so please explain why you think this is "fine." $\endgroup$
    – whuber
    Nov 29, 2019 at 17:32
  • 2
    $\begingroup$ +1 or +0.1 or +0.01 turns out to be very arbitrary and very hard to generalize in predictive modeling in my experience. $\endgroup$
    – AdamO
    Jun 22, 2023 at 15:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.