# Parameter estimation when the likelihood function does not exist

The observations $$Z_1,Z_2\cdots$$ are i.i.d. We have

$$Z_k = \sum_{i=1}^\infty \frac{X_{ki}}{2^k}.$$

where the $$X_{ki}$$'s are i.i.d. with a Bernouilli$$(p)$$ distribution. If $$p=\frac{1}{2}$$ then $$Z_k$$ is Uniform on $$[0, 1]$$. Otherwise $$Z_k$$ does not have a density, see here.

Yet the solution is trivial in this case: $$p$$ is just the proportion of binary digits of $$Z_k$$ that are equal to $$1$$. So you can directly estimate this proportion by looking at the binary digits of your observations. But how would you do if you were to use standard statistical techniques to solve this problem? You could simulate the $$Z_k$$'s using various values for $$p$$ and see the one that provides the best fit with your data. But are there any techniques, other than non-parametric ones, that could solve this problem in a "traditional" way?

Likewise, how would you test (say) $$p=0.75$$ vs. $$p\neq 0.75$$? The density for the null hypothesis does not exist because the CDF is nowhere differentiable, it seems.

Distribution of $$Z_k$$

First, note that $$E(Z_k) =p$$ and $$\mbox{Var}(Z_k) =p(1-p)/3$$.

We assume that $$0< p <1$$. Let us denote as $$S$$ the set of normal numbers in $$[0, 1]$$. This set has Lebesgue measure equal to $$1$$. However, the set of non-normal numbers in $$[0, 1]$$ is dense in $$[0, 1]$$ despite the fact that its Lebesgue measure is $$0$$.

If $$p=\frac{1}{2}$$, then $$Z_k$$ is uniform on $$S$$, and thus, does not even a density in the classical sense, since $$S$$ if full of holes: for instance, none of the rational numbers in $$[0, 1]$$ belong to $$S$$ because they are periodic numbers, thus their $$X_{ki}$$'s are not independent in this case.

If $$p\neq \frac{1}{2}$$, then $$Z_k$$'s support domain has Lebesgue measure $$0$$ yet it is dense in $$[0, 1]$$. Its CDF, when $$p=0.75$$, is pictured below.

If it had a density, it would like this:

Moments of $$Z^k$$

Let $$F$$ denote the distribution, $$Z$$ any of the $$Z_k$$'s, and $$X$$ any of the $$X_{ki}$$'s. The following functional equation must be satisfied: $$F_Z = F_{(X+Z)/2}$$. Thus we have $$E(Z^n) = 2^{-n} E((X+Z)^n)$$ which leads to the following recursion formula for the moments: $$E(Z^n)=\frac{p}{2^n-1} \sum_{i=1}^n \frac{n!}{i!(n-i)!}E(Z^{n-i}).$$ This yields

$$E(Z) = p \\ E(Z^2) = \frac{p}{3}(1+2p)\\ E(Z^3) = \frac{p}{7}(1+4p+2p^2)\\ E(Z^4) = \frac{p}{105}(7+46p + 44p^2+8p^3)$$

More generally, if $$s\geq 0$$ then

$$E(Z^s) = \frac{1}{2^s} E((X+Z)^s) \Rightarrow E(Z^s) = \frac{p}{2^s-1+p}E((1+Z)^s).$$

Example

Let $$Z_k =\{ 2^k \pi/4\}$$ be the observations, that is, the data. Here the brackets represent the fractional part function. I estimated the first few moments using the first 1,000 $$Z_k$$'s. The result is $$0.506$$ for the mean, and $$0.799$$ for the variance. The theoretical values are $$1/2$$ and $$1/12$$ if $$p=1/2$$. We can thus conclude that $$p$$ is not statistically different from $$1/2$$. Note that $$p$$ can be estimated, using the moment method, by averaging the $$Z_k$$'s. Also here the $$Z_k$$'s are auto-correlated. The theoretical lag-1 autocorrelation is $$1/2$$.

• I think you are framing the problem in an unhelpful way. The real estimation problem is: "Given a sample of infinite size from a $\text{Bernoulli}(p)$ distribution, how do I estimate $p$?" Simply because you have chosen to transform your sample into something that doesn't have a distribution doesn't mean that we have to work with the transformed sample. As you yourself have pointed out, we can just work with the original sample and the problem induced by your transformation goes away. So why not do that? – jbowman Dec 5 '19 at 21:51
• No discrete distribution has a density--but they all produce likelihoods. Ergo, lack of a density is not a barrier to performing MLE. As a practical matter, all real statistical problems can be solved if necessary by adding a tiny absolutely continuous variable to the data (thereby representing infinitesimal measurement error). See stats.stackexchange.com/questions/248476. – whuber Dec 5 '19 at 22:02
• @JBowman: In some cases, it might not be possible to easily reverse the transformation and get rid of it. My example here is probably the simplest case of some kind of chaotic process. In practice you observe the $Z_k$'s, and the $X_{ki}$ are an hidden layer. – Vincent Granville Dec 5 '19 at 22:07
• You can't get from a countably infinite sum of discrete random variables to a continuous random variable. OTOH, if a density were required in order to estimate parameters, we wouldn't be able to use computers to do so. – jbowman Dec 5 '19 at 22:19
• To test $H_0:p=3/4,$ I would be inclined to use a sequential probability ratio test. It will terminate almost surely, obviating any need to inspect all of the bits of the $Z_k.$ This exploits the observation by @jbowman that the $Z_k$ are merely a way to encode countable sequences of binary values: and that's the basic setup of a great many Binomial tests. – whuber Dec 5 '19 at 22:29