What is the equivalent of linear Gaussian noise but for discrete data

Question

A common framework in stats and ML is that our data is noisy observation of an underlying true value, e.g.

$$ \begin{aligned} \mathbf{x} &= \mathbf{z} + \boldsymbol{\varepsilon}, \\ \boldsymbol{\varepsilon} &\sim \mathcal{N}(0, \sigma^2 \mathbf{I}), \\ \mathbb{E}[\mathbf{x}] &= \mathbf{z}. \end{aligned} $$

This linear-Gaussian assumption is everywhere (see A Unifying Review of Linear Gaussian Models).

However, what if $\mathbf{x}$ is discrete. Are there noise models in this case? I don't find anything interesting when I search for this online.

Answer 1

If you reformulate your assumption, to say that our observations are realisations of a normally distributed random variable, then, of course, there are analogies in discrete cases:

• a coin flip is a realisation of a Bernoulli-distributed random variable;
• the number of heads in a series of coin flips is a realisation of a random variable with a binomial distribution;
• the number of hits a website gets in an hour is a realisation of a random variable with a Poisson distribution;

etc.

But, if you are looking for additive error, then, I believe, it is highly uncommon. Not that you couldn't construct one, but I cannot remember having ever seen one in practice.

Answer 2

One option would be to model $\boldsymbol{x}$ as arising from, e.g., a Poisson distribution with mean parameter $\boldsymbol{z}$, where $\log(\boldsymbol{z})$ follows a Gaussian distribution centered at the desired mean.

Answer 3

It is not common to my knowledge to see discrete models which are of the form you described on the observed space, but some discrete models can be motivated from latent processes of the form that you described.

For example, let:

$Z^\star = \mu + \epsilon$ where the star is used to denote that it is not observed, or latent. Then the variable:

$$ X = 1\{Z^\star >0 \}$$

is a binary variable. If $\epsilon$ is logistically distributed then you get

\begin{align*} P(X =1) = \frac{1}{1 + \exp(-\mu)} \end{align*}

This is one way to motivate logistic regression. Similarly if $\epsilon$ has a gaussian distribution you get a probit.

Another class of models of a similar form are random utility choice models. Here we assume typically that a class of say J choices each has an associated random variable, $X_1, X_2,...,X_J$, which typically represents the utility of value to the agent choosing.

We suppose that the agent makes the choice with the highest utility, i.e $\arg max_{j \in 1,2,\dots,J} (X_1,\dots,X_J)$ and each $X_i$ has the form:

$X_i = \mu_i + \epsilon$, where $\mu_i$ is the structural part of the value and $\epsilon$ is gumbel distributed noise with scale parameter $\sigma$.

Then if $Y = \arg max_{j \in 1,2,\dots,J} (X_1,\dots,X_J)$ is a random variable taking discrete values $1,\dots, J$ representing the observed choice

\begin{align*} P(Y =j) = \frac{exp(\frac{\mu_j}{\sigma})}{\sum_{i=1}^Jexp(\frac{\mu_i}{\sigma})} \end{align*}

The $\mu_i$'s themselves can be modelled as functions of other covariates of the choice or choice maker and this leads to slightly different models different models. Different choices of error also leads to different models (but in this case will not have convenient closed forms to work with).

These are just a few examples of models with what I believe to be the flavour of model you are looking for. It would not be impossible to construct variable of the form you described directly on the observed space, but I believe there are larger literatures where we imagine a continuous noisy latent space that then gets transformed to the discrete space. The reason for this is that discrete distributions are not location families and thus composing a mean function with an additive discrete error gives you a different distribution than the error.