5
$\begingroup$

I want to discretize the Truncated Normal to describe the spread in the Grades of students in a class sitting the same exam.

Let an arbitrary number of students sit the same exam. The grades are usually said to be normally distrubutted (they couldn't possibly be distributed with the truncated normal and much less by a simply normal because the grade are bound from both ends and are discrete).

The minimum grade is a parametre a (even negative grades are allowed)

The maximum grade is a parametre b

The minimum difference possible between two grades is a parametre c

x (measured in X axis) is the grade

y (measured in Y axis) is the probability of a student achieving X grade or the frequency that students in the class achieve grade X.

μ is a transformation of the mean, it is the mode if μ lies between a and b and is also the mean if μ lies halfway between a and b

σ is a transformation of the SD, it is the SD when $a\to-\infty\cap b\to\infty \cap c=0$

When $c=0$ we have a Truncated Normal

When $a\to-\infty\cap b\to\infty \cap c=0$ we have a Normal.

I want my distribution (f(x,μ,σ,a,b,c)) to meet some criteria

1 Being supported on the terms of an arithmetic progression. Both the initial and final terms and the common difference are parametres

2 Being unimodal in all cases but 1. Being bimodal in case the mode is equidistant from 2 values in the support.

3 Having probabilites strictly decreasing as X distances itself from the mode. The decrease in probability should be equal for x equally distant from the mode. The change in the difference should first decrease and then increase after reaching a quasi-inflection point. The change in the difference should also be equal for x equally distant from the mean. I.e if the mode coincides with the midrange the distribution is symmetrical while if the mode is closer to one bound the distribution is symmetrical for all values closer than the nearest to the mode bound (short of like the Truncated Normal if you fold it at the mode all the values that are closer that the nearest bound coincide 1 to 1 and only what is further away does not coincide because there is nothing left for them to correspond to)

Hopefully the first 3 criteria are all compatible with each other.

4 If the above three criteria allow for it the distribution should belong to the Natural Exponential Family (better) or the Exponential Family (Plan B)

5 The distribution should be the Maximum entropy distribution that the above 4 criteria allow for (ignore criterion 4 if the first 3 make it impossible for the distribution to belong to the Exponential Family)

Such that the probability mass assigned to any point is proportional to the kernel of the normal distribution (leaving the mean and the standard error a parametre) calculated at that point.

E.g when a=0, b=25 and c=1

$f(0,\mu,\sigma, 0, 25, 1)\propto\mathrm{e}^{-\frac{{\mu}^2}{2{\sigma}^2}}$

$f(1,\mu,\sigma, 0, 25, 1)\propto\mathrm{e}^{-\frac{\left({\mu}-1\right)^2}{2{\sigma}^2}}$

$f(2,\mu,\sigma, 0, 25, 1)\propto\mathrm{e}^{-\frac{\left({\mu}-2\right)^2}{2{\sigma}^2}}$

$f(3,\mu,\sigma, 0, 25, 1)\propto\mathrm{e}^{-\frac{\left({\mu}-3\right)^2}{2{\sigma}^2}}$

.

.

.

$f(25,\mu,\sigma, 0, 25, 1)\propto\mathrm{e}^{-\frac{\left({\mu}-25\right)^2}{2{\sigma}^2}}$

of course

$\displaystyle \sum_{x=0}^{25} f(x,\mu,\sigma, 0, 25, 1) = 1$

or more generally

$\displaystyle \sum_{x=a}^b f(x,\mu,\sigma, a, b, c)=1$

What is the closed form solution (probably in terms of Theta functions)

Edit.

I am hoping the probability mass function I will get will be a conjugate prior (in the same probability family) that a posterior predictive distribution would be if we started with a discrete uniform prior (with a common difference c between each adjacent value in the support) where the random variable would be the individual students performance and each grade would be assigned the same frequency (if there are say 25 values in the support, 25 different grades possible, it would assign a probability of 0.04 for each grade) and the observations would be the individual performance of every student in the class (in the end we will have the performance of the class not simply their mean but an histogram). The posterior would be a distribution assigning a probability that a student randomly chosen scored X given that the class scored as the histogram shows.

I don't understand the difference between

$$P(X = x\vert \mu,\sigma,a,b,c) = \frac{e^{-\frac{(x-\mu)^2}{2\sigma^2}}}{\sum_{y \in \Omega} e^{-\frac{(y-\mu)^2}{2\sigma^2}}}$$

and

$$p_X(x) = \frac{\Phi \Big( \frac{x - \mu + 1/2}{\sigma} \Big) - \Phi \Big( \frac{x - \mu - 1/2}{\sigma} \Big)}{\Phi \Big( \frac{m - \mu + 1/2}{\sigma} \Big) - \Phi \Big( \frac{1/2 - \mu}{\sigma} \Big)} \quad \quad \quad \text{for } x=1,...,m,$$

Whuber said that they are not closed form solutions, because they involve an unevaluated sum. There is no closed form solution that doesn't involve Theta functions. I don't understand this either.

$\endgroup$
27
  • 5
    $\begingroup$ There is little justification in defining a pmf this way when compared with other functions from $\{0,...,y\}$ to $(0,1)$. The Normal density function is a density for a continuous variate and its values at a finite number of values such as the above integers cannot be interpreted as probabilities. In addition, most of the nice properties of a Normal distribution do not transfer to this new distribution. $\endgroup$
    – Xi'an
    Dec 9, 2020 at 19:44
  • $\begingroup$ @Xi'an The justification is simple I hate approximations. My data are bound both from below and above and much more significantly are discrete. I don't want a distribution which is easy to work with I want an acurate distribution (or the closest possible to that) even if I end up with something hard to work with. What other distributions f(μ,σ,x,z,y) are there (which are both discrete, and bounded from both sides). What properties do those distributions have that my constructed distributtion would lack? $\endgroup$ Dec 9, 2020 at 20:19
  • 5
    $\begingroup$ In which sense an integer-Normal distribution is less of an approximation? more accurate? And what properties are you looking for in your model? $\endgroup$
    – Xi'an
    Dec 9, 2020 at 21:01
  • 6
    $\begingroup$ This question is confusing. You repeatedly state you "hate approximations," but ironically you have yet to characterize the quantities you wish to compute! All you have said explicitly is "I need a discrete distribution." Could you characterize what you need in clear, mathematical terms so that people can know what might constitute an "approximation" and how to evaluate it? What do you mean by the "general problem" of which this might be one instance? $\endgroup$
    – whuber
    Dec 9, 2020 at 22:21
  • 1
    $\begingroup$ That helps clarify things, thank you. However, you seem to be posing a very broad question: namely, how to construct parametric families of discrete distributions supported on a specified set (such as the integers from 0 to 25). And of course you don't want to construct the family based on the data: that would make correct inference impossible. On what basis, then, do you want to construct the distributional family?? $\endgroup$
    – whuber
    Dec 10, 2020 at 14:30

3 Answers 3

5
$\begingroup$

The other answer here uses the normal density values at the exact points. Another similar method would be to take the normal probabilities across intervals centred on those points. In the latter case, taking the support to be $X = 1,...,m$ you get:

$$p_X(x) = \frac{\Phi \Big( \frac{x - \mu + 1/2}{\sigma} \Big) - \Phi \Big( \frac{x - \mu - 1/2}{\sigma} \Big)}{\Phi \Big( \frac{m - \mu + 1/2}{\sigma} \Big) - \Phi \Big( \frac{1/2 - \mu}{\sigma} \Big)} \quad \quad \quad \text{for } x=1,...,m,$$

where $\Phi$ is the CDF of the standard normal distribution. You can then adjust to an arbitrary arithmetic progression by taking the appropriate linear transformation. This has the first three properties you stipulated in your question.

$\endgroup$
8
  • $\begingroup$ What does it mean to "take the normal probabilities"? Does this pmf belong to either the natural exponential family or the exponential family? Is the entropy maxed out given the constraints? If we calculated the bayesian posterior given a discrete uniform distribution (with constant c) and normally distributed data would we end up with a posterior of this family; would this pmf be a conjugate prior to the posterior calculated with a discrete unifrom and normally distributed data? How much of a difference would it make to use this pmf as a prior rather than a discrete uniform (with constant c)? $\endgroup$ Nov 6, 2021 at 13:17
  • $\begingroup$ @SextusEmpiricus Does it belong to the Natural Exponential family too? $\endgroup$ Nov 6, 2021 at 13:39
  • $\begingroup$ @George I don't believe it does. Neither does the truncated normal distribution (if you do not fix $\sigma, a, b$ and $c$). The motivation/advantage of the distribution here would be that the cumulative distribution is relatively simple to express, it is exactly the same as the continuous truncated normal distribution. $\endgroup$ Nov 6, 2021 at 13:47
  • $\begingroup$ What is the difference between $$p_X(x) = \frac{\Phi \Big( \frac{x - \mu + 1/2}{\sigma} \Big) - \Phi \Big( \frac{x - \mu - 1/2}{\sigma} \Big)}{\Phi \Big( \frac{m - \mu + 1/2}{\sigma} \Big) - \Phi \Big( \frac{1/2 - \mu}{\sigma} \Big)} \quad \quad \quad \text{for } x=1,...,m,e$$ and $$P(X = x\vert \mu,\sigma,a=0,b,c=1) = \frac{e^{-\frac{(x-\mu)^2}{2\sigma^2}}}{\sum_{y \in \Omega} e^{-\frac{(y-\mu)^2}{2\sigma^2}}}$$ are they the same distribution just differently formulated? $\endgroup$ Nov 6, 2021 at 16:11
  • 1
    $\begingroup$ @Xi'an you are right, silly mistake. I was erroneously thinking of the categorical distribution, which is in the exponential family, where the parameters are $p_1, p_2, p_3, \cdots, p_n$, and thought that would capture any other discrete distribution. But, then the parameterization is entirely changed. $\endgroup$ Nov 13, 2021 at 8:34
5
+100
$\begingroup$

When you use the values of the normal density then you get automatically the first four conditions satisfied

$$P(X = x\vert \mu,\sigma,a,b,c) = \frac{e^{-\frac{(x-\mu)^2}{2\sigma^2}}}{\sum_{y \in \Omega} e^{-\frac{(y-\mu)^2}{2\sigma^2}}}$$

where $\Omega = \lbrace a,a+c,a+2c,\dots b-c,c \rbrace$ all the values in the support.


Maximum entropy

The fifth condition, maximum entropy, is also fulfilled since the maximum entropy distribution with constraints $\sum_{\forall x} x p(x) = \mu$ and $\sum_{\forall x} x^2 p(x) = \text{var}$ must be of the form $$P(X=x) = ce^{\lambda_1 x + \lambda_2 x^2}$$ which is like the distribution above (e.g. see that with $\lambda_1 = \mu/\sigma^2$ and $\lambda_2 = -1/(2\sigma^2)$ you get the above).


We can also prove it explicitly by considering the Kullback-Leiber divergence or Gibbs inequality with another distribution $f(x)$, and our distribution $g(x)$

$$\begin{array}{} - \sum_{\forall x} f(x)\log f(x) &\leq& - \sum_{\forall x} f(x)\log g(x) \\ &\leq& - \sum_{\forall x} f(x)\log \left(ce^{\lambda_1 x + \lambda_2 x^2} \right)\\ &\leq & - \sum_{\forall x} f(x)\left(\log c + \lambda_1 x + \lambda_2 x^2 \right)\\ &\leq & - \sum_{\forall x} g(x)\left(\log c + \lambda_1 x + \lambda_2 x^2 \right) \\ &\leq& - \sum_{\forall x} g(x)\log g(x) \end{array}$$

This second last step where we switch from $f(x)$ to $g(x)$ is because of the constraints $$\begin{array}{} \sum_{\forall x} f(x) &=& \sum_{\forall x} g(x) &=& 1 \\ \sum_{\forall x} xf(x) &=& \sum_{\forall x} xg(x) &=& \mu \\ \sum_{\forall x} x^2f(x) &=& \sum_{\forall x} x^2g(x) &=& \text{var} \end{array}$$ with these constraints we can rewrite $$\sum_{\forall x} f(x)\left(\log c + \lambda_1 x + \lambda_2 x^2 \right) = \sum_{\forall x} g(x)\left(\log c + \lambda_1 x + \lambda_2 x^2 \right)$$

So we must have that the entropy of $f(x)$ must be smaller than the entropy of $g(x)$.

$\endgroup$
40
  • $\begingroup$ What does it mean to "use the values of the normal density"? Does this pmf belong to the natural exponential family or just the exponential family? If we calculated the bayesian posterior given a discrete uniform distribution (with constant c between values in the support) and normally distributed data would we end up with a posterior of this family? Would this pmf be a conjugate prior to the posterior calculated with a discrete unifrom (with constant c) and normally distributed data? How much of a difference would it make to use this pmf as a prior rather than a discrete uniform (with c)? $\endgroup$ Nov 6, 2021 at 13:24
  • 2
    $\begingroup$ @GeorgeNtoulos "use the values of the normal density": it means that we make the PMF of the discrete distribution equal to the PDF of the continuous distribution, with only a multiplicative constant as difference. $\endgroup$ Nov 6, 2021 at 13:52
  • $\begingroup$ The pmf belongs to a two-parameter exponential family (or five-parameters if you include $a,b$ and $c$). When you fix the variance then it will also belong to the natural exponential family. $\endgroup$ Nov 6, 2021 at 13:55
  • 2
    $\begingroup$ Indeed my distribution is similar to Tyrel Stokes' solution. The only difference is that the parameters $a,b,c$ are incorporated and I have shown (intuitively) that the distribution is the maximum entropy distribution. The distribution by Ben is different, but approximately it is very close because with small steps (relative to $\sigma$) you get that $$\Phi \Big( \frac{x - \mu + 1/2}{\sigma} \Big) - \Phi \Big( \frac{x - \mu - 1/2}{\sigma} \Big) \approx \phi \Big( \frac{x - \mu}{\sigma}\Big)/\sigma $$ $\endgroup$ Nov 6, 2021 at 16:25
  • 1
    $\begingroup$ What is the likelihood function of the observations (and what do you observe?) based on which you update the prior into the posterior? $\endgroup$ Nov 6, 2021 at 18:53
4
$\begingroup$

The most natural way would be as follows:

Let $X$ have support $0,1,...T$, where $T \in \{2,3,...,\}$ (i.e T can be taken as the limit to $\infty$, since the infinite sum can be bounded by the integral over the kernel of a gaussian pdf, but it will be intractable to work with in practice).

$$f_X(X=x;\mu, \sigma) = \frac{exp(-\frac{(\mu -x)^2}{2\sigma^2})}{\sum_{y=0}^Texp(-\frac{(\mu -y)^2}{2\sigma^2})}$$

This is a special case of the softmax function ($\frac{exp(z_i)}{\sum_{i=1}^Nexp(z_i)})$, which is commonly used to map the reals to the interval $(0,1)$ with the property that the elements sum to 1. There are many choice models which take this form, such as the multinomial logit, with $T < \infty$.

I agree with @Xi'an, however, that defining a pmf this way is unlikely to be justified. It is hard to imagine a principled construction of what this would represent and as Xi'an stated, the properties of a gaussian that make it nice to work with are not inherited by this new model.

$\endgroup$
21
  • $\begingroup$ The justification is simple I hate approximations. My data are bound both from below and above and much more significantly are discrete. I don't want a distribution which is easy to work with I want an acurate distribution (or the closest possible to that) even if I end up with something hard to work with. What other distributions f(μ,σ,x,z,y) are there (which are both discrete, and bounded from both sides). What properties do those distributions have that my constructed distributtion would lack? $\endgroup$ Dec 9, 2020 at 20:22
  • $\begingroup$ reading the other comments, are you modelling the responses to each question i for data set j? Or the number of correct responses in each data set j? $\endgroup$ Dec 9, 2020 at 23:28
  • $\begingroup$ Most probably the second. There are i students (and i exams) that each received the same 25 questions (different each year). The results are positively dependent (students who know/believe that they will do well tend to sit the exams more often, so on years with better teaching and/or a smarter cohort the sample is larger and the mean is larger too because of self-choice students themselves choose to sit the exam or not). I consider each year a different population (different students unless some failed and choose to retake it and different 25 question). $\endgroup$ Dec 10, 2020 at 7:17
  • $\begingroup$ I want to infer 2 things a) how good was the performance of the specific student (my metric is [Their score / (The mean of all other students + The standard error of all other students)] and I take the weighted harmonic mean of the "Standardized" score between all their subjects. b I want to make inference on the Teaching that year. the mean might have dropped either because the test makers gave difficult questions or because the teachers taught poorly that year (or the cohort were stupid). $\endgroup$ Dec 10, 2020 at 7:24
  • $\begingroup$ You could use a binomial model, where p_i is the probability of success for each class. If you wanted to try and separate the effects of teaching and the effect of the cohort it will be tricky. The two main approaches would be to find covariates with sufficient information to control for the strength of the class (maybe mark from previous year or gpa generally) and model p_i as an inverse logit which is a function of a parameter for the class and then the covariate information. The other typical route would be to model the selection process itself and use a weighted estimator. $\endgroup$ Dec 11, 2020 at 23:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.