The correct way to form a PDF from probabilities of means (to calculate Entropy later)

Question

The Question

I have a probabilistic belief. I have 4 normal distributions, and I believe in each one a different amount. As follows: $$ P(M=ceramic)=0.9987 \\ P(M=aluminium)=0.0013 \\P(M=plastic)=0 \\ P(M=steel)=0 $$

How do I translate my whole belief into one single gaussian, that could be used for entropy calculation later?

As if I want to put the "best mean" for my belief on the pic above, how would I mathematically move the mean and change the deviation? The discrete probabilities are easy to work with bayesian inference wise, but not in the context of information entropy. Alternatively, is there any way to compute the discrete entropy based on the probabilities of materials?

To make some choices for actions (e.g. to weigh the object - get density, or squeeze the object - get elasticity) I want to minimize the entropy. The action that gains the most information about the object should be chosen.
For continuous PDFs the integral for entropy looks like: $$ H(x) = -\int f(x) \log{f(x)} dx $$ where the $f(x)$ represents the PDF. For normal distribution, there even exists a closed form solution: $$ H(x) = -\frac{1}{2}log(2\pi e \sigma^2) $$ but the only thing, that is a variable here is the $\sigma$. My Bayesian inference only updates the probabilities. It merely updates my belief in how likely I think the object might be made of for example steel. I was thinking of creating a multimodal distribution, but that is not in close form, as I understand...

---

Context of the problem

More detailed description of the problem context follows:

Outline

1. Observe object
2. Output probabilities for materials > The prior
3. Based on a Bayesian network simulate how would measurements update the prior
4. Compute the Entropy difference for every simulation output
5. Choose the action with the highest information gain (highest entropy reduction)

---

The problem introduction

1. and 2. part

I have a vision system, that observes an object and based on previously seen data (neural network) outputs range of probabilities for known materials. For the sake of simplicity, let's say the output looks like this:

$$P(M=ceramic)=0.7 \\ P(M=aluminium)=0.15 \\P(M=plastic)=0.1 \\ P(M=steel)=0.05 $$ where $M$ stands for a Material variable.
To use these probabilities in the 3rd part of the outline with the Bayesian network (with which I will not go to great details) I create Gaussians based on some ground truth information from material tables.

---

3. part

I do the inference based on Bayes theorem (showcased for steel): $$ P(M=steel|\hat{\varepsilon}) = \frac{P(\hat{\varepsilon} | M=steel)P(M=steel)}{\sum_{i \in I}P(\hat{\varepsilon} | M=material_i)P(M=material_i)} $$ where $\hat{\varepsilon}$ stands for the measurement with noise (error or std deviation) and $I$ is the set of materials. (More on how I updated in the Appendix A.)
I compute this for every material and have one step of the Bayesian inference and it yields updated probabilites given in 1. and 2. step. After some measurement the probabilities look, for example, like this: $$ P(M=ceramic)=0.9987 \\ P(M=aluminium)=0.0013 \\P(M=plastic)=0 \\ P(M=steel)=0 $$

---

Thank you

for diving into my problem. I really appreciate your time. I tried to go only as deep as necessary, not to miss any important details.

Appendix A

I have marginalized over the true value of measurement $\varepsilon$ to get the $P(\hat{\varepsilon}|M=material)$ for the Bayes theorem with $$ P(\hat{\varepsilon}|M=material) = \int_I P(\hat{\varepsilon}|\varepsilon)P(\varepsilon | M=material) d\varepsilon $$ which has a closed form solution for Normal distributions - sampling from a "widened" Gaussian, where the "widening" looks like: $$ N_x(\mu_{\text{prior}},\sigma^2_x) = N_{\text{prior}}(\mu_{\text{prior}}, \sigma^2_{\text{prior}} + \sigma^2_{\text{meas}}) $$ So I sample the new PDF $N_x$ at $\mu_{meas}$ and get the function value, which represents the measurement -> the aforementioned $P(\hat{\varepsilon}|M=material)$. I remember this technique from the physics labs, but cannot source it and am not able to assess it's validity. But I assume it is correct.

Answer 1

The most useful way to express the belief is via a Gaussian Mixture Model.

How do I translate my whole belief into one single gaussian, that could be used for entropy calculation later?

Summing the belief into one simple gaussian would inevitably mean a loss of information. (Intuitively: I have 4 gaussians, with 2 parameters each, so 8 parameters in total. Representing a distribution of 8 parametets with a single gaussian (2 parameters) means simplyfing.) So a Gaussian Mixture Model is the thing needed. The Differential entropy can be computed numerically (so not in closed form) as a: $$H(X) = -\sum_{x \in X} \text{bin} \times f(x) \times \log_2f(x)$$ Where:
$X$ - the whole support of the distribution
$x$ - the disreticized point on the x axis of the distribution
$\text{bin}$ - the size of the bin (you are computing an integral, so the bin is the size of the step)
$f(x)$ - your distribution (also know as likelihood function)

Gaussian Mixture Model (GMM)

is a multimodal and possibly multivariate distribution of weighted gaussians. In my example, I weighted the gaussians by their probability and simply summed them up.

Updating the GMM with measurements

Updating of the Gaussian Mixture Model with incoming measurements is as simple as computing the product of the GMM and likelihood (the measurement) and then normalizing.