Let data $\mathbf{X}\in \mathbb{R}^d$ come from one of the $K$ possible sources $\mathsf{S}\in \{1,2,...,K\}$. The true $\mathsf{S}$ is unknown but it is fixed. The main task is to infer the true value of $\mathsf{S}$ by observing a sequence of $\mathbf{X}$s. We have an uninformative prior on the true $\mathsf{S}$ (i.e. uniform). We also have access to an emprical classifier $\mathcal{F}: \mathbf{X} \rightarrow [0,1]^K$ that produce a softmax output reciving $\mathbf{X}$.

As an example, we are observing an audio signal stream and we want to infer the age of the speaker which is something in range $\mathsf{S}\in \{18,19,...,60\}$. We already have an age classifier, $\mathcal{F}$, that has been trained on a public dataset (not including the current speaker's data).

Or, let's say:

  • We receive $\mathbf{X}_1$ (at time $t=1$) and our previously trained classifier produces a guess about $\mathsf{S}_1$.
  • Then, we receive $\mathbf{X}_2$, and we again use the classifier to guess $\mathsf{S}_2$.
  • Then, we receive $\mathbf{X}_3$, and so on.

So, at time $t=T$ we have $\mathsf{S}_1, \mathsf{S}_2, ..., \mathsf{S}_T$.

Now, the questions:

  1. By having multiple consecutive observations $\mathbf{X}_1, \mathbf{X}_2, \mathbf{X}_3,...$ (e.g. second by second audio signals), how can I improve my belief about the true $\mathsf{S}$ (e.g. speaker's age) over time, leveraging the available $\mathcal{F}$ (e.g. audio based age classifier):

  2. Is there any framework to calcualte some sort of confidnece interval over the predicted value and narrowing it down by obseving more and more data?

    • A reference to a related subject or a link to the state-of-the-art method/article is also appreciated.

Question 1 : Your model gives you $\mathbf{X}_i = \mathbb{P}(S = s | data[t_i:t_{i+1}])$. I think you want $\mathbb{P}(S = s | data[1:T])$. According to the Bayes rule :

$$\mathbb{P}(S = s | data[1:T]) = \mathbb{P}(data[1:T]| S = s)\mathbb{P}(S = s)/\mathbb{P}(data[1:T]).$$

We have that $data[1:T] = \cup_{i=1}^N data[t_i:t_{i+1}]$. We may want to assume that $data[t_i:t_{i+1}]$ are independent and independent conditionally on S (*). $$\mathbb{P}(data[1:T]) = \prod_{i=1}^N \mathbb{P}(data[t_i:t_{i+1}])$$ $$\mathbb{P}(data[1:T]| S = s) = \prod_{i=1}^N \mathbb{P}(data[t_i:t_{i+1}]| S = s).$$

Therefore, we have $$\mathbb{P}(S = s | data[1:T]) = \mathbb{P}(S = s)\prod_{i=1}^N \mathbb{P}(data[t_i:t_{i+1}]| S = s)/ \mathbb{P}(data[t_i:t_{i+1}]).$$

We use again the Bayes rule : $$\mathbb{P}(S = s | data[1:T]) = \mathbb{P}(S = s)\prod_{i=1}^N \mathbb{P}(S = s|data[t_i:t_{i+1}])/ \mathbb{P}(S=s) = \frac{\prod_{i=1}^N \mathbf{X}_i(s)}{\mathbb{P}(S=s)^{N-1}}.$$

Now we add your assumption on uniform prior on S : $\mathbb{P}(S = s)=1/K$ $$\mathbb{P}(S = s | data[1:T]) = K^{N-1} \prod_{i=1}^N \mathbf{X}_i(s).$$

This is weird because it is not well normalized... Maybe I did some mistakes. But anyway, what I would do is to compute : $\hat{p}(s) = \prod_{i=1}^N \mathbf{X}_i(s)$, and then normalize on $s$ to have $\mathbb{P}(S = s | data[1:T])$.

Question 2 : Once you have $\mathbb{P}(S = s | data[1:T])$, you can estimate the probability of any set of values $\mathcal{S}$ by summing $\sum_{s \in \mathcal{S}}\mathbb{P}(S = s | data[1:T])$. In particular, you can have the probability of a range of value around the mean, ie a confidence interval.

Of course, your result will be super-dependent on the quality of your model $\mathcal{F}$. For instance, if your model output some inacurrate $0$ in the $\mathbf{X}_i$ values, it will totally screw up your predictor. This should not happen in a correctly train model but you may want to replace extremely low value by a $\epsilon$ "just in case".

(*): These assumptions may be inaccurate. A more correct way to proceed is to retrain $\mathcal{F}$ to have a sequential model that gives directly $\mathbb{P}(S =s | data[1:T])$ instead of $\mathbb{P}(S =s | data[t_i:t_{i+1}])$ (like LSTM or this kind of stuff). If this is not possible, I guess you need assumptions of this kind to compute more or less meaningful probabilities.

  • $\begingroup$ I don't understand why $\mathbb{P}(S = s | data[t_i:t_{i+1}])$ is replaced with $X_i(s)$? What do you mean by $X_i(s)$? is it output of $\mathcal{F}$ for $\mathbf{X}_i$? $\endgroup$ – moh Nov 6 '19 at 18:29
  • $\begingroup$ Because $X_i$ is the output of $\mathcal{F}$ when you input your data stream $data[t_i , t_{i+1}]$. Your model is trained to give a belief of this form if I understand well. $\endgroup$ – Julien S. Nov 6 '19 at 18:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.