Weights Update - Ensemble Models I must identify if a data point is an outlier or not in a dataset (we don't have labels). I have different unsupervised models to identify the outlier. Then, I normalize the outlier score and I combine them via a weight average. According to the fact that I don't have information about their accuracy I use the same weight for each models.
Now, suppose that I have a small fraction of the dataset with also the label.
How can I update the weights according to the new information?
This is what I think that could work, but I don't have reference to say that for sure.
Imagine that I have 4 models. The probability of the model I assume that is the weight, so 0.25.
$$ P(Model) = 0.25  $$
Then, the likelihood:
$$ P(Outlier | Model) $$
that are the normalized outlier score.
And so, the posterior is:
$$ P(Model | Outlier) = P(Outlier | Model) P(Model)$$
My question is: can I sum all the different posterior probability for each observation and then normalize w.r.t. the sum of the other models?
I give you an example in python of my thinking:
import numpy as np

def bayesian_update(anomaly, weight, prob):
    #inizialization vector
    posterior = np.zeros(len(anomaly))
    
    
    for i in range(len(anomaly)):
        #if the labled indicate an anomaly
        if anomaly[i] == 1:
            posterior[i] = prob[i] * weight
        #if the labled indicate a non anomaly
        else:
            posterior[i] = (1-prob[i]) * weight
    return posterior



np.random.seed(0)
n_observations = 100
n_models = 4

# 
models_probs = np.random.rand(n_observations, n_models)


#a way to indicate that the first model (colomn) is better
anomaly = np.where(models_probs[:, 0] > 0.5, 1, 0)

posterior_sum = np.zeros(n_models)

for i in range(n_models):
    posterior_sum[i] = np.sum(bayesian_update(anomaly, 0.25, models_probs[:, i]))
    
new_weight = posterior_sum/np.sum(posterior_sum)

print(new_weight)



```

 A: Have you considered a stacking approach where you build a meta predictor using the 4 model outputs as features? In your case I would consider building a Naïve Bayes classifier and then predicting an outlier based on the maximum a-posteriori hypothesis:
\begin{align}
P(Outlier|m_1,...,m_4) &\propto P(Outlier) \prod_{m=1}^4 P(M_i=m_i|Outlier) \\
P(\neg Outlier|m_1,...,m_4) &\propto P(\neg Outlier) \prod_{m=1}^4 P(M_i=m_i|\neg Outlier)
\end{align}
where $m_i$ is the output of the $i$-th unsupervised outlier detection model for a given new point. You predict "Outlier" whenever $P(Outlier|m_1,...,m_4) > P(\neg Outlier|m_1,...,m_4)$.
From your labelled training set (assumed to be representative) you need to:

*

*estimate the prior probabilities $P(Outlier)$ and $P(\neg Outlier)$

*estimate $P(M_i=1|Outlier)$ which is the sensitivity (true positive rate) of predictor $i$ (and hence also find $P(M_i=0|Outlier)=1-P(M_i=1|Outlier)$)

*estimate $P(M_i=0|\neg Outlier)$ which is the specificity (true negative rate) of predictor $i$ (and hence also find $P(M_i=1|\neg Outlier)=1-P(M_i=0|\neg Outlier)$).

So, for example, if the 4 predictors predict [0,1,0,1], you would predict "outlier" if:
$$P(Outlier) \cdot (1-Sens_1) \cdot Sens_2 \cdot (1-Sens_3) \cdot Sens_4
>
P(\neg Outlier) \cdot Spec_1 \cdot (1-Spec_2) \cdot Spec_3 \cdot (1-Spec_4).
$$
As you can see, compared to a simple model average, this approach takes into account the breakdown in terms of sensitivity and specificity rather than just the model's accuracy.

If your models $m_i$ provide a score, rather than a binary value, you can use pretty much the same approach by changing the conditional probabilities into $P(M_i\ge m_i|Outlier)$ (etc), which can be estimated by counting how many of the "Outlier" examples in the labelled training set provided a score at least as high as the current sample (and same for the "$\neg$Outliers").
