I have machine learning models that have a some possible post processing possibilities. I would like to use multi armed bandits to select the post processing that optimizes a continuous KPI. I have hundreds of models and I don't know what the distribution of the KPI is. It could potentially be different for different models and the optimisation should be done in real time with no human intervention.
How can I approach this problem? Under what conditions can I suppose the KPI is normally distributed? I'm looking for any guidance or keywords that can point me to a direction.