Let's say a set training samples like D from a discrete distribution like p(x) over a discrete variable vector like x is available. We don't have any prior knowledge about the form of p(x). We are given two different full estimators for p(x). Let's call them f(x) and g(x). How can I prefer f or g?
What is my application? (I the case you ask why)
I work on building a pipeline for anomaly detection application. I deal with a non-stationary data with unknown form of prior distribuion. I have two proposals for the entire pipeline and trying to develop a criteria for comparison. Both learn how to make probabilities of each observation. I have a test set which can measure the inferred probability. Intuitively, the pipeline which produces the probability more accurately is desirable.
While we have different metrics in probability theory for comparing two distributions like different divergences, or hypothesis tests like kolmogorov-smirnov test for comparing two distribution, I don't how to make a fair comparison in the case of unknown p.
What is not my application?
- I can't compare two estimators using synthetic data. Why? Making a representative synthetic data is not very feasible.
- I can't assume that the form is fixed to any known form and then shrink the problem to comparing model parameter estimation comparison. Why? Since I run the pipeline in different environments (data-set) each one differs.