My Question

Let's say a set training samples like D from a discrete distribution like p(x) over a discrete variable vector like x is available. We don't have any prior knowledge about the form of p(x). We are given two different full estimators for p(x). Let's call them f(x) and g(x). How can I prefer f or g?

What is my application? (I the case you ask why)

I work on building a pipeline for anomaly detection application. I deal with a non-stationary data with unknown form of prior distribuion. I have two proposals for the entire pipeline and trying to develop a criteria for comparison. Both learn how to make probabilities of each observation. I have a test set which can measure the inferred probability. Intuitively, the pipeline which produces the probability more accurately is desirable.

While we have different metrics in probability theory for comparing two distributions like different divergences, or hypothesis tests like kolmogorov-smirnov test for comparing two distribution, I don't how to make a fair comparison in the case of unknown p.

What is not my application?

  1. I can't compare two estimators using synthetic data. Why? Making a representative synthetic data is not very feasible.
  2. I can't assume that the form is fixed to any known form and then shrink the problem to comparing model parameter estimation comparison. Why? Since I run the pipeline in different environments (data-set) each one differs.


  • $\begingroup$ Notation is a little confusing. So p(x) is a distribution right? And f(x) is what? A single input function or a vector valued input to accommodate all data? What is its output, a complete distribution? $\endgroup$ – Cagdas Ozgenc May 13 at 18:22
  • $\begingroup$ p(x) denotes the probability distribution over x and as I said, f(x) and g(x) are two estimators for p(x). $\endgroup$ – Mike Zadeh May 13 at 19:49
  • $\begingroup$ So f(x) is a full density estimator? And x is vector valued input, i.e. list of training data? You mention that f is unknown but I think you mean p unknown. If my assumptions are correct then given equally likely hypothesis you compare $\sum log p_f(x) >? \sum log p_g(x)$ over test data where $p_f(x)$ is output of f(x) on training data $\endgroup$ – Cagdas Ozgenc May 13 at 20:08
  • $\begingroup$ Thanks, I edited my question. x is a vector of discrete random variable conform a distribution like p(x) defined on domain of x. We have a training set like D for learning f(x) and g(x) and we have a disjoint test set for running our comparison between f and g. Regarding you solution is Pf(x) is the estimated distribution (my f(x)). If that is correct, it is like assuming 1/N for probability of x in test set and comparing the cross-entropy. Is that what you meant? $\endgroup$ – Mike Zadeh May 13 at 23:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.