1
$\begingroup$

I recently got dataset for 37000 households with declared income and a few dozens of other variables of various types: continuous, discrete, binary.

The task is to automatically (unsupervised) evaluate credibility of declared income based on the remaining variables: evaluate if it agrees with statistics of the sample.

Approach I have used ( https://arxiv.org/pdf/1812.08040 , general slides):

  1. Normalize the income to uniform distribution on [0,1] using empirical distribution function like in copula theory. Thanks of it, modeled density of conditional distribution of this variable seems a proper way to evaluate credibility (?) Here are examples of pairwise dependencies of such normalized variables - would be rho=1 if independent, inhomogeneity allows to predict different conditional distribution of income based on the second variable (e.g. for 70 year old, extreme value is less credible): enter image description here

  2. To combine predictions from multiple variables, I just used linear regression: of cumulant-like polynomial coefficients of predicted variable, as linear combinations of features of the remaining variables (e.g. their contribution to j-th moment) - works nicely, here are some predicted densities: enter image description here

I would like to compare it with some standard approach, but don't know what to use?

KDE doesn't seem useful here (?) - what other ML methods can be used to model such complex conditional continuous probability distributions?

|cite|improve this question
$\endgroup$
1
+50
$\begingroup$

What you might try is estimating quantile regression to predict, let's say, 1st and 9th decile (or other n-tiles) of declared income given all the other information. Then, for the test data, you can treat realized income outside of predcited ntile range as unusual (procedure sort of similar to hypothesis testing). In fact, quantile regression is available in the context of many machine learning models (glms, random forets, boosting trees and so on).

Another thing you can try are Vine Copulas to estimate multivariate probability density. Then, compute marginal distribution of your income and compare it to realized values.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ Thanks, it would correspond to splitting above square diagrams into smaller squares and modeling their densities independently. One its issue are discontinuities, e.g. 1 parameter would allow to split into left and right half, while using polynomial we have linear behavior of density - smoothing. Another problem is lack of representants for some combinations, what is solved by above polynomials as mixed moments - interpolating between. Similar technique is context mixing used in data compression. Thanks for vine copulas, I will have to take a look - what other multi-parameter copula are used? $\endgroup$ – Jarek Duda Jan 17 '19 at 9:32
  • $\begingroup$ Which of the two approaches you're refferring to? For the case of quantile regression I wouldn't think for a while about probability estimation - I'd treat it more like task to predict low/high quantiles as opposed to expected value (like in most cases of machine learning models). First approach is rather simplistic, but might act like a good baseline. Imagine boosted trees with significantly pruned trees as weak learners - you will be quite sure to have enough observations in each of the leaves and, thus, have some understanding how might look underlying distribution for one single subgroup $\endgroup$ – user2280549 Jan 17 '19 at 14:13
  • $\begingroup$ As for Vine copulas it is a mixture of two-dimensional copulas (all widely-used like Archimedean copulas might be useful to utilized in that particular context) $\endgroup$ – user2280549 Jan 17 '19 at 14:15
  • $\begingroup$ Thanks, quantile regression is really nice, I didn't know it. For now I prefer MSE polynomial fitting I use - it directly gives entire density, has better interpretability (uses mixed moments), and is much cheaper (least squares vs linear programing), but I will have to compare their predictions in some future. Regarding credibility evaluation, it is more complicated than 0.05 and 0.95 quantiles pointing tails. Instead, I think it should be rather conditional density for copula-like normalization (diagrams at the top), which can be e.g. multimodal - with less credible values between maxima. $\endgroup$ – Jarek Duda Jan 18 '19 at 22:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.