This problem is encountered by me lot of times.

Let's say I have a set of data points (x1, x2,....,xn) whose mean and standard deviation can be calculated. However these data points are non-normally distributed. Now, given a new data point (X_new), how do I infer how likely it is to observe this new data point being drawn from the same population as the above obtained data points.

Per my understanding, if the underlying distribution is normal in that case we can use the mean and standard deviation obtained from the observed sample to make inference about the new data point's likelihood of belonging to same distribution. However, I am not sure how do we proceed if we are not sure about the underlying distribution being normal or if they are non-normal?

Use case: I am working on a Business use case where I have historical performance of Items being sold on my website (univariate distribution of #units sold). Now given the sales of any item, I want to establish if their sale is significantly higher compared to most of the other items. The histogram of the sales is right skewed with 5% of items contributing to more than 80% sale and hence the distribution is non normal. Due to non-normality of the distribution I can not use the mean sales and the standard deviation to estimate the likelihood of any single item's sale being as high as observed and am not sure what should be my approach here.

  • $\begingroup$ Do you have a distribution that matches better your population? $\endgroup$
    – PC1
    Jul 4 at 19:31
  • $\begingroup$ Not really. But as mentioned in my use case, observations in sales world are mostly skewed (mostly follows 80-20 kind of pareto), so I fear if the underlying distribution can ever be normally distributed. $\endgroup$ Jul 4 at 19:33
  • $\begingroup$ If you can presume that you know your distribution, computing the incremental entropy could provide more insightful information. $\endgroup$
    – PC1
    Jul 4 at 19:36
  • $\begingroup$ SO basically, if I can approximate a density function basis the observed data. And then understanding the increase in the entropy on observing the new data point? is that what you are suggesting? could you please share some refrences for the same? $\endgroup$ Jul 4 at 19:39
  • $\begingroup$ Depending on the support for your distribution, there are many options to approximate it. The metalog distribution can be used for example. Then you have a closed form to compute the incremental entropy for each new data point. $\endgroup$
    – PC1
    Jul 4 at 20:25


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.