4
$\begingroup$

I've been researching different methods to compare two distributions for equality, or inequality rather. I want to compare actual user performance against projected performance, after a particular "tweak" is made in a user interface. I project the performance using simulations. Each run (whether simulated or actual) generates a "number" at the end to indicate "performance".

I can run as many simulations as I want. However, I only have a few users (between 10-30, depending on how users are divided) so few data points (again, 10-30). Assuming I can't get any more user data, the distribution of actual user performance data is represented by a small number of data points.

I know about skewness, kurtosis, and the Kolmogorov–Smirnov test, but are these sufficient to test for equality, even with such low-resolution data? If not, are there other existing tests for distributions with few samples? Or unevenly distributed samples?

Improve this question
$\endgroup$
3
  • $\begingroup$ If you have only a few values, you could try the Wilcoxon-Mann-Whitney-Test. With this one, you can check whether two distributions belong to the same population. see en.wikipedia.org/wiki/Mann%E2%80%93Whitney_U_test $\endgroup$
    – Regenschein
    Dec 15, 2014 at 10:30
  • 2
    $\begingroup$ Try the specialist stats site, crossvalidated.stackexchange.com, this is probably more on-topic there than on "data science". $\endgroup$
    – Spacedman
    Dec 15, 2014 at 15:43
  • 1
    $\begingroup$ This question appears to be off-topic because it belongs on CrossValidated $\endgroup$
    – Spacedman
    Dec 15, 2014 at 15:43

2 Answers 2

Reset to default
1
$\begingroup$

The distributions are not going to be identical (but with small samples you may well be unable to pick that they differ). I suspect that a goodness-of-fit test doesn't actually answer the question of interest (does it really matter if they're very, very slightly different, for example, no matter how small that difference might be?)

With few data points you probably need to focus on getting as much power as you can. This will involve carefully identifying alternatives of interest and identifying a test with good power against those sorts of alternatives.

If you can get a lot of values for the simulated distribution (enough to treat it as effectively a population), you could perhaps consider an Anderson-Darling test or a Cramer-von Mises; whether one of these has better power than a Kolmogorov-Smirnov depends on what alternatives you're more interested in power against. [If you can identify more specific alternatives still you should be able to do better than any of those.]

Improve this answer
$\endgroup$
0
$\begingroup$

I would obtain the distribution CDF from simulations, then use it as theoretical in KS test, one-sample. In this case the actual data size will be your sample size is KS test.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.