I have over a 100 observations of human decisions. If my hypothesis is correct, they should all be a fixed number, or around that fixed number (allowing for small errors around that fixed point). How can I test that the real distribution is different from a distribution that is essentially just that number, with small mistakes or deviations around it?

  • 2
    $\begingroup$ You cannot, because you haven't made any quantitative characterization of what you're trying to test. If you could state what "small" means, then you might be able to make some progress. $\endgroup$
    – whuber
    Aug 6 '18 at 12:31
  • $\begingroup$ Sounds like tests/estimates of location (errors "around that fixed point" & scale ("small" errors) are what you want; but to justify a specific test you need to explain what assumptions you can make about those errors. $\endgroup$ Aug 6 '18 at 14:02

Is there some reason you can't do a t-test comparing the mean value of your decisions to a fixed value. In other words, test the null hypothesis that $H_0: \mu = c$, where c is whatever your predicted value is? You would have to make sure you have sufficient power to reject that null hypothesis if it is false.

Alternatively -- perhaps better -- you could use a Bayesian test. This will allow you to determine a probability that the null hypothesis is actually true. See Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009). Bayesian t tests for accepting and rejecting the null hypothesis. Psychonomic Bulletin & Review, 16(2), 225–237. https://doi.org/10.3758/PBR.16.2.225

  • $\begingroup$ Interesting alternative interpretation of a somewhat vague question: I took "fixed number" and "that number" to be the same population means, and thought (I guess along with commenters) that the issue is about a possible difference in population variances. Maybe OP can clarify. $\endgroup$
    – BruceET
    Aug 8 '18 at 8:16
  • $\begingroup$ I guess it is somewhat vague. If your interpretation is correct - and rereading it now I suspect it probably is - then I think your first two solutions (which will, essentially, give the same result) are the way to go. If it were me, I'd prefer the first solution (confidence interval) because of the asymmetry of null hypothesis testing. $\endgroup$ Aug 8 '18 at 17:32
  • 1
    $\begingroup$ I have read the question several times without increased enlightenment. (+1) $\endgroup$
    – BruceET
    Aug 8 '18 at 17:36

Just to get started with some specifics, here are three of approaches that might be helpful. Somehow (as in @whuber's Comment), you need to develop a 'standard' of what you mean by a 'small' variance. I hope your situation is similar enough to one of the three scenarios below that this Answer will help you to do that.

CI for a single variance. First, you might make a confidence interval for the variance $\sigma^2$ or the standard deviation $\sigma$ of the population from which your $n = 100$ observations were sampled at random. I assume that the population is normal.

Consider $n = 100$ observations with $S^2 = \frac{1}{n-1}\sum_{i=1}^n (X_i - \bar X)^2 = 9.943.$ For normal data, one has $Q= \frac{(n-1)S^2}{\sigma^2} \sim \mathsf{Chisq}(df=n-1).$ Thus one can find values $L$ and $U$ such that $P\left(L < \frac{(n-1)S^2}{\sigma^2} < U\right) = 0.95,$ so that a 95% CI for $\sigma^2$ is of the form $\left(\frac{(n-1)S^2}{U}, \frac{(n-1)S^2}{L}\right).$ The figure below shows $L = 73.36$ and $U = 128.4.$

enter image description here

In our example, a 95% CI for $\sigma^2$ is $(7.664, 13.417)$ and, taking square roots of endpoints, a 95% CI for $\sigma$ is $(2.77, 3.67).$ [Computations below are from R, but you could also use printed tables of chi-squared distributions.]

99*9.942/qchisq(c(.975,.025), 99)
[1]  7.664248 13.416624
sqrt(99*9.942/qchisq(c(.975,.025), 99))
[1] 2.768438 3.662871

With these results, you can ponder whether the variability is greater than you anticipated, "allowing for small errors around a fixed point."

Test of a sample variance against a known population variance. Second, you may have done previous research on this kind of human decision making and you may believe that 'small' means a standard deviation around $\sigma = 2.5.$ Looking at the CI for the population standard deviation just above, you should already be suspicious that the variability in my fake data is not 'small' because $\sigma = 2.5$ is not 'small'. But if you want to do a formal test of the hypothesis $H_0: \sigma^2 \le 6.25$ against the one-sided alternative $H_1: \sigma^2 > 6.26.$

Assuming that $H_0$ is true the test statistic $Q = \frac{(n-1)S^2}{6.25}$ has $Q \sim \mathsf{Chisq(99)}$ and the critical value for a test at the 5% level is $c = 125.0.$ Because $Q = 157.6 > 125.0,$ you would reject $H_0$ in favor of the right-sided alternative at the 5% level of significance. (The P-value of this test is about 0.00017.)

qchisq(.96, 99)
[1] 124.955
1 - pchisq(157.5, 99)
[1] 0.0001680777

F-test to compare two sample variances. Third, if you have a control group in which you suspect variability is 'small' and a treatment group in which you suspect variability may be larger, then you can do a standard F-test to see if variances differ. This kind of test comparing two sample variances has notoriously bad power, especially for small sample sizes. But if you have sample sizes around 100, you should be able to make meaningful distinctions.

My fake data x above were sampled from a normal population with $\sigma_X^2 = 9.$ Suppose you had a control group with $n = 100$ observations y from a population with $\sigma_Y^2 = 6.25.$ Sample variances are $S_X^2 = 9.942,\, S_Y^2 = 6.364.$

var(x); var(y)
[1] 9.942942
[1] 6.36421

Then in R, the F-test of $H_0: \sigma_X^2/\sigma_Y^2= 1$ against $H_1: \sigma_X^2/\sigma_Y^2 > 1,$ might look like this:

var.test(x, y, alte="greater")

        F test to compare two variances

data:  x and y
F = 1.5623, num df = 99, denom df = 99, p-value = 0.01373
alternative hypothesis: true ratio of variances is greater than 1
95 percent confidence interval:
 1.120698      Inf
sample estimates:
ratio of variances 

The figure below show the density function of $\mathsf{F}(99,99)$ with critical value $c = 1.394$ for a one-sided test at the 5% level (dotted red line). The area under the density curve to the right of the observed value $F = S_X^2/S_Y^2 = 1.562$ is $0.0137.$

enter image description here

Notice that comparison of a sample variance with a known population variance (constant) is different than comparing two sample variances to assess whether the corresponding population variances differ.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.