0
$\begingroup$

I've thought quite a lot on large sample size inference where the strong law of large numbers is easily validated. In my case however, I'm trying to infer the sign and magnitude of an outcome where the noise-to-signal is quite large in comparison to my available sample size.

How do I properly test for goodness of fit? I possess a very small sample size (n = 16). I cannot possibly get more data as it simply doesn't exist.

I'm fitting a generalized linear model with Gaussian errors for which I obtained the following p-values with summary(my.glm.fit) using R's glm() function for fitting.

intercept   0.34172  
slope       0.00734
  • Using Cook's D gives one data point that is problematic. If anyone knows about its validity with small samples, please speak out!
  • I don't think a Durbin–Watson test statistic is appropriate for n = 16. Even if my data has almost surely no auto-correlation, I have very sound reasons to think it doesn't. What could I use instead?
  • Normality tests for small sample size...is Lilliefors OK? or should I go with a Bayesian test?

Anything else you can point out? Anything else missing in my maybe too-broad question? I could provide more details if that's of any interest to you guys. Partial answers are also welcome. Thanks.

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ I wouldn't bother formally testing normality anyway. It doesn't answer the question that your really need to worry about (which is not 'is my data normal?' - it almost certainly isn't normal, you don't need a test to tell you that, it's 'how badly could my inference be impacted by the kind of non-normality I have?', which the test doesn't tell you about). $\endgroup$
    – Glen_b
    Commented Jun 19, 2014 at 6:02

1 Answer 1

Reset to default
2
$\begingroup$

I'm assuming here that you are pretty happy with your model, and believe that there are no serious lack-of-fit (LoF) issues. LoF shows up in the residuals and will well-and-truly mess up any and every test you might use that is based on the residuals. Your use of influence measures causes me to question the validity of that assumption.

Goodness of fit tests are fair weather friends:

  1. They have abundant power in large samples, to the point that they can detect differences you really don't care about.
  2. They don't have very much power in small sample situations.

In other words, they work best when you don't need them and don't work well when you need them most.

Cook's D and related measures (like Belsley's DFFITS) are not goodness of fit statistics. They are influence statistics, trying to measure how important a given point is in your result. The thing to remember about them is that although they marginally have the derived distribution, they do not have the distribution jointly. Use the suggested cutoffs as rough guides rather than as commandments. The thing Cook's D tells you to do is to check the point out and be aware of its impact on your conclusions.

According to Madansky (Prescriptions for Working Statisticians), the Shapiro-Wilk test is the best overall compromise test of Normality. Shapiro, Wilk (and later Chen's) work bear this out. In the last ten years, some work with alternative tests (based on measures of skewness and kurtosis) have slightly better properties than the S-W test. They have not hit the mainstream yet.

There aren't any good tests of autocorrelation for tiny samples that I am aware of. I suppose D-W is as good as any.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you Dennis, your insight is very much appreciated. I'm not particularly happy about my model. In fact, I'm very suspicious of the results; which is why I'm trying to compute test statistics and other measures of fit to help me judge its validity. I'm hoping this is just sample size bias, but I see some heteroskedasticity AND non-linearities in the tails. I can live with non-linearities; they produce conservative results and improve the performance of my system. But perceived heteroskedasticity due to a flagged data point from Cook's D makes me question the linear model fit I have. $\endgroup$
    – not.so.quanty
    Commented Jun 19, 2014 at 13:45
  • $\begingroup$ Provided that you randomized properly, there is no such thing as "sample size bias". You cannot reliably measure tail behavior with n = 16. It just isn't happening -- you can't reasonably expect to get much beyond the 5th and 95th percentiles, absent a miracle. As Goldberger said, the cure for micronumerosity is to take larger samples. The diagnostics might tell you that your model is inadequate, but whether the problems are due to what the diagnostic is nominally looking for or due to lack of fit is an open question. $\endgroup$
    – Dennis
    Commented Jun 20, 2014 at 1:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.