# What are "poor finite sample properties"?

In MacKinlay’s (1997) well-known article about event study methodologies, he states that there are two ways event clustering can be accommodated in event studies. Event clustering means that event windows of some observations overlap in calendar-time, resulting in overstated significance, because observations are no longer independent. One methodology to counter event clustering is the calendar-portfolio approach and the other is “to analyze the abnormal returns without aggregation. One can consider testing the null hypothesis having no impact using unaggregated security by security data… The basic approach is an application of a multivariate regression model with dummy variables…” (MacKinlay, 1997, p. 27).

The two drawbacks of using this approach of a regression model with dummy variables are that: “frequently the test statistic will have poor finite sample properties except in special cases and often the test will have little power against economically reasonable alternatives” (MacKinlay, 1997, p. 27).

Presume I do an event study with 2000 events. Because of event clustering in my data, I decide to use a regression model with dummy variables as suggested by MacKinlay. Because I run the regression with dummy variables (fixed effects) per firm, I reduce the degrees of freedom in my regression to 200.

Is this what MacKinlay refers to as “poor finite sample properties”?

Reference:

MacKinlay, A.C. (1997). Event Studies in Economics and Finance. Journal of Economic Literature, 35 (1), 13 - 39.

In the context of hypothesis tests, poor finite sample properties usually mean that the actual rejection rate of the test differs from the nominal one.

Recall that the nominal one is the level at which you are testing, often 5%. That means that you expect to reject 5% of all correct null hypotheses. If the actual rejection frequency differs from the nominal one, we have what is known as a size distortion.

Often, this distortion arises because we compute critical values for our test based on asymptotic considerations (often, the central limit theorem), i.e., an asymptotic distribution of the test statistic under the null (typically because, except in special cases, the finite sample distribution of the test statistic is not known or not tractable). That is, the distribution that the test statistic would follow if the null is true and if we have infinitely many observations. Clearly, in practice, we don't, so the statistic will follow its (unknown) finite sample distribution.

[So the answer to your question is basically yes, many dummies will lead to a low effective sample size (a.k.a. eat up degrees of freedom) - there no such thing as a unique threshold for "small" sample sizes. E.g., in problems with many parameters, such as long regressions, more data is needed for good inference than in simpler problems.]

Hence, its quantiles will differ from the quantiles of the asymptotic distribution, where the latter are used to get critical values. Thus, the test statistic will not exceed the critical value (i.e., reject) as often as expected from the nominal level.

Here is a simple example (a little Monte Carlo study that simulates the actual rejection rate of a test):

mean(replicate(10000, abs(t.test(rchisq(10, df=1)-1)$statistic) > qnorm(.975)))  What does the code do? It • simulates 10 r.v.s from a (very skewed and) demeaned $$\chi^2_1$$-distribution (recall that the mean of a $$\chi^2_k$$ distribution is $$k$$, so that subtracting 1 here produces a mean zero distribution) • finds the usual t-statistic for the (correct) null that the expected value of the distribution from which we sample is zero (that is what t.test does without specifying further options) • checks if the (absolute value of the) test statistic is larger than the 5%-critical value of a two-sided t-test (qnorm(.975)) (hence, if we reject at nominal level 5%) based on the normal distribution that the t-statistic follows asymptotically (see e.g. Is there a version of the central limit theorem which lets me replace Var(X) with the sample variance?), • it does that 10,000 times, producing 1s for rejections and 0s for acceptances • and takes an average to produce a finite-sample rejection rate. Depending on the seed, this code will return values of around 0.17, where, if the test had good finite-sample properties, we would see 0.05. In other words, instead of rejecting every 20th correct null hypothesis, as one might think it does based on the nominal level, it rejects every 6th, thus leading us to reject the null (quite far) too often. [In practice, information such as the one used here that we sample from a certain distribution is not available, so that we will not easily be able to work with more accurate critical values. If I had run the code with, say, draws from a symmetric mean-zero such as a t-distribution with small degrees of freedom, we would have still seen some size distortion, but much less.] [Is a size distortion that leads to rejecting too infrequently bad, too - after all, it leads to a smaller type I-error rate? I'd say yes, because rejection rates do not change discontinuously at the margin of null and alternative - hence, if you reject too infrequently if the null is true, you'll also reject less frequently when it is false, i.e., you'll have low power.] If you replace 10 in the code with, say, 1000, you will get values much closer to the nominal level 0.05, illustrating that the problem is a finite-sample one. Trying this for different $$n$$ would produce something like this: n <- c(10,20,50,100,200,500,1000,10000) mcn <- lapply(n, function(i) mean(replicate(10000, abs(t.test(rchisq(i, df=1)-1)$statistic) > qnorm(.975))))
plot(n, unlist(mcn), type="o", col="salmon", lwd=2, ylim=c(0,0.2), ylab="rejection rate")
abline(h=0.05, lty=2)