Aggregate-level difference-in-difference analysis Thank you in advance. I am new to difference-in-difference (DID) analysis.
I want to conduct a research project examining an educational policy shock to local school districts.
I have aggregated student performance data at the school-district level (2 years prior to the policy shock and 2 years after the policy shock; 5 years of data in total). The number of school districts with the policy shock is small (n = 20) and the treatment to control ratio is not yet determined. For example, with a 1:1 ratio for DID, the total sample will be 40 school districts.
Is the sample size (e.g., 20 + 20 = 40) large enough for a DID analysis? Are such data appropriate for a DID analysis?
Any suggestions and/or reference articles are appreciated.
 A: I will tackle your questions and concerns from the bottom of your post and work my way up.

Are such data appropriate for a DID analysis?

Yes.
You're well within the realm of the "classical" difference-in-differences (DiD) approach. Your data is 'aggregated up' to the district level and treatment begins at the same time for all treated districts. The equation should look something like the following:
$$
\text{Performance}_{dt} = \alpha + \gamma \text{District}_{d} + \lambda \text{Post}_{t} + \delta (\text{District}_{d} \times \text{Post}_{t}) + \theta X_{dt} + \epsilon_{dt}.
$$
In the "classical" case, $\text{District}_{d}$ is a dummy indexing your treatment group (i.e., treated districts). $\text{Post}_{t}$ is a dummy indexing post-treatment years in both treatment and control groups. $X_{dt}$ is a vector of time-varying covariates at the district level. Your estimate of $\delta$ is your treatment effect.
But caveats abound! Now I will address your other concerns.

Is the sample size (e.g., 20 + 20 = 40) large enough for a DID analysis?

The number of districts is neither large nor small in my estimation. You will more than likely need to make some finite sample adjustments to deal with your standard errors. Review this guide for more information. Note, if your policy shock is at the district level then it should influence all schools nested within that district. If some schools within the district decided to opt out of the new policy, then there is another layer of variation you can exploit. I only ask because aggregation reduces outcome variation in your performance data. This is inevitable, though.

I have aggregated student performance data at the school-district level (2 years prior to the policy shock and 2 years after the policy shock; 5 years of data in total)

It appears you whittled down your sample of districts because you do not have enough pre-event observations for some treated districts. This is unfortunate because the number of district-year observations before the policy shock is scanty. I would argue two periods is insufficient to demonstrate a clear parallelism in the group trends. Three or more periods is desirable, and enough to support claims of trend equivalence in the pre-period.
Not to belabor the point, but you will have to make the case that the group trends were moving in tandem prior to the policy shock. This is not something to overlook. Did a subset of superintendents opt into the policy/program to improve student performance? If so, was it because some schools were already performing lower than average on certain performance metrics? Suppose a district experiences a drop in performance in the year immediately before policy adoption. This effect might be transitory, and as such we might observe better performance in a subsequent period(s) even in the absence of treatment. In other words, the improved performance in the "post-period" is interpreted as a program effect when, in reality, the 'low performing' districts were simply 'rebounding' from a very bad year. One could also make the case that students and teachers would respond more favorably to the policy. In other words, they might have more motivation to improve.
I should note that my musings are more methodological than statistical. If the policy was truly "random" then you have less to worry about. But in most practical applications, selection bias is a major concern. Again, you will have to make a very strong case that the "soon to be treated" school districts were not already on a different trajectory in the years prior to the policy/program.

Student performance data [is] at the school-district level are available from 2010 to 2018, and different school district[s] may implement the policy between 2010 and 2018.

The foregoing quote is from the comments section. If this means that different districts may opt into the policy in different years, then you cannot use the "classical" equation outlined above. In settings where the adoption period is 'staggered' over the years, you must proceed with a "generalized" DiD framework. This suggestion, though, appears to be outside the scope of your analysis.
You might also find this paper by Zhou and colleagues (2016) an interesting read.
