Here I am not talking about medical research, where there is placebo effect need to be controlled with the blind design. In social science studies, we almost never do blind experiment. In this case, if we have data from all the subjects, do we really need to have a randomly selected control group to be compared to?
Say, if I have a teaching strategy to improve 5th grade students' math achievement and I want to test its effect. Student math achievement is measured by statewise standard assessments at every end of school year and all data will be extracted from district database directly. Which of the following is better:
randomly select 20 schools from 200 schools in a district, and then randomly assign 10 schools to treatment group (use the new teaching strategy) and 10 schools to control group (do business as usual). Compare the control and treatment.
randomly select 10 treatment schools from the same district. Compare the treatment with the rest of the district (control on previous achievement and demographics)?
randomly select 10 treatment schools from the same district. Compare the treatment with 10 randomly selected schools (post hoc) from the same district (control on previous achievement and demographics)?
Is method 1 a better design and why?