In observational studies, when randomization is not an option (everyone is treated), it is necessary to estimate a counterfactual of what would have happened in the absence of treatment.
Once you estimate this counterfactual, how do you evaluate if it's significant or not?
Can you use hypothesis testing methods like in a randomized controlled trial (RCT)/AB Test: $t$-test, $z$-test, deriving a $p$-value, etc.?
Or is there another way to do it since the treatment is not randomized?
In observational studies, when randomization is not an option (everyone is treated), it is necessary to estimate a counterfactual of what would had happened in the absence of treatment.
It is not always the case that "everyone is treated" in the absence of randomization. In observational studies where we observe multiple units over time, we could estimate a counterfactual if a subset of units were never exposed to some intervention. Say you wish to assess the effects of a county level policy on violent crime. Now suppose it impacts a subset of counties in one region of the United States but not others. In practice, what applied researchers often find post-hoc is treated and non-treated counties more than likely differ based upon their characteristics (e.g., residential population size, geography, unemployment rate, etc.). Some differences represent time-constant attributes of each county, while others might change over time if observed over a long time horizon. Moreover, there might be baseline differences in overall crime between exposed and unexposed jurisdictions. In other words, the mean crime rate might be higher in the adopter counties relative to the non-adopters. A popular technique to overcome many of these concerns is difference-in-differences. Randomizing a population level "treatment" takes care of most of these concerns, but we often find it unfeasible in practice, or even unethical.
Now that you have this counterfactual, how do you evaluate if it's significant or not?
Traditional regression techniques should get the job done.
Can you use hypothesis testing methods like in an RCT/AB Test: t-test, z-test, deriving p-value, etc.
Yes.
We don’t normally eschew traditional hypothesis testing when employing quasi-experimental approaches.
Or is there another way to do it since it is not randomized?
It often depends upon how treatment is assigned. In health policy arenas, individuals might be assigned to receive some exposure (i.e., treatment) on the basis of a "quantitative measure"; often this is some explicit measure of need or merit. Suppose a school lunch program is assigned to children whose household income falls below the poverty line; this precise demarcation line is used to divide students into treatment and control groups. By comparing individuals on either side of this cusp, we can endeavor to estimate the average treatment effect. A regression discontinuity design is a good candidate in this setting.
The purpose of illustrating the foregoing examples is to help you understand that treatment assignment matters. In uncontrolled environments (e.g., policing), interventions will invariably be aimed at jurisdictions with a demonstrable need. When randomization fails in the application of a treatment, we often exploit these popular quasi-experimental approaches.
