I would like to measure the change of a policy in a community, that is, check whether there was a change after the new policy was applied. The old policy required that people had to give away a right in order to participate in a community (similarly as academic journals work, where authors usually are asked to give away their copyright). The new policy dropped that requirement.
The old policy could have refrained some people to contribute (or participate) to the community.
The hypothesis would be:
- $H_0$: The change in the policy does not produce any effect in contribution (participation)
- $H_1$: The change in the policy produces an effect in contribution (participation)
It seems to me I have to use Chi-square (with a contingency matrix) because I would like to measure changes in the population (contributions or participants).
The following aspects makes me hesitate on how to approach the problem:
- There are participants that contribute before and after the policy change. Therefore, the groups before and after are not independents. (Does this matter for the research design?)
- The participation level is not homogeneous, neither before nor after. Because these contributions happen in a period of time (years), before the change there were participants who contributed for a short period of time, others for a longer period and other that still contribute.
Questions:
- ¿Is Chi-Square the most appropriate approach for this problem? If not, what would be an adequate approach?
- Does it makes sense to be concerned about the previous aspects listed? If so, how should they be approached?
- Is there anything else I should consider? (something that could invalidate my analysis or something that could improve it)
For what it is worth, after analyzing one community and if the null hypothesis ($H_0$) were rejected, I would try to analyze if there are changes in the level of contributions of the active participants in both before and after periods. After that, I would analyze other communities with similar policy changes and compare them (if applicable).
Finally, it is an ex post facto study.
I have looked into another questions, but they seem different:
- How to detect a significant change in time series data due to a "policy" change? It is about a change in one person behaviour, not level of participation in a population.
- What test should I use to determine if a policy change had a statistically significant impact on website registrations? It might be, but the problem measure there is only registrations, not actual participation or contributions.
