I would like to measure the change of a policy in a community, that is, check whether there was a change after the new policy was applied. The old policy required that people had to give away a right in order to participate in a community (similarly as academic journals work, where authors usually are asked to give away their copyright). The new policy dropped that requirement.

The old policy could have refrained some people to contribute (or participate) to the community.

The hypothesis would be:

  • $H_0$: The change in the policy does not produce any effect in contribution (participation)
  • $H_1$: The change in the policy produces an effect in contribution (participation)

It seems to me I have to use Chi-square (with a contingency matrix) because I would like to measure changes in the population (contributions or participants).

The following aspects makes me hesitate on how to approach the problem:

  • There are participants that contribute before and after the policy change. Therefore, the groups before and after are not independents. (Does this matter for the research design?)
  • The participation level is not homogeneous, neither before nor after. Because these contributions happen in a period of time (years), before the change there were participants who contributed for a short period of time, others for a longer period and other that still contribute.


  • ¿Is Chi-Square the most appropriate approach for this problem? If not, what would be an adequate approach?
  • Does it makes sense to be concerned about the previous aspects listed? If so, how should they be approached?
  • Is there anything else I should consider? (something that could invalidate my analysis or something that could improve it)

For what it is worth, after analyzing one community and if the null hypothesis ($H_0$) were rejected, I would try to analyze if there are changes in the level of contributions of the active participants in both before and after periods. After that, I would analyze other communities with similar policy changes and compare them (if applicable).

Finally, it is an ex post facto study.

I have looked into another questions, but they seem different:

  • $\begingroup$ How is your response measured? $\endgroup$
    – Glen_b
    Commented Apr 20, 2013 at 3:23
  • $\begingroup$ There is no questionnaire. All contributions are recorded (person, date, contribution, etc.). Contributions are tipically in source code. However, participants give away their rights only once, before making the first contribution (the waiver is the any further contribution.) $\endgroup$
    – gpoo
    Commented Apr 20, 2013 at 3:34
  • $\begingroup$ You misunderstand. WHat is being compared before and after? Don't just say the name of the variable (like 'participation'), tell me how you measure it. Is it a count? A rate or proportion? $\endgroup$
    – Glen_b
    Commented Apr 20, 2013 at 3:43
  • $\begingroup$ Contributions are usually focused, so they could be counted as times a participant has submitted a contribution or the size/complexity of the contribution. The records allow both. $\endgroup$
    – gpoo
    Commented Apr 20, 2013 at 3:48
  • $\begingroup$ When I say participants is because that also is counted. The change in the policy is to attract more participants (increase de population of the community). $\endgroup$
    – gpoo
    Commented Apr 20, 2013 at 3:50


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.