How can I test if a recruitment effort was "worth it"? I am currently working on a study that involved recruiting a lot of people over a very long period of time. Participants were recruited through invitations sent by post, if they did not respond to the first letter, they would receive a second one, then a third. 
For some reason, during one wave (out of 100), they tried to knock on doors, convincing people to come join. This "door to door" attempt would only happen if the person contacted did not respond to the third letter.
How can I determine if this attempt was "worth it"(statistically)? or in other words, how can I come up with a conclusion that would either recommend this attempt to be done on a larger scale or not.
So far I have reported that the total Response rate was 12% before this "door to door" approach and became 14% after. I also measured the response rate over various subgroups to see who got affected the most. But the question remains unanswered, were these extra "2 percent" worth recruiting? 
P.S- I do not have data with regards to cost, I just have data of all of those who decided to participate.
 A: Ideally, to ensure your comparison is reliable, you would need to make sure you had controlled for other possible differences between your samples, to avoid confounding effects.
So, if the door to door approach is to be done following the third letter, you would need to have a control group (do not knock on door) and a treatment group (knock on door). Preferably, to control for socio-economic effects, in the same neighbourhood, at random. Then you could compare what happened to the recruitment rate between control and treatment groups.
Note that you might want to try this in a number of neighbourhoods (in case the first neighbourhood you choose is unusual in some way) and perhaps also to test the effect of the sampling effort (e.g. do you go back at a later time if the door knock is not answered?).
What statistical test you then use will depend on the form of your data.
If you do not have a control group, you cannot distinguish the knocking on door effect from the effect of time, word of mouth from neighbours etc. You might be able to get figures on what percentage of people who answered the door signed up immediately, but you would not know how many of them would have signed up anyway, eventually, without this approach.
