How to determine if a (binary) variable has a statistically significant effect on a response variable? To be more specific, assume the following scenario:
A customer advertising for a job on your site wants to know whether buying an extra ad-product or not will increase the number of applicants. You have access to data of previous customers who bought the product, and the resulting number of job applicants their advertisement got. Let’s assume that you have $n$ such data points. In addition, you have $N$ data points of customers who did not buy the product and the number of applicants their ads got. We can assume that the number of customers who bought the ad-product is small compared to the number of customers who did not, i.e., $n<<N$.
The data is heterogeneous, containing information about where the job is located (e.g. which city/county), which industry (engineer, designer, teacher, etc.). You also have information regarding the number of page views each ad got, and so on. In all, you have $p$ features corresponding to each observation (the number of job-applicants). The response variables takes discrete values $\geq 0$.
Given the above scenario, what is the recommended approach to determine if buying the extra ad-product has any (statistically significant) effect on the expected number of job applicants? For clarity let's assume that "significant" means p-value $p<0.05$. In summary, I would like to test the hypothesis 
$H_1 = \{$ Buying the ad-product has an effect on the number of job applicants $\}$ 
against the null hypothesis 
$H_0 = \{$ Buying the ad-product has no effect on the number of job applicants $\}$ 
and determine if $H_1$ should be rejected at the mentioned significance level. 
Ideally, I would also like to be able to say something about the conditional probability $P($ "Num. job applicants" $> n_0 \,|\,$  "Bought ad product" $)$, for some number of applicants $n_0$, but I consider that secondary.
Thoughts about possible methods:
1) Use a bootstrap procedure to augment the "bought ad-product" data set (assuming the sample observations are iid and representative of the underlying population). For each bootstrap iteration, perform a regression analysis and determine the coefficient corresponding to the binary "bought ad-product" variable. From these, build a confidence interval that contains 95% of the coefficient observations and check whether it is consistent with zero or not, at that level. This possibly represents a dead-end, since all I would prove by following this procedure is that the variable is correlated with the response, correct?
2) Fit a regression model to all the data with customers who did not buy the ad-product. This represents the model under the $H_0$ assumption. Second, bootstrap the data with few data points (customers who bought the ad-product) and for each bootstrap-iteration fit a model (now including the binary "bought ad-produtct" variable). "Aggregate" all the models (not sure how) and then compare the resulting model $H_1$ to $H_0$. I am quite uncertain as to how this comparison should be performed now, but am looking into it as we speak.
I have knowledge about statistics, but am not a statistician, so I would greatly appreciate any helpful comments and inputs. Links to useful information are also very welcome. Thank you very much for your time.
Regards,
M
 A: This issue is similar to this one: Finding significant independent variables for categorical dependent variable
Except that you may want to use some of the variables you mentioned as random effects (location?).
A: The question is quite old, but if you are still interested in the problem, here is a suggestion that elaborates on the comments by Dave and Frank Harrel.
Ordinary linear regression is for continuous unlimited response variables. For responses that are non-negative counts, a special case of "generalized linear models" (GLM) has been specifically devised: Poisson regression. One way to answer your research question is thus:

*

*Fit a GLM from the Poisson (or "quasi-Poisson", see below) family to your data. In R, this is fit <- glm(y ~ ., family=poisson, data=...)


*As your predictor variable of interest is only binary, you can simply check whether its coefficient is significant, e.g. in R with summary(fit). For the more general case of more than two category levels, you can do a (type III) ANOVA, e.g. in R with drop1(fit, test="Chisq").
There are some things to check, though. The Poisson regression fit might be poor due to "overdispersion", which can be circumvented by a "quasi-Poiisson" family. As this is a generalization of Poisson regression and includes ordinary Poison regression as a special case, you can generally try it.
The other, more serious, caveat is that your predictor of interest might be itself predictable from other predictor variables. In  other words: you have multicollinearity in your data. A simple test for collinearity is to check whether the "variance inflation factor" (VIF) of your variabale of interest is suspiciously high (greater than 5 or greater than 10).
