# How to determine if a (binary) variable has a statistically significant effect on a response variable?

To be more specific, assume the following scenario:

A customer advertising for a job on your site wants to know whether buying an extra ad-product or not will increase the number of applicants. You have access to data of previous customers who bought the product, and the resulting number of job applicants their advertisement got. Let’s assume that you have $n$ such data points. In addition, you have $N$ data points of customers who did not buy the product and the number of applicants their ads got. We can assume that the number of customers who bought the ad-product is small compared to the number of customers who did not, i.e., $n<<N$.

The data is heterogeneous, containing information about where the job is located (e.g. which city/county), which industry (engineer, designer, teacher, etc.). You also have information regarding the number of page views each ad got, and so on. In all, you have $p$ features corresponding to each observation (the number of job-applicants). The response variables takes discrete values $\geq 0$.

Given the above scenario, what is the recommended approach to determine if buying the extra ad-product has any (statistically significant) effect on the expected number of job applicants? For clarity let's assume that "significant" means p-value $p<0.05$. In summary, I would like to test the hypothesis

$H_1 = \{$ Buying the ad-product has an effect on the number of job applicants $\}$

against the null hypothesis

$H_0 = \{$ Buying the ad-product has no effect on the number of job applicants $\}$

and determine if $H_1$ should be rejected at the mentioned significance level.

Ideally, I would also like to be able to say something about the conditional probability $P($ "Num. job applicants" $> n_0 \,|\,$ "Bought ad product" $)$, for some number of applicants $n_0$, but I consider that secondary.

1) Use a bootstrap procedure to augment the "bought ad-product" data set (assuming the sample observations are iid and representative of the underlying population). For each bootstrap iteration, perform a regression analysis and determine the coefficient corresponding to the binary "bought ad-product" variable. From these, build a confidence interval that contains 95% of the coefficient observations and check whether it is consistent with zero or not, at that level. This possibly represents a dead-end, since all I would prove by following this procedure is that the variable is correlated with the response, correct?

2) Fit a regression model to all the data with customers who did not buy the ad-product. This represents the model under the $H_0$ assumption. Second, bootstrap the data with few data points (customers who bought the ad-product) and for each bootstrap-iteration fit a model (now including the binary "bought ad-produtct" variable). "Aggregate" all the models (not sure how) and then compare the resulting model $H_1$ to $H_0$. I am quite uncertain as to how this comparison should be performed now, but am looking into it as we speak.

I have knowledge about statistics, but am not a statistician, so I would greatly appreciate any helpful comments and inputs. Links to useful information are also very welcome. Thank you very much for your time.

Regards,

M

• I would be interested to hear why you want to do all of this bootstrap work instead of just doing the regression and looking at the p-value for the indicator variable's parameter. (The indicator variable is the binary variable that says whether or not a customer bought the extra ad product.) – Dave Nov 25 '19 at 19:12