Measuring the level of effect a continuous variable has on a binary outcome

Question

I'm working with auction data, and I'd like to better understand how much the bid amount effects whether or not the bid is won. I have evidence to believe that the bid amount is not a reliable predictor, but I can't figure out how to quantify this.

A major hurdle in this specific case is that the win rate is extremely low, because it's likely that many of the auctions aren't actually "winnable", but are really just means for gauging the value of the items for sale. Logistic models I train (bid amount as predictor, with outcome 0 for loss and 1 for win) predict 0 for all validation examples. As such, I've been using log-loss to validate the model, and found that training on bid amount yields only a marginally better log-loss value than training on random normal variables. I'm looking for a single number that expresses whether or not the bid amount is actually predictive, so I know whether or not I should keep exploring this path or try another approach.

Answer 1

You state that the logistic models predict 0 for all validation examples. I assume you are taking a classification perspective on logistic regression where you are rounding a probability less than 0.5 of winning down to 0 = "lost" and a probability above 0.5 up to 1 = "won". I suggest you examine the probability of wining as a function of item price.

I think you are interested in measuring the effect size of item price on probability of winning. In logistic regression it is common to look at the odds ratio. The odds ratio tells you the change in odds of success (e.g., winning) for a one unit change in the predictor (e.g., item price). You probably also want to account for the uncertainty in the estimate. Therefore, I suggest you calculate the 95% confidence interval for the odds ratio

$ (OR_{LL}, OR_{UL}) = exp\left(\hat{\beta_1} \pm z_{1-\frac{\alpha}{2}}s_{\hat{\beta_1}}\right) $

where $\hat{\beta_1}$ is the estimated coefficient of item price in your model.