# Visualizing relationship between independent variable and binary response

I have ~14.000 observations with an independent variable, interactions, and a dependent variable, accuracy. The accuracy can be either 0 (misclassification) or 1 (true classification). There is a weak, but statistically significant relationship between the two as seen in the figure below (whiskers specify 95% confidence interval).

I would like to have a smooth plot across the possible values of the interactions instead of binning them as done in the previous figure. I've tried using GAM in R to do that, but I end up with the following figure, which is clearly wrong. I've also tried logistic regression, which just ended up being a straight line, thus not capturing the convergence around interactions = 100.

How can I get a smooth plot of the relationship between the two values that captures the initial rise in accuracy and then the convergence around interactions=100? It would be preferred if a confidence interval could be inferred as well. The data can be found at codeshare.io. Interactions above 400 are not interesting so they can be left out if needed.

• Have you tried a lowess? If you need a CI, try a fractional polynomial as well. – Dimitriy V. Masterov Jun 18 '15 at 0:59

I can't speak to the modeling (except to guess that the bend near 100 is too sharp to be captured by a logistic curve), but a visualization idea is to continue your binning idea to the extreme. Consider a bin for every possible interactions value which extends some fixed amount on each side. Compute the mean and CI for each of those bins. But instead of plotting 100s of interval bars, plot the means as a connected line and the upper and lower CI bounds as an area.
More on the moving bins: For each interaction value, say 57, I looked at the interval +/25, which would be [32 .. 82). For all the values in that range (3071 for this example) I computed the mean and Std Error. Each interval may have a different count, but the SE is taking the number into account. Other methods like Loess typically look at weighted intervals of equal count. I don't know the statistical merits either way, but the graph can at least be used to suggest a non-linear function that's better than a logistic curve.