0
$\begingroup$

I have 7 bins of binary data according to a variable x as follows:

x1: N=1000, success = 324, failure=676

x2: N=1000, success = 444, failure= 556 .... ....

x7: N=...., success = ..., failure = ...

I only have 7 x values (7 bins of data) and I want to fit a probability distribution to this data based on the fractions of successes (approximated as the probability of success) for each value of x.

For example say if I know the probabilities of success for x1 = 1.0, x2 = 2.0, and x3 = 3.0 but based on this I want to estimate the probability of x = 1.5 or x=2.5 having a successful outcome.

I thought logistic regression since the outcome of each experiment is a binary varibale... but since the x values are not evenly spread (0.8,1.0,1.5,2.0,3.0,5.0,8.0) the logistic regression model is as good as random according to the sklearn AUC and the Brier score is higher than it should be for a well calibrated model.

It is evident from the actual experiment I am investigating that the probabilities of success between, say x1 and x2 should increase in a roughly linear fashion as a function of x. That is, larger x values give higher probability of success.

Is it extremely crude to just interpolate the probabilities between the points (interpolate between p(x1) and p(x2) and so on)? Or is it wise to do a linear regression with probabilities? Or else how should one fit a probability distribution in an experiment like this?

$\endgroup$
  • 1
    $\begingroup$ "All models are wrong, some models are useful" <- of course it is incorrect, and crude, but you don't cite any of the arguments supporting the approach. Certainly there are some, otherwise such a question is moot. Defend your approach and cite its limitations. That's good science. $\endgroup$ – AdamO Sep 25 '17 at 15:50
0
$\begingroup$

Perhaps this approach will prove useful. As I understand it, your data can be described as follows: \begin{equation} \{(n_i,y_i,x_i)\}_{i=1}^m , \end{equation} where $n_i$ is the number of trials, $y_i$ is the number of successes, and $x_i$ the associated value. In the question, $m = 7$. For future reference, let $n = (n_1, \ldots, n_m)$, $y = (y_1, \ldots, y_m)$, and $x = (x_1, \ldots, x_m)$.

Let $\theta(x_i;\phi)$ denote the probability of success as a function of $x_i$ where $\phi$ is a vector of adjustable parameters. The likelihood for the unknown parameters $\phi$ can be expressed as \begin{equation} p(y|n,x,\phi) = \prod_{i=1}^m \textsf{Binomial}\big(y_i|n_i, \theta(x_i;\phi)\big) , \end{equation} where $\textsf{Binomial}\big(y_i|n_i,\theta(x_i;\phi)\big)$ is the density function for the binomial distribution. At this point one can maximize the likelihood with respect to $\phi$. Alternatively, given a prior for $\phi$, one can compute the posterior distribution.

But what is a suitable functional form for $\theta(x_i;\phi)$? Whatever it is, it must keep $\theta(x_i;\phi)$ positive over the appropriate range for $x_i$. It may be possible to "get away" with a linear function: \begin{equation} \theta(x_i;\phi) = \phi_0 + \phi_1\,x_i , \end{equation} but this may not work. The logistic function will maintain positivity, but the monotonicity may not be appropriate. A flexible functional form (such as a spline) subject to restrictions on the coefficients might work. Any knowledge you have about the appropriate shape of the relation could prove useful in this regard.

$\endgroup$
  • $\begingroup$ This is extremely helpful, thank you! So it seems that a logistic function could be appropriate then, since I do know that the probability does indeed increase monotonically. I think a linear function won't work since some values of xi will give probabilities of, say less than 0.1. So if I wanted to determine a value of xi lower than that for which I have data I presume that is where I will get into trouble (negative value of probability). $\endgroup$ – none Sep 27 '17 at 15:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.