# Regression model where output is a probability

I am trying to fit a model, and have the suspicion that what I am doing is not quite right. The data tracks what proportion of people made a decision, and what factors were active when they made their decision, i.e. something like this:
[1,0,1,0, 23%]
[1,1,0,1, 41%]
etc...
I also know how big each group is. The goal is to predict the percentage, based on the binary input. My initial thought was, the model cannot be a straight linear combinations, if only because the output is bounded. It is not exactly a logistic regression either, because the output is not a label, but the average for each group. My first take has been to transform the output in a fashion similar to logistic regression, into log (p/(1-p)), and fit a linear regression. This has given me some decent results, but I have a nagging feeling this is not quite right. Besides the output transformation, I am also concerned that treating the input as numbers, when they really represent binary values, is probably not the best way to go.
So my question is, if this is not the right way to go, are there models that address this specific type of situation? What should I be looking for?

[Edit for clarification] From the comments / responses it seems my description of the data was a bit lacking, so here is a bit more, as well as why I am uncertain about using logistic regression. I'll illustrate on similar data. Suppose products had a set of binary characteristics, and various products were presented to customers, and the result (buy/no buy) was recorded. Then the raw dataset would look like:
1, 1, 0, .. 0, 1
1, 1, 1, .. 1, 0
where each row is a specific product, and what the customer did. Now I can aggregate these by identical products, which will have the same characteristics, and simply record the proportion of buys, as well as the number of customers that were presented with that choice. This is essentially what I have.
I could disaggregate back into the original dataset, and run a logistic regression on it, but the groups themselves are very large, and of very different sizes. On top of that, I have 2 problems. First, I could reconstruct synthetic groups that have the same proportion as the original (i.e. if 4% purchased, construct 4 Buy, 96 No Buy rows), but the the Buy/No Buy ratios are pretty small, which would mean reconstructing large groups to properly approximate. Second, the groups have very different size, and I believe the group composition in the complete sample should have similar composition to the original one, which would mean creating potentially very large groups. Which is why I was essentially wondering if there was a way to work off the much smaller dataset directly, without having to reconstruct an artificial gigantic dataset.
My current approach has been to use a gradient descent approach, weighting observations by the group sizes, but I was wondering if there was a smarter way to handle this!

• I'm struggling to understand why you don't think logistic regression is appropriate. The response for logistic regression is a count of $x$ successes out of $n$ trials, which is what you say you have, & the estimated coefficients give you predicted probabilities for any predictor pattern, which is what you say you want. I'm wondering if you think that because logistic regression can be used for classification (i.e. give a "label" as an "output"), that's all it does. – Scortchi - Reinstate Monica Mar 19 '14 at 20:07
• You are correct. I realize that a logistic does largely what I want. The problem I have is, I believe the output of a logistic has to be categorical, and to do that I would need to explode the groups back into individuals (i.e. 43% would give me 43 "yes", 57 "no") - and the groups are large, which is why I was hope I could use the group aggregate results. Am I thinking about this incorrectly? – Mathias Mar 19 '14 at 20:21
• You would need to express the data as e.g. 43 "yes" & 57 "no", though I can't imagine there's software for which an array of counts of "yes" & "no" for each predictor pattern would be unsuitable - you wouldn't have to make that 100 rows. And I don't know what you mean by the output being categorical. – Scortchi - Reinstate Monica Mar 19 '14 at 20:34
• If you have the individual data and how they map into groups you could do a hierarchical logistic regression. – John Mar 19 '14 at 21:09
• I think I now know what you mean by "output": the response, or dependent variable. So there is no problem in expressing it as as a two-level categorical (i.e. dichotomous) variable, you have the group sizes to be able to do so. – Scortchi - Reinstate Monica Mar 19 '14 at 21:30

You can convert this into a logistic regression without having to blow the data set up to the size of the groups, you just have to use weights. For that first group, you would split that into [1,0,1,0,1] (last column is the response) and the weight would be [(# in that group) * 0.23], and [1,0,1,0,0] with a weight of [(# in that group) * 0.77]. See the R documentation for GLM with the parameter "weights" for how to execute that in R. T

After that, it's a straightforward logistic regression. This is equivalent to the binomial regression that someone else has suggested.

• Thanks! I looked at data.princeton.edu/R/glms.html, and the example is pretty much exactly what you describe and what I want. I had just never seen logistic expressed that way. Now I need to dig into the R implementation a bit, and see how the fitting is done! – Mathias Mar 20 '14 at 19:58

If I understand correctly, you're really trying to measure event counts among groups of different sizes. You could use a Poisson regression and add an offset for the size of each group. See When to use an offset in a Poisson regression? and poisson vs logistic regression for more explanation.

Not sure if this will be better from a predictive standpoint. I have had good predictive success using linear models or GBMs with ratios as the dependent variable. It would be a problem if you have many ratios close to 0 or 1 as a linear reg will start predicting ratios outside of that range.

After seeing your update, I would disaggregate the data and use the group as a factor variable in a logistic reg. What's wrong with a large dataset?

• You could be right, but there's a lot in the question suggesting the response is binomial. – Scortchi - Reinstate Monica Mar 19 '14 at 21:38
• Thank you! The point about low ratios is an important one, as most of the them are very low. – Mathias Mar 19 '14 at 21:43
• GBM = "generalized boosted model"? Or "geometric Brownian motion"? or something else? – Glen_b Mar 19 '14 at 23:24
• I meant "generalized boosted model". These have performed much better than linear reg on the ratio prediction problems that I have had. – wcampbell Mar 20 '14 at 1:49

I think you should look at a Binomial regression model. Instead of modeling the proportions (percentages) you would model the counts of the decisions (still only one row for each group.) I.e. a generalized linear model with a Binomial rather than the usual Bernoulli likelihood. Here is a description of how it can be done in R.

• Of which logistic regression is a type. – Scortchi - Reinstate Monica Mar 19 '14 at 21:17
• Yes - my point was that what is special in this particular data is that it is binomial rather than binary. – Mikkel N. Schmidt Mar 20 '14 at 5:51
• And a good point it is - just clarifying for the OP or other readers. – Scortchi - Reinstate Monica Mar 20 '14 at 12:00

You should be able to work with only one row of data for each group :

With these two bits of information: "The data tracks what proportion of people made a decision", and "I also know how big each group is", you can turn the data into a count of success and trials (or fails): simply multiply the size of the group by the proportion in each group to get the number of successes, and subtract from the group size to get the number of fails. You can then fit a logistic regression. R, SAS, and (I assume) other packages can fit a logistic model with data specified as counts of successes and fails or trials.

For example in R the response variable can be "a two-column matrix with the columns giving the numbers of successes and failures" (quoting the documentation for the glm() function). Alternately, in R you can just fit the model with the response variable as a proportion and use the weights vector to specify the number of trials (group size).

"I am also concerned that treating the input as numbers, when they really represent binary values."

You could tell the software that these are categorical variables (e.g. converting them into factors in R or using a class statement in SAS), but with binary variables this is not strictly neccessary (categories get converted into binary dummy variables "under the hood" anyway). It might make the code clearer though.