I was wondering if anyone has come across a similar question to the following.

I have data of the form $s_{x,y}, t_{x,y}$ (successes and trials) for varying groups with $x \in X, y \in Y$. I also have estimates $p_x$ for $x \in X$, which were trained on this data with $y$ marginalized out. My goal is to produce multipliers $m_y$ for $y \in Y$, such that $p_{x,y} \approx m_y p_x$. Of course, my data is big and sparse and both $x$ and $y$ will be multilevel in actuality.

I am currently entertaining two ideas, both GLMs. One which involves an inverse link that is a scaled up expit (inverse logit) function with range on $(0, 1/max(p_x))$. The other involves using a log inverse link. In both cases I would still use binomial logloss and some reasonable regularization.

These both feel very unnatural, and I would be shocked if I'm the first person to have asked this question and tried to answer it. I am aware that even attempting this places unusual implicit assumptions on the data generating mechanism, but let's look past that for now, if you don't mind.


Note: If you think this is hopeless or an ill-posed question please let me know why.

  • $\begingroup$ When you say your $p_x$ were "trained on the data with $y$ marginalized out", do you mean $p_x = \left(\sum_{y \in Y} s_{x,y}\right) / \left(\sum_{y \in Y} t_{x,y}\right)$? Or are there covariates associated with the group $x$ and you trained a binomial model to output the probabilities? $\endgroup$
    – Ben Kuhn
    Apr 22 '15 at 18:11
  • $\begingroup$ Also, when you want $p_{x,y} \approx m_y p_x$, do you literally require that the $p_x$ are the output from your marginalized model above? Or do you just want to express $p_{x,y}$ as a product of a term only depending on $x$ and a term only depending on $y$ (neither of which need necessarily be probabilities in the general case)? $\endgroup$
    – Ben Kuhn
    Apr 22 '15 at 18:13
  • $\begingroup$ Ben, thanks for your interest. To answer your first question, $p_x$ is not a simple ratio. As I mentioned my data is sparse, and what I have called $x$ and $y$ are actually hierarchies of variables. $p_x$ is the output of model, you can think of it as output of a logistic regression model. When I say it is trained on data with $y$ marginalized out I mean that it was given $\sum_{y \in Y}s_{x,y}$ as successes for $x$, and similarly trials. As for the second question I do want to use the given $p_x$ and only define new $m_y$, which need no probabilistic interpretation. $\endgroup$ Apr 22 '15 at 18:42
  • $\begingroup$ For what it's worth, unlike you, I wouldn't be shocked if you were the first person to ask this question, since assuming a multiplicative structure on probabilities is not a very natural thing to do (compared to, e.g., a multiplicative structure on odds ratios). The latter would make finding a "nice" model much easier, I think. $\endgroup$
    – Ben Kuhn
    Apr 22 '15 at 22:18
  • $\begingroup$ For instance, the same reason that using a scaled-up expit function feels unnatural to you (namely, that you have to do weird things to avoid getting probabilities above 1) is the same reason that people typically use logistic regression (which assumes multiplicative effects on the odds ratio) rather than log-regression, i.e. assuming multiplicative effects on the probability. Furthermore, the latter model, and your proposed structure, are also not invariant if you reverse your definition of "success." $\endgroup$
    – Ben Kuhn
    Apr 22 '15 at 22:26

If you have a multiplicative structure on the odds ratio--that is, $\frac{p_{x,y}}{1-p_{x,y}} = m_y \frac{p_x}{1-p_x}$--you can do a binomial hierarchical GLM with $\frac{p_x}{1-p_x}$ as a covariate.

If you absolutely positively need a multiplicative structure on the probability, consider just writing down your loss function and using a constrained optimization routine to optimize it directly. This allows you to encode per-variable restrictions like $\forall y: m_y \in [0, 1/(\max_x p_{x,y})]$ rather than having to fiddle awkwardly with the scaling of a link function. (If you want to allow your model to output probabilities above 1, then decide what you're going to do with them--e.g., clamp them to 1?--and just include that in your loss function as well.)

  • $\begingroup$ Thank you very much for your response. I really wanted to see if there was anything out there for this type of problem. I will probably still use the scaled up link, because I would like regularize the effects, preferably on the log or logit scale. In particular I appreciate the intuition about treating success and failure asymmetrically, so I need to let that marinate for this decision. Thanks again. $\endgroup$ Apr 23 '15 at 2:56
  • $\begingroup$ Cool. Sorry I don't have more helpful things to say. FYI, you can also just include a regularization term in a loss function that you optimize directly, if scaling the link function ends up restricting the range of the $m_y$ too much. $\endgroup$
    – Ben Kuhn
    Apr 23 '15 at 17:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.