8
$\begingroup$

When modeling rare events with logistic regression, oversampling is a common method to reduce computation complexity (i.e., keep all the rare positive cases but just a subsample of negative cases). After model fitting, adding a offset to the intercept term is a common method to correct the event probability to reflect the original sample proportion. The offset is equal to log( r1*(1-p1) / (1-r1)*p1 ), where r1 is the proportion of rare events in the oversampled data and p1 is the proportion in the original data. What is the equivalent formula with multinomial logistic regression, where 1 or more classes is oversampled?

$\endgroup$

2 Answers 2

2
$\begingroup$

Off the cuff, I presume one could proceed as in logistic regression: a generalisation to $K>2$ categories and base category $K$ would be to set the $i$-th correction term to be $$\log \frac{(r_i p_K)}{(r_K p_i)}$$ corresponding to the $i$ vs $K$ contrast. For $K=2$, $p_1$ is as before and $p_K = p_2 = 1-p_1$, so it reduces to $$\log \frac{r_1 (1-p_1)}{(1-r_1) p_1}.$$

However, I'd be happy to be corrected on this one.

$\endgroup$
0
$\begingroup$

@conjugateprior's answer is correct. Let's verify the answer.

Let $(X,Y)$ be a pair of random random variables, where $X$ takes values in $\mathbb{R}^N$ (for some $N$), and $Y$ is categorical with values in $\{1,2,\dots, K\}$.

Warning: to deal with conditional probabilities intelligibly, we will abuse notation and consider probabilities of the form $P(X = x)$. A complete argument would make use of densities with respect to a reference measures, and Radon-Nikodym derivatives.

Let $r_j = P(Y=j)$ for $j=1,\dots,K$.


Suppose $Y\,\mid\,X$ satisfies a multinomial logistic model, i.e.: supose there are arrays $\beta_{j0}, \beta_{j1}$ ($j=1,\dots,K$), of the appropriate dimensions), such that

$$ P(Y=j \mid X=x) = G~ e^{\beta_{j0} + \beta_{j1} \cdot x} $$

with $\beta_{K0} = 0$ and $\beta_{K1} = 0$, "$\cdot$" the dot product in $\mathbb{R}^N$, and $G$ a constant (with respect to $j$) that makes $P(..\mid X=x)$ a probability. i.e.:

$$ G(x) = \frac{1}{1 + \sum_{t < K} e^{\beta_{j0} + \beta_{j1} \cdot x} }. $$


Let's now consider another random variable $(X,Y_S)$ ($S$ is for synthetic) which is similar to $(X,Y)$ in the sense that:

$$ \tag{*} P_S(X = x \mid Y_S = j) = P(X = x \mid Y = j) $$ for all $x, j$. Let $p_j = P_S(Y_S = j)$.



Factoid: $Y_S\,\mid\,X$ also satisfies a multinomial logistic model.

Proof:

$$ \begin{align*} P_S(Y_S = j \mid X=x) =& \frac{ P_S( Y_S = j ~\& ~ X = x )} { P_S( X = x )} = \frac{ P_S( X = x \mid Y_S = j)\ p_j} { P_S( X = x )} \\ & \\ =& \frac{ P( X = x \mid Y = j)\ p_j} { P_S( X = x )} = \frac{ P( X = x ~\&~ Y = j)/r_j } { P_S( X = x )}\ p_j \\ & \\ =& \frac{ P( X = x ) P(Y = j \mid X =x) } { P_S( X = x )} \frac{1}{r_j} \ p_j \\ & \\ =& \left[ \frac{ P( X = x ) } { P_S( X = x )} \ G(x)~ \frac{p_K}{r_K}\right] \frac{p_j}{r_j} \frac{r_K}{p_K} e^{\beta_{j0} + \beta_{j1} \cdot x} \\ & \\ =& \widetilde{G}(x)~ e^{\tilde{\beta}_{j0} + \beta_{j1} \cdot x} \end{align*} $$

where $$ \tilde{\beta}_{j0} = \ln\!\left( \frac{p_j}{r_j} \frac{r_K}{p_K} \right) + \beta_{j0}. $$


Application to oversampling

Suppose $\{ (x_1,y_1), (x_2, y_1),\dots , (x_n, y_n) \}$ are iid observations drawn from of a random pair that satisfies a multinomial logistic model. Consider a subsample $\mathcal{S} = \{ (x_i,y_i) \}_{ i \in F }$ where $\{ x_i \mid y_i = j , i \in F \}$ is drawn at random from $\{ x_i \mid y_i = j \}_{1 \le i \le n}$. Because of the subsampling schema, condition (*) is satisfied, i.e.: the conditional distribution of $X|(Y=j)$ is the subsample is the same as in the original sample.

It follows from the Factoid that $Y\,\mid\,X$ satisfies a multinomial logistic model in the subsample. Furthermore, the model coefficients are the same, except for the constant terms that satisfy

$$ \tilde{\beta}_{j0} = \ln\!\left( \frac{P(Y=j)/P(Y=k)}{P_S(Y=j)/P_S(Y=k)} \right) + \beta_{j0}. $$


Therefore, if we estimate the constant coefficient in the subsample (i.e.: if we estimate $\tilde{\beta}_{j0}$), we can estimate $\beta_{j0}$ using:

$$ \beta_{j0} = \ln\!\left( \frac{P_S(Y=j)/P_S(Y=k)}{P(Y=j)/P(Y=k)} \right) + \tilde{\beta}_{j0}. $$


$\endgroup$
4
  • $\begingroup$ You can use \mid in place of ~\Big |~ and the edit is in that vein. $\endgroup$ Commented Jul 4 at 6:07
  • 3
    $\begingroup$ Why would over sampling ever be used in the first place? There is no problem with highly imbalanced data, and predicted values with retain their validity if you don’t over sample. Computational complexity is not an issue as far as I can see. $\endgroup$ Commented Jul 4 at 12:10
  • $\begingroup$ About oversampling: in the case of logistic regression, I thought the standard deviation of the coefficient estimates are dominated by the size^(-1) of the minority class, but I've never gone though the computation. I've came to this posting searching for oversampling, because I am fitting a model of consumer baskets; when I observe the baskets week after week, the overwhelming majority of consumers have empty baskets. In my model, I am imputing the fractional part of consumption (to avoid computing an integral in high dimensions), which is computationaly expensive. $\endgroup$ Commented Jul 5 at 19:41
  • $\begingroup$ Hence, I am considering sampling the empty baskets. Becuase I am sampling cases based on the model target, I am assessing the changes one introduces in the model "parameters" $\endgroup$ Commented Jul 5 at 19:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.